# Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language
Prompt

Yongqi Wang, Ruofan Hu11footnotemark: 1, Rongjie Huang, Zhiqing Hong, Ruiqi
Li,
Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao
Zhejiang University
{cyanbox, 3200102312, rongjiehuang<EMAIL_ADDRESS>Equal contribution.

###### Abstract

Recent singing-voice-synthesis (SVS) methods have achieved remarkable audio
quality and naturalness, yet they lack the capability to control the style
attributes of the synthesized singing explicitly. We propose Prompt-Singer,
the first SVS method that enables attribute controlling on singer gender,
vocal range and volume with natural language. We adopt a model architecture
based on a decoder-only transformer with a multi-scale hierarchy, and design a
range-melody decoupled pitch representation that enables text-conditioned
vocal range control while keeping melodic accuracy. Furthermore, we explore
various experiment settings, including different types of text
representations, text encoder fine-tuning, and introducing speech data to
alleviate data scarcity, aiming to facilitate further research. Experiments
show that our model achieves favorable controlling ability and audio quality.
Audio samples are available at http://prompt-singer.github.io.

Prompt-Singer: Controllable Singing-Voice-Synthesis with Natural Language
Prompt

Yongqi Wang††thanks:  Equal contribution., Ruofan Hu11footnotemark: 1, Rongjie
Huang, Zhiqing Hong, Ruiqi Li, Wenrui Liu, Fuming You, Tao Jin, Zhou Zhao
Zhejiang University {cyanbox, 3200102312, rongjiehuang<EMAIL_ADDRESS>

##  1 Introduction

Singing-voice-synthesis (SVS) systems (Chen et al., 2020; Huang et al., 2021;
Liu et al., 2022; Zhang et al., 2022b, c, 2023b; Hong et al., 2023), which aim
to generate high-fidelity singing voices given lyrics and pitch notes, have
made significant advancements in improving audio quality and naturalness in
recent years, facilitating music composition and development of entertainment
industries. However, it hasn’t been fully studied to control the style
attributes of synthesized singing, such as speaker timbre, vocal range and
energy. Despite that some works use fixed speaker IDs Huang et al. (2021);
Zhang et al. (2022c) or reference speech/singing segments Shen et al. (2023);
Huang et al. ; Huang et al. (2023d) to provide information on singer identity
or other style attributes, these mechanisms are not user-friendly and lack the
ability to control specific acoustic attributes explicitly.

An ideal approach to controlling the style of generated singing voices is to
use natural language instructions as style prompts, as it can not only achieve
precise control over specific attributes with certain descriptions, but also
simplify user interaction, which may bring convenience to non-professional
users such as musicians and video creators. However, applying natural language
style prompts in singing-voice-synthesis faces several challenges:

  * •

Decoupling Melody and Vocal Range. In real-life situations, different speakers
(e.g. an elderly man and a little girl) may sing the same song within
different vocal ranges. However, pitch annotations in SVS data are each tied
to a specific singer in a certain vocal range. This coupling nature makes it
challenging to generate singing voices with consistent vocal range and timbre
to the prompt together with an accurate melody aligned with given pitch notes.

  * •

Textual Representation. Despite that some works have explored connecting text
representations with music, speech and general audio concepts Elizalde et al.
(2023a, b); Wu et al. (2023), there is no text representation tailored for
singing style descriptions, and the optimal choice of prompt representation
for this task remains unknown.

  * •

Data Scarcity. Due to the requirement of fine-grained annotations, existing
SVS datasets Liu et al. (2022); Wang et al. (2022); Huang et al. (2021); Zhang
et al. (2022a) are small in scale, typically consisting of only a few hours or
tens of hours of singing data. This not only causes limited data diversity but
also poses more challenges to learning the correlation between natural
language descriptions and data distribution.

In this paper, we propose Prompt-Singer, the first controllable SVS model with
natural language prompts to control the singer gender, vocal range and volume.
Considering the outstanding performance of recent spoken LLMs Borsos et al.
(2023); Wang et al. (2023); Huang et al. (2023d); Yang et al. (2023b) in terms
of generation and in-context learning capabilities, we adopt a decoder-only
transformer with a multi-scale hierarchy for conditional generation of
discrete codec units of the singing, together with a unit vocoder for waveform
reconstruction. To address the challenges mentioned above, we 1) design a
decoupled pitch representation with a vocal range factor and a speaker-
independent melody sequence, enabling voice range controlling while
maintaining melodic accuracy; 2) investigate various text encoders for prompt
encoding, as well as fine-tuning the encoders to seek the optimal textual
representation for this task; 3) introduce speech data to alleviate data
scarcity, and evaluate the model performance under different levels of low-
resource singing data combined with speech data. Experiments show that our
method achieves favorable style controlling accuracy on the three attributes,
while keeping good audio quality and melodic accuracy. Our contributions are
summarized as follows:

  * •

We propose the first controllable SVS model with natural language prompts to
control the singer gender, vocal range, and volume of the generated singing
voice.

  * •

We design a pitch representation for SVS that decouples voice range and
melody, which enables prompt-conditioned voice range manipulation while
keeping melodic accuracy.

  * •

We investigate different text representations and fine-tune the text encoders
to seek optimal text representation for the prompt in this task.

  * •

We alleviate data scarcity by introducing speech data, which boosts prompt-SVS
performances in low-resource scenarios.

##  2 Related Works

###  2.1 Singing Voice Synthesis

Singing-voice-synthesis aims to generate human-like singing voices from lyrics
and pitch notes, and recent deep-learning-based models have achieved
remarkable progress in synthesized voice quality. Several works Chen et al.
(2020); Zhang et al. (2022c, 2023b); Huang et al. (2022) adopt generative
adversarial networks for high-fidelity SVS. Diffsinger Liu et al. (2022)
adopts a shallow diffusion mechanism to enhance the quality of the generated
mel-spectrogram. VISinger Zhang et al. (2022b) proposes an end-to-end
architecture based on a variational autoencoder. UniSinger Hong et al. (2023)
proposes a unified framework for multiple singing-voice-related tasks based on
representation disentanglement and cross-modality information matching.
However, it has not been fully studied to control the style of generated
singing. Previous multi-singer systems Huang et al. (2021); Zhang et al.
(2022c) use a fixed group of IDs to indicate singer identities. NaturalSpeech
2 Shen et al. (2023) and Make-A-Voice Huang et al. (2023d) use a reference
singing or speech clip to provide holistic style information. Currently, there
is a lack of fine-grained controllable methods for SVS.

###  2.2 Instruct-guided Voice Generation

Inspired by the success in text, image and audio generation guided with
natural language instructions Brown et al. (2020); Ramesh et al. (2021); Kreuk
et al. (2022); Huang et al. (2023a, b, c), some recent works have explored
using text prompts to govern the stylistic attributes in voice synthesis.
PromptTTS Guo et al. (2023) incorporates style features from a fine-tuned BERT
into a TTS backbone with attention. InstructTTS Yang et al. (2023a) achieves a
text-controlled expressive TTS system with cross-modal representation
learning. PromptTTS 2 Leng et al. (2023) employs a variational network to
generate reference acoustic features conditioned on text features. PromptVC
Yao et al. (2023) and PromptSpeaker Zhang et al. (2023a) investigate text-
prompted voice conversion and speaker-embedding generation separately.
However, due to the data scarcity and the demand for precise pitch
controlling, research on natural-language-instructed SVS is currently lacking.

Figure 1: The pipeline of generating and fetching prompt sentence for training
data.

##  3 Prompt Generation and Fetching

Our goal is to control the singer gender, vocal range and volume in singing-
voice-synthesis with natural language prompts. Since there is no available
dataset for this task, we utilize normal SVS datasets and design a method for
generating a prompt sentence for each data item. We introduce this process in
this section.

Considering the high cost of manual annotation, we utilize a large language
model (GPT 3.5 Turbo) to generate prompt sentences. The prompt generation
mainly consists of 3 stages: 1) attribute categorization; 2) keyword and
sentence template generation and 3) prompt sentence assembling.

Figure 1(a) and (b) demonstrate the process of the first two stages.
Initially, we categorize the audio based on different attributes. The two
gender categories, male and female, are pre-annotated in the datasets. For
volume, we build three categories of “low”, “medium”, and “high”, indicating
the amplitude root mean square (RMS) ranges of $[0.02,0.04]$, $[0.07,0.10]$
and $[0.16,0.20]$, respectively. Additionally, we can rescale audio into
different ranges dynamically during training. For vocal range, we set two
categories of “high” and “low”, and use the average F0 of the voiced part as
the criterion for classification, with the threshold being 125 Hz for male
singers and 305 Hz for female singers.

After categorization, we use the LLM to generate a set of 4-7 synonyms for
each category as the keywords. We further utilize the LLM to generate prompt
sentence templates for each single attribute, where each template contains a
placeholder to be replaced with the keywords (such as Generate a song by a
[gender] singer). We also generate a small number of prompt sentences
targeting specific categories (such as Could you synthesize a song that’s as
powerful as a thunderstorm? for large volume). We obtain approximately 50
sentence templates for each attribute after manual selection. These single-
attribute templates can be further combined to create multi-attribute
templates by prompting the LLM. We provide sample sentence templates and
keywords in Appendix A.

The prompt sentence assembling stage takes place dynamically during training.
Figure 1(c) illustrates the pipeline of fetching a prompt sentence. We first
obtain the pre-annotated labels for the data item, and in order to make the
model adaptable to prompts with varying numbers of attributes, one or two
labels are randomly dropped with probabilities $p_{1}$ and $p_{2}$. We then
randomly fetch a keyword and a sentence template from the pre-generated sets,
and replace the placeholder with the keyword to get the final prompt sentence.
Note that we do not control vocal range independently in the absence of
gender, as its boundary is different for male and female. We use pre-generated
specific prompts for each sample in the evaluation for fair comparison.

Figure 2: Model architecture of Prompt-Singer and the multi-scale transformer.

##  4 Prompt-Singer

In this section, we introduce the model design of Prompt-Singer. The overall
architecture of our model is illustrated in Figure 2(a). It is primarily
composed of two sub-modules: 1) the multi-scale transformer, which generates
discrete acoustic units conditioned on inputs of natural language prompt,
lyrics with duration, and pitch information; and 2) the unit vocoder, which
maps the generated acoustic units to an audio waveform.

In the following subsections, we introduce the input and output
representations of the model in Section 4.1 to 4.3, model architecture in
detail in Section 4.5 and 4.6, together with our method for data scarcity
alleviation in Section 4.4.

###  4.1 Voice Representation

The acoustic units used as the prediction targets of the transformer are
generated by SoundStreamZeghidour et al. (2021), a neural codec with an
encoder-decoder architecture and a residual vector quantizer (RVQ). Such a
codec model can produce discrete compressed representations of audio by
employing a convolutional encoder followed by the RVQ, and these
representations can be used to reconstruct waveforms with the decoder. An
acoustic unit sequence can be represented as
$\mathbf{a}=[a_{1}^{1},a_{1}^{2},...,a_{1}^{C},a_{2}^{1},...,a_{T}^{C}],a_{i}^{j}\in\\{0,1,...,K_{a}-1\\},\forall
1\leq i\leq T,1\leq j\leq C$, with $T,C,K_{a}$ being number of frames, number
of residual codebooks and codebook size.

###  4.2 Textual Representation

The textual input for our model comprises two components: 1) lyrics, which
correspond to the content of the generated song, and 2) natural language
prompt, which controls the style of the singing. We introduce their
representations in this subsection.

For lyrics, we first phonemize the text and obtain corresponding phoneme-level
duration in seconds from dataset annotations or a forced-alignment tool
McAuliffe et al. (2017). We then convert the duration to frame level based on
a preset frame rate, and regulate the length of the phoneme sequence with this
duration by duplicating phonemes. We set the frame rate of phonemes to be the
same as acoustic units, making it easier for the model to learn the length
alignment. The regulated phoneme sequence is then embedded by a look-up table
(LUT) and fed to the transformer.

For the natural language prompt, we utilize a parameter-frozen text encoder to
extract a semantic representation, followed by a linear layer for mapping its
dimension to fit the transformer. To explore the impact of different text
representations on style controlling, we attempt three types of encoders in
our experiments: 1) BERT Devlin et al. (2018), a widely-used self-supervised
text encoder trained with masked language modeling and next sentence
prediction; 2) FLAN-T5 Chung et al. (2022), the encoder of a unified text-to-
text transformer fine-tuned with instructions; and 3) CLAP Wu et al. (2023), a
text encoder through contrastive pretraining on natural language and audio. We
compare BERT and FLAN-T5 of different sizes, as well as CLAP pretrained on two
different datasets. We also fine-tune BERT-large and FLAN-T5 large using
prompts and corresponding labels. We fine-tune BERT with multi-label
prediction and have FLAN-T5 predict the label sequence corresponding to the
prompt in a text-to-text manner. Note that the prompts used in the evaluation
are not included in fine-tuning.

###  4.3 Decoupled Pitch Representation

According to the equal temperament theory Von Helmholtz (1912), humans’
perception of musical intervals corresponds to the logarithmic distance of
frequencies. This means if we multiply the fundamental frequency (F0) of the
voiced part of singing by a factor (equivalent to adding an offset in the
logarithmic domain), we can adjust the vocal range without changing the
melody. Based on this principle, we decompose F0 into two components: 1)
$\bar{f_{0}}$, which is the average value of the voiced part of F0, indicting
the vocal range; and 2)
$\mathbf{\tilde{f_{0}}}=[\tilde{f_{0}^{1}},\tilde{f_{0}^{2}},...,\tilde{f_{0}^{T}}]$,
where we rescale the voiced part of the original F0 sequence to have a
specific mean value (230Hz, in our practice), indicating vocal-range-invariant
melody information. This simple yet effective representation creates an
information bottleneck, forcing the model to extract melodic and vocal range
information from the rescaled F0 sequence and average F0 factor, respectively.
In our practice, we round $\mathbf{\tilde{f_{0}}}$ and $\bar{f_{0}}$ into
integers, and use an LUT to embed them before feeding them to the transformer
backbone. Both $\mathbf{\tilde{f_{0}}}$ and $\bar{f_{0}}$ share the same
embedding space.

###  4.4 Alleviating Data Scarcity

Considering that both speech and singing are human voices in different forms,
it is intuitive that they share some commonalities in style characteristics
and distributions. Based on this point, we incorporate text-to-speech (TTS)
data into the training of the prompt SVS task to alleviate data scarcity.
Specifically, we employ the same methods as for singing to phonemize the text
and generate prompts, and use an off-the-shelf tool to extract pitch from the
speech, finally obtaining data items in the same format as SVS data.

Furthermore, we explore the feasibility of substituting speech data for
singing data in low-resource scenarios. We evaluate the model performance
under compositions of varying amounts of low-resourced SVS data with abundant
TTS data, with experiment results presented in Section 5.5.

###  4.5 Multi-Scale Transformer Architecture

The end-to-end differentiable multi-scale transformer architecture Yu et al.
(2024); Yang et al. (2023b) has exhibited remarkable capabilities in audio
synthesis and modeling intrinsic relationships between acoustic and other
modalities, as well as high efficiency of generating long sequences based on
sub-quadratic self-attention. In this work, we utilize a multi-scale
transformer derived from UniAudio Yang et al. (2023b) to serve as the backbone
of our model. It is a decoder-only transformer with a hierarchical structure
to facilitate the modeling of long sequences. This module aims to generate
discrete acoustic units of singing voices conditioned on natural language
prompts, lyrics phonemes, phoneme durations and vocal-range agnostic melody
representation, together with the vocal-range factor as intermediate output.
During training, the conditional inputs and target outputs are concatenated
into a single sequence and fed to the transformer, which models the
correlation using next-token-prediction with cross-entropy loss calculated on
the target output part. During inference, the model predicts the range factor
and acoustic units conditioned on the prefix input sequence autoregressively,
which can be formulated as:

$\displaystyle
P_{cond}\left(\mathbf{a}\right)=P_{cond}\left(\bar{f_{0}}\right)\cdot\prod_{t=1}^{T}\prod_{c=1}^{C}P_{AR}\left(\mathbf{a}_{t}^{c}\right)$
(1) $\displaystyle
P_{cond}\left(*\right)=p\left(*\mid\mathbf{E}_{\mathcal{P}}(\mathcal{P}),L,\mathbf{d},\mathbf{\tilde{f_{0}}};\theta_{AR}\right)$
(2) $\displaystyle
P_{AR}\left(\mathbf{a}_{t}^{c}\right)=p\left(\mathbf{a}_{t}^{c}\mid\mathbf{a}_{<t},\mathbf{a}_{t}^{<c},\mathbf{E}_{\mathcal{P}}(\mathcal{P}),L,\mathbf{d},\mathbf{\tilde{f_{0}}},\bar{f_{0}};\theta_{AR}\right)$
(3)

where $\mathbf{a}$, $\mathbf{E}_{\mathcal{P}}$, $\mathcal{P}$, $L$,
$\mathbf{d}$, $\mathbf{\tilde{f_{0}}}$, $\bar{f0}$ and $\theta_{AR}$ indicate
acoustic units, prompt encoder, prompt, lyrics, durations, melody
representation, vocal-range factor and model parameters, respectively, and
$t$, $c$ indicate temporal and codebook indices of the acoustic unit. Consider
the process of the transformer predicting the vocal range factor, which is
formulated by

$P_{cond}\left(\bar{f_{0}}\right)=p\left(\bar{f_{0}}\mid\mathbf{E}_{\mathcal{P}}(\mathcal{P}),L,\mathbf{d},\mathbf{\tilde{f_{0}}};\theta_{AR}\right),$
(4)

as we assume that the average F0 value is independent of the lyrics, duration
and melody, this formula indicates our model’s capability to control the vocal
range through natural language prompts. The predicted vocal range information
is further taken as a condition for singing acoustic unit generation.

The hierarchical structure of the multi-scale transformer is illustrated in
Figure 2(b). This structure is formed by a global and a local transformer,
both of which are decoder-only transformers. For a temporal position $t$,
embeddings $z^{1:n_{q}}_{t}$ of acoustic units from different codebooks are
concatenated and fed to the global transformer for inter-frame correlation
modeling. The output hidden feature $h_{t}$ is generated autoregressively
conditioned on $h_{1:t-1}$. This hidden feature is then split according to the
original shape of the embeddings, projected by a linear layer, and added to
the input embeddings of the local transformer as a frame-level context. The
local transformer predicts acoustic units of different codebooks inside a
frame autoregressively. For non-acoustic modalities, each item is repeated
$n_{q}$ times to fit this modeling mechanism, with $n_{q}$ being the number of
codebooks.

###  4.6 Unit Vocoder

When the acoustic unit generation finishes, the generated units need to be
mapped to a high-fidelity audio waveform. Due to the compressive nature of the
codec, reconstructing audio from acoustic units of limited codebooks with the
decoder may result in degraded perceptual quality. Instead of using the codec
decoder directly, we adopt a GAN-based unit vocoder for singing voice
reconstruction, aiming to generate audio of higher quality and richer details.
Specifically, our vocoder is derived from BigVGAN Lee et al. (2022), with a
generator built from a set of LUTs that embed the discrete units, and a series
of blocks composed of transposed convolution and a residual block with dilated
layers. Multi-period and multi-resolution discriminators (MPD, MRD) are used
for adversarial training.

##  5 Experiments

###  5.1 Datasets

We combine 4 SVS datasets for our task, including M4Singer, Opencpop,
Opensinger and PopCS, forming a multi-singer singing dataset of 127 hours. For
speech data, we utilize 4 Mandarin TTS corpora, including AISHELL-3, Biaobei,
THCHS-30 and a subset of DidiSpeech, totaling approximately 179 hours. We
provide details of these datasets in Appendix B.

We phonemize the lyrics with PyPinyin111https://github.com/mozillazg/python-
pinyin, and extract F0 from raw audios with harvest Morise et al. (2017). We
separately select 2% of the singing data randomly for validation and testing,
with the remaining used for training.

###  5.2 Model Configurations

The global transformer has 20 layers with 320M parameters, while the local
transformer has 6 layers with 100M parameters. Both of them share the same
hidden dimension of 1152. For acoustic units, we train a SoundStream model for
24k audio, with 12 quantization levels, a codebook size of 1024 and a
downsampling rate of 480. We use the first 3 quantization levels as the
acoustic units, and the unit vocoder is trained to reconstruct 24k audios from
acoustic units of 3 codebooks. The label dropping probability $p_{1}$ and
$p_{2}$ are both set to 0.05. Detailed structure and hyper-parameters of the
model are appended in Appendix C.

###  5.3 Experiment Settings

ID | Model | Gender (F/M) | Volume | Range | R-FFE | MOS | RMOS
---|---|---|---|---|---|---|---
Prompt-Singer with Pre-trained Text Encoders |  |
1 | FLAN-T5 small | 76.7 / 78.1 | 92.0 | 79.1 | 0.11 | 3.75 $\pm$ 0.08 | 3.27 $\pm$ 0.09
2 | FLAN-T5 base | 82.2 / 79.5 | 92.4 | 80.8 | 0.12 | 3.79 $\pm$ 0.07 | 3.39 $\pm$ 0.07
3 | FLAN-T5 large | 83.1 / 80.8 | 92.7 | 82.6 | 0.12 | 3.83 $\pm$ 0.08 | 3.43 $\pm$ 0.08
4 | FLAN-T5 XL | 83.4 / 80.4 | 92.6 | 82.9 | 0.11 | 3.84 $\pm$ 0.06 | 3.46 $\pm$ 0.08
5 | BERT-base | 80.8 / 80.1 | 93.9 | 80.1 | 0.10 | 3.81 $\pm$ 0.06 | 3.42 $\pm$ 0.07
6 | BERT-large | 84.9 / 80.9 | 94.3 | 78.9 | 0.09 | 3.78 $\pm$ 0.08 | 3.44 $\pm$ 0.08
7 | CLAP-general | 82.2 / 79.5 | 94.1 | 80.3 | 0.12 | 3.83 $\pm$ 0.07 | 3.43 $\pm$ 0.06
8 | CLAP-speech/music | 82.2 / 78.1 | 94.2 | 80.8 | 0.11 | 3.85 $\pm$ 0.09 | 3.38 $\pm$ 0.08
Prompt-Singer with Fine-tuned Text Encoders |  |
9 | FLAN-T5 large finetuned |  87.7 / 86.3 | 94.4 | 84.7 | 0.12 | 3.89 $\pm$ 0.07 | 3.62 $\pm$ 0.08
10 | BERT-large finetuned | 86.3 / 83.6 | 94.9 | 79.8 | 0.10 | 3.90 $\pm$ 0.07 | 3.60 $\pm$ 0.08
Non-controllable SVS models and Ground Truth |  |
11 | FFT-Singer | / | / | / | 0.17 | 3.67 $\pm$ 0.08 | /
12 | Diffsinger | / | / | / | 0.09 | 3.86 $\pm$ 0.07 | /
13 | Ground Truth | 98.0 / 97.0 | / | / | / | 4.09 $\pm$ 0.06 | /
Table 1: Results on different text representations, including percentage accuracies of the three attributes, rescaled f0-frame error (R-FFE) and mean-opinion-scores of audio quality (MOS) and relevance to the prompt (RMOS). ID | SVS Data | TTS Data | Gender (F/M) | Volume | Range | R-FFE | MOS | RMOS
---|---|---|---|---|---|---|---|---
1 | ✓ | ✗ | 75.3 / 65.8 | 87.6 | 78.7 | 0.11 | 3.68 $\pm$ 0.08 | 3.37 $\pm$ 0.08
2 | ✓ | ✓ | 87.7 / 86.3 | 94.4 | 84.7 | 0.12 | 3.89 $\pm$ 0.07 | 3.62 $\pm$ 0.08
3 | 10min | 100h | 65.8 / 65.6 | 78.3 | 80.9 | 0.29 | 3.06 $\pm$ 0.09 | 2.89 $\pm$ 0.09
4 | 1h | 100h | 71.2 / 64.4 | 84.8 | 81.2 | 0.25 | 3.34 $\pm$ 0.08 | 3.03 $\pm$ 0.09
5 | 10h | 100h | 76.7 / 68.5 | 88.6 | 81.6 | 0.23 | 3.28 $\pm$ 0.08 | 3.17 $\pm$ 0.09
6 | 100h | 100h | 86.2 / 80.5 | 92.5 | 82.3 | 0.12 | 3.75 $\pm$ 0.08 | 3.45 $\pm$ 0.08
Table 2: Experiment results on data scarcity alleviation in low resource
scenarios.

As we are investigating a new task with no previous work to compare with, our
experiments mainly focus on exploring different settings within our framework,
including different text representations and different training data
compositions, together with ablation studies. The settings of various text
representations are presented in table 1. As described in Section 4.2, we
experimented with encoders of different types, parameter sizes, and pre-
training data as well as fine-tuning the encoders. We also provide the results
of ground truth and two non-controllable SVS models in table 1 as baselines of
singing quality: 1) FFT-Singer, which generates mel-spectrograms through
stacked feed-forward transformer blocks; and 2) DiffsingerLiu et al. (2022),
an SVS model based on the diffusion probabilistic model.

In table 2, we compare the results of incorporating speech data for training
or not, together with a series of low-resource data configurations with SVS
data varying from 10 minutes to 100 hours paired with speech data of a fixed
quantity of 100 hours. The ablation studies are described in a dedicated
subsection.

###  5.4 Metrics

We employ both subjective and objective metrics to measure the controlling
ability and singing voice quality of the models. For objectives metrics, we
calculate the percentage accuracy for each attribute, where we train a gender
classifier and use amplitude RMS and average F0 of the voiced part for volume
and range evaluation. We mainly use single-attribute prompts for evaluation
with an additional gender attribute for vocal range, and multi-attribute
evaluation is conducted in ablation studies. We also calculate R-FFE for
melodic accuracy between the synthesized and reference singing, which is
F0-frame-error (FFE) with the voiced part of F0 rescaled to have an average of
230Hz to eliminate the impact of vocal range. For subjective metrics, we use
crowd-sourced human evaluation via Amazon Mechanical Turk, where raters are
asked to rate scores on 1-5 Likert scales on singing voice quality and the
relevance between synthesized singing and the prompt. We report the mean-
opinion-scores of quality (MOS) and relevance (RMOS) with 95% confidence
intervals (CI) in the tables. Details of evaluation metrics are provided in
Appendix D.

###  5.5 Results and Analysis

We can draw two basic conclusions from the results in table 1: 1) Generally,
our models (1-10) exhibit favorable attribute controlling accuracies, with the
best values being 87.7 / 86.3, 94.9 and 84.7 for the three attributes,
together with competitive audio quality and melodic accuracy to non-
controllable baselines (1-10 v.s. 11-13), with the best R-FFE and MOS being
0.09 and 3.90. This indicates the effectiveness of our model design on the
task of controllable SVS. 2) The accuracies on volume are higher than gender
and vocal range by a salient margin, with the values varying between 7.4 and
15.4 across different models. We speculate that this is because the random
amplitude scaling in training allows the data with different volumes to be
expanded to a large scale (somewhat similar to data augmentation), while the
quantities and diversities of gender and range are limited by the training
datasets. This, from one perspective, confirms that data scarcity makes
learning the correlation between prompt and style attributes difficult.

ID | Model | Gender (F/M) | Volume | Range | R-FFE | RMOS
---|---|---|---|---|---|---
Ablation on Decoupled Pitch Representation
1 | Factor: ✓ Rescale: ✓ | 87.7 / 86.3 | 94.4 | 84.7 | 0.12 | 3.62 $\pm$ 0.08
2 | Factor: ✗ Rescale: ✓ | 78.1 / 63.0 | 91.3 | 76.1 | 0.11 | 3.34 $\pm$ 0.09
3 | Factor: ✗ Rescale: ✗ | 64.4 / 58.9 | 91.6 | 72.3 | 0.08 | 2.75 $\pm$ 0.09
Ablation on Different Prompted Attribute Numbers
4 | Attribute Num: 1 | 87.7 / 86.3 | 94.4 | / | 0.12 | 3.67 $\pm$ 0.08
5 | Attribute Num: 2 | 84.3 / 82.9 | 93.4 | 84.7 | 0.11 | 3.58 $\pm$ 0.08
6 | Attribute Num: 3 | 81.2 / 80.7 | 93.0 | 82.4 | 0.11 | 3.52 $\pm$ 0.07
Table 3: Results of ablation studies.

####  5.5.1 Evaluation on Text Representations

We have the following further observations from the results in table 1: 1)
Fine-tuning the text encoders leads to a considerable improvement in
controlling accuracy (3 vs. 9 & 6 vs.10), with the improvements being 4.6 /
5.5, 1.7 and 2.1 for FLAN-T5 large, and 1.4 / 2.7, 0.6 and 0.9 for BERT-large.
This indicates that aligning the text representations with the labels, which
have a much simpler distribution, helps the model learn their correlation with
singing style. Nevertheless, using only the pre-trained text encoders already
yields quite good results. 2) Generally, larger model sizes bring better
results (1-4 & 5-6). However, such a tendency between 3 and 4 is less
significant compared to 1-2 and 2-3, suggesting that text encoder parameters
beyond a certain size are no longer a bottleneck for model performance. 3)
Different types of text encoders exhibit varying controlling capabilities over
different attributes. For instance (1-4 vs. 5-8), the FLAN-T5 family shows
weaker control over volume compared to CLAP and BERT, with an accuracy gap of
1.2-2.3. However, the large and xl models outperform CLAP and BERT in vocal-
range controlling accuracy by 1.8-4.0. This may be related to differences in
the models’ pretraining methods and data. We choose the fine-tuned FLAN-T5
large model for subsequent experiments.

####  5.5.2 Evaluation on Data Scarcity Alleviation

From the results of different data compositions in table 2, we have the
following observations: 1) Introducing speech data leads to a comprehensive
improvement in controlling accuracies and generation quality, with the cost
being a slight increase in R-FFE of 0.01 (1 vs. 2). This is because the
additional speech data increases the quantity and diversity of the training
data, aiding the network in modeling the correlation between prompt and
acoustic style. However, due to the difference in the distributions of singing
melody and speech prosody, both of which are manifested in pitch variation,
the speech data may have a negative impact on modeling singing melody, causing
the slight increase in R-FFE. 2) In the low resource scenarios (3-6), we find
that there is a drastic decline in the singing audio quality, melody accuracy
as well as the accuracy on gender with the decrease in the quantity of SVS
data. In contrast, the changes in volume and vocal range are relatively
gradual, yielding acceptable results of 88.6 and 81.6 even with 10 hours of
singing data. This suggests that, while speech data helps improve controlling
accuracy and audio quality, it still cannot substitute for singing data in
modeling certain vocal characteristics. In conclusion, introducing speech data
effectively enhances the performance of controllable SVS, but it is still
necessary to have a sufficient amount of singing data to ensure synthesis
quality and melody accuracy.

###  5.6 Ablation Studies

We mainly focus on validating the effectiveness of our decoupled pitch
representation and multi-attribute prompting mechanism in the ablation
studies, and the results are presented in table 3.

For pitch representation (1-3), we first remove the vocal range factor from
the sequence, and then eliminate the rescaling on the input F0. We can see
that when removing the range factor, there is a drastic drop of 9.6 / 23.3,
3.1 and 8.6 in accuracies, accompanied by an RMOS decrease of 0.28. This
indicates that explicitly predicting the vocal range factor facilitates vocal
range and gender control greatly. When we continue to eliminate the input F0
rescaling, the accuracies on gender and range as well as RMOS further decline
by 13.7 / 4.1, 3.8 and 0.59, respectively, which indicates that the vocal
range information contained in the original F0 interferes with the model’s
modeling of the correlation between prompt and singing style. We also observe
that removing the range factor and input F0 rescaling leads to an improvement
in melodic accuracy. This suggests that the decoupling mechanism may cause
some loss of pitch information. Despite this, our model keeps a satisfactory
melodic accuracy with the decoupled pitch representation.

We further examine the model’s controlling effectiveness under multi-attribute
prompts. The results of 4-6 in table 3 show that there is a slight decrease in
accuracies and RMOS as the attribute number increases, with the drop being 3.4
/ 3.4, 1.0, 0.09 from 1 to 2 attributes, and 3.1 / 2.2, 0.4, 2.3, 0.06 from 2
to 3. We suggest that this is because the conditional distribution of acoustic
style with respect to controlling signals of multiple attributes is more
complicated to be modeled. Nevertheless, our model shows favorable performance
on prompts with both single and multiple attributes.

##  6 Conclusion

In this paper, we propose Prompt-Singer, the first singing-voice-synthesis
method with the ability of style control using natural language prompts. We
adopt a multi-scale decoder-only transformer for generating acoustic units of
singing, followed by a unit-vocoder for audio reconstruction. We design a
decoupled pitch representation for vocal range modification with an accurate
melody kept. Furthermore, we investigate various experiment settings,
including different text representations, fine-tuning the text encoders, and
using speech data to boost performance in low-resource scenarios.

In future works, we plan to introduce more style attributes in controllable
SVS, such as emotion, rhythm and more detailed singer information. We hope our
work will facilitate the development of the SVS community.

##  7 Limitations and Potential Risks

Despite that our model achieves remarkable controlling capability and audio
quality on prompt singing-voice-synthesis, it still has two major limitations:
1) Due to the simplicity and inflexibility of our existing prompt generation
pipeline, the generated prompt texts may suffer from distributional bias,
manifested mainly as grammatical errors, unnatural expressions, and
restrictions in expressive capacity and diversity. We suggest that a potential
solution is to pass the assembled prompt sentences through the LLM once more
for refinement and synonymous sentence generation to improve accuracy and
expressiveness. 2) Due to the utilization of large-scale models (including the
text encoders and the transformer backbone) along with an autoregressive
generation paradigm, our model entails relatively high computational overhead,
resulting in considerable inference latency. We discuss the relationship
between inference latency and the length of the generated audio in appendix E.

Besides, misuse of our model for singing voice generation may lead to
copyright issues. We will add some constraints to guarantee people who use our
code or pre-trained model will not use the model in illegal cases.

## Acknowledgements

This work is supported by National Key R&D Program of China under Grant
No.2022ZD0162000, National Natural Science Foundation of China under Grant No.
62222211 and No.62072397.

## References

  * Borsos et al. (2023) Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al. 2023.  Audiolm: a language modeling approach to audio generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2020) Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie-Yan Liu. 2020.  Hifisinger: Towards high-fidelity neural singing voice synthesis.  _arXiv preprint arXiv:2009.01776_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dong Wang (2015) Zhiyong Zhang Dong Wang, Xuewei Zhang. 2015.  Thchs-30 : A free chinese speech corpus.
  * Elizalde et al. (2023a) Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023a.  Clap learning audio concepts from natural language supervision.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Elizalde et al. (2023b) Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. 2023b.  Natural language supervision for general-purpose audio representations.
  * Guo et al. (2021) Tingwei Guo, Cheng Wen, Dongwei Jiang, Ne Luo, Ruixiong Zhang, Shuaijiang Zhao, Wubo Li, Cheng Gong, Wei Zou, Kun Han, et al. 2021.  Didispeech: A large scale mandarin speech corpus.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6968–6972. IEEE.
  * Guo et al. (2023) Zhifang Guo, Yichong Leng, Yihan Wu, Sheng Zhao, and Xu Tan. 2023.  Prompttts: Controllable text-to-speech with text descriptions.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Hong et al. (2023) Zhiqing Hong, Chenye Cui, Rongjie Huang, Lichao Zhang, Jinglin Liu, Jinzheng He, and Zhou Zhao. 2023.  Unisinger: Unified end-to-end singing voice synthesis with cross-modality information matching.  In _Proceedings of the 31st ACM International Conference on Multimedia_ , pages 7569–7579.
  * Huang et al. (2023a) Jiawei Huang, Yi Ren, Rongjie Huang, Dongchao Yang, Zhenhui Ye, Chen Zhang, Jinglin Liu, Xiang Yin, Zejun Ma, and Zhou Zhao. 2023a.  Make-an-audio 2: Temporal-enhanced text-to-audio generation.  _arXiv preprint arXiv:2305.18474_.
  * Huang et al. (2021) Rongjie Huang, Feiyang Chen, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao. 2021.  Multi-singer: Fast multi-singer singing voice vocoder with a large-scale corpus.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pages 3945–3954.
  * Huang et al. (2022) Rongjie Huang, Chenye Cui, Feiyang Chen, Yi Ren, Jinglin Liu, Zhou Zhao, Baoxing Huai, and Zhefeng Wang. 2022.  Singgan: Generative adversarial network for high-fidelity singing voice generation.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pages 2525–2535.
  * Huang et al. (2023b) Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. 2023b.  Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models.  In _International Conference on Machine Learning_ , pages 13916–13932. PMLR.
  * Huang et al. (2023c) Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2023c.  Audiogpt: Understanding and generating speech, music, sound, and talking head.  _arXiv preprint arXiv:2304.12995_.
  * (17) Rongjie Huang, Yi Ren, Jinglin Liu, Chenye Cui, and Zhou Zhao.  Generspeech: Towards style transfer for generalizable out-of-domain text-to-speech.  In _Advances in Neural Information Processing Systems_.
  * Huang et al. (2023d) Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Luping Liu, Zhenhui Ye, Ziyue Jiang, Chao Weng, Zhou Zhao, and Dong Yu. 2023d.  Make-a-voice: Unified voice synthesis with discrete representation.  _arXiv preprint arXiv:2305.19269_.
  * Kreuk et al. (2022) Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022.  Audiogen: Textually guided audio generation.  _arXiv preprint arXiv:2209.15352_.
  * Lee et al. (2022) Sang-gil Lee, Wei Ping, Boris Ginsburg, Bryan Catanzaro, and Sungroh Yoon. 2022.  Bigvgan: A universal neural vocoder with large-scale training.  _arXiv preprint arXiv:2206.04658_.
  * Leng et al. (2023) Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, et al. 2023.  Prompttts 2: Describing and generating voices with text prompt.  _arXiv preprint arXiv:2309.02285_.
  * Liu et al. (2022) Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao. 2022.  Diffsinger: Singing voice synthesis via shallow diffusion mechanism.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 36, pages 11020–11028.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_ , volume 2017, pages 498–502.
  * Morise et al. (2017) Masanori Morise et al. 2017.  Harvest: A high-performance fundamental frequency estimator from speech signals.  In _INTERSPEECH_ , pages 2321–2325.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR.
  * Shen et al. (2023) Kai Shen, Zeqian Ju, Xu Tan, Yanqing Liu, Yichong Leng, Lei He, Tao Qin, Sheng Zhao, and Jiang Bian. 2023.  Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers.  _arXiv preprint arXiv:2304.09116_.
  * Shi et al. (2020) Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, and Ming Li. 2020.  Aishell-3: A multi-speaker mandarin tts corpus and the baselines.  _arXiv preprint arXiv:2010.11567_.
  * Von Helmholtz (1912) Hermann Von Helmholtz. 1912.  _On the Sensations of Tone as a Physiological Basis for the Theory of Music_.  Longmans, Green.
  * Wang et al. (2023) Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. 2023.  Neural codec language models are zero-shot text to speech synthesizers.  _arXiv preprint arXiv:2301.02111_.
  * Wang et al. (2022) Yu Wang, Xinsheng Wang, Pengcheng Zhu, Jie Wu, Hanzhao Li, Heyang Xue, Yongmao Zhang, Lei Xie, and Mengxiao Bi. 2022.  Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis.  _arXiv preprint arXiv:2201.07429_.
  * Wu et al. (2023) Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023.  Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.
  * Yang et al. (2023a) Dongchao Yang, Songxiang Liu, Rongjie Huang, Guangzhi Lei, Chao Weng, Helen Meng, and Dong Yu. 2023a.  Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt.  _arXiv preprint arXiv:2301.13662_.
  * Yang et al. (2023b) Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, et al. 2023b.  Uniaudio: An audio foundation model toward universal audio generation.  _arXiv preprint arXiv:2310.00704_.
  * Yao et al. (2023) Jixun Yao, Yuguang Yang, Yi Lei, Ziqian Ning, Yanni Hu, Yu Pan, Jingjing Yin, Hongbin Zhou, Heng Lu, and Lei Xie. 2023.  Promptvc: Flexible stylistic voice conversion in latent space driven by natural language prompts.  _arXiv preprint arXiv:2309.09262_.
  * Yu et al. (2024) Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2024.  Megabyte: Predicting million-byte sequences with multiscale transformers.  _Advances in Neural Information Processing Systems_ , 36.
  * Zeghidour et al. (2021) Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. 2021.  Soundstream: An end-to-end neural audio codec.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 30:495–507.
  * Zhang et al. (2022a) Lichao Zhang, Ruiqi Li, Shoutong Wang, Liqun Deng, Jinglin Liu, Yi Ren, Jinzheng He, Rongjie Huang, Jieming Zhu, Xiao Chen, et al. 2022a.  M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus.  _Advances in Neural Information Processing Systems_ , 35:6914–6926.
  * Zhang et al. (2022b) Yongmao Zhang, Jian Cong, Heyang Xue, Lei Xie, Pengcheng Zhu, and Mengxiao Bi. 2022b.  Visinger: Variational inference with adversarial learning for end-to-end singing voice synthesis.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7237–7241. IEEE.
  * Zhang et al. (2023a) Yongmao Zhang, Guanghou Liu, Yi Lei, Yunlin Chen, Hao Yin, Lei Xie, and Zhifei Li. 2023a.  Promptspeaker: Speaker generation based on text descriptions.  _arXiv preprint arXiv:2310.05001_.
  * Zhang et al. (2022c) Zewang Zhang, Yibin Zheng, Xinhui Li, and Li Lu. 2022c.  Wesinger: Data-augmented singing voice synthesis with auxiliary losses.  _arXiv preprint arXiv:2203.10750_.
  * Zhang et al. (2023b) Zewang Zhang, Yibin Zheng, Xinhui Li, and Li Lu. 2023b.  Wesinger 2: fully parallel singing voice synthesis via multi-singer conditional adversarial training.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 1–5. IEEE.

##  Appendix A Sample Prompt Keywords and Sentence Templates

We list the keywords for each category in table 4, and provide some samples of
prompt sentence templates in table 6.

Category | Keywords
---|---
Gender
female | woman, lady, girl, female, lass, miss, madam
male | man, boy, guy, gentleman, male, sir
Volume
high | loud, ringing, booming, thunderous,
deafening, roaring
medium | moderate, average, intermediate,
middle-range
low | quiet, slight, twittering, hushed, whispering
Vocal Range
high | sharp, treble, shrill, whistling,
shrieking, high-pitched
low | deep, low, bass, thick, low-pitched
Table 4: Prompt keywords for each category.

##  Appendix B Dataset Statistics

In table 5, we list the statistics of the datasets used. F and M in the
Speakers column indicate the numbers of female and male speakers or singers.

Dataset |  Hours |  Speakers
---|---|---
SVS datasets
M4Singer Zhang et al. (2022a) | 29.8 | F:10 M:10
Opencpop Wang et al. (2022) | 5.3 | F:1
Opensinger Huang et al. (2021) | 86.5 | F:49 M:28
PopCS Liu et al. (2022) | 5.9 | F:1
TTS datasets
AISHELL-3 Shi et al. (2020) | 86.4 | F:176 M:42
Biaobei 222https://www.data-baker.com/open_source.html | 11.8 | F:1
THCHS-30 Dong Wang (2015) | 34.2 | F:31 M:9
Didispeech Guo et al. (2021) | 47.0 | F:198 M:202
Table 5: Statistics of training datasets. Single-Attribute Templates
---
Do you have any songs with a [gender] lead singer?
Can you create a song sung by a [gender] vocalist?
I’m searching for a song featuring a [gender] singer.
I need a song with a [volume] voice that resonates.
Play me a song with a [volume] voice.
I’d like to listen to a song with a [volume] voice.
I need a song where every note is gentle and delicate. (for low volume)
Kindly provide me with a song that features a voice of balanced volume,
pleasing to the ears. (for medium volume)
Give me a song with a voice that shakes the ground with its thunderous vocals!
(for high volume)
Double-Attribute Templates
Can you find me a song with a [gender] singer and a [volume] voice?
I would like to hear a song with a [volume] voice and if possible, a [gender]
voice.
Synthesize a new song with a [volume] voice and a [gender] lead singer.
Need a [pitch] pitch song sung by a [gender] vocalist.
Generate a song featuring a [gender] vocalist with a unique use of [pitch]
pitch.
A [gender] voice with a [pitch] pitch is what I’m looking for.
Create an enchanting song sung by a [gender] vocalist in the [pitch] pitch.
Create a [gender] artist’s song with a [volume] voice, softly mesmerizing with
its gentle tone. (for low volume + any gender)
Generate a [gender] artist singing at just the right volume. (for medium
volume + any gender)
Can you generate a [gender]-sung song with a [volume] voice that balances
softness and loudness? (for medium volume + any gender)
I’m looking for a song with a [gender] singer and a voice that’s as powerful
as a thunderstorm. (for high volume + any gender)
Triple-Attribute Templates
Explore [gender] [volume] songs with emotive [pitch] pitch.
Synthesize a song with a [pitch] pitch and a [volume] voice, preferably
[gender].
Design a [gender] singer’s song with a [volume] voice and [pitch] pitch.
Showcasing superb [pitch] pitch, create a [volume] song by a [gender] artist.
Generate a song with stunning [pitch] harmonies and a [gender] singer with a
[volume] voice.
Can you compose a song with a [gender] vocalist and [volume] volume, while
incorporating the singer’s unique use of [pitch] pitch?
Generate a song featuring [gender] vocals, delicately whispered with [volume]
voice and [pitch] harmony. (for low volume + any gender / vocal range)
Compose a [pitch]-keyed song with a [volume] voice that balances softness and
loudness, sung by a [gender] singer. (for medium volume + any gender / vocal
range)
Craving a [gender] artist’s song with a [volume] voice that exudes energy and
power and a [pitch] note that creates a memorable hook! (for high volume + any
gender / vocal range)
Table 6: Sample prompt sentence templates.

##  Appendix C Model Settings

We illustrate the architecture of the global transformer in Figure 3. The
local transformer shares the same structure as the global one with two
differences: 1) the local transformer has no positional embedding, and 2)
there is a linear lm-head appended to the top of it for token prediction. We
also list the model hyper-parameters of Prompt-Singer in Table 7. The multi-
scale transformer is trained with 6 NVIDIA-V100 gpus for about 4-5 days, and
the vocoder is trained with 4 NVIDIA-V100 gpus for a week.

Figure 3: Structure of Global Transformer Hyperparameter | Prompt-Singer
---|---
Global Transformer | Layers | 20
Hidden Dim | 1,152
Attention Headers | 16
FFN Dim | 4,608
Number of Parameters | 320.07M
Local Transformer | Layers | 6
Hidden Dim | 1,152
Attention Headers | 8
FFN Dim | 4,608
Number of Parameters | 100.13M
Unit Vocoder | Upsample Rates | [6,5,2,2,2,2]
Hop Size | 480
Upsample Kernel Sizes | [12,9,4,4,4,4]
Number of Parameters | 125.43M
Table 7: Hyperparameters of Prompt-Singer.

##  Appendix D Evaluation Metrics

###  D.1 Objective Evaluation

Figure 4: Soft-margin accuracy curve of high vocal-range of male. Figure 5:
Soft-margin accuracy curve of medium volume.

For gender controlling accuracy, we train an open-source gender
classifier333https://github.com/x4nth055/gender-recognition-by-
voice/tree/master with our singing and speech data. The performance of the
classifier on the test set is provided as ground-truth accuracy in line 13 of
table 1.

For controlling accuracies on volume and vocal range, considering that the
values of generated singing may slightly deviate from the boundaries used for
categorization, we adopt a soft-margin mechanism for accuracy calculation.
Specifically, we take the accuracy of data falling within the correct range as
100, and calculate the accuracy with $100*\exp{(-k\epsilon)}$ for data outside
the correct range, where $\epsilon$ is the error between the data value and
the boundary, and $k$ is a hyper-parameter controlling the decay rate of
accuracy at the margins, with larger $k$ corresponding to faster decay. We
take accuracy curves of high vocal-range of male and medium volume as examples
and illustrate them in Figure 4 and 5, respectively. We set $k$ to 120, 150
and 180 for high, medium and low volume, and 0.2 for vocal range accuracy.

###  D.2 Subjective Evaluation

For each evaluated model, we mix all generated results together and randomly
select 220 items with their corresponding prompts for subjective evaluation.

Our subjective evaluation tests are crowd-sourced and conducted via Amazon
Mechanical Turk. For audio quality evaluation, we ask the testers to examine
the audio quality and naturalness and ignore the content. For prompt-style
relevance, we instruct the testers to evaluate the relevance between the
natural language prompt and the singing style while ignoring the content. The
testers rate scores on 1-5 Likert scales. We provide screenshots of the
testing interfaces in Figure 6 and 7. Each data item is rated by 4 testers,
and the testers are paid $8 hourly.

Figure 6: Screenshot of MOS testing. Figure 7: Screenshot of RMOS testing.

##  Appendix E Inference Efficiency

To give an intuitive impression of our model’s inference efficiency, we
visualize the relationship between model inference latency and the length of
the generated audio in Figure 8, including the acoustic unit generation stage
with two types of text encoder, together with the wave reconstruction stage.
The inference is conducted on a single NVIDIA-V100 GPU. It can be observed
that the major latency comes from the transformer backbone, and it increases
with the length of the sequence; on the other hand, the latency of the non-
autoregressive vocoder is minimal and not significantly affected by the
sequence length.

(a) Latency of acoustic unit generation

(b) Latency of wave reconstruction

Figure 8:  Inference latency at varying lengths of generated audio.


# ParrotTTS: Text-to-Speech synthesis by exploiting
self-supervised representations

Saiteja Kosgi 1 Neil Kumar Shah1,2 Vishal Tambrahalli 1
Neha Sherin1 Vineet Gandhi1
1Kohli Centre on Intelligent Systems, IIIT Hyderabad
2TCS Research, Pune
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Text-to-speech (TTS) systems are modelled as mel-synthesizers followed by
speech-vocoders since the era of statistical TTS that is carried forward into
neural designs. We propose an alternative approach to TTS modelling referred
to as ParrotTTS borrowing from self-supervised learning (SSL) methods.
ParrotTTS takes a two-step approach by initially training a speech-to-speech
model on unlabelled data that is abundantly available, followed by a text-to-
embedding model that leverages speech with aligned transcriptions to extend it
to TTS. ParrotTTS achieves competitive mean opinion scores on naturalness
compared to traditional TTS models but significantly improves over the
latter’s data efficiency of transcribed pairs and speaker adaptation without
transcriptions. This further paves the path to training TTS models on
generically trained SSL speech models. Speech samples from ParrotTTS can be
found at https://parrottts.github.io/tts/

##  1 Introduction

Vocal learning forms the first phase of infants starting to talk Locke (1996,
1994). In this phase, the learning happens by simply listening to
sounds/speech. Studies show that vocal learning begins in the final trimester
of pregnancy; the normally developing fetus can hear its mother’s voice within
the womb Kolata (1984). Several studies show that the best way to promote
language development for babies is to talk to them. It is hypothesized Kuhl
and Meltzoff (1996) that infants listening to ambient language store
perceptually derived representations of the speech sounds they hear, which in
turn serve as targets for the production of speech utterances. Interestingly,
in this phase, the infant has no conception of text or linguistic rules, and
speech is considered sufficient to influence speech production Kuhl and
Meltzoff (1996). Eventually, if parrots can talk without understanding
language, there is no reason human infants should need to possess grammatical
capability either to comprehend and produce speech Locke (1994).

Figure 1:  (a) Traditional TTS and (b) Proposed TTS model

We propose a novel design for text-to-speech synthesis called ParrotTTS that
follows a similar learning process. Our idea mimics the-step approach, with
the first learning to produce sounds capturing the whole gamut of phonetic
variations. It is attained by learning quantized representations of sound
units in a self-supervised manner. The second phase builds on top of the first
by learning a mapping from text to the quantized representations (embeddings).
This step uses paired text-speech data. The two phases are analogous to first
learning to talk followed by learning to read.

Our proposed ParrotTTS is illustrated in Figure 1(b) distinguishing it from
traditional design in Figure 1(a). The self-supervised module learns discrete
speech representations using raw audio data from multiple speakers without
aligned transcriptions similar to Wav2Vec 2.0 Baevski et al. (2020) or Hubert
Hsu et al. (2021). The SSL module includes a speech-to-embedding (STE) encoder
trained on masked prediction task to generate the intermediate representation
of audio input. An embedding-to-speech (ETS) decoder is independently trained
to invert embeddings to synthesize audio waveforms and is additionally
conditioned on speaker identity. This learning to talk is the first of the
two-step training pipeline.

In the subsequent learning to read step, a separate text-to-embedding (TTE)
encoder is trained to generate embeddings from text (or equivalent phonetic)
inputs. This step requires labeled speech with aligned transcriptions.
However, the data requirement in this step is very low in terms of volume and
number of speakers. We show that transcribed samples from even a single
speaker suffices to learn phonetic mapping (TTE) sufficiently well for
generalization on a large number of speakers. Further, the decoder ETS can be
conditioned on speaker identity to change the voice of rendered speech. In our
model, the speech embeddings can be obtained either from the text (using TTE)
or directly from audio (using STE), providing a unified model for speech
synthesis, of which we limit the scope of this work to only text-to-speech.

Overall, the restructuring of learning components has effectively changed the
data dependence equation in our favor, cutting down the amount of transcribed
data needed by leveraging abundant raw audio to achieve similar speech
quality. This further makes it easy to extend the model to de novo voices
unseen in initial training by independently fine-tuning the ETS decoder module
on untranscribed audio from the corresponding speakers. Also, the ParrotTTS’
components are functionally different from that of traditional synthesizer-
vocoder design. This offers several other benefits.

  1. 1.

For instance, our speech embedding has lower variance than that of Mel frames
reducing the complexity to train TTE and increasing capacity of downstream
ETS. We observe that, for example, our embeddings are speaker agnostic,
requiring ETS conditioning on speaker identity for speaker adaptation.

  2. 2.

Speaker agnostic speech embeddings paired with independently trained STE
disentangled speaker handling from content. This enabled adaptation to novel
voices with untranscribed speech alone. The data requirement is placed between
zero-shot methods that use speaker-embedding but are poor in quality and
traditional TTS requiring fully transcribed speech while its quality matches
the latter.

  3. 3.

Segregation of functions pushed acoustic handling into ETS module towards the
end that directly infers the speech signal without going through Mel frames.
This bypasses potential vocoder generalization issues Kim et al. (2021)
similar to FastSpeech2s Ren et al. (2020).

  4. 4.

Reduced complexity helps in stabler training of TTE encoder for either
autoregressive or non-autoregressive choice. For example, we observe at least
eight-fold faster convergence in training iterations of our TTE module
compared to that of Ren et al. (2020) and Wang et al. (2017).

The main contribution of this work is the novel ParrotTTS architecture
detailed in Section 3. It redesigns the standard synthesizer-vocoder neural
TTS to leverage self-supervised learning from which the various benefits
listed above flow. We train multiple models of the proposed ParrotTTS approach
with different choices and study their effects like the quality of rendered
speech, data efficiency, word-error rates upon transcription of speech output,
etc., see Section 4. Experimental results reported in Section 5 consistently
point to the competitive or superior performance of ParrotTTS relative to the
current state-of-the-art for TTS. While these observations are of significant
value to practitioners in evaluating the adoption of ParrotTTS approach for
speech synthesis, numerous questions need further investigation. We conclude
in Section 6 with a discussion of these questions and the related topics that
need further exploration to better understand the proposed approach.

##  2 Related work

TTS systems have been studied for decades now, with the concatenative
statistical models from earlier attempts (Hunt and Black, 1996; Cohn and
Zellou, 2020) being increasingly replaced by neural variants in recent years
Oord et al. (2016). We specifically review the popular and better-performing
supervised models in Section 2.1 and their unsupervised counterparts in
Section 2.2. These references help understand data challenges for TTS training
and how their quality is observed to vary with the degree of supervision.
Towards the end of this section, we review the self-supervised learning
approach that ParrotTTS leverages with pointers to its application in other
domains.

###  2.1 Supervised TTS

A typical neural TTS model has an acoustic synthesizer that generates
frequency-domain Mel-spectrogram frames. The synthesizer has an encoder that
maps text or phonetic inputs to hidden states, followed by a decoder that
generates Mels from the hidden states. Predicted Mel frames contain all the
necessary information to reconstruct speech (Griffin and Lim, 1984) and an
independently trained vocoder (Oord et al., 2016; Kong et al., 2020)
transforms them into time-domain waves. Mel predicting decoders could be
autoregressive models (Wang et al., 2017; Valle et al., 2020; Shen et al.,
2018) that generate the Mel frames in sequential order. It conditions the
generation of a Mel frame at any time instant on all preceding predictions and
the encoder output using attention modules Graves (2013). In contrast, non-
autoregressive or parallel models (Ren et al., 2019, 2020; Łańcucki, 2021)
predict intermediate features like duration, pitch, and energy for each
phoneme. Mel frames of all time instants are then generated simultaneously
from these predicted intermediate features. Non-autoregressive models are
quicker at inference and robust to word skipping or repetition errors Ren et
al. (2020).

The quality and quantity of transcribed audio used in TTS training are known
to impact the quality of speech rendered. Public data with about $24$ hours of
studio recorded audio is known to train reasonable quality single-speaker
models (Ito and Johnson, 2017). This becomes more demanding in a multi-speaker
setting requiring sufficient per-speaker audio to learn all voices well Veaux
et al. (2017). Speaker conditioning of the decoder is commonly achieved by
one-hot-encoding of those seen at train time. Alternatively, speaker
embeddings (Jia et al., 2018) could be used for decoder conditioning which in
theory could render speech for de novo voices not part of the training set.
However, speech rendered through this method is known to be of poorer quality
and naturalness, especially for speakers not sufficiently represented in the
train set (Tan et al., 2021).

###  2.2 Raw-audio for TTS

Unsupervised speech synthesis Ni et al. (2022) does not require transcribed
text-audio pairs for the TTS acoustic modeling. They typically employ
unsupervised automatic speech recognition (ASR) model (Baevski et al., 2021;
Liu et al., 2022a) to transcribe raw speech to generate pseudo labels.
However, their performance tends to be bounded by the performance of the
unsupervised ASR model, which still has to close a significant gap compared to
supervised counterparts Baevski et al. (2021). Furthermore, switching to a
multi-speaker setup worsens quality relative to fully supervised models Liu et
al. (2022b).

Some prior works have looked at adapting TTS to novel speakers using
untranscribed audio Yan et al. (2021); Luong and Yamagishi (2019); Taigman et
al. (2017). Unlike ours, these methods require a large amount of paired data
from multiple speakers during initial training. Some of these Luong and
Yamagishi (2019); Taigman et al. (2017) jointly train the TTS pipeline and the
modules for speaker adaptation but the model convergence gets tricky. In
contrast, ParrotTTS benefits from the disentanglement of linguistic content
from speaker information, making adaptation easier.

Figure 2: Schematic diagram of the proposed model.

###  2.3 Self-supervised learning

Self-supervised learning (SSL) methods have become increasingly popular in
numerous applications owing to their ability to leverage copious amounts of
unlabeled data to learn large models that can be fine-tuned for multiple tasks
later. They are reported to achieve results better than supervised models
trained on fewer labeled samples and have found applications in computer
vision He et al. (2022), natural language processing Devlin et al. (2018);
Vaswani et al. (2017) and audio processing Schneider et al. (2019). Mask
prediction, temporally contrastive learning, next-step prediction, etc., are
some common techniques to train SSL models. Wav2vec2 Baevski et al. (2020),
Hubert Hsu et al. (2021) are popular SSL models for speech processing and ASR
Baevski et al. (2020), phoneme segmentation Kreuk et al. (2020), and spoken
language modeling Lakhotia et al. (2021), speech resynthesis Polyak et al.
(2021) are tasks that gained from leveraging them. In the same spirit, our
work explores SSL, specifically pre-trained Hubert Hsu et al. (2021), for TTS.
To the best of our knowledge, there are no known TTS models trained on SSL,
and our efforts fill this gap.

##  3 ParrotTTS architecture

As mentioned earlier, ParrotTTS has three modules; two encoders, STE and TTE
that map audio and text respectively to embedding, and a decoder ETS that maps
the embedding to the speech signal. Our speech encoder-decoder choices are
borrowed from Polyak et al. (2021). The speech encoder STE is HuBERT Hsu et
al. (2021) that maps input audio clip to discrete vectors with entries called
HuBERT units. Our speech decoder ETS is a modified version of HiFiGan Kong et
al. (2020). Text encoder TTE is an encoder-decoder architecture, and we
experiment with both autoregressive (AR) and non-autoregressive (NAR) choices
for the same. We give architectural details of these three modules below.

###  3.1 Speech encoder STE

The self-supervised HuBERT model we use for our STE is pre-trained on large
raw audio data on masked prediction task very similar to the BERT model for
text Devlin et al. (2018) to learn “combined acoustic and language model over
the continuous inputs” of speech. It borrows the base architecture from
Wav2vec 2.0 Baevski et al. (2020) with convolutions on raw inputs followed by
a few transformer layers, however, replaces its contrastive loss with a BERT-
like classification. The “noisy” classes are derived by clustering MFCC
features of short speech signals. Encoder input is audio signal
$X=(x_{1},....x_{T})$ sampled at a rate of $16$kHz. Let $E_{r}$ denote the
raw-audio encoder, and its output be,

$\mathbf{h}_{r}=(h_{1},....,{h_{\widehat{T}}})\coloneqq E_{r}(X),$

Where $\widehat{T}=T/320$ indicating downsampling and each
$h_{i}\in\\{1,\dots,K\\}$ with $K$ being a number of clusters in HuBERT’s
clustering step, set to $100$ in our experiments.

###  3.2 Speech decoder ETS

We use a modified version of HiFiGAN Kong et al. (2020) vocoder for our ETS to
decode from $\mathbf{h}=(\mathbf{h}_{r},\mathbf{h}_{s})$ to speech, where
$\mathbf{h}_{s}$ is the one-hot speaker embedding. It has a generator $G$ and
a discriminator $D$. $G$ runs $\mathbf{h}$ through transposed convolutions for
upsampling to recover the original sampling rate followed by residual block
with dilations to increase the receptive field to synthesize the signal,
$\widehat{X}\coloneqq G(\mathbf{h})$.

The discriminator distinguishes synthesized $\widehat{X}$ from the original
signal $X$ and is evaluated by two sets of discriminator networks. Multi-
period discriminators operate on equally spaced samples, and multi-scale
discriminators operate at different scales of the input signal. Overall, the
model attempts to minimize $D(X,\widehat{X})$ over all its parameters to train
ETS.

###  3.3 Text encoder TTE

The third module we train, TTE is a text encoder that maps phoneme sequence
$P=(p_{1},\dots,p_{N})$ to embedding sequence
$\mathbf{h}_{p}=(h_{1},\dots,h_{\widehat{N}})$. We train a sequence-to-
sequence architecture to achieve this $\mathbf{h}_{p}\coloneqq E_{p}(P)$.
$E_{p}$ initially encodes $P$ into a sequence of fixed dimensional vectors
(phoneme embeddings), conditioned upon which its sequence generator produces
variable dimensional $\mathbf{h}_{p}$. Embedding $\mathbf{h}_{p}$ is intended
to mimic $\mathbf{h}_{r}\coloneqq E_{r}(X)$ extracted from the audio $X$
corresponding to the text $P$. Hence, the requirement of transcribed data
$(X,P)$ to derive the target $\mathbf{h}_{r}$ for training TTE by optimizing
over the parameters of $E_{p}$.

One could model $E_{p}$ to generate $\mathbf{h}_{p}$ autoregressively one step
at a time, which we refer to as AR-TTE model. See Figure 2(b) for an
illustration. Input phoneme sequence is encoded through a feed-forward
transformer block that stacks self-attention layers Vaswani et al. (2017) and
1D convolutions similar to FastSpeech2 Ren et al. (2019). Decoding for
$\mathbf{h}_{p}$ uses a transformer module with self-attention and cross-
attention. Future-masked self-attention attends to ground truth at train and
to previous decoder predictions at inference. Cross-attention attends to
phoneme encoding in both cases.

Alternatively, for a non-autoregressive choice of $E_{p}$, the model NAR-TTE
determines the output length $\widehat{N}$ followed by a step to
simultaneously predict all $\widehat{N}$ entries of $\mathbf{h}_{p}$. Figure
2(c) depicts NAR-TTE where the input phoneme sequence encoding is similar to
that of AR-TTE. The duration predictor and length regulator modules are
responsible for determining $\widehat{N}$ followed by the decoding step to
generate $\mathbf{h}_{p}$.

| Model |  MOS $\uparrow$ |  WER $\downarrow$
---|---|---|---
Traditional TTS | SS-FastSpeech2 | 3.87 | 4.52
SS-Tacotron2 | 3.90 | 4.59
FastSpeech2-SupASR | 3.78 | 4.72
Tacotron2-UnsupASR | 3.50 | 11.3
ParrotTTS | AR-TTE${}_{\text{LJS}}$+SS-ETS | 3.85 | 4.80
NAR-TTE${}_{\text{LJS}}$+SS-ETS | 3.86 | 4.58
NAR-TTE${}_{\frac{1}{2}\text{LJS}}$+SS-ETS | 3.81 | 6.14
Table 1:  Subjective and objective comparison of studied TTS models in the
single speaker setting.

##  4 Experiments

We train multiple models of the ParrotTTS under different settings and
benchmark them against comparable models in the literature. Specifically, we
train single-speaker and multi-speaker models to evaluate naturalness,
intelligibility, and speaker adaptability. Naturalness is measured by mean-
opinion scores (MOS) from human judgments. Intelligibility is measured by
word-error rates from an ASR model on the rendered speech output. Speaker
adaptability is measured using Equal-Error-Rate from a pre-trained speaker
verification system. We perform these experiments with both autoregressive and
non-autoregressive choices of TTE.

###  4.1 ParrotTTS training

We use two public data sets for our experiments. LJSpeech Ito and Johnson
(2017) provides about 13k high-quality English transcribed audio clips
totaling about 24 hours from a single speaker. Data are split into two, with
512 samples set aside for validation and the remaining available for model
training. VCTK Veaux et al. (2017) with about 44 hours of transcribed speech
from 108 different speakers is used for the multi-speaker setup. It has a
minimum, average, and maximum of $7$, $22.8$, and $31$ minutes per speaker
speech length, respectively. All audio samples are resampled to $16$kHz before
use.

STE training. We use $12$ layer transformer model for HuBERT trained for two
epochs on $960$ hour-long LibriSpeech corpus Panayotov et al. (2015) as our
STE module to extract $\mathbf{h}_{r}$ embeddings. The model splits each $T$
seconds long audio into units of $T/320$ seconds and maps each of the obtained
units to a $768$ dimensional vector. The vectors are drawn from the network’s
activation units on the sixth layer similar to that of Lakhotia et al. (2021).
Continuous vectors are then discretized to $\mathbf{h}_{r}$ embeddings using a
codebook made from applying $k$-means (with $k$ set to $100$) to $100$ hour
subset of the data called LibriSpeech-clean Panayotov et al. (2015).

TTE training. We use LJSpeech to train two different TTE encoder modules;
TTE${}_{\textsc{LJS}}$ that uses all the data from our LJSpeech train set and
a second, TTE${}_{\frac{1}{2}\textsc{LJS}}$ with only half the data. This is
used to understand the effect of training data size on our metrics. All
variants of TTE we experiment with are trained only on samples from the single
speaker in LJSpeech data.

Text converted to phoneme sequence as described by Sun et al. (2019) are
inputs with $\mathbf{h}_{r}$ targets extracted from STE for training.
Additionally, NAR-TTE requires phonetic alignment to train the duration
predictor. We use Montreal forced-aligner McAuliffe et al. (2017) to generate
them for its training. Unlike standard TTS systems that predict Mel
spectrograms, TTE generates discrete units. Hence, we replace the mean-square
error loss used in Mels with cross-entropy with as many classes as clusters in
the discretization codebook.

ETS training. We train a single-speaker ETS, SS-ETS using only speech clips
from LJSpeech since its training does not require transcriptions. Similarly,
our multi-speaker ETS, MS-ETS decoder model uses only raw audio of all
speakers from VCTK data Veaux et al. (2017). So only embeddings
$\mathbf{h}_{r}$ extracted from VCTK audio clips are used along with one-hot
speaker vector $\mathbf{h}_{s}$. We emphasize that VCTK data were used only in
training the multi-speaker-ETS module, and the TTE has not seen any from this
set.

###  4.2 Comparison to prior art

Single Speaker TTS. We compare against state-of-the-art TTS models from the
literature of both kinds; Tacotron2 Wang et al. (2017) from among
autoregressive models and FastSpeech2 Ren et al. (2020) from the non-
autoregressive models. Both models are trained using the ground truth
transcripts of LJspeech and referred to as SS-Tacotron2 and SS-FastSpeech2.

We additionally trained an unsupervised version of FastSpeech2 by replacing
the ground truth transcripts with transcriptions obtained from the ASR model.
FastSpeech2-SupASR uses supervised ASR model Radford et al. (2022) to generate
the transcripts while Tacotron2-UnsupASR Ni et al. (2022) alternatively uses
unsupervised ASR Wav2vec-U 2.0 Liu et al. (2022a). We compare against three
variants of ParrotTTS;

  1. 1.

AR-TTE${}_{\text{LJS}}$+SS-ETS that is autoregressive TTE trained on full
LJSpeech with single speaker ETS,

  2. 2.

NAR-TTE${}_{\text{LJS}}$+SS-ETS that pairs TTE with non-autoregressive
decoding trained on full LJSpeech with single speaker ETS, and

  3. 3.

NAR-TTE${}_{\frac{1}{2}\text{LJS}}$+SS-ETS that uses TTE with non-
autoregressive decoding trained on half LJSpeech with single speaker ETS.

Multi-speaker TTS. In the multi-speaker setting, we compare against a fully
supervised Fastspeech2 baseline trained on VCTK with all its speakers using
the entire paired audio-transcript data that we refer to as MS-FastSpeech2. We
borrow the TTE module trained on LJSpeech and use the raw audio of VCTK to
train the multi-speaker ETS module. We refer to this multi-speaker variant of
our ParrotTTS model as NAR-TTE${}_{\text{LJS}}$+MS-ETS that uses non-
autoregressive decoding for TTE similar to the FastSpeech2 baseline trained on
LJSpeech alone and multi-speaker ETS trained on VCTK alone.

For a fair comparison, we also curate a multi-speaker TTS baseline using a
combination of single-speaker TTS and a voice cloning model. We use
FastSpeech2 trained on LJspeech with state-of-the-art voice cloning model
Polyak et al. (2021) in our experiments and refer to this model as VC-
FastSpeech2. We also compare against multi-speaker TTS trained by obtaining
pseudo labels from a supervised ASR called MS-FastSpeech2-SupASR. In all
multi-speaker experiments, we use one-hot encoding for speaker identity.
Additionally, we also report numbers from GT-Mel+Vocoder that converts ground
truth Mels from actual audio clip back to speech using a vocoder Kong et al.
(2020) for a perspective of best achievable with ideal Mel frames.

Model | VCTK Transcripts |  MOS $\uparrow$ |  WER $\downarrow$ |  EER $\downarrow$
---|---|---|---|---
GT-Mel+Vocoder | Yes | 4.12 | 2.25 | 2.12
MS-FastSpeech2 | Yes | 3.62 | 5.32 | 3.21
MS-FastSpeech2-SupASR | No | 3.58 | 6.65 | 3.85
VC-FastSpeech2 | No | 3.41 | 7.44 | 8.18
NAR-TTE${}_{\text{LJS}}$+MS-ETS | No | 3.78 | 6.53 | 4.38
Table 2:  Comparison of the studied multi-speaker TTS models on the VCTK
dataset. The second column suggests if the corresponding method uses the
ground truth VCTK transcripts while training.

###  4.3 Evaluation metrics

Naturalness is measured by mean opinion scores (MOS) from subjective listening
tests on a five-point Likert scale, with $1$ being “completely unnatural”
speech to $5$ indicating “completely natural” output. We randomly sample five
clips per model from the validation set for each of our forty subjects who are
proficient English speakers. They are asked to make quality judgments by
rating the naturalness of the synthesized speech samples. The average rating
of MOS is calculated and reported. Intelligibility is measured by the word
error rate of ASR transcriptions of rendered speech. We use pre-trained
Whisper small model Radford et al. (2022) for this.

We validate the speaker adaptability by reporting Equal Error Rate (EER) from
a pre-trained speaker verification network. Specifically, we use the
verification model proposed in Desplanques et al. (2020) trained on VoxCeleb2
Chung et al. (2018) with a $0.8$% EER on the test split of VoxCeleb1 Chung et
al. (2018).

##  5 Results

Quantitative and qualitative results evaluating the proposed ParrotTTS system
are shown in Tables 1 and 2 for single-speaker and multi-speaker models,
respectively.

###  5.1 Single-speaker TTS

Naturalness and intelligibility. As shown in Table 1, ParrotTTS is competitive
to state-of-the-art in the single-speaker setting. In the autoregressive case,
our AR-TTE${}_{\textsc{LJS}}$+SS-ETS has a statistically insignificant drop
(of about $0.05$ units) on the MOS scale relative to the Tacotron2 baseline.
The non-autoregressive case has a similar observation (with a $0.01$ drop) on
MOS in our NAR-TTE${}_{\textsc{LJS}}$+SS-ETS model relative to FastSpeech2.
This empirically establishes that the naturalness of the speech rendered by
ParrotTTS is on par with the currently established methods. The WER scores
show a similar trend with a statistically insignificant drop (of under
$0.2$pp111Percentage points abbreviated as pp.) among the autoregressive and
non-autoregressive model classes.

Supervision and data efficiency. In the study to understand how the degree of
supervision affects TTS speech quality, we see a clear drop by $0.28$ MOS
units in moving from the FastSpeech2-SupASR model that employs supervised ASR
for transcriptions to Tacotron2-UnsupASR model using unsupervised ASR. Despite
some modeling variations, this is generally indicative of the importance of
clean transcriptions on TTS output quality, given that all other models are
within $0.05$ MOS units of each other.

The data requirement for TTS supervision needs to be understood in light of
this impact on output quality, and we show how ParrotTTS helps cut down on
this dependence. TTE is the only module that needs transcriptions to train,
and we show that by reducing the size of the train set by half in NAR-
TTE${}_{\frac{1}{2}\textsc{LJS}}$+SS-ETS the MOS is still comparable to that
of the model trained on all data NAR-TTE${}_{\textsc{LJS}}$+SS-ETS (with only
about $0.04$ units MOS drop). Finally, the MOS numbers of FastSpeech2-SupASR,
need to be read with some caution since the supervised ASR model used,
Whisper, is itself trained with plenty of transcriptions (spanning over $600$k
hours) from the web, including human and machine transcribed data achieving
very low WERs on various public and test sets. So, the machine transcriptions
used in FastSpeech2-SupASR are indeed very close to ground truth.

Figure 3: Visualization of attention between output units and phonemes. (a)
Evolution of attention matrix with training steps. (b) Attention loss plotted
against training steps.

###  5.2 Multi-speaker TTS

Naturalness and intelligibility. Table 2 summarizes results from our multi-
speaker experiments. Among all methods listed in it, NAR-
TTE${}_{\textsc{LJS}}$+MS-ETS clearly outperform all other models ranking only
below re-synthesizing from ground truth Mels, GT-Mel+Vocoder. Interestingly,
ParrotTTS fares even better than MS-FastSpeech2, which is, in turn, better
than other models that ignore transcripts at the train, namely, MS-
FastSpeech2-SupASR and VC-FastSpeech2. On the WER metric for intelligibility,
ParrotTTS is about $1$pp behind supervised MS-FastSpeech2 but fares better
than the other two models that discard VCTK transcripts for training.

Speaker adaptability. VC-FastSpeech2 is the closest in terms of experimental
setup since it is limited to transcriptions from LJSpeech for training similar
to ours, with VCTK used only for adaptation. In this case, EER of NAR-
TTE${}_{\textsc{LJS}}$+MS-ETS is about twice as good as that of VC-
FastSpeech2. However, improvements are visible when VCTK transcripts are part
of training data but remain under $1$pp relative to ParrotTTS while GT-
Mel+Vocoder continues to dominate the scoreboard leaving room for further
improvement.

###  5.3 Stabler training and faster inference

In Figure 3, we compare training profiles of Tacotron2 and AR-TTE keeping
batch size the same. As visualized in Figure 3(a), the attention matrix in
Tacotron2 takes about $20$k iterations to stabilize with an anti-diagonal
structure and predict a phoneme-aligned Mel sequence. AR-TTE, in contrast, is
about ten times faster at predicting a discrete HuBERT unit sequence that
aligns with input phonemes taking only about $2$k iterations to arrive at a
similar-looking attention plot. While the snapshots are illustrative, we use
the guided-attention loss described by Tachibana et al. (2018) as a metric to
quantify the evolution of the attention matrix through training steps. As
shown in Figure 3(b), the loss dives down a lot sooner for ParrotTTS relative
to its Tacotron2 counterpart. In a similar comparison, we observe that NAR-TTE
converges ($20$k steps) about eight times faster than FastSpeech2 ($160$k
steps).

We suppose that the faster convergence derives from the lower variance of
discrete embeddings in ParrotTTS as opposed to the richness of Mels that are
complete with all acoustic variations, including speaker identity, prosody,
etc. The output speech is independent of inputs given the Mel-spectrogram
unlike ParrotTTS embeddings that further need cues like speaker identity in
later ETS module. We hypothesize that segregating content mapping away from
learning acoustics like speaker identity helps improve training stability,
convergence, and data efficiency for the TTE encoder.

The proposed NAR-TTE system also improves inference latency and memory
footprint, which are crucial factors for real-world deployment. On NVIDIA RTX
$2080$ Ti GPU, we observe ParrotTTS serves 15% faster than FastSpeech2,
reducing the average per utterance inference time to 11ms from 13 ms.
Furthermore, the TTE module uses $17$M parameters in contrast to $35$M
parameters of the Mel synthesizer module in Fastspeech2.

##  6 Conclusion, limitations and future work

In this work, we proposed ParrotTTS, a fast, high quality, and efficient to
train TTS system. The two-stage learning process of ParrotTTS is designed to
leverage untranscribed speech data and the corresponding self-supervised
embeddings. We show that even when trained using transcribed data of a single
speaker from the LJSpeech dataset, ParrotTTS can synthesize speech in 108
different voices of the VCTK corpus. In terms of naturalness of speech,
ParrotTTS outperforms the established prior art and alternative baselines by a
noticeable margin in the multi-speaker setup. On single speaker benchmarks,
ParrotTTS provides competitive performance compared to the prior art. Overall,
our work paves the way for further explorations towards exploiting SSL in TTS
models.

Our experiments are limited to a single language. A deeper study exploring
multiple languages, effects of background noise, accents, and other
demographic variations is left for future work. The current pre-trained HuBERT
model skips prosody information Kharitonov et al. (2021), so the model has no
levers to control these prosodic variations. We want to study ways to bring
prosodic controllability into ParrotTTS. Further, it would be essential to
improve TTE training to use noisy samples that the current model does not work
well with to leverage weak supervision to scale.

##  7 Ethical Considerations

Our research is founded on ethical considerations. We are excited about the
potential of text-to-speech synthesis to push the frontier of technology, such
as in accessibility (giving voice to the voiceless), human-computer
interaction, telecommunications, and education. However, there is the
potential for misuse. Notably, multi-speaker text-to-speech systems have
raised concerns about unethical cloning. Our experiments limit to publicly
available datasets, and our method is not intended for synthesizing someone’s
voice without their permission. Another potential misuse is creating an audio
file of someone supposedly speaking words they never actually uttered. We are
keenly aware of these negative consequences. While the benefits outweigh the
concerns at this point, we firmly believe that the research community should
proactively continue to identify methods for detecting and preventing misuse.

Our approach is trained on western speech data and has yet to be validated on
different languages or people with speech impediments. As such, the dataset
and results are not representative of the population. A deeper understanding
of this issue requires future studies in tandem with linguistic and socio-
cultural insights.

## References

  * Baevski et al. (2021) Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2021.  Unsupervised speech recognition.  _Advances in Neural Information Processing Systems_ , 34:27826–27839.
  * Baevski et al. (2020) Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in Neural Information Processing Systems_ , 33:12449–12460.
  * Chung et al. (2018) Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. 2018.  Voxceleb2: Deep speaker recognition.  _arXiv preprint arXiv:1806.05622_.
  * Cohn and Zellou (2020) Michelle Cohn and Georgia Zellou. 2020.  Perception of concatenative vs. neural text-to-speech (tts): Differences in intelligibility in noise and language attitudes.  In _Proceedings of Interspeech_.
  * Desplanques et al. (2020) Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck. 2020.  Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification.  _arXiv preprint arXiv:2005.07143_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Graves (2013) Alex Graves. 2013.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_.
  * Griffin and Lim (1984) Daniel Griffin and Jae Lim. 1984.  Signal estimation from modified short-time fourier transform.  _IEEE Transactions on acoustics, speech, and signal processing_ , 32(2):236–243.
  * He et al. (2022) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16000–16009.
  * Hsu et al. (2021) Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021.  Hubert: Self-supervised speech representation learning by masked prediction of hidden units.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 29:3451–3460.
  * Hunt and Black (1996) Andrew J Hunt and Alan W Black. 1996.  Unit selection in a concatenative speech synthesis system using a large speech database.  In _1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings_ , volume 1, pages 373–376. IEEE.
  * Ito and Johnson (2017) Keith Ito and Linda Johnson. 2017.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/.
  * Jia et al. (2018) Ye Jia, Yu Zhang, Ron Weiss, Quan Wang, Jonathan Shen, Fei Ren, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, Yonghui Wu, et al. 2018.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis.  _Advances in neural information processing systems_ , 31.
  * Kharitonov et al. (2021) Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, et al. 2021.  Text-free prosody-aware generative spoken language modeling.  _arXiv preprint arXiv:2109.03264_.
  * Kim et al. (2021) Jaehyeon Kim, Jungil Kong, and Juhee Son. 2021.  Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech.  In _International Conference on Machine Learning_ , pages 5530–5540. PMLR.
  * Kolata (1984) Gina Kolata. 1984.  Studying learning in the womb: behavioral scientists are using established experimental methods to show that fetuses can and do learn.  _Science_ , 225(4659):302–303.
  * Kong et al. (2020) Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033.
  * Kreuk et al. (2020) Felix Kreuk, Joseph Keshet, and Yossi Adi. 2020.  Self-supervised contrastive learning for unsupervised phoneme segmentation.  _Interspeech_.
  * Kuhl and Meltzoff (1996) Patricia K Kuhl and Andrew N Meltzoff. 1996.  Infant vocalizations in response to speech: Vocal imitation and developmental change.  _The journal of the Acoustical Society of America_ , 100(4):2425–2438.
  * Lakhotia et al. (2021) Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021.  On generative spoken language modeling from raw audio.  _Transactions of the Association for Computational Linguistics_ , 9:1336–1354.
  * Łańcucki (2021) Adrian Łańcucki. 2021.  Fastpitch: Parallel text-to-speech with pitch prediction.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6588–6592. IEEE.
  * Liu et al. (2022a) Alexander H Liu, Wei-Ning Hsu, Michael Auli, and Alexei Baevski. 2022a.  Towards end-to-end unsupervised speech recognition.  _arXiv preprint arXiv:2204.02492_.
  * Liu et al. (2022b) Alexander H Liu, Cheng-I Jeff Lai, Wei-Ning Hsu, Michael Auli, Alexei Baevskiv, and James Glass. 2022b.  Simple and effective unsupervised speech synthesis.  _Interspeech_.
  * Locke (1994) John L Locke. 1994.  Phases in the child’s development of language.  _American Scientist_ , 82(5):436–445.
  * Locke (1996) John L Locke. 1996.  Why do infants begin to talk? language as an unintended consequence.  _Journal of child language_ , 23(2):251–268.
  * Luong and Yamagishi (2019) Hieu-Thi Luong and Junichi Yamagishi. 2019.  A unified speaker adaptation method for speech synthesis using transcribed and untranscribed speech with backpropagation.  _arXiv preprint arXiv:1906.07414_.
  * McAuliffe et al. (2017) Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_ , volume 2017, pages 498–502.
  * Ni et al. (2022) Junrui Ni, Liming Wang, Heting Gao, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa-Johnson. 2022.  Unsupervised text-to-speech synthesis by unsupervised automatic speech recognition.  _arXiv preprint arXiv:2203.15796_.
  * Oord et al. (2016) Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_.
  * Panayotov et al. (2015) Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015.  Librispeech: an asr corpus based on public domain audio books.  In _2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5206–5210. IEEE.
  * Polyak et al. (2021) Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021.  Speech resynthesis from discrete disentangled self-supervised representations.  _Interspeech_.
  * Radford et al. (2022) Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022.  Robust speech recognition via large-scale weak supervision.  _OpenAI Blog_.
  * Ren et al. (2020) Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020\.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_.
  * Ren et al. (2019) Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019\.  Fastspeech: Fast, robust and controllable text to speech.  _Advances in Neural Information Processing Systems_ , 32.
  * Schneider et al. (2019) Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019.  wav2vec: Unsupervised pre-training for speech recognition.  _Interspeech_.
  * Shen et al. (2018) Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018\.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions.  In _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 4779–4783. IEEE.
  * Sun et al. (2019) Hao Sun, Xu Tan, Jun-Wei Gan, Hongzhi Liu, Sheng Zhao, Tao Qin, and Tie-Yan Liu. 2019.  Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion.  In _Proc. Interspeech 2019_ , pages 2115–2119.
  * Tachibana et al. (2018) Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara. 2018.  Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention.  In _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 4784–4788. IEEE.
  * Taigman et al. (2017) Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nachmani. 2017.  Voiceloop: Voice fitting and synthesis via a phonological loop.  _arXiv preprint arXiv:1707.06588_.
  * Tan et al. (2021) Xu Tan, Tao Qin, Frank Soong, and Tie-Yan Liu. 2021.  A survey on neural speech synthesis.  _arXiv preprint arXiv:2106.15561_.
  * Valle et al. (2020) Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. 2020.  Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis.  _arXiv preprint arXiv:2005.05957_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Veaux et al. (2017) Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. 2017.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit.
  * Wang et al. (2017) Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017.  Tacotron: Towards end-to-end speech synthesis.  _arXiv preprint arXiv:1703.10135_.
  * Yan et al. (2021) Yuzi Yan, Xu Tan, Bohan Li, Tao Qin, Sheng Zhao, Yuan Shen, and Tie-Yan Liu. 2021\.  Adaspeech 2: Adaptive text to speech with untranscribed data.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6613–6617. IEEE.