# DiffProsody: Diffusion-based Latent Prosody Generation for Expressive Speech
Synthesis with Prosody Conditional Adversarial Training

Hyung-Seok Oh, Sang-Hoon Lee, and Seong-Whan Lee This work was supported by
Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government (MSIT) (No. 2019-0-00079,
Artificial Intelligence Graduate School Program (Korea University) and No.
2021-0-02068, Artificial Intelligence Innovation Hub) and Voice&Avatar, Naver
Cloud, Seongnam, Korea. (Corresponding author: Seong-Whan Lee.)Hyung-Seok Oh,
Sang-Hoon Lee, and Seong-Whan Lee are with the Department of Artificial
Intelligence, Korea University, Seoul 02841, South Korea (e-mail:
hs_oh@korea.ac.kr;sh_lee@korea.ac.kr;sw.lee@korea.ac.kr).

###### Abstract

Expressive text-to-speech systems have undergone significant advancements
owing to prosody modeling, but conventional methods can still be improved.
Traditional approaches have relied on the autoregressive method to predict the
quantized prosody vector; however, it suffers from the issues of long-term
dependency and slow inference. This study proposes a novel approach called
DiffProsody in which expressive speech is synthesized using a diffusion-based
latent prosody generator and prosody conditional adversarial training. Our
findings confirm the effectiveness of our prosody generator in generating a
prosody vector. Furthermore, our prosody conditional discriminator
significantly improves the quality of the generated speech by accurately
emulating prosody. We use denoising diffusion generative adversarial networks
to improve the prosody generation speed. Consequently, DiffProsody is capable
of generating prosody 16 times faster than the conventional diffusion model.
The superior performance of our proposed method has been demonstrated via
experiments.

###### Index Terms:

text-to-speech, speech synthesis, denoising diffusion model, prosody modeling,
generative adversarial networks

##  I Introduction

Recent advancements in neural text-to-speech (TTS) models have significantly
enhanced the naturalness of synthetic speech. In several studies[1, 2, 3, 4,
5, 6, 7], prosody modeling has been leveraged to synthesize speech that
closely resembles human expression. Prosody, which encompasses various speech
properties, such as pitch, energy, and duration, plays a crucial role in the
synthesis of expressive speech.

In some studies[8, 9], reference encoders have been used to extract prosody
vectors for prosody modeling. A global style token (GST)[10] is an
unsupervised style modeling method that uses learnable tokens to model and
control various styles. Meta-StyleSpeech[11] proposes the application of style
vectors extracted using a reference encoder through a style-adaptive layer
norm. Progressive variational autoencoder TTS[12] presents a method for
gradual style adaptation. A zero-shot method for speech synthesis that
comprises the use of a normalization architecture, speaker encoder, and
feedforward transformer-based architecture[13] was proposed. Despite the
intuitive and effective nature of using a reference encoder, these methods
cannot reflect the details of prosody in their synthetic speech without
ground-truth (GT) prosody information.

Recently, methods for inferring prosody from text in the absence of reference
audio have been developed [14, 15]. FastPitch[16], for instance, synthesizes
speech under text and fundamental frequency conditions. FastSpeech 2[3] aims
to generate natural, human-like speech by using extracted prosody features,
such as pitch, energy, and duration, through an external tool and introduce a
variance adaptor module that predicts these features. Some studies[17, 18]
have proposed hierarchical models through the design of prosody features at
both coarse and fine-grained levels. However, the separate modeling of
prosodic features may yield unnatural results owing to their inherent
correlation.

Some studies have predicted a unified prosody vector, thus enhancing the
representation of prosody, given the interdependence of prosody features.
Text-predicted GST[19] is a method for modeling prosody without reference
audio by predicting the weight of the style token from the input text. [20]
proposed a method for paragraph-based prosody modeling by introducing a
paragraph encoder. Gaussian-mixture-model-based phone-level prosody
modelling[21] is a method for sampling reference prosody from Gaussian
components. [22] proposed a method for modeling prosody using style,
perception, and frame-level reconstruction loss. There are also studies in
which prosody is modeled using pre-trained language models[23, 24, 25, 26].
ProsoSpeech [27] models prosody with vector quantization (VQ) using large
amounts of data and predicts the index of the codebook using an autoregressive
(AR) prosody predictor. However, when predicting prosody vectors, an AR
prosody predictor encounters challenges related to long-term dependencies.

To address these issues, we propose DiffProsody, a novel approach that
generates expressive speech by employing a diffusion-based latent prosody
generator (DLPG) and prosody conditional adversarial training.

The primary contributions of this work are as follows:

  * •

We propose a diffusion-based latent prosody modeling method that can generate
high-quality latent prosody representations, thereby enhancing the
expressiveness of synthetic speech. Furthermore, we adopted denoising
diffusion generative adversarial networks (DDGANs) to reduce the number of
timesteps, resulting in speeds that were 2.48× and 16× faster than those of
the AR model and denoising diffusion probabilistic model (DDPM) [28],
respectively.

  * •

We propose prosody conditional adversarial training to ensure an accurate
reflection of prosody using the TTS module. A significant improvement in
smoothness, attributable to vector quantization, was observed in the generated
speech.

  * •

Objective and subjective evaluations demonstrated that the proposed method
outperforms comparative models.

The implementation111https://github.com/hsoh0306/DiffProsody of proposed
method and audio samples222https://prml-lab-speech-
team.github.io/demo/DiffProsody/ for various datasets, such as
VCTK333https://datashare.ed.ac.uk/handle/10283/2651 and LibriTTS[29], are
available online.

##  II Related works

###  II-A Non-autoregressive text-to-speech

Traditional TTS models function autoregressively. This implies that the
spectrogram generates one frame at a time, with each frame conditioned on the
preceding frames. Despite the high-quality speech that this approach can
produce, it has a drawback in terms of speed owing to the sequential nature of
the generation process. To address this issue, non-autoregressive TTS (NAR-
TTS) models have been proposed as an alternative for parallel generation.
These models have the advantage of simultaneously generating the entire
spectrogram, thus resulting in a significant acceleration of the speech
synthesis. FastSpeech[2] and FastSpeech 2[3] serve as examples of NAR-TTS
models that can synthesize speech at a much faster rate than their AR
counterparts while maintaining a comparable level of quality. For parallel
generation, these models require phoneme-level durations. FastSpeech uses an
AR teacher model to obtain durations through knowledge distillation. The
phoneme-level input is scaled to the frame-level using a length regulator, and
a transformer-based network [30] is used to generate the entire utterance at
once. FastSpeech 2 addresses some of the disadvantages of FastSpeech by
extracting the phoneme duration from forced alignment as the training target
instead of relying on the attention map of the AR teacher model and
introducing more variation information in speech as conditional inputs. In
contrast to these models that use an external aligner,[31, 32, 33, 34] are
parallel TTS models that use an internal aligner to model duration. These
parallel-generation models exhibit faster and more robust generation than the
AR models. In this study, we adopted a transformer-based NAR model with a
simple structure to focus on prosody modeling.

###  II-B Generative adversarial networks

Generative adversarial networks (GANs)[35] are generative models in which
generative and discriminative networks compete against each other. The
objective of the generative network is to create samples that closely resemble
the true data distribution, whereas the discriminative network strives to
differentiate between the data sampled from the true and generated
distributions.

These two networks play a minimax game with the following value function
$V(D,G)$:

$\min_{G}\max_{D}V(D,G)=\mathbb{E}_{\mathbf{x}\sim
p_{data}(\mathbf{x})}[\log{(D(\mathbf{x}))}]\\\ +\mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}(\mathbf{z})}[\log{(1-D(G(\mathbf{z})))}],$ (1)

where $G$ represents the generative network, and $D$ represents the
discriminative network. The training process involves minimizing $D(1-G(z))$
to recognize $D$ as real for $G$, and maximizing $log(D(x))$ for $D$ to learn
the likelihood of real data. The conditional generation model was obtained by
introducing condition $c$ into both $G$ and $D$. In this study, we
incorporated adversarial training conditions into prosody features to generate
expressive speech.

###  II-C Denoising diffusion models

The denoising diffusion model is a generative model that gradually collapses
data into noise and generates data from noise. The processes of collapsing and
denoising data are called the forward and reverse processes, respectively. The
forward process gradually collapses data $\mathbf{x}_{0}$ into noise over the
$T$-step, with a predefined variance schedule $\beta_{t}$.

$q(\mathbf{x}_{1:T}|\mathbf{x}_{0})=\prod_{t=1}^{T}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}),$
(2)

where
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}I)$.
The reverse process is defined as follows:

$p_{\theta}(\mathbf{x}_{0:T})=p(\mathbf{x}_{T})\prod_{t=1}^{T}p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}).$
(3)

The reverse process is driven by the $\theta$ denoising model parameterized by
a. The denoising model was optimized for a variational bound on the negative
log-likelihood.

$\mathbb{E}[-\log{p_{\theta}(\mathbf{x}_{0})}]\leq\mathbb{E}_{q}[-\log\frac{p_{\theta}(\mathbf{x}_{0:T})}{q(\mathbf{x}_{1:T}|x_{0})}]\\\
=\mathbb{E}_{q}[-\log{p(\mathbf{x}_{T})}-\sum_{t\geq
1}\log{\frac{p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})}{q(\mathbf{x}_{t}|\mathbf{x}_{t-1})}}]:=L.$
(4)

Figure 1: Framework of DiffProsody. (a) Overall architecture including TTS and
prosody modeling with prosody conditional adversarial training; (b) Prosody
modeling by vector quantization with prosody encoder and diffusion-based
latent prosody generator; (c) Text encoder that models text at the phoneme-
level and word-level; (d) Prosody encoder that models the word-level target
prosody; (e) Prosody conditional discriminator for adversarial training. DP
represents a duration predictor, and LR represents a length regulator. In the
first stage, the TTS and prosody encoder are trained jointly, and in the
second stage, a diffusion-based latent prosody generator (DLPG) is trained
using the output of the pre-trained prosody encoder as a target. In inference,
the TTS module synthesizes speech conditioned on the prosody vector generated
by DLPG.

In the DDPM[28], the denoising distribution
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ is assumed to comprise a
Gaussian distribution. Moreover, it has been demonstrated that the diffusion
model can generate a diverse range of complex distributions, provided that a
sufficient number of iterations are performed. Proceeding with only a small
step at a time is possible by setting the denoising distribution to a Gaussian
distribution, which implies that a considerable number of timesteps are
required. DDGAN[36] was proposed by modeling the denoising distribution with a
non-Gaussian multimodal distribution to reduce the sampling step. It predicts
$\mathbf{x}_{0}$ with the generator $G_{\theta}$, which models the implicit
distribution, in contrast to the denoising network of the DDPM, which predicts
noise. In the DDGAN, the conditional probability
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ is defined as follows:

$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t}):=\int{p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t})q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0})}d\mathbf{x}_{0}\\\
=\int{p(\mathbf{z})q(\mathbf{x}_{t-1}|\mathbf{x}_{t},\mathbf{x}_{0}=G_{\theta}(\mathbf{x}_{t},\mathbf{z},t))}d\mathbf{z},$
(5)

where $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ represents the implicit
distribution imposed by the generator $G_{\theta}(\mathbf{x},\mathbf{z},t)$,
which outputs $\mathbf{x}_{0}$ given $\mathbf{x}_{t}$ and the latent variable
$\mathbf{z}\sim p(\mathbf{z}):=\mathcal{N}(\mathbf{z};0,\mathbf{I})$. In the
DDGAN, the denoising distribution
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ is modeled as a complex
multimodal distribution, in contrast to the unimodal distribution in the DDPM.
Sampling $\mathbf{x}_{0}$ with small timesteps is possible by leveraging the
complicated $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$. We introduced this
model to reduce the sampling timesteps while maintaining its ability to
generate diffusion models.

###  II-D Prosody modeling

Although the pronunciation capabilities of TTS models have seen significant
advancements, they still fail to replicate the naturalness inherent in human
speech. Various studies have proposed methods for prosody modeling to address
this limitation. One such method is the reference-based approach[8, 9, 10, 11,
37], which is a type of expressive speech synthesis that extracts styles from
the reference audio. This approach is particularly beneficial for style
transfers, but can result in unnatural results when the text and reference
audio do not align well. However, there are methods that directly model
prosody properties such as pitch, energy, and duration[3, 16, 38, 39]. These
methods offer the advantages of explainability and controllability because
they directly use and model prosody features. In the context of
ProsoSpeech[27], the authors argue that prosodic features are interdependent
and that modeling them separately can result in unnatural outcomes. To address
this, they proposed the modeling of prosody as a latent prosody vector (LPV)
and introduced an AR prosody predictor to obtain the LPV. In this study, we
adopt a similar latent vector approach to model prosody. The use of pre-
trained language models was also suggested[23, 24, 25, 26] . These models
comprise the use of models that have been pre-trained on large datasets, such
as BERT[40] or GPT-3[41].

##  III DiffProsody

The proposed method, called DiffProsody, aims to enhance speech synthesis by
incorporating a diffusion-based latent prosody generator (DLPG) and prosody
conditional adversarial training. The overall structure and process of
DiffProsody are presented in Figure 1. In the first stage, we trained a TTS
module and a prosody encoder using a text sequence and a reference Mel-
spectrogram as inputs. The prosody conditional discriminator evaluates the
prosody vector from the prosody encoder and the Mel-spectrogram from the TTS
module to provide feedback on their quality. In the second stage, we train a
DLPG to sample a prosody vector that corresponds to the input text and
speaker. During inference, the TTS module synthesizes speech without relying
on a reference Mel-spectrogram. Instead, it uses the output of a DLPG. This
facilitates the generation of expressive speech that accurately reflects the
desired prosody.

Figure 2: Training a diffusion-based latent prosody generator. We adopt the
design of DDGANs[36] to shorten the diffusion timestep. The generator
$G_{\theta}$ takes speaker hidden representation $\mathbf{h}_{spk}$ and text
hidden representation$\mathbf{h}_{txt}$, timestep $t$, and noisy data
$\mathbf{x}_{t}$ as input to generate $\mathbf{x}_{0}^{\prime}$, and the
disriminator $D_{\phi}$ determines which of $\mathbf{x}_{t-1}^{\prime}$
obtained by posterior sampling on $\mathbf{x}_{0}^{\prime}$ and
$\mathbf{x}_{t-1}$ obtained by forward process on $\mathbf{x}_{0}$ is
compatible with $\mathbf{x}_{t}$ at $t$ timestep.

###  III-A Text-to-speech module

The TTS module is designed to transform text into Mel-spectrograms using
speaker and prosody vectors as conditions. The overall structure of the model
is presented in Figure 1a. The TTS module comprises a text encoder and a
decoder. The text encoder processes the text at both the phoneme and word
levels, as illustrated in Figure 1c. The input text, denoted as
$\mathbf{x}_{txt}$, is converted into a text hidden representation
$\mathbf{h}_{txt}$, by the phoneme encoder $E_{p}$ and word encoder $E_{w}$.
The $E_{p}$ takes the phoneme-level text $\mathbf{x}_{ph}$ and the $E_{w}$
takes as input the word-level text $\mathbf{x}_{wd}$. The $\mathbf{h}_{txt}$
is then obtained as the element-wise sum of the outputs of
$E_{p}(\mathbf{x}_{ph})$ and $E_{w}(\mathbf{x}_{wd})$ expanded to the phoneme-
level.

$\mathbf{h}_{txt}=E_{p}(\mathbf{x}_{ph})+expand(E_{w}(\mathbf{x}_{wd})),$ (6)

where $expand$ is an operation that expands the word-level features to the
phoneme-level. Obtaining the quantized prosody vector $\mathbf{z}_{pros}$
involves using $\mathbf{h}_{txt}$ and speaker hidden representation
$\mathbf{h}_{spk}$ as inputs for the prosody module. In addition,
$\mathbf{h}_{spk}$ is acquired using a pre-trained speaker encoder. We use
Resemblyzer444https://github.com/resemble-ai/Resemblyzer, an open-source model
trained with generalized end-to-end loss (GE2E)[42], to extract
$\mathbf{h}_{spk}$.

During the first stage of training, a prosody encoder is employed, which
receives the target Mel-spectrogram. In the inference,
$\mathbf{z}_{pros}^{\prime}$ is obtained by inputting $\mathbf{h}_{txt}$ and
$\mathbf{h}_{spk}$ into a DLPG, and this is performed without a reference Mel-
spectrogram. Finally, the information related to the text, speaker, and
prosody is combined by expanding the latent vectors $\mathbf{h}_{txt}$,
$\mathbf{h}_{spk}$, and $\mathbf{z}_{pros}$ to the phoneme-level and then
performing an element-wise summation.

$\mathbf{h}_{total}=\mathbf{h}_{txt}+\mathbf{h}_{spk}+\mathbf{z}_{pros}.$ (7)

The phoneme duration is modeled using the duration predictor $DP$. The goal of
the $DP$ is to predict the phoneme duration at the frame-level based on the
input variable $\mathbf{h}_{total}$.

$dur^{\prime}=DP(\mathbf{h}_{total}).$ (8)

In addition, there is a length regulator $LR$ that expands the input variable
to the frame-level using the phoneme duration $dur$. The expanded
$\mathbf{h}_{total}$ is then transformed to Mel-spectrogram
$\mathbf{y}^{\prime}$ by $D_{mel}$.

$\mathbf{y}^{\prime}=D_{mel}(LR(\mathbf{h}_{total},dur)).$ (9)

For TTS modeling, we use two types of losses: the mean square error (MSE) and
structural similarity index (SSIM) loss. These losses aid in accurately
modeling the TTS. For the duration modeling, we use the MSE loss.

$\mathcal{L}_{rec}=\mathcal{L}_{MSE}(\mathbf{y},\mathbf{y}^{\prime})+\mathcal{L}_{SSIM}(\mathbf{y},\mathbf{y}^{\prime}).$
(10)

$\mathcal{L}_{dur}=\mathcal{L}_{MSE}(dur,dur^{\prime}).$ (11)

###  III-B Prosody module

Figure 1b presents the prosody module, which includes a prosody encoder
$E_{pros}$ that derives a prosody vector from a reference Mel-spectrogram, a
DLPG that produces a prosody vector using text and speaker hidden states, and
a codebook $Z=\\{\mathbf{z}_{k}\\}_{k=1}^{K}\in\mathbb{R}^{K\times d_{z}}$,
where $K$ represents the size of the codebook and $d_{z}$ is the dimension of
the codes. During the training of $E_{pros}$, instead of a full-band Mel-
spectrogram, we used a low-frequency band Mel-spectrogram to alleviate
disentanglement, as in the case of ProsoSpeech[27]. Figure 1d presents the
structure of $E_{pros}$, which comprises two convolutional stacks and a word-
level pooling layer. To extract the target prosody, $E_{pros}$ uses the lowest
$N$ bins of the target Mel-spectrogram $\mathbf{y}_{[0:N]}$, along with the
$\mathbf{h}_{txt}$ and $\mathbf{h}_{spk}$, as its inputs. The output of this
process is a prosody vector, $\mathbf{h}_{pros}\in\mathbb{R}^{L\times d_{z}}$,
where $L$ is the word-level length of the input text.

$\mathbf{h}_{pros}=E_{pros}(\mathbf{y}_{[0:N]},\mathbf{h}_{txt},\mathbf{h}_{spk}).$
(12)

During the inference stage, the prosody vector $\mathbf{h}_{pros}^{\prime}$ is
obtained using the prosody generator trained in the second stage.

$\mathbf{h}_{pros}^{\prime}=DLPG(\mathbf{h}_{txt},\mathbf{h}_{spk}).$ (13)

The $DLPG$ process is described in section III-D. To obtain the discrete
prosody token sequence $\mathbf{z}_{pros}\in\mathbb{R}^{L\times d_{z}}$, the
vector quantization layer $Z$ maps each prosody vector
$\mathbf{h}_{pros}^{i}\in\mathbb{R}^{d_{z}}$ to the nearest element of the
codebook entry $\mathbf{z}_{k}\in\mathbb{R}^{d_{z}}$.

$\mathbf{z}_{pros}^{i}=\operatorname*{arg\,min}_{\mathbf{z}_{k}\in
Z}||\mathbf{h}_{pros}^{i}-\mathbf{z}_{k}||_{2}\text{ for }i=1\text{ to }L,$
(14)

where $\mathbf{z}_{pros}^{i}$ is $i$-th element of $\mathbf{z}_{pros}$. In the
first stage, the TTS module is trained jointly with the codebook $Z$ and
prosody encoder $E_{pros}$.

$\mathcal{L}_{vq}=||sg[\mathbf{h}_{pros}]-\mathbf{z}_{pros}||_{2}^{2}+\beta||\mathbf{h}_{pros}-sg[\mathbf{z}_{pros}]||_{2}^{2},$
(15)

where $sg[\cdot]$ denotes the stop-gradient operation. Moreover, we employ an
exponential moving average (EMA)[43] to enhance the learning efficiency by
applying it to codebook updates.

###  III-C Prosody conditional adversarial training

For our prosody adversarial training, we develop prosody conditional
discriminators (PCDs) that handle inputs of varying lengths. This design is
inspired by multi-length window discriminators[44]. The PCD structure is
presented in Figure 1e. The PCD is designed to accept a Mel-spectrogram $y$
and a quantized prosody vector $\mathbf{z}_{pros}$ as inputs, and its role is
to determine whether these input features are original or generated. The PCD
comprises two lightweight convolutional neural networks (CNNs) and fully
connected layers. One of the CNNs is designed to receive only the Mel-
spectrogram, whereas the other is designed to receive a combination of
$\mathbf{z}_{pros}$ and $y$. To match the corresponding PCD, the length of
each Mel-spectrogram and the extended $\mathbf{z}_{pros}$ are randomly
clipped. For our objective function, we adopt the least square GAN loss[45]:

$\mathcal{L}_{D}=\sum_{i}[\mathbb{E}[(PCD^{i}(\mathbf{y}^{\prime},\mathbf{z}_{pros}))^{2}]\\\
+\mathbb{E}[(PCD^{i}(\mathbf{y},\mathbf{z}_{pros})-1)^{2}]],$ (16)
$\mathcal{L}_{G}=\sum_{i}\mathbb{E}[(PCD^{i}(\mathbf{y}^{\prime},\mathbf{z}_{pros})-1)^{2}],$
(17)

where $\mathcal{L}_{D}$ denotes the training goal of the discriminators and
$\mathcal{L}_{G}$ represents the feedback on the TTS module. The final object
$\mathcal{L}_{TTS}$ of the TTS module is as follows:

$\mathcal{L}_{TTS}=\mathcal{L}_{rec}+\mathcal{L}_{dur}+\mathcal{L}_{vq}+\lambda_{1}\mathcal{L}_{G},$
(18)

where $\lambda_{1}$ corresponds to the weight of the adversarial loss.

###  III-D Diffusion-based latent prosody generator

We propose a new module called the DLPG, which leverages the powerful
generative capabilities of diffusion models. In addition, we introduce a DDGAN
framework[36], which enables faster sampling by reducing the number of
required timesteps. Figure 2 presents the training process for the DLPG.
During training, the DLPG aims to generate target $\mathbf{h}_{pros}$, which
is extracted from the prosody encoder trained in the first stage. The DLPG is
trained to produce $\mathbf{h}_{pros}^{\prime}$ based on the
$\mathbf{h}_{spk}$ and $\mathbf{h}_{txt}$. In the diffusion model, we set
$\mathbf{x}_{0}$ as the target $\mathbf{h}_{pros}$. The DLPG generator
$G_{\theta}$ directly generates $\mathbf{x}^{\prime}_{0}$.

$\mathbf{x}^{\prime}_{0}=G_{\theta}(\mathbf{x}_{t},t,\mathbf{h}_{spk},\mathbf{h}_{txt}),$
(19)

where $t$ is timestep of diffusion process. To ensure adversarial training,
$\mathbf{x}^{\prime}_{t-1}$ is derived from $\mathbf{x}_{t}$ and
$\mathbf{x}^{\prime}_{0}$ using posterior sampling
$q(\mathbf{x}^{\prime}_{t-1}|\mathbf{x}_{t},\mathbf{x}^{\prime}_{0})$.
Subsequently, a time-dependent discriminator $D_{\phi}$ determines the
compatibility of $\mathbf{x}_{t-1}$ (obtained from the forward processing of
$\mathbf{x}_{0}$) and $\mathbf{x}^{\prime}_{t-1}$ (generated through posterior
sampling of $\mathbf{x}_{0}^{\prime}$) with respect to $t$ and
$\mathbf{x}_{t}$, conditioned on $\mathbf{h}_{spk}$ and $\mathbf{h}_{txt}$.
The objective function of $G_{\theta}$ is then defined as follows:

$\mathcal{L}_{G_{\theta}}^{adv}=\sum_{t\geq
1}\mathbb{E}[(D_{\phi}(\mathbf{x}^{\prime}_{t-1},\mathbf{x}_{t},t,\mathbf{h}_{txt},\mathbf{h}_{spk})-1)^{2}],$
(20)

$\mathcal{L}_{G_{\theta}}^{rec}=L_{MAE}(\mathbf{x}_{0},\hat{\mathbf{x}}_{0}),$
(21)

where $\mathcal{L}_{G_{\theta}}^{adv}$ corresponds to the adversarial loss,
and $\mathcal{L}_{G_{\theta}}^{rec}$ denotes the reconstruction loss of
$G_{\theta}$. The total generator loss $\mathcal{L}_{G_{\theta}}$ is expressed
as follows:

$\mathcal{L}_{G_{\theta}}=\mathcal{L}_{G_{\theta}}^{rec}+\lambda_{2}\mathcal{L}_{G_{\theta}}^{adv},$
(22)

where $\lambda_{2}$ is the weight of adversarial loss
$\mathcal{L}_{G_{\theta}adv}$. The objective function of the $D_{\phi}$ is as
follows:

$\mathcal{L}_{D_{\phi}}=\sum_{t\geq
1}[\mathbb{E}[D_{\phi}(\mathbf{x}^{\prime}_{t-1},\mathbf{x}_{t},t,\mathbf{h}_{txt},\mathbf{h}_{spk})^{2}]\\\
+\mathbb{E}[(D_{\phi}(\mathbf{x}_{t-1},\mathbf{x}_{t},t,\mathbf{h}_{txt},\mathbf{h}_{spk})-1)^{2}]].$
(23)

The DLPG leverages the DDGAN framework to achieve stable and high-quality
results in only a few timesteps. This process involves the $G_{\theta}$, which
iteratively generates $\mathbf{x}^{\prime}_{0}$ $T$ times during the
inference. We set $\mathbf{x}_{T}$ to follow a normal distribution. The
$\mathbf{h}_{pros}^{\prime}$ is obtained as the final
$\mathbf{x}^{\prime}_{0}$ of the reverse process. The final prosody vector
$\mathbf{z}_{pros}$ is then derived through the vector quantization of
$\mathbf{h}_{pros}^{\prime}$.

###  III-E Inference

Here is the step-by-step process of generating a Mel-spectrogram using the
trained TTS module and DLPG.

  1. 1.

The $\mathbf{h}_{txt}$ is extracted from the text encoder, and
$\mathbf{h}_{spk}$ is extracted from the pre-trained speaker encoder.

  2. 2.

The DLPG generates a $\mathbf{h}_{pros}^{\prime}$ with $\mathbf{h}_{txt}$ and
$\mathbf{h}_{spk}$ as inputs.

  3. 3.

$\mathbf{h}_{pros}^{\prime}$ is mapped to a codebook, denoted as $Z$, in the
VQ layer to obtain the prosody vector, which is denoted as
$\mathbf{z}_{pros}$.

  4. 4.

The decoder $D_{mel}$ generates a Mel-spectrogram $\mathbf{y}^{\prime}$ using
$\mathbf{h}_{txt}$, $\mathbf{h}_{spk}$ and $\mathbf{z}_{pros}$. This process
involves the expansion $\mathbf{h}_{txt}$, $\mathbf{h}_{spk}$, and
$\mathbf{z}_{pros}$ to the frame-level. The phoneme-duration is predicted by
the duration predictor.

  5. 5.

The Mel-spectrogram $y^{\prime}$ was converted to a raw waveform using a pre-
trained vocoder.

##  IV Experimental result and discussion

###  IV-A Experimental setup

We conducted experiments using the VCTK dataset, a multispeaker English
dataset consisting of audio clips of 44,200 sentences recorded by 109 speakers
for approximately 400 sentences. A total of 2,180 audio clips were randomly
selected with 20 sentences per speaker, which constituted the test set.
Furthermore, 545 audio clips randomly selected from five sentences per speaker
were used as the verification sets, and the remainder were used as the
training sets. We sampled audio at 22,050 Hz and then transformed it to an
80-bin Mel-spectrogram using an STFT with a window length of 1,024 and hop
size of 256. The text was converted into phoneme sequences for text input
using a grapheme-to-phoneme tool555https://github.com/Kyubyong/g2p. We
extracted the phoneme duration using the Montreal Forced Aligner[46] tool.
Furthermore, we used the AdamW optimizer[47] with $\beta_{1}=0.9$ and
$\beta_{2}=0.98$. The learning rates for the TTS and latent prosody generator
training were set as $5\times 10^{-4}$ and $2\times 10^{-4}$, respectively.
Throughout the training process, the batch size used was 48. The TTS and
prosody encoder were updated in 160k steps, and the latent prosody generator
was updated in 320k steps. In the experiment, all the audio was synthesized
using the official implementation of the HiFi-
GAN666https://github.com/jik876/hifi-gan[48] and a pre-trained model. We
trained the TTS module and prosody encoder for approximately 16 h and the DLPG
for 7 h using a single NVIDIA RTX A6000 GPU.

###  IV-B Implementation details

The number of layers, hidden size, filter size, and kernel size of the feed-
forward transformer blocks of the phoneme encoder, word encoder, and decoder
were set as 4, 192, 384, and 5, respectively. The extracted speaker embedding
was projected onto 192 dimensions, and the number of dimensions of the prosody
vector was 192. The structure of the prosody encoder follows that of
ProsoSpeech[27]. For VQ, we set the size of codebook and dimension of code as
128 and 192, respectively. and updated it using EMA with a decay rate of
0.998. The codebook was initialized as the center of the k-means clustering
after 20k steps of TTS training. The number of Mel-spectrogram bins used in
the prosody encoder $N$ was set as 20. We set multiple PCDs to receive
different input sizes, such as[49, 50, 51]. The PCD comprises two 2D
convolution stacks and three fully connected layers. The convolution stacks in
the PCD consist of three 2D convolutions: LeakyReLU, BatchNorm, and a linear
layer with a multilength window size of [32, 64, 128]. The latent prosody
generator consists of 20 residual blocks with a hidden size of 384. The
prosody discriminator is configured with four convolution layers and hidden
dimensions of 384. We investigated $\lambda_{1}$ and $\lambda_{2}$ between
0.001 and 1.0 and set $\lambda_{1}$ and $\lambda_{2}$ as 0.01 and 0.05,
respectively. The number of timesteps used in the training and inference was
set as four.

###  IV-C Comparative studies

We developed a prosody model from previous works, our proposed model, and the
model of the ablation study for performing comparative experiments. All the
models produced an 80-bin Mel-spectrogram and synthesized speech using the
same vocoder.

####  IV-C1 GT

GT audio is human-recorded.

####  IV-C2 GT (vocoded)

The generated audio was obtained by converting the GT Mel-spectrogram using
HiFi-GAN V1[48].

####  IV-C3 FastSpeech 2

FastSpeech 2[3] is a speech-synthesis model that directly predicts prosody
features (pitch and energy). We trained FastSpeech 2 using an open-source
implementation777https://github.com/NATSpeech/NATSpeech. For the purpose of a
fair comparison, we used a text encoder at the phoneme and word-levels.

####  IV-C4 ProsoSpeech

ProsoSpeech[27] is a speech synthesis model that models prosody features as
latent prosody vectors and predicts them using an AR predictor. We implemented
the model by following the hyperparameters in the study and used the model
that provided the best performance.

####  IV-C5 DiffProsody

The proposed model was implemented using a text encoder and decoder with the
same structure as FastSpeech 2 and ProsoSpeech. We used a prosody encoder with
the same structure as that of ProsoSpeech. In the first stage, a prosody
conditional discriminator was added. In the second stage, a DLPG was used
instead of an AR predictor.

####  IV-C6 DiffProsody (AR)

DiffProsody (AR) is a model in which a DiffProsody trained with an AR prosody
predictor to estimate prosody vectors. It has the same structure as the
ProsoSpeech prosody predictor.

####  IV-C7 DiffProsody (DDPM)

DiffProsody (DDPM) is a model in which a DiffProsody trained with a DDPM
framework to estimate a prosody vector. It has the same structure as
DiffProsody’s diffusion-based latent generator.

####  IV-C8 DiffProsody (w/o PCD)

DiffProsody (w/o PCD) is a model in which a DiffProsody trained without the
assistance of a prosody conditional discriminator.

####  IV-C9 DiffProsody (w/o VQ)

DiffProsody (w/o VQ) is a model in which a DiffProsody trained without a VQ
layer in the prosody encoder.

####  IV-C10 DiffProsody N

DiffProsody $N$ is a model in which a DiffProsody trained with the lowest $N$
Mel-bins in the prosody encoder.

###  IV-D Subjective metrics

A subjective assessment is conducted to confirm the effectiveness of the
proposed method. To measure the level of naturalness, we used the mean opinion
score (MOS). For this evaluation, we employed Amazon Mechanical Turk (MTurk),
a crowdsourcing service, to gather feedback from 20 native Americans. MOS was
assessed using a 5-point scale, and confidence intervals were calculated at a
95% level. For the evaluation, 100 samples were randomly selected from the
test set.

###  IV-E Objective metrics

For realizing an objective evaluation, we calculated the equal error rate
(EER) using a pre-trained speaker verification
model888https://github.com/clovaai/voxceleb_trainer [52]. We used the pre-
trained wav2vec 2.0[53] to the compute character error rate (CER) and word
error rate (WER).

For the prosodic evaluation, we computed the average differences in utterance
duration (DDUR)[54, 55], pitch error (RMSE${}_{f_{0}}$) (in cents),
periodicity error (RMSEperiod) and F1 score of the voiced/unvoiced
classification (F1v/uv). We used
torchcrepe999https://github.com/maxrmorrison/torchcrepe to extract the pitch
and periodicity features for evaluation. In addition, we measured the
Kullback–Leibler (KL) divergence of log f0 and log energy to compare the
distributions of the prosody features in the generated audio. Finally, we
calculated the real-time factor (RTF) to compare the generation speeds.

####  IV-E1 Average differences in the utterance duration (DDUR)

We obtained $DDUR$ by calculating the mean absolute error of the difference in
the duration of each utterance.

$DDUR=\frac{1}{N}\sum_{i=1}^{N}|dur_{i}-dur_{i}^{\prime}|,$ (24)

where $dur_{i}$ denotes the duration of the $i$-th GT utterance and
$dur_{i}^{\prime}$ denotes the duration of the $i$-th generated utterance.

####  IV-E2 Pitch error

To measure the pitch error RMSE${}_{f_{0}}$, we aligned the pitches extracted
in hertz with dynamic time warping (DTW)[56] and calculated the root mean
square (RMSE) in cents, defined as $1200\log_{2}(y/\hat{y})$ for pitch of the
GT speech $y$ and pitch of the generated speech $\hat{y}$. We measured the
portion wherein both the GT speech and generated speech were voiced.

$RMSE_{f_{0}}=\sqrt{\frac{1}{T}\sum_{i=1}^{T}(1200\log_{2}(y_{i}/y_{i}^{\prime}))^{2}}.$
(25)

####  IV-E3 Periodicity error

We measured the periodicity error RMSEperiod by root mean square between the
periodicities $\psi$ aligned with the DTW.

$RMSE_{period}=\sqrt{\frac{1}{T}\sum_{i=1}^{T}(\psi_{i}-\psi_{i}^{\prime})^{2}},$
(26)

where $\psi_{i}$ means the $i$-th periodicity value.

####  IV-E4 F1 score of voiced/unvoiced

We obtained voiced/unvoiced flags from the aligned pitches using DTW and
calculated the F1 score (F1v/uv) between them. We defined a match between the
GT voiced flag $u$ and generated voiced flag $u^{\prime}$ as a true positive
(TP), a match between the GT unvoiced flag $uv$ and generated voiced flag
$v^{\prime}$ as a false positive (FP), and a match between the GT voiced flag
$v$ and generated unvoiced flag $uv^{\prime}$ as a false negative (FN).

$TP=\sum_{i}^{n}[v_{i}=v_{i}^{\prime}],$ (27)
$FP=\sum_{i}^{n}[uv_{i}=v_{i}^{\prime}],$ (28)
$FN=\sum_{i}^{n}[v_{i}=uv_{i}^{\prime}],$ (29)

where $n$ is the length of the sequence, and $[a_{i}=b_{i}]$ is a function
that returns 1 if the $i$-th element has the same value, and 0 otherwise. The
precision and recall are defined as follows:

$precision=\frac{TP}{TP+FP},recall=\frac{TP}{TP+FN}.$ (30)

We then calculated the F1 score as follows:

$F1_{v/uv}=\frac{2}{recall^{-1}+precision^{-1}}.$ (31)

####  IV-E5 KL divergence of log f0 / log energy

We also measured the KL D to analyze the pitch and energy distribution. We
first extracted the pitch and energy in log-scale. We then binned the entire
range into 100 bins and applied a kernel density estimation[57] to each bin to
calculate the KL D for a smoothed distribution. The KL divergence for feature
$x$ is defined as follows:

$KLD_{x}=\frac{1}{N}\sum_{i=1}^{N}(KLD(KDE(x_{i}),KDE(x_{i}^{\prime})),$ (32)

where $N$ is the number of bins, $x_{i}$ is the probability in the $i$-th bin
of the distribution of $x$, $KDE$ is the kernel density estimator function,
and $KLD$ is the KL divergence calculation.

TABLE I: Objective and subjective evaluation of previous methods

Method |  MOS (CI) ($\uparrow$) |  CER (%) ($\downarrow$) |  WER (%) ($\downarrow$) |  DDUR ($\downarrow$) |  EER (%) ($\downarrow$) |  RMSEf0 ($\downarrow$) |  RMSEperiod ($\downarrow$) |  F1v/uv ($\uparrow$) |  RTF ($\downarrow$) | Params
---|---|---|---|---|---|---|---|---|---|---
GT | 4.24 ($\pm$ 0.02) | 0.63 | 1.53 | - | 2.184 | - | - | - | - | -
GT (Vocoded) | 4.17 ($\pm$ 0.03) | 0.76 | 1.74 | - | 2.539 | 22.80 | 0.336 | 0.7923 | - | -
FastSpeech2 | 3.84 ($\pm$ 0.03) | 1.81 | 4.89 | 0.305 | 8.685 | 88.83 | 0.486 | 0.6848 | 0.015 | 27M
ProsoSpeech | 3.97 ($\pm$ 0.04) | 1.68 | 3.82 | 0.296 | 7.200 | 59.66 | 0.483 | 0.6874 | 0.149 | 59M
DiffProsody |  4.03 ($\pm$ 0.03) | 0.90 | 2.55 | 0.295 | 5.404 | 54.60 | 0.475 | 0.6952 | 0.054 | 53M

###  IV-F Evaluation results

Table I lists the MOS results and objective evaluations. The results
demonstrate that the proposed model DiffProsody surpasses the other models in
terms of both subjective and objective metrics. In particular, our model
achieved a superior MOS for the subjective metrics, with a p-value of less
than 0.05. It also exhibited outstanding performance in terms of the DDUR,
EER, RMSE${}_{f_{0}}$, RMSEperiod, and F1v/uv, which suggests that the speech
generated by DiffProsody closely resembles the target prosody. Furthermore,
our model displayed a lower WER and CER, thus indicating its capability for
synthesizing speech with a more accurate pronunciation.

(a) Ground truth

(b) FastSpeech2

(c) ProsoSpeech

(d) DiffProsody

Figure 3: Comparison of the visualized spectrogram and pitch contour. The red
box indicates that the proposed model is more similar to the GT.

Figure 3 presents the Mel-spectrogram and pitch contour of speech from each
model. The red box in the figure indicates that the DiffProsody model was more
similar to the GT. To further evaluate the prosody of the generated speech, we
examined pitch and energy distributions.

Figure 5 presents the histogram distribution for log f0 (pitch), and Figure 5
presents the histogram distribution for log energy. In both the figures, the
blue bars represent the distribution of the GT features, and the orange bars
represent the distribution of the generated features. The results indicate
that the proposed model aligns more closely with the GT distribution than the
other models. Table II lists the KL divergence values for comparison.
ProsoSpeech exhibited a better performance in f0 than FastSpeech 2, but not in
the case of energy. However, DiffProsody outperformed the comparison model in
terms of f0 and energy.

###  IV-G Latent prosody generation method

In Table III, DiffProsody (AR) is a model comprising the use of an AR prosody
predictor. DiffProsody (DDPM) is a model comprising the use of DDPM (100
timesteps). DiffProsody is a model comprising the use of DDGAN (four
timesteps). The AR predictor has the same structure as the prosody predictor
in ProsoSpeech, but it does not use context encoders, and the denoising
network in the DDPM has the same structure as generator $g_{\theta}$ in the
DDGAN. We also conducted a 7-point comparative mean opinion score (CMOS)
evaluation to compare the latent prosody generation methods and measured the
RTF to compare the generation speeds. DDGAN achieved $-$0.172 and $+$0.015
CMOS compared with AR and DDPM, respectively. The objective metrics showed
that the DDPM and DDGAN outperformed the AR. The DDGAN achieved results nearly
identical to those of the DDPM for all the metrics. The experimental results
showed that the diffusion models performed better than the AR models, as
reported in[58]. Furthermore, DDGAN can generate high-quality prosody vectors
such as DDPM in only four timesteps. According to the RTF results, the DDGAN
produces a 2.7$\times$ and 16$\times$ faster prosody than the AR and DDPM,
respectively.

(a) FastSpeech2

(b) ProsoSpeech

(c) DiffProsody

Figure 4: Histogram visualization of log f0, where the blue bars represent the
GT distribution and orange bars represent the generated distribution. The
distribution of the proposed model overlaps to a greater extent with the GT
distribution than the other comparison models.

(a) FastSpeech2

(b) ProsoSpeech

(c) DiffProsody

Figure 5: Histogram visualization of log energy, where the blue bars represent
the GT distribution and orange bars represent the generated distribution. The
proposed model’s distribution has a greater overlap with the GT distribution
compared to the other models being compared. TABLE II: KL Divergence of log f0
and energy

Number of Mel-bins | KL divergence $\log{f0}$ | KL divergence $\log{energy}$
---|---|---
FastSpeech 2 | 0.00574 | 0.02375
ProsoSpeech | 0.00542 | 0.02657
DiffProsody | 0.00343 | 0.01547

Figure 6 presents the traces of the WER, CER, and EER results for the speech
generated by the DLPG with the prosody vector generated by each denoising
iteration. The red and blue lines represent the results obtained using the
DDPM and DDGAN, respectively. We observed that the DDGAN has a larger error
rate than the DDPM in the early stages and that the midpoint and final stages
of each model have almost the same value. This implies that the DDGAN
compresses the timesteps of the DDPM by modeling the denoising process as a
multimodal non-Gaussian distribution.

TABLE III: Objective and subjective evaluations of ablation study models

Method | CMOS |  CER (%) ($\downarrow$) |  WER (%) ($\downarrow$) |  DDUR ($\downarrow$) |  EER (%) ($\downarrow$) |  RMSEperiod ($\downarrow$) |  RMSEperiod ($\downarrow$) |  F1v/uv ($\uparrow$) |  RTF ($\downarrow$) | Params
---|---|---|---|---|---|---|---|---|---|---
DiffProsody | 0.0 | 0.90 | 2.55 | 0.295 | 5.404 | 54.60 | 0.475 | 0.6952 | 0.054 | 53M
DiffProsody (AR) |  $-$0.172 | 1.41 | 3.73 | 0.308 | 6.947 | 65.63 | 0.486 | 0.6797 | 0.134 | 59M
DiffProsody (DDPM) |  $+$0.015 | 0.96 | 2.76 | 0.282 | 5.263 | 54.04 | 0.477 | 0.6933 | 0.871 | 53M
DiffProsody (w/o PCD) | - | 1.19 | 3.39 | 0.306 | 5.762 | 56.09 | 0.477 | 0.6921 | 0.054 | 53M
DiffProsody (w/o VQ) | - | 1.41 | 3.73 | 0.308 | 7.306 | 62.03 | 0.482 | 0.6783 | 0.054 | 53M

(a) WER

(b) CER

(c) EER

Figure 6: Comparison of objective evaluation results based on diffusion
timesteps when using the DDPM and DDGAN framework in DLPG. The blue line is
the result for the DDGAN and the red line is the result for the DDPM. TABLE
IV: CMOS results for prosody conditional discriminator

Method | reference | CMOS |
---|---|---|---
DiffProsody | ✓ | 0.0 |
DiffProsody (w/o PCD) |  |  $-$0.171 |
DiffProsody | ✓ | 0.0 |
ProsoSpeech |  |  $-$0.183 |
DiffProsody (w/o PCD) | ✓ | 0.0 |
ProsoSpeech |  |  $-$0.065 |

###  IV-H Prosody conditional adversarial training

We conducted CMOS to assess the effectiveness of PCD. The CMOS values were
used to measure the degree of PCD preference in comparison with the reference
model. Our evaluation involved the analysis of three different models:
DiffProsody using PCD and a diffusion-based model, DiffProsody without PCD
(referred to as DiffProsody (w/o PCD)) using a diffusion-based model, and
ProsoSpeech using an AR prosody predictor without PCD. The results presented
in Table IV clearly indicate that models comprising the use of the PCD are
preferred over those trained without the PCD. Furthermore, by comparing the
CMOS results of DiffProsody (w/o PCD) and ProsoSpeech, we can infer that the
diffusion-based method is preferable to the method employed by ProsoSpeech. To
provide a more comprehensive analysis, we also incorporated the objective
metrics for DiffProsody (w/o PCD) in Table III. These objective metric results
demonstrate that DiffProsody outperformed DiffProsody (w/o PCD) in all
aspects. However, DiffProsody (w/o PCD) still received a better objective
evaluation than ProsoSpeech. In addition, when comparing DiffProsody (AR) with
ProsoSpeech, DiffProsody (AR) consistently achieved higher scores on the
majority of the objective evaluations. These findings validate that both PCD
and DLPG significantly improve the model performance.

###  IV-I Prosody encoder evaluation

In this section, the effectiveness of the prosody encoder is evaluated. We
focus on two main aspects: the impact of the number of Mel-bins used in the
reference Mel-spectrogram and the role of the vector quantization layer.
Detailed results of these evaluations are presented in the following
subsections.

Figure 7: Comparison of Mel-spectrograms at iteration 0 (noise) in diffusion
with DiffProsody trained on the number of bins $N$ in the Mel spectrogram. The
Mel-spectrogram exhibits a gradual collapse as $N$ increases. This occurs when
disentangling has failed, and the prosody vector is responsible for the
linguistic information.

####  IV-I1 Impact of the number of Mel-bins

We conducted an experiment to compare the performance of DiffProsody when
trained using various numbers of Mel-bins. The results, including the EER,
CER, and WER scores, are presented in Table V. We examined the performance of
10, 20, 30, 40, 60, and 80 (full-band) Mel-bins. The findings indicated that
as the number of bins exceeded 20 (baseline), the EER tended to increase,
while no significant difference was observed in terms of the CER and WER. It
should be noted that the model with 10 bins outperformed the baseline (20
bins) in terms of the WER but yielded higher EER results. Figure 7 illustrates
the Mel-spectrogram at the initial iteration (noise) during the diffusion
timestep of the DLPG, which was trained using various numbers of Mel-bins. As
the number of Mel-bins used for the training was increased, the results of the
Mel-spectrogram progressively smoothened and eventually collapsed. This
phenomenon occurs because the large amount of information in the reference
forces the model to reconstruct the Mel-spectrogram by leveraging the prosody
vectors. Through this experiment, we found that, as $N$ increases, linguistic
information becomes increasingly entangled. Consequently, it is reasonable to
employ 20 Mel-bins as the input to the prosody encoder for realizing effective
prosody modeling.

(a) DiffProsody (w/o VQ)

(b) DiffProsody

Figure 8: Visualization of a Mel-spectrogram trace synthesized with latent
generation for each diffusion timestep. (a) DiffProsody (w/o VQ) trained
without vector quantization in prosody encoder; (b) DiffProsody. (a) shows
that, in contrast (b), the early stages of diffusion produce a completely
collapsed Mel-spectrogram. This is caused by the inclusion of linguistic
information in the prosody vector.

####  IV-I2 Vector quantization layer analysis

Figure 8 presents a Mel-spectrogram trace synthesized using the prosody vector
generated for each diffusion step of the DLPG. We compared two versions of
DiffProsody: one trained without the vector quantization layer (Figure 8a;
defined as DiffProsody (w/o VQ)) and the other trained with the vector
quantization layer (Figure 8b; defined as DiffProsody). In the case of
DiffProsody (w/o VQ), the early steps exhibited a completely distorted Mel-
spectrogram, but there was a significant recovery in the middle steps.
Conversely, the initial step of DiffProsody exhibits a smoothed but slightly
distorted Mel-spectrogram that gradually returns to its original state over
the subsequent steps. This phenomenon is also related to prosody
disentangling, and we confirmed that the prosody disentangling failed in
DiffProsody (w/o VQ). This experiment demonstrated that vector quantization
plays a crucial role in effective prosody disentangling. The last row of Table
III presents the objective evaluation results for DiffProsody (w/o VQ).
DiffProsody (w/o VQ) performed worse for all the objective measurements. These
results provide an objective assessment of how the failure to properly
disentangle prosody affects the overall performance of the system.

TABLE V: Objective evaluation based on the number of Mel-bins

Number of Mel-bins | EER | CER | WER
---|---|---|---
DiffProsody (10) | 5.71 | 0.95 | 2.49
DiffProsody (20) | 5.40 | 0.90 | 2.55
DiffProsody (30) | 6.26 | 1.0 | 2.58
DiffProsody (40) | 6.56 | 0.98 | 2.59
DiffProsody (60) | 6.20 | 0.97 | 2.52
DiffProsody (80) | 7.11 | 0.98 | 2.67

##  V Conclusion

In this study, a novel technique called DiffProsody is proposed, the aims of
which is to synthesize high-quality expressive speech. Through prosody
conditional adversarial training, we observed significant improvements in
speech quality with a more pronounced display of expressive prosody. In
addition, our DLPG successfully generated expressive prosody. Our proposed
method outperformed comparative models in terms of producing accurate and
expressive prosody, as evidenced by the prosody evaluation metrics. Moreover,
our method demonstrated superior accuracy in pronunciation, as indicated by
the CER and WER evaluations. The KL divergence and histogram analysis further
support the claim that DiffProsody yields a more accurate prosody distribution
than the other models. Furthermore, we successfully reduced the sampling speed
while maintaining the expected performance by introducing DDGAN.

##  VI Future works

Despite the importance of vector quantization for disentangling, it has been
observed that this approach can negatively affect model performance. This
problem is expected to be addressed with the introduction of methods such as
residual vector quantizers[59, 60, 61]. Moreover, we acknowledge the
limitations of attempting to model prosody using the TTS dataset. It has been
suggested that using a language model pre-trained on a large dataset, such as
HuBERT[62], could result in significant improvements. In the future, we plan
to extend the latent diffusion method to include controllable emotional
prosody modeling[63].

## References

  * [1] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu, “Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 4779–4783.
  * [2] Y. Ren, Y. Ruan, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “FastSpeech: Fast, Robust and Controllable Text to Speech,” in _Advances in Neural Information Processing Systems_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 2019. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2019/file/f63f65b503e22cb970527f23c9ad7db1-Paper.pdf
  * [3] Y. Ren, C. Hu, X. Tan, T. Qin, S. Zhao, Z. Zhao, and T.-Y. Liu, “FastSpeech 2: Fast and High-Quality End-to-End Text to Speech,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=piLPYqxtWuA
  * [4] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron: Towards End-to-End Speech Synthesis,” in _Proc. Interspeech 2017_ , 2017, pp. 4006–4010.
  * [5] Y. Ren, J. Liu, and Z. Zhao, “PortaSpeech: Portable and High-Quality Generative Text-to-Speech,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 13 963–13 974. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2021/file/748d6b6ed8e13f857ceaa6cfbdca14b8-Paper.pdf
  * [6] J. Liu, C. Li, Y. Ren, F. Chen, and Z. Zhao, “DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 10, pp. 11 020–11 028, Jun. 2022. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/21350
  * [7] S.-H. Lee, H.-R. Noh, W.-J. Nam, and S.-W. Lee, “Duration Controllable Voice Conversion via Phoneme-Based Information Bottleneck,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 1173–1183, 2022.
  * [8] R. Skerry-Ryan, E. Battenberg, Y. Xiao, Y. Wang, D. Stanton, J. Shor, R. Weiss, R. Clark, and R. A. Saurous, “Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 4693–4702. [Online]. Available: https://proceedings.mlr.press/v80/skerry-ryan18a.html
  * [9] G. Xu, W. Song, Z. Zhang, C. Zhang, X. He, and B. Zhou, “Improving Prosody Modelling with Cross-Utterance Bert Embeddings for End-to-End Speech Synthesis,” in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6079–6083.
  * [10] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor, Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 5180–5189. [Online]. Available: https://proceedings.mlr.press/v80/wang18h.html
  * [11] D. Min, D. B. Lee, E. Yang, and S. J. Hwang, “Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation,” in _Proceedings of the 38th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 7748–7759. [Online]. Available: https://proceedings.mlr.press/v139/min21b.html
  * [12] J.-H. Lee, S.-H. Lee, J.-H. Kim, and S.-W. Lee, “PVAE-TTS: Adaptive Text-to-Speech via Progressive Style Adaptation,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 6312–6316.
  * [13] N. Kumar, A. Narang, and B. Lall, “Zero-Shot Normalization Driven Multi-Speaker Text to Speech Synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 1679–1693, 2022.
  * [14] S. Seshadri, T. Raitio, D. Castellani, and J. Li, “Emphasis Control for Parallel Neural TTS,” in _Proc. Interspeech 2022_ , 2022, pp. 3378–3382.
  * [15] H. Bae and Y.-S. Joo, “Enhancement of Pitch Controllability using Timbre-Preserving Pitch Augmentation in FastPitch,” in _Proc. Interspeech 2022_ , 2022, pp. 6–10.
  * [16] A. Łańcucki, “Fastpitch: Parallel Text-to-Speech with Pitch Prediction,” in _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021, pp. 6588–6592.
  * [17] T. Raitio, J. Li, and S. Seshadri, “Hierarchical Prosody Modeling and Control in Non-Autoregressive Parallel Neural TTS,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7587–7591.
  * [18] M. Chen, X. Tan, B. Li, Y. Liu, T. Qin, sheng zhao, and T.-Y. Liu, “AdaSpeech: Adaptive Text to Speech for Custom Voice,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=Drynvt7gg4L
  * [19] D. Stanton, Y. Wang, and R. Skerry-Ryan, “Predicting Expressive Speaking Style from Text in End-To-End Speech Synthesis,” in _2018 IEEE Spoken Language Technology Workshop (SLT)_ , 2018, pp. 595–602.
  * [20] L. Xue, F. K. Soong, S. Zhang, and L. Xie, “ParaTTS: Learning Linguistic and Prosodic Cross-Sentence Information in Paragraph-Based TTS,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 2854–2864, 2022.
  * [21] C. Du and K. Yu, “Phone-Level Prosody Modelling with GMM-Based MDN for Diverse and Controllable Speech Synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 190–201, 2022.
  * [22] R. Liu, B. Sisman, G. Gao, and H. Li, “Expressive TTS Training with Frame and Style Reconstruction Loss,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 1806–1818, 2021.
  * [23] B. Stephenson, L. Besacier, L. Girin, and T. Hueber, “BERT, can HE predict contrastive focus? Predicting and controlling prominence in neural TTS using a language model,” in _Proc. Interspeech 2022_ , 2022, pp. 3383–3387.
  * [24] H.-W. Yoon, O. Kwon, H. Lee, R. Yamamoto, E. Song, J.-M. Kim, and M.-J. Hwang, “Language Model-Based Emotion Prediction Methods for Emotional Speech Synthesis Systems,” in _Proc. Interspeech 2022_ , 2022, pp. 4596–4600.
  * [25] P. Makarov, S. Ammar Abbas, M. Łajszczak, A. Joly, S. Karlapati, A. Moinet, T. Drugman, and P. Karanasou, “Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody,” in _Proc. Interspeech 2022_ , 2022, pp. 3368–3372.
  * [26] J.-S. Hwang, S.-H. Lee, and S.-W. Lee, “PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling,” _arXiv preprint arXiv:2306.07489_ , 2023.
  * [27] Y. Ren, M. Lei, Z. Huang, S. Zhang, Q. Chen, Z. Yan, and Z. Zhao, “Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7577–7581.
  * [28] J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf
  * [29] H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu, “LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech,” in _Proc. Interspeech 2019_ , 2019, pp. 1526–1530.
  * [30] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you Need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017\. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [31] J. Kim, S. Kim, J. Kong, and S. Yoon, “Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 8067–8077. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/5c3b99e8f92532e5ad1556e53ceea00c-Paper.pdf
  * [32] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, and M. Kudinov, “Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech,” in _Proceedings of the 38th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. Meila and T. Zhang, Eds., vol. 139. PMLR, 18–24 Jul 2021, pp. 8599–8608. [Online]. Available: https://proceedings.mlr.press/v139/popov21a.html
  * [33] J. Kim, J. Kong, and J. Son, “Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 5530–5540.
  * [34] S.-H. Lee, S.-B. Kim, J.-H. Lee, E. Song, M.-J. Hwang, and S.-W. Lee, “HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis,” in _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 16 624–16 636. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/69c754f571806bf15add18556ff39b4f-Paper-Conference.pdf
  * [35] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Nets,” in _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
  * [36] Z. Xiao, K. Kreis, and A. Vahdat, “Tackling the Generative Learning Trilemma with Denoising Diffusion GANs,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=JprM0p-q0Co
  * [37] X. An, F. K. Soong, and L. Xie, “Disentangling Style and Speaker Attributes for TTS Style Transfer,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 646–658, 2022.
  * [38] J. Lee, W. Jung, H. Cho, and J. Kim, “PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTS,” _arXiv preprint arXiv:2302.12391_ , 2023.
  * [39] Y. Ju, I. Kim, H. Yang, J.-H. Kim, B. Kim, S. Maiti, and S. Watanabe, “TriniTTS: Pitch-controllable End-to-end TTS without External Aligner,” in _Proc. Interspeech 2022_ , 2022, pp. 16–20.
  * [40] J. D. M.-W. C. Kenton and L. K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in _Proceedings of NAACL-HLT_ , 2019, pp. 4171–4186.
  * [41] T. Brown, B. MaWnn, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language Models are Few-Shot Learners,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 1877–1901. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * [42] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized End-to-End Loss for Speaker Verification,” in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2018, pp. 4879–4883.
  * [43] A. van den Oord, O. Vinyals, and k. kavukcuoglu, “Neural Discrete Representation Learning,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/7a98af17e63a0ac09ce2e96d03992fbc-Paper.pdf
  * [44] Y. Ren, X. Tan, T. Qin, Z. Zhao, and T.-Y. Liu, “Revisiting Over-Smoothness in Text to Speech,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2022, pp. 8197–8213.
  * [45] X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley, “Least Squares Generative Adversarial Networks,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [46] M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi,” in _Proc. Interspeech 2017_ , 2017, pp. 498–502.
  * [47] I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7
  * [48] J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 17 022–17 033. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/c5d736809766d46260d816d8dbc9eb44-Paper.pdf
  * [49] S.-H. Lee, H.-W. Yoon, H.-R. Noh, J.-H. Kim, and S.-W. Lee, “Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 14, pp. 13 198–13 206, May 2021. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/17559
  * [50] Z. Ye, Z. Zhao, Y. Ren, and F. Wu, “SyntaSpeech: Syntax-Aware Generative Adversarial Text-to-Speech,” _International Joint Conferences on Artificial Intelligence (IJCAI)_ , 2022.
  * [51] L. Zhang, Y. Ren, L. Deng, and Z. Zhao, “HiFiDenoise: High-Fidelity Denoising Text to Speech with Adversarial Networks,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7232–7236.
  * [52] J. weon Jung, Y. Kim, H.-S. Heo, B.-J. Lee, Y. Kwon, and J. S. Chung, “Pushing the limits of raw waveform speaker recognition,” in _Proc. Interspeech 2022_ , 2022, pp. 2228–2232.
  * [53] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations,” in _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, Eds., vol. 33. Curran Associates, Inc., 2020, pp. 12 449–12 460. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2020/file/92d1e1eb1cd6f9fba3227870bb6d7f07-Paper.pdf
  * [54] J.-X. Zhang, Z.-H. Ling, L.-J. Liu, Y. Jiang, and L.-R. Dai, “Sequence-to-Sequence Acoustic Modeling for Voice Conversion,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 3, pp. 631–644, 2019.
  * [55] J.-X. Zhang, Z.-H. Ling, and L.-R. Dai, “Non-Parallel Sequence-to-Sequence Voice Conversion With Disentangled Linguistic and Speaker Representations,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 540–552, 2020.
  * [56] M. Müller, “Dynamic time warping,” _Information retrieval for music and motion_ , pp. 69–84, 2007.
  * [57] G. R. Terrell and D. W. Scott, “Variable Kernel Density Estimation,” _The Annals of Statistics_ , pp. 1236–1265, 1992.
  * [58] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical Text-Conditional Image Generation with Clip Latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [59] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An End-to-End Neural Audio Codec,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 30, pp. 495–507, 2022.
  * [60] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “AudioLM : a Language Modeling Approach to Audio Generation,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , pp. 1–11, 2023.
  * [61] J.-S. Hwang, S.-H. Lee, and S.-W. Lee, “HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models,” _arXiv preprint arXiv:2306.06814_ , 2023.
  * [62] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [63] C.-B. Im, S.-H. Lee, S.-B. Kim, and S.-W. Lee, “EMOQ-TTS: Emotion Intensity Quantization for Fine-Grained Controllable Emotional Text-to-Speech,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 6317–6321.

|  Hyung-Seok Oh received the B.S. degree in Computer Science and Engineering
from Konkuk University, Seoul, South Korea, in 2021. He is currently working
toward the integrated master’s and Ph.D. degree with the Department of
Artificial Intelligence, Korea University, Seoul, South Korea. His research
interests include artificial intelligence and audio signal processing.
---|---
|  Sang-Hoon Lee received the B.S. degree in life science from Dongguk
University, Seoul, South Korea, in 2016 and received the Ph.D. degree in Brain
and Cognitive Engineering from Korea University, Seoul, South Korea, in 2023.
He is currently a postdoctoral researcher in AI Research Center, Korea
University, Seoul, South Korea. His current research interests include
artificial intelligence and audio signal processing.
---|---
|  Seong-Whan Lee (Fellow, IEEE) received the B.S. degree in computer science
and statistics from Seoul National University, South Korea, in 1984, and the
M.S. and Ph.D. degrees in computer science from the Korea Advanced Institute
of Science and Technology, South Korea, in 1986 and 1989, respectively. He is
currently the Head of the Department of Artificial Intelligence, Korea
University, Seoul. His current research interests include artificial
intelligence, pattern recognition, and brain engineering. He is a Fellow of
the International Association of Pattern Recognition (IAPR) and the Korea
Academy of Science and Technology.
---|---