# A Variational Prosody Model for Mapping the Context-Sensitive Variation of
Functional Prosodic Prototypes

Branislav Gerazov<EMAIL_ADDRESS>Gérard Bailly
<EMAIL_ADDRESS>Omar Mohammed<EMAIL_ADDRESS>Yi
Xu<EMAIL_ADDRESS>and Philip N. Garner<EMAIL_ADDRESS>Faculty of
Electrical Engineering and Information Technologies, Ss. Cyril and Methodius
University in Skopje, Skopje, Macedonia  Univ. Grenoble-Alpes, CNRS, Grenoble-
INP, GIPSA-lab, Grenoble, France  Department of Speech, Hearing and Phonetic
Sciences, University College London, London, UK  Idiap Research Institute,
Martigny, Switzerland

###### Abstract

The quest for comprehensive generative models of intonation that link
linguistic and paralinguistic functions to prosodic forms has been a
longstanding challenge of speech communication research. Traditional
intonation models have given way to the overwhelming performance of deep
learning (DL) techniques for training general purpose end-to-end mappings
using millions of tunable parameters. The shift towards black box machine
learning models has nonetheless posed the reverse problem – a compelling need
to discover knowledge, to explain, visualise and interpret. Our work bridges
between a comprehensive generative model of intonation and state-of-the-art DL
techniques. We build upon the modelling paradigm of the Superposition of
Functional Contours (SFC) model and propose a Variational Prosody Model (VPM)
that uses a network of variational contour generators to capture the context-
sensitive variation of the constituent elementary prosodic contours. We show
that the VPM can give insight into the intrinsic variability of these prosodic
prototypes through learning a meaningful prosodic latent space representation
structure. We also show that the VPM is able to capture prosodic phenomena
that have multiple dimensions of context based variability. Since it is based
on the principle of superposition, the VPM does not necessitate the use of
specially crafted corpora for the analysis, opening up the possibilities of
using big data for prosody analysis. In a speech synthesis scenario, the model
can be used to generate a dynamic and natural prosody contour that is devoid
of averaging effects.

###### keywords:

Prosody modelling, latent space, variational encoding, deep learning, prosody
decomposition

##  1 Introduction

The quest for comprehensive generative models of intonation that link
linguistic and paralinguistic functions to prosodic forms has been a
longstanding challenge of speech communication research. The first
quantitative models proposed in the early 70s by Fujisaki (1971), Gårding
(1983) and consorts, the transcription frameworks for intonation and rhythm,
e.g. TOBI (Silverman et al., 1992), INTSINT (Hirst, 2005), etc., and the more
complex generative models of intonation originally proposed at the turn of the
century by Bailly and Holm (2005) and Xu (2005), were focusing on extracting
elementary prosodic atoms or _geons_ (Biederman, 1987) that syntactically
combine to form the observed prosodic contours. The quest for building blocks
and organisation principles of intonation systems is fundamental for
understanding how oral languages are built, learnt and evolve. Modelling
constraints also spare data requirements – we certainly experience few samples
of the extraordinary varieties of expressive emotions, e.g. the Baron-Cohen et
al. (2004) taxonomy comprises 412 auditory-visual patterns, but we master most
of them on a large arsenal of textual supports.

Starting at the turn of the century, within the paradigm of statistical
parametric text-to-speech (TTS) synthesis, explicit prosody models gave way to
machine learning techniques based on Hidden Markov Models (Yoshimura et al.,
1999; Obin et al., 2011) and later Deep Learning (DL) (Zen et al., 2013; Zen,
2015; Yin et al., 2016; Wang et al., 2017a). Today, state-of-the-art TTS
systems are built on top of the overwhelming performance of DL techniques for
training general purpose end-to-end mappings using millions of tunable
parameters (Wang et al., 2017b; Arik et al., 2017; Sotelo et al., 2017;
Taigman et al., 2018; Shen et al., 2018). The success of these black box
machine learning models has nonetheless now posed the reverse problem – a
compelling need to discover knowledge, to explain, visualise, and interpret.
For example, one of the key challenges is the disentanglement of style from
textual content and its control (Wang et al., 2018; Skerry-Ryan et al., 2018;
Hsu et al., 2018; Ma et al., 2019).

On the other side, there is still a need to identify, analyse and catalogue
the immense variability of prosody in language, even in well studied languages
like English (Goodhue et al., 2016). This necessitates the creation of
specially tailored corpora that can be tedious to design and create,
especially if the prosodic variability depends on the interaction between
multiple linguistic and paralinguistic dimensions (Liu and Xu, 2005).

Our work bridges between a comprehensive generative model of intonation and
state-of-the-art DL techniques. We build upon the modelling paradigm of the
Superposition of Functional Contours (SFC) model (Bailly and Holm, 2005, 2002;
Morlec et al., 2001), by designing a deep neural network architecture
(Goodfellow et al., 2016) that incorporates variational encoding (Kingma and
Welling, 2013; Zhao et al., 2017) and is jointly trained using backpropagation
(Kingma and Ba, 2014). Our proposed Variational Prosody Model (VPM) has the
capability of capturing context-sensitive variations of the constituent
elementary prosodic atoms, i.e. multiparametric prosodic prototypes, or
_clichés_ (Fónagy et al., 1983) that encode given linguistic and
paralinguistic functions onto units of variable size, or scope. These
prototypes are multiparametric because they feed a multiparametric prosodic
score that includes intonation, rhythm, but can also include function-specific
eye, head and body movements, etc. Moreover, the VPM is able to learn a
meaningfully structured prosodic latent space representation of this
variability.

We compare here the modelling power of the VPM to the original SFC and the
weighted SFC (WSFC) (Gerazov et al., 2018b), which adjoins a weighting module
to each contour generator for scaling its contribution in the decomposition,
as well as a baseline deep model based on the Merlin speech synthesiser
prosody module Wu et al. (2016). We demonstrate that the proposed deep
architecture of the VPM while bringing added value, gives comparable if not
better modelling performance than our previous models. Finally, we show that
within the function-specific prosodic latent spaces, the VPM is able to
capture spatiotemporal variations of the _clichés_ that go beyond modelling
their mean (SFC) or simple amplitude scaling (WSFC). This can be used in the
exploration of context-specific prosodic phenomena. Moreover, the fact that it
is based on decomposing prosody into its constituents, allows the use of the
VPM on large unconstrained natural speech data, eliminating the need of
specially designed corpora. We also believe this variational modelling power
of the VPM can readily be used in a speech synthesis scenario to generate rich
and natural sounding prosodic contours.

##  2 The SFC and WSFC

The SFC modelling paradigm supposes that multiple functions acting on multiple
units are transmitted via prosody using a simple channel sharing procedure.
Thus, the prosody of the utterance is simply performed by overlapping-and-
adding all contributing multiparametric prototypes. The problem of decomposing
prosody into these elementary patterns is ill-posed because the SFC does not
impose any a priori constraints on the spatiotemporal patterns such as
bandwidth or shape. In the SFC, these _clichés_ , in fact, emerge from
statistical modelling – an iterative analysis-by-synthesis training process is
used to train function-specific contour generators (CGs) (Morlec, 1997), shown
in Fig. 1. The SFC has been successfully used to model different functions
acting at various linguistic levels, including: attitudes (Morlec et al.,
2001), grammatical dependencies (Morlec et al., 1998), cliticisation (Bailly
and Holm, 2002), focus (Brichet and Aubergé, 2004), as well as tones in
Mandarin (Chen et al., 2004).

Each of the contour generators is trained to encode a specific functional
contour across different scopes. The scope designates the number of rhythmic
units that the contour spans, e.g. syllables or inter perceptual centre groups
(Campbell, 1992). In the SFC, the CGs are implemented as a single shallow
neural network that takes as input the absolute and relative position of the
current RU in the linguistic function’s scope w.r.t. its boundaries and anchor
point, encoded through input ramps shown in Fig. 1. Based on this input, the
CG then outputs the prosodic contour for the function, one RU at a time. The
prosodic contour comprises pitch targets for each RU’s vowel nucleus, and a
duration modification coefficient based on the average RU duration that solely
depends on its phonetic constituents (Bailly and Holm, 2005).

Figure 1: The SFC contour generator (CG) generating a function’s prosody
contour for a given scope one rhythmic unit at a time. The shown function has
two subscopes divided by an anchor point (A).

Figure 2: Comparison of the contour generator architectures (left to right):
the WSFC weighted contour generator (WCG), and the VPM RU-based variational
contour generator (VCG) and the variational recurrent contour generator
(VRCG).

The Weighted SFC model (Gerazov et al., 2018b), introduces a weighted contour
generator (WCG) that incorporates a weight module responsible for scaling its
contribution in the decomposition, shown on the left in Fig. 2. This is
similar to the Mixture of Experts (ME) model proposed by Jacobs et al. (1991),
in that here the experts are the contour generators and the gates are the
weighting modules. The weight module in the WCG is itself a shallow neural
network that computes the prosodic contribution of the contour given the
context of its scope in the utterance. Here, the context of the utterance can
be arbitrarily defined, e.g. does it coincide with emphasis, or what is the
attitude in the utterance.

By virtue of the WCG, the WSFC can model the prosodic prominence of the
_clichés_ in an utterance, such as the impact of the attitude and emphasis on
the prominence of coinciding prototype contours (Gerazov et al., 2018b). This
added degree of freedom also gives WSFC a slightly improved modelling
performance.

In both the SFC and the WSFC, the CGs are trained within an analysis-by-
synthesis loop. In the synthesis part, the CGs are used to generate the
prototype contours, which are then summed to form the utterance’s prosodic
contour reconstruction. In the analysis part, the reconstruction is subtracted
from the original prosodic contour, and the error is distributed among the
constituent CGs by adding it to their previous outputs. These adjusted
contours are used as new targets for training the CGs with backpropagation.
The training loop (synthesis, error distribution and CGs training) is
typically iterated a dozen times. Consistency of prototype contours obtained
at convergence mainly depends on the statistical coverage of maximally
independent locations and sizes of overlapping functions and scopes. In the
SFC the error is distributed equally to all the CGs, while in the WSFC their
contribution, through their weight coefficients, is taken into account.

##  3 The Variational Prosody Model

The Variational Prosody Model (VPM) follows the SFC modelling paradigm but
introduces two new features: _i_) it introduces variational encoding to map a
prosodic latent space able to model the context-sensitive variations of the
prosodic prototypes, surpassing the one-dimensional modelling of prominence in
the WSFC. Moreover, _ii_) it integrates all the contour generators within a
single network architecture. This allows the joint training of all of the
contour generators, thus eliminating the need of an analysis-by-synthesis loop
with its ad hoc distribution of errors.

###  3.1 Variational Autoencoders

An autoencoder (AE) is a deep neural network built to learn an efficient data
encoding scheme, primarily used for data reduction (Cottrell and Munro, 1988;
Deng et al., 2010), but can also be used for denoising (Lu et al., 2013) and
learning meaningful signal representations (Socher et al., 2011; Liou et al.,
2014; Ap et al., 2014; Obin and Beliao, 2018). The generic AE architecture
comprises: an _encoder_ that receives the input signal and transforms it
through a bottleneck layer to a latent low-dimensional representation, also
called the latent code, and a _decoder_ which regenerates the input signal
from the latent representation, giving the autoencoder its name. The encoder
and decoder are usually built from feedforward neural networks, but sequence-
to-sequence models use recurrent neural networks that allow for variable
length input/output signals, e.g. in machine translation (Cho et al., 2014;
Sutskever et al., 2014).

Variational autoencoders (VAEs), whose architecture resembles the variational
contour generator (VCG) architecture of the VPM shown in the centre of Fig. 2,
differ from classical AEs in that their latent representation is probabilistic
and thus continuous, allowing for random sampling and interpolation (Kingma
and Welling, 2013). A variational encoder maps an input vector $\mathbf{x}$
into a latent space representation $\mathbf{z}$ using an encoder neural
network with parameters $\phi$ that outputs $q_{\phi}(\mathbf{z|x})$, i.e. a
probability distribution of the hidden representation conditioned on the
input. In fact, $q_{\phi}(\mathbf{z|x})$ is an approximation of the
intractable true posterior $p_{\theta}(\mathbf{z|x})$, which we assume takes a
multivariate Gaussian form with a diagonal covariance matrix, i.e. for a given
input data point $\mathbf{x}^{(i)}$:

$q_{\phi}(\mathbf{z|x}^{(i)})=\mathcal{N}(\mathbf{z};\bm{\mu}^{(i)},\bm{\sigma}^{2(i)}\mathbf{I})$
(1)

Thus the output of the encoder network, for a given input $\mathbf{x}$ is a
vector of $N$ means and $N$ variances, where $N$ is the chosen dimension of
the latent space representation $\mathbf{z}$. We can then sample the posterior
distribution using the reparametrisation trick:

$\mathbf{z}^{(i,l)}=\bm{\mu}^{(i)}+\bm{\sigma}^{(i)}\circ\bm{\epsilon}^{(l)}\,\,,\quad\text{where}\,\bm{\epsilon}^{(l)}\sim\mathcal{N}(\mathbf{0,I}).$
(2)

The obtained sample can then be passed through the decoder neural network with
parameters $\theta$, which models $p_{\theta}(\mathbf{x|z})$, and outputs an
approximation of the original input vector $\mathbf{x}$. In fact, the
parameters of the encoder and decoder networks $\phi$ and $\theta$ are trained
using backpropagation and gradient descend so that the VAE reproduces as close
as possible its input. As a by-product of this process, the VAE learns the
$q_{\phi}(\mathbf{z|x})$, structuring the latent space representation.

###  3.2 Variational Contour Generators

We explored two types of contour generators for the VPM: the RU-based
variational contour generator (VCG) and the variational recurrent contour
generator (VRCG), both shown in Fig. 2. The variational encoding introduced in
the contour generators of the VPM is reminiscent of the one used in VAEs.
Unlike the classic VAE architecture though, we do not use the output prosodic
contour as input; thus they are not autoencoders per se. Instead, we input the
RU position and the function’s context and train the contour generators to
learn a context-specific latent space representation of the shape of each
prosodic contour. In the recurrent contour generator, which can be built with
long short-term memory (LSTM) cells, the prosodic space mapping is
additionally decoupled from the RU position within the scope, and only depends
on the function’s context.

###  3.3 VPM architecture

The architecture of the VPM is intrinsically dynamic in that the combination
of contour generators used depends on the linguistic functions in the
utterance, i.e. for a given utterance, the VPM recruits the contributing
contour generators and overlaps and adds their outputs accordingly. A static
structure of the VPM architecture can be imposed though, shown in Fig. 3. The
static architecture improves training in two ways: _i_) it gives the freedom
to choose an arbitrary batch size, i.e. the number of samples of data used for
a single update of the VPM parameters, one that is independent of the number
of occurrences of a particular linguistic function combination. This means
that having a small number of occurrences of a particular combination will not
result in a smaller batch size, producing an imbalance of the influence on
parameter updates among the combinations. And _ii_) the shared graph structure
across combinations allows the batches to contain a random set of different
contour combinations, resulting in a better parameter update per batch.

The static architecture comprises copies of the contour generators necessary
to cover all possible overlapping function combinations found within a
dataset. A mask is then used for each RU to take into account only the active
contour generators, both for contour prediction in the forward pass, and for
weight updates in the backpropagation.

Figure 3: Static form of the Variational Prosody Model architecture, shown
here built with RU-based VCGs. The function mask is used at training time to
solely activate the CGs that contribute to a given RU position. Contrary to
the original SFC, the repartition of the modelling error is performed by
gradient descent.

###  3.4 VPM training

The loss function used to train the VPM comprises the Mean Square Error (MSE)
of the utterance’s prosodic contour reconstruction and the Maximum Mean
Discrepancy (MMD) term for regularisation, as proposed in InfoVAEs (Zhao et
al., 2017):

$\displaystyle\begin{split}\mathcal{L}&=\mathcal{L}_{\text{MSE}}+\lambda\mathcal{L}_{\text{MMD}}\\\
&=\frac{1}{N}\sum_{n=0}^{N-1}(f-\hat{f})^{2}+\lambda
D_{\text{MMD}}(q_{\phi}(\mathbf{z})||p(\mathbf{z}))\,.\end{split}$ (3)

Figure 4: Structure of the prosodic latent space for the left-dependency
function contour (DG) from Morlec obtained with a VPM using RU-based VCGs, for
different values of the regularisation coefficient $\lambda$ (left to right):
0.01, 1, and 100. The X-s and the ellipses show the means and standard
deviations of the latent space distributions corresponding to each unique
input combination. The plots also show the contours generated when sampling
the prosodic latent space. These graphs show that $\lambda$ controls the
compactness of the latent representation

Here, $f$ and $\hat{f}$ are the original and reconstructed prosody contours
and $N$ is the number of prosody samples per rhythmic unit. In our case we use
$N-1$ pitch targets and a duration coefficient. Specifically, we use $N=6$,
i.e. 5 $f_{0}$ pitch targets per vocalic nucleus, evenly distributed
throughout its duration. $D_{\text{MMD}}$ is the Maximum-Mean Discrepancy
(MMD) divergence measure between the marginal inference distribution on the
latent space $q_{\phi}(\mathbf{z})$ and the prior $p(\mathbf{z})$, and
$\lambda$ is the regularisation coefficient. The MMD quantifies the distance
between two distributions by comparing all their moments, and can be
calculated using the kernel trick:

$\displaystyle D_{\text{MMD}}(p||q)=$
$\displaystyle\,\,\mathbb{E}_{p(\mathbf{z}),p(\mathbf{z}^{\prime})}[k(\mathbf{z},\mathbf{z}^{\prime})]-2\,\mathbb{E}_{q(\mathbf{z}),p(\mathbf{z}^{\prime})}[k(\mathbf{z},\mathbf{z}^{\prime})]$
$\displaystyle+\mathbb{E}_{q(\mathbf{z}),q(\mathbf{z}^{\prime})}[k(\mathbf{z},\mathbf{z}^{\prime})]\,.$
(4)

We have used the MMD for regularisation instead of the Kullback–Leibler
divergence (KLD) as proposed in the original Evidence Lower Bound (ELBO)
criterion (Kingma and Welling, 2013), because it gave poor training results
for the VPM. In fact, it has been established that the KLD on the posterior
$q_{\phi}(\mathbf{z|x})$, with a powerful enough decoder, leads the VAE to
ignore the latent code, resulting in a latent space identical to the prior
$p(\mathbf{z})$ (Chen et al., 2016). This issue in VAE training cannot be
mitigated by simply scaling the KLD term as suggested in $\beta$-VAE (Higgins
et al., 2016), and has been addressed in various ways in literature: by
limiting the power of the decoder (Chen et al., 2016), regularisation
scheduling strategies for the KLD term (Bowman et al., 2015), and the use of
divergence measures on the $q_{\phi}(\mathbf{z})$ as in InfoVAEs (Zhao et al.,
2017).

##  4 Experiments and results

We have designed our experiments to test five hypotheses: _i_) the VPM is a
plausible model that can decompose the prosodic contour into its constituent
prototype contours, _ii_) the VPM network architecture, because of its joint
contour generator optimisation, is able to outperform the analysis-by-
synthesis based SFC and WSFC when using the same contour generator structure,
while maintaining the same level of performance with a standard deep model,
_iii_) the VPM is able to capture contour prominence as well as the WSFC,
_iv_) the VPM, unlike the SFC and WSFC, is able to capture context-specific
variance in the shape of the extracted prototype contours, and _v_) the VPM is
also able to capture variance in the shape of the extracted prototype contours
that is not context-specific.

All of the models used in these experiments have been implemented in the
scientific Python ecosystem using the PyTorch deep learning library (Paszke et
al., 2017) and the Scikit-Learn machine learning library (Pedregosa et al.,
2011). The code is available as free software on
GitHub.111https://github.com/gerazov/prosodeep

###  4.1 Databases

To test the four hypotheses, we use three databases in our experiments:

  * 1.

Morlec – a database of 6 attitudes in French: declaration, question,
exclamation, incredulous question, suspicious irony and obviousness, totalling
1932 utterances from one speaker (Morlec et al., 2001),

  * 2.

Liu – a database of declarations and five question types in Chinese that
include emphasis at three different positions (Liu and Xu, 2005). We use the
first speaker with 76 carrier sentences using a single tone each, recorded 5
times, totalling 380 utterances, and

  * 3.

Chen – a database of read Chinese from a single female speaker comprising 110
carrier utterances ranging from 6 to 38 syllables in length (Chen et al.,
2004).

###  4.2 Hyperparameters

The most important hyperparameters in training the VPM are the dimension of
the latent space, the number of hidden layers and their size, and the
regularisation coefficient $\lambda$ in the loss function (3). In our
experiments we use a two-dimensional prosodic latent space, both because it
offers ample modelling power and because it is favourable for visualisation
and exploration. The number of hidden layers and their size also impacts the
modelling power of the system, if it is too small the model will have trouble
handling the data complexity and if it is too high the model will have a
tendency to overfit the training data. Also, the larger the number of layers
and the larger their size, the more data the model needs to train all of its
parameters.

Figure 5: Decomposition of the melody (left column) and RU duration
modification coefficient (centre column) of the French utterance _“Les gamins
coupaient des rondins.”_ from Morlec, with the VPM-VRCG for declaration (DC,
top row) and incredulous question (DI, bottom row) into constituent prototype
contours: attitude (DC and DI), left- and right- dependencies (DG, DD), and
clitics (XX). The VPM successfully captures the strongly reduced prominence of
XX, DG and DD, when solicited in the DI context. Decomposition of the
intonation of two Chinese utterances (right column): _“Huā shéng néng ín qǐ
īng iòu ér guò mǐn.”_ from Chen (top) and _“Lǐ Mǐn fǎn gǎn _Liǔ Yǔ_ diǎn huǒ
qǔ nuǎn?”_ from Liu (bottom) with the VPM-VRCG, a declaration (DC) and a
question (QS) with the four Chinese tones (C1 - C4), word boundaries (WB), and
emphasis (EM). The VPM captures the correct shapes of the tones and properly
captures the effect emphasis has on them.

The regularisation coefficient can compact the prosodic latent space, but it
can also hinder learning meaningful contours by forcing the mapped
distributions to overlap. This process is illustrated in Fig. 4, which shows
the changes in the prosodic latent space mappings of the unique input
combinations for the left-dependency functional contour (DG) in Morlec as a
function of the regularisation coefficient for a VPM with RU-unit based VCGs.
We can clearly see the convergence of the mapped distributions in the latent
space as $\lambda$ increases, both from their relative dispersal, as well as
the scale of the latent space axes. Also, a $\lambda$ of 100 can be seen to
collapse the latent space structure into the prior, thus precluding learning
of a latent space that would adequately represent the variability of the
elementary contours.

To obtain the best value for these as well as other hyperparameters we
conducted a grid-search over the Morlec data, as it has the sufficient
prosodic variety that we want our model to handle. We evaluated the model’s
performance using a validation set and found that we obtained best performance
for a single hidden layer with a size of 32 units, a $\lambda$ value of 0.3,
$\tanh$ activation function, a $L^{2}$ norm weight decay of 0.0001, a batch
size of 256 RUs, and a learning rate of 0.001 with the Adam optimizer (Kingma
and Ba, 2014). For the recurrent contour generators we chose a batch size of 8
utterances. The RU position ramps were not normalised because we found that
normalisation in the range 0.01 – 0.99 (Wu et al., 2016), did not lead to
performance gains for both Morlec, as well as Chen, which contains longer
utterances. These hyperparameter values were used in our further experiments.

We empirically verified that LSTM cells work better than gated recurrent units
(GRUs) and vanilla recurrent neural networks (RNNs). In addition, we
determined that the best way to bias the LSTM with the latent space sample is
to feed the sampled value as an initial input step, similar to image
captioning applications (Vinyals et al., 2015) and as shown in Fig. 2.

###  4.3 Baseline

The baseline system used for comparison was built using the benchmark
intonation module in the Merlin speech synthesis system (Wu et al., 2016). We
used the basic deep neural network (DNN) model and the single-direction
recurrent model based on LSTM cells. We chose the latter as it outperformed
the bi-directional LSTM models for modelling intonation in their benchmark. In
the original Merlin implementation the DNN comprises 6 feedforward hidden
layers, 1024 units each, while the LSTM model comprises five feedforward
hidden layers of 1024 each, with 512 units for the last one. Their output is
based on some 416 features containing the answers to 416 binary and numerical
questions about the context of the phone, all normalised to 0.01 – 0.99. In
addition, there are 6 frame and state position features and 3 state and phone
duration features.

To make a fair comparison with the VPM, in our implementation we feed the two
Merlin models with the information about the type of overlapping functions at
each RU, and the RU positions within the scope of each of these functions. To
avoid problems with the dataset size we ran a grid search on the baseline
model hyperparameters and chose the best performing configuration on the
Merlin dataset. We found that for the DNN model, two hidden layers of 256
units each, gave the best results. For the LSTM model it was one hidden layer
with 1024 units. The rest of the hyperparameters for the models were kept the
same as the benchmark: $\tanh$ activation, $10^{-5}$ $L^{2}$ regularisation,
and a learning rate of 0.002.

###  4.4 Plausibility

The plausibility of the proposed VPM can be qualitatively observed in the
example decomposition of the two French utterances and two Chinese utterances
in Fig. 5. The intonation decomposition of the example utterances is shown in
the left and right columns, while the centre column plots the duration
coefficients for the French utterances. In these decompositions, the latent
prosodic space of the VPM was sampled at the means of the distributions.

Figure 6: WRMSE distribution for the different models for Morlec and the
$p$-values for these distributions between each pair of models.

We can see that the model successfully extracts the general shape of the
prototype pitch contours from the intonation contour, specifically the shapes
of the four Chinese tones. Also, we can see that the model correctly captures
the central speed up and then the phrase final slow down of the speech rate
for the incredulous question through the attitude contour itself, and not via
the contributing syntactic contours. Finally, and most importantly, we can see
that the difference in prominence of the syntactic contours between the two
attitudes in French, and the tone contours in context to the position of
emphasis in the second Chinese example, is successfully captured by the VPM
through its variance encoding mechanism. This is similar to the results
obtained with the WSFC that has an explicit prominence weighting mechanism
(Gerazov et al., 2018b). All of this confirms our initial hypothesis that the
VPM is a plausible prosody model able to decompose the prototype contours.

Figure 7: Prosodic latent space of the clitic functional contour (XX) in
Morlec captured by the VPM-VCG (left) demonstrates that a full blown contour
is only generated in context of the declaration and exclamation attitudes,
while it is largely diminished for the rest: question, suspicious irony,
obviousness, and incredulous question. Prosodic latent space of the left-
dependency function contour (DG) in Morlec captured by the VPM-VRCG (right)
shows that based on attitude context declaration and exclamation elicit full-
blown contours, while they are largely diminished for the rest.

###  4.5 Performance

We assess the performance of the proposed VPM architecture by comparing its
intonation reconstruction performance to that of the SFC and WSFC models, as
well as the two baseline models on the Morlec. As a metric we have used a
weighted version of the root mean square error (WRMSE) to take into account
only the error in pitch within the nuclei. To evaluate the impact on
performance of the joint contour generator optimisation through
backpropagation (BP), the VPM network architecture was also fitted with SFC
contour generators (BP-SFC), WSFC contour generators (BP-WCG), as well as
recurrent contour generators (BP-RSFC). The VPM in turn was evaluated both
with the RU-based VCGs (VPM), and the recurrent VRCGs (VRPM).

To maintain the overall model complexity we chose to keep the number of hidden
units across the models to 32, except for the baseline models for which we
kept the best performing parameters for the grid-search, i.e. two layers with
256 units for the base-DNN, and a single layer of 1024 units for the base-
LSTM. Since there are 13 different contour types in the Morlec data, the
equivalent number of hidden units for the contour generator based models is
around 416 units (32 hidden units $\times$ 13 contour generators). This number
is indicative of the number of units that output the contour, but does not
include the hidden units in the weight module of the WSFC, the encoder in the
VCG, and the encoder and decoder in VRCG. All models were trained using early
stopping based on a validation set, with a $\Delta$WRMSE threshold of
$10^{-4}$ and a patience of 20 epochs, and then evaluated on a separate test
set. The latent space was sampled at the mean of each mapping for this
evaluation.

Table 1 shows the obtained WRMSEs for the different models. We can see that
the VPM architecture when used with the SFC and WSFC contour generators
outperforms the analysis-by-synthesis loop based training. We can also see
that the two varieties of the VPM generally outperform all of the other
models. The exception to this is that BP-WSFC outperforms the VRPM. We believe
this is due to: _i_) both the efficiency of capturing prominence by the
weighted contour generators and prominence being the dominant prosodic
variation in the prototypes in Morlec, and _ii_) the suboptimal extraction of
the mean contours in the VPM induced by the random sampling procedure within
training. This effect is reduced when using the VCGs, where we have a spread
of mappings for each RU position combination, as can be seen in Fig. 4. This
more complex utilisation of the prosodic latent space, leads to larger
modelling capacity, as is reflected in the better performance of the VCGs over
the VRCGs.

Table 1: Weighted RMSE in semitones for the reconstructed pitch obtained for
Morlec with the VPM compared to other models.

Model | WRMSE
---|---
mean $\pm$ standard deviation
SFC | $1.83\pm 0.90$
BP-SFC | $1.76\pm 0.85$
BP-RSFC | $1.77\pm 0.81$
WSFC | $1.76\pm 0.85$
BP-WSFC | $1.69\pm 0.76$
base-DNN | $1.76\pm 0.87$
base-LSTM | $1.83\pm 0.94$
VPM |  1.68 $\pm$ 0.75
VRPM | $1.75\pm 0.83$

The distribution of the WRMSE within the test set for each model is shown in
the boxplot in Fig. 6. To analyse the statistical significance of the
difference in results obtained with the different models, we used a post-hoc
analysis of Wilcox’s Robust Repeated-Measures Analysis of Variance (RM-ANOVA)
(Wilcox, 2011). The obtained $p$-values are shown in the heatmap in Fig. 6. We
can see that although for some models we get statistically significant
differences in the performance result, e.g. for the SFC and the VPM, most of
the results are not significantly different, e.g. for the VRPM. This leads us
to believe, that the models share a similar level of performance. Even so, the
presented findings confirm our second hypothesis that the joint contour
generator training leads to performance benefits, and that the VPM performs on
a par with, if not slightly better than, a standard deep baseline.

###  4.6 Prominence

The ability of the VPM to capture prominence due to attitude context in Morlec
has already been shown in the example decomposition shown in Fig. 5. Here, we
reaffirm this observation by exploring the prosodic latent space of the clitic
contour (XX) solicited in the 6 different attitudes, shown to the left in Fig.
7, and of the left-dependency contour (DG) shown to the right. In both cases,
we can clearly see that the declaration and exclamation attitudes map to areas
in the latent space where the prototypes are fully realised. On the other
hand, all of the other attitudes map to areas where these contours are largely
diminished and close to 0. In fact, we can argue that there is a “prominence”
vector in these latent spaces captured by the VPM. Thus the VPM captures
gradience as well as WSFC, but it is also able to model more subtle variations
of shape. Note that, since the DG prototype contour has a larger scope, we
have to use the variational recurrent contour generators that decouple the
latent space from the scope position ramps.

To analyse the impact of emphasis, i.e. narrow focus, on Chinese tones we will
use the Liu database. Even though the carry-over effect has a significant
impact on modelling performance (Gerazov et al., 2018a), the imposed uniform
structure of the utterances in Liu precludes training tonal prototypes for an
expanded scope. Thus we trained tonal prototypes with a single RU scope. To
demonstrate the usability of the VPM in a transfer learning scenario, we first
pretrained the tonal prototype contour generators on the Chen data, which
contains a natural tone distributions, and then fine-tuned them on the Liu
data.

The structure of the latent space for Tone 3 is shown in Fig. 8. Note that
this latent space corresponds to the decomposition shown in the bottom left
plot of Fig. 5. Since the focus in the Liu data always falls on two
consecutive RUs, i.e. first and last names, four emphasis contexts are
considered for conditioning the VPM: no emphasis (None), pre-emphasis (EMp) on
the first of the two RUs, on-emphasis (EM) on the second RU, and post-emphasis
continuation (EMc). From the plot we can see that the VPM has successfully
captured the increase in amplitude of the pitch movement in EM, as identified
in literature (Liu and Xu, 2005). Moreover, both results we obtained in
modelling prominence are in line with our findings with the WSFC (Gerazov et
al., 2018b), thus, validating our third hypothesis that the VPM is able to
model prominence.

Figure 8: Prosodic latent space of the Tone 3 prototype contour in Liu, in
context of no-, pre-, on- and post-emphasis, as extracted by the VPM-VCG. The
VPM captures prominence in the on-emphasis context, as well as shape variation
in the pre- and post-emphasis contexts. Figure 9: Sampling (blue circles)
around the input mapping (X) in the prosodic latent space of Tone 2 obtained
with the VPM-VCG for Chen. The latent space does capture variation in the
contour’s shape, albeit a small one. Note that the mapping is not centred in
the latent space due to the use of a small regularisation coefficient
$\lambda$, see Fig. 4 for comparison.

Figure 10: Final analysis-by-synthesis targets for Tone 2 in Chen from the SFC
(top row), and contours generated by sampling the latent space with the VPM-
VCG (bottom row), with the variation explained by the first (centre column)
and second (right column) PCA components for both sets of contours. The VPM
successfully captures the different components of variation in the contour’s
shape. The amplitude difference is due to the approximative nature of the SFC
targets, as well as the local averaging effect dependent on random sampling of
the latent space during VPM training.

###  4.7 Exploring variation

What is more interesting in Fig. 8, as well as in the decomposition shown in
the bottom left plot of Fig. 5, is that the VPM has managed to capture a
variation of the prototype shape of Tone 3 conditioned on emphasis.
Specifically, we can see that post-emphasis the contour’s slope flattens out,
reflecting reduced pitch dynamics in line with post-focus compression (Xu,
1999). Moreover, Tone 3 can be seen to transform from its usual low tone into
a rising tone similar to Tone 2 in the first focused RU preceding the second
focused RU (pre-emphasis). Since this data is made up of single tone
utterances, the captured change in shape is in fact the Chinese tonal sandhi,
i.e. a phenomenon in which a Tone 3 preceding another Tone 3 changes into a
Tone 2 (Xu, 1997). These observations confirm the added value brought by the
variational encoding scheme and our fourth hypothesis that the VPM can model
context-specific variation in the prosodic prototypes.

Since the variational encoder of the VPM is fed by the function’s context, its
latent space is made to capture variations that can be explained by our
contextual features. These features do not entirely reflect the structure of
the manifold encoded in the prosodic shapes. However, due to the random
sampling around the variational encoder mappings in the latent space during
training, the VPM has the potential to capture a part of the within context
variation.

To evaluate this, we trained single scope, i.e. one RU long, tone prototype
contours with no context. These contour generators effectively see a single
input feature combination, which their variational encoders map to a single
point in the prosodic latent space. We now sample around this point with a
normal distribution using the output standard deviation by the encoder, as
shown for Tone 2 in Fig. 9, and generate prosodic contours, shown in the
bottom left corner plot in Fig. 10.

For comparison, we use the final targets obtained in the last step of the
analysis-by-synthesis loop for the SFC, shown for Tone 2 from Chen in the top
left plot in Fig. 10. Note here that these final targets are not perfect
representations of the prototypical contours themselves, as coinciding
contours could jeopardise the error distribution in the SFC.

Fig. 10 also shows the variation captured with the first two components of a
Principal Component Analysis (PCA) based decomposition for multiples of the
standard deviation in the transform domain. We can see that the target
contours exhibit a variation of 81% that can be explained by the first PCA
component, 13% by the second, and 6% by the rest of the PCA components. On the
other hand, 88% of the variation captured with the VPM can be explained by the
first PCA component, and 12% by the second. Since we have limited the prosodic
latent space to two dimensions, the VPM cannot capture the variation in the
data explained by the higher PCA components.

These results reaffirm the VPM’s ability to capture prototype shape variations
beyond simple amplitude scaling, compared to the WSFC model. Moreover, we can
see that even in the absence of context based conditioning of the latent
space, the VPM still manages to capture a part of the variation in the data
through the random sampling process. However, this process has its limitations
that take the form of a localised averaging effect. Namely, during training,
succeeding samples are mapped randomly in the latent space, thus different
contour shapes may be mapped within each other’s vicinity that will result
with the model learning an average contour between them. The effects of this
averaging can be seen in the reduced amplitude of the VPM generated contours
in Fig. 10. Nevertheless, these results confirm our final hypothesis.

##  5 A unified prosodic latent space

The contour generators in the VPM can also be extended to encode different
function prototypes within a single “unified” prosodic latent space. For
instance, one can think of mapping an attitude prosodic space that will
encompass all of the different attitude functions in the data, or a syntactic
prosodic space with the different dependency functions, or a morphological
prosodic space that would encode tone and focus for example. This might lead
to insights of prototype shape distribution within one level of the prosodic
hierarchy, and might be closer to how humans structure prosody. We give here
only a demonstration of this concept in the form of an attitude prosodic
latent space shown in Fig. 11. The latent space is obtained by representing
all attitude prototypes with a single contour generator that is conditioned on
the attitude type.

Figure 11: Attitude prosodic latent space obtained with the VPM-VRCG for the
Morlec database showing the 6 attitudes present in the data: declaration (DC),
question (QS), exclamation (EX), suspicious irony (SC), obviousness (EV), and
incredulous question (DI).

##  6 Conclusions

The proposed Variational Prosody Model uses a deep architecture of variational
(recurrent) contour generators to decompose the prosody into its underlying
_cliché_ prototype contours and captures a well structured prosodic latent
space of their context-specific variation. The VPM plausibility has been
demonstrated across two very different languages, and it has been shown to
outperform previous state-of-the-art decompositional prosody models, and
perform on a par to standard deep models. The prime contribution of the VPM is
its incorporation of variational encoding that has been shown as beneficial
towards the exploration of the underlying context-specific spatiotemporal
variation of the constituent prototype contours in prosody. Another
prospective use of the VPM, that has not been explored in this paper, is its
application in speech synthesis, where we believe that the modelled
variability can be used to generate a more dynamic and natural prosody that is
not as affected by averaging effects.

##  7 Acknowledgements

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No 745802 “ProsoDeep: Deep understanding and modelling of the
hierarchical structure of Prosody”.222https://gerazov.github.io/prosodeep

## References

  * Ap et al. (2014) Ap, S.C., Lauly, S., Larochelle, H., Khapra, M., Ravindran, B., Raykar, V.C., Saha, A., 2014.  An autoencoder approach to learning bilingual word representations, in: Advances in Neural Information Processing Systems, pp. 1853–1861.
  * Arik et al. (2017) Arik, S.O., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Ng, A., Raiman, J., et al., 2017.  Deep voice: Real-time neural text-to-speech.  arXiv preprint arXiv:1702.07825 .
  * Bailly and Holm (2002) Bailly, G., Holm, B., 2002.  Learning the hidden structure of speech: from communicative functions to prosody.  Cadernos de Estudos Linguisticos 43, 37–54.
  * Bailly and Holm (2005) Bailly, G., Holm, B., 2005.  SFC: a trainable prosodic model.  Speech communication 46, 348–364.
  * Baron-Cohen et al. (2004) Baron-Cohen, S., Golan, O., Wheelwright, S., Hill, J., 2004\.  Mind reading: The interactive guide to emotions.
  * Biederman (1987) Biederman, I., 1987.  Recognition-by-components: a theory of human image understanding.  Psychological review 94, 115\.
  * Bowman et al. (2015) Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S., 2015\.  Generating sentences from a continuous space.  arXiv preprint arXiv:1511.06349 .
  * Brichet and Aubergé (2004) Brichet, C., Aubergé, V., 2004\.  La prosodie de la focalisation en français: faits perceptifs et morphogénétiques.  Journées d’Etudes sur la Parole, Nancy-France , 33–36.
  * Campbell (1992) Campbell, W.N., 1992.  Syllable-based segmental duration.  Talking machines: Theories, models, and designs , 211–224.
  * Chen et al. (2004) Chen, G.P., Bailly, G., Liu, Q.F., Wang, R.H., 2004\.  A superposed prosodic model for chinese text-to-speech synthesis, in: Chinese Spoken Language Processing, 2004 International Symposium on, IEEE. pp. 177–180.
  * Chen et al. (2016) Chen, X., Kingma, D.P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., Abbeel, P., 2016\.  Variational lossy autoencoder.  arXiv preprint arXiv:1611.02731 .
  * Cho et al. (2014) Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014.  Learning phrase representations using rnn encoder-decoder for statistical machine translation, in: EMNLP.  URL: arXivpreprintarXiv:1406.1078.
  * Cottrell and Munro (1988) Cottrell, G.W., Munro, P., 1988\.  Principal components analysis of images via back propagation, in: Visual Communications and Image Processing’88: Third in a Series, International Society for Optics and Photonics. pp. 1070–1078.
  * Deng et al. (2010) Deng, L., Seltzer, M.L., Yu, D., Acero, A., Mohamed, A.r., Hinton, G., 2010\.  Binary coding of speech spectrograms using a deep auto-encoder, in: Eleventh Annual Conference of the International Speech Communication Association.
  * Fónagy et al. (1983) Fónagy, I., Bérard, E., Fónagy, J., 1983.  Clichés mélodiques.  Folia linguistica 17, 153–186.
  * Fujisaki (1971) Fujisaki, H., 1971.  A generative model for the prosody of connected speech in japanese.  Annual Report of the Engineering Research Institute, University of Tokyo 30, 75–80.
  * Gårding (1983) Gårding, E., 1983.  A generative model of intonation, in: Prosody: Models and measurements. Springer, pp. 11–25.
  * Gerazov et al. (2018a) Gerazov, B., Bailly, G., Xu, Y., 2018a.  The significance of scope in modelling tones in chinese, in: Tonal Aspects of Languages.
  * Gerazov et al. (2018b) Gerazov, B., Bailly, G., Xu, Y., 2018b.  A weighted superposition of functional contours model for modelling contextual prominence of elementary prosodic contours, in: INTERSPEECH.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y., 2016\.  Deep learning. volume 1.  MIT press Cambridge.
  * Goodhue et al. (2016) Goodhue, D., Harrison, L., Su, Y.C., Wagner, M., 2016\.  Toward a bestiary of english intonational contours.  The Proceedings of the North East Linguistics Society (NELS) 46, 311–320.
  * Higgins et al. (2016) Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A., 2016\.  beta-vae: Learning basic visual concepts with a constrained variational framework .
  * Hirst (2005) Hirst, D.J., 2005.  Form and function in the representation of speech prosody.  Speech Communication 46, 334–347.
  * Hsu et al. (2018) Hsu, W.N., Zhang, Y., Weiss, R.J., Zen, H., Wu, Y., Wang, Y., Cao, Y., Jia, Y., Chen, Z., Shen, J., et al., 2018.  Hierarchical generative modeling for controllable speech synthesis.  arXiv preprint arXiv:1810.07217 .
  * Jacobs et al. (1991) Jacobs, R.A., Jordan, M.I., Nowlan, S.J., Hinton, G.E., 1991\.  Adaptive mixtures of local experts.  Neural computation 3, 79–87.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  CoRR abs/1412.6980.  URL: http://arxiv.org/abs/1412.6980, arXiv:1412.6980.
  * Kingma and Welling (2013) Kingma, D.P., Welling, M., 2013\.  Auto-encoding variational bayes.  CoRR abs/1312.6114.  URL: http://arxiv.org/abs/1312.6114, arXiv:1312.6114.
  * Liou et al. (2014) Liou, C.Y., Cheng, W.C., Liou, J.W., Liou, D.R., 2014\.  Autoencoder for words.  Neurocomputing 139, 84–96.
  * Liu and Xu (2005) Liu, F., Xu, Y., 2005.  Parallel encoding of focus and interrogative meaning in mandarin intonation.  Phonetica 62, 70–87.
  * Lu et al. (2013) Lu, X., Tsao, Y., Matsuda, S., Hori, C., 2013.  Speech enhancement based on deep denoising autoencoder., in: Interspeech, pp. 436–440.
  * Ma et al. (2019) Ma, S., Mcduff, D., Song, Y., 2019.  A generative adversarial network for style modeling in a text-to-speech system, in: International Conference on Learning Representations.  URL: https://openreview.net/forum?id=ByzcS3AcYX.
  * Morlec (1997) Morlec, Y., 1997.  Génération multiparamétrique de la prosodie du français par apprentissage automatique.  Ph.D. thesis.
  * Morlec et al. (2001) Morlec, Y., Bailly, G., Aubergé, V., 2001.  Generating prosodic attitudes in french: data, model and evaluation.  Speech Communication 33, 357–371.
  * Morlec et al. (1998) Morlec, Y., Rilliard, A., Bailly, G., Aubergé, V., 1998\.  Evaluating the adequacy of synthetic prosody in signalling syntactic boundaries: methodology and first results, in: Proceedings of the first International Conference on Language Resources and Evaluation. Granada, Spain, pp. 647–650.
  * Obin and Beliao (2018) Obin, N., Beliao, J., 2018.  Sparse coding of pitch contours with deep auto-encoders, in: Speech Prosody.
  * Obin et al. (2011) Obin, N., Lacheret, A., Rodet, X., 2011.  Stylization and trajectory modelling of short and long term speech prosody variations, in: Interspeech.
  * Paszke et al. (2017) Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A., 2017\.  Automatic differentiation in pytorch .
  * Pedregosa et al. (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E., 2011\.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research 12, 2825–2830.
  * Shen et al. (2018) Shen, J., Pang, R., Weiss, R.J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al., 2018.  Natural tts synthesis by conditioning wavenet on mel spectrogram predictions, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE. pp. 4779–4783.
  * Silverman et al. (1992) Silverman, K., Beckman, M., Pitrelli, J., Ostendorf, M., Wightman, C., Price, P., Pierrehumbert, J., Hirschberg, J., 1992\.  Tobi: A standard for labeling english prosody, in: Second international conference on spoken language processing.
  * Skerry-Ryan et al. (2018) Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., Saurous, R.A., 2018.  Towards end-to-end prosody transfer for expressive speech synthesis with tacotron, in: Dy, J., Krause, A. (Eds.), Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholmsmässan, Stockholm Sweden. pp. 4693–4702.  URL: http://proceedings.mlr.press/v80/skerry-ryan18a.html.
  * Socher et al. (2011) Socher, R., Pennington, J., Huang, E.H., Ng, A.Y., Manning, C.D., 2011.  Semi-supervised recursive autoencoders for predicting sentiment distributions, in: Proceedings of the conference on empirical methods in natural language processing, Association for Computational Linguistics. pp. 151–161.
  * Sotelo et al. (2017) Sotelo, J., Mehri, S., Kumar, K., Santos, J.F., Kastner, K., Courville, A., Bengio, Y., 2017.  Char2wav: End-to-end speech synthesis .
  * Sutskever et al. (2014) Sutskever, I., Vinyals, O., Le, Q.V., 2014.  Sequence to sequence learning with neural networks, in: Advances in neural information processing systems, pp. 3104–3112.
  * Taigman et al. (2018) Taigman, Y., Wolf, L., Polyak, A., Nachmani, E., 2018\.  Voiceloop: Voice fitting and synthesis via a phonological loop .
  * Vinyals et al. (2015) Vinyals, O., Toshev, A., Bengio, S., Erhan, D., 2015\.  Show and tell: A neural image caption generator, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156–3164.
  * Wang et al. (2017a) Wang, X., Takaki, S., Yamagishi, J., 2017a.  An rnn-based quantized f0 model with multi-tier feedback links for text-to-speech synthesis, in: Proc. Interspeech, pp. 1059–1063.
  * Wang et al. (2017b) Wang, Y., Skerry-Ryan, R., Stanton, D., Wu, Y., Weiss, R.J., Jaitly, N., Yang, Z., Xiao, Y., Chen, Z., Bengio, S., et al., 2017b.  Tacotron: Towards end-to-end speech synthesis.  arXiv preprint arXiv:1703.10135 .
  * Wang et al. (2018) Wang, Y., Stanton, D., Zhang, Y., Skerry-Ryan, R., Battenberg, E., Shor, J., Xiao, Y., Ren, F., Jia, Y., Saurous, R.A., 2018.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  arXiv preprint arXiv:1803.09017 .
  * Wilcox (2011) Wilcox, R.R., 2011.  Introduction to robust estimation and hypothesis testing.  Academic press.
  * Wu et al. (2016) Wu, Z., Watts, O., King, S., 2016.  Merlin: An open source neural network speech synthesis system.  Proc. SSW, Sunnyvale, USA .
  * Xu (1997) Xu, Y., 1997.  Contextual tonal variations in mandarin.  Journal of phonetics 25, 61–83.
  * Xu (1999) Xu, Y., 1999.  Effects of tone and focus on the formation and alignment of f0contours.  Journal of phonetics 27, 55–105.
  * Xu (2005) Xu, Y., 2005.  Speech melody as articulatorily implemented communicative functions.  Speech communication 46, 220–251.
  * Yin et al. (2016) Yin, X., Lei, M., Qian, Y., Soong, F.K., He, L., Ling, Z.H., Dai, L.R., 2016\.  Modeling f0 trajectories in hierarchically structured deep neural networks.  Speech Communication 76, 82–92.
  * Yoshimura et al. (1999) Yoshimura, T., Tokuda, K., Masuko, T., Kobayashi, T., Kitamura, T., 1999.  Simultaneous modeling of spectrum, pitch and duration in hmm-based speech synthesis, in: Sixth European Conference on Speech Communication and Technology.
  * Zen (2015) Zen, H., 2015.  Acoustic modeling in statistical parametric speech synthesis–from hmm to lstm-rnn.  Proc. MLSLP .
  * Zen et al. (2013) Zen, H., Senior, A., Schuster, M., 2013.  Statistical parametric speech synthesis using deep neural networks, in: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE. pp. 7962–7966.
  * Zhao et al. (2017) Zhao, S., Song, J., Ermon, S., 2017.  Infovae: Information maximizing variational autoencoders.  arXiv preprint arXiv:1706.02262 .