# Style Equalization: Unsupervised Learning of Controllable
Generative Sequence Models

Jen-Hao Rick Chang  Ashish Shrivastava  Hema Swetha Koppula  Xiaoshuai Zhang
Oncel Tuzel  https://apple.github.io/ml-style-equalization

###### Abstract

Controllable generative sequence models with the capability to extract and
replicate the style of specific examples enable many applications, including
narrating audiobooks in different voices, auto-completing and auto-correcting
written handwriting, and generating missing training samples for downstream
recognition tasks. However, under an unsupervised-style setting, typical
training algorithms for controllable sequence generative models suffer from
the training-inference mismatch, where the same sample is used as content and
style input during training but unpaired samples are given during inference.
In this paper, we tackle the training-inference mismatch encountered during
unsupervised learning of controllable generative sequence models. The proposed
method is simple yet effective, where we use a style transformation module to
transfer target style information into an unrelated style input. This method
enables training using unpaired content and style samples and thereby mitigate
the training-inference mismatch. We apply style equalization to text-to-speech
and text-to-handwriting synthesis on three datasets. We conduct thorough
evaluation, including both quantitative and qualitative user studies. Our
results show that by mitigating the training-inference mismatch with the
proposed style equalization, we achieve style replication scores comparable to
real data in our user studies.

Style transform, generative model, controllable

,

##  1 Introduction

The goal of controllable generative sequence models is to generate sequences
containing target content in a target style. With the capability to select
speaker voices, multi-speaker text-to-speech models have been successfully
adopted in many voice assistants (Gibiansky et al., 2017; Ping et al., 2018;
Hayashi et al., 2020). Many applications, however, require style
controllability beyond selecting speaker voices. To perfectly reconstruct a
speech example, we need to replicate not only the speaker’s voice but also
other aspects of style, including but not limited to prosody, intonation
dynamics, background noise, echo, and microphone response appeared in the
given sample. To utilize synthetic data to analyze failures or biases of a
downstream recognizer, we need a style representation that models the style
distribution beyond speaker identity. In these applications, style represents
all information (except the content) to exactly reconstruct a sample, as
illustrated in Fig. 1a. To model the temporal dynamics contained in input
samples (_e.g_., speech and handwriting), our style representation is a time-
varying sequence, instead of a fixed vector. To capture the large variation of
style, we learn the style representation in an unsupervised manner from a
reference sample, rather than using a few human-annotated attributes.

Figure 1: Controllable generative models with sample-level style control. (a)
The information contained in a sample can be divided into content (_i.e_., the
text) and style (_i.e_., all other information besides content). (b) Our goal
during inference is to generate samples containing target content A in the
style of sample B. Notice that sample B generally contains a different
content. (c) There exists a training-inference mismatch when learning these
models in typical unsupervised training of controllable generative models.
During training, the same sample is used as content input and style input,
whereas during inference, content and style inputs are from different samples,
_i.e_., the reference style sample contains a different content than the
target content. The mismatch leads to incorrect content generation during
inference. (d) To mitigate the training-inference mismatch, the proposed style
equalization takes unpaired samples as input during both training and
inference. It transforms the style of sample B to that of A by estimating
their style difference.

Our goal is to learn a controllable generative sequence model that controls
its style with a reference example (_e.g_., an existing audio) and controls
the content with a content sequence (_e.g_., text), as shown in Fig. 1b. Our
training dataset $\mathcal{X}$ is composed of $\left\\{({\bf x}^{i},{\bf
c}^{i})\right\\}_{i=1,\dots,n}$, where ${\bf x}^{i}{=}\left[{x}^{i}_{1}\
\dots\ {x}^{i}_{T_{i}}\,|\,{x}^{i}_{t}\in{\mathbb{R}}^{d}\,\right]$ is the
i-th sample and ${\bf c}^{i}{=}\left[{c}^{i}_{1}\ \dots\
{c}^{i}_{N_{i}}\,|\,{c}^{i}_{j}\in{\mathbb{R}}^{m}\right]$ is the
corresponding content sequence. In general, ${\bf x}^{i}$ and ${\bf c}^{i}$
have different lengths, _i.e_., $T_{i}\neq N_{i}$, and we do not assume to
have the alignment between them. For example, in text-to-speech synthesis,
${\bf x}^{i}$ is the mel-spectrogram of an audio sample, ${\bf c}^{i}$ is the
corresponding phonemes of the spoken words. While there exist methods, _e.g_.,
(McAuliffe et al., 2017; Kotani et al., 2020), that can segment phonemes from
an mel-spectrogram or characters from a handwriting sequence, we choose not to
rely on these information and directly learn the alignment using attention.
This allows the learned models to deal with ambiguous phoneme/character
boundaries (_e.g_., cursive handwriting). We also do not assume to have any
style supervision, including speaker or attribute labels, nor any grouping of
the data based on style.

While the unsupervised setting requires only the essential information
(_i.e_., samples and their content), it makes learning a controllable
generative sequence model difficult. The main challenge is the mismatch
between the inputs used during training and inference. As shown in Fig. 1c,
during inference we pair arbitrary content A and reference sample B as inputs
(_i.e_., nonparallel setting). However, due to the lack of ground truth
containing content A and in the style of B, during training we pair content A
and sample A (_i.e_., parallel setting). In other words, we train the model
under the parallel setting but we use the model in the non-parallel setting
during inference. Due to the training-inference mismatch, a well-performing
model during training may perform poorly during inference. If a generative
model learns to utilize the content information in the style example, during
inference the generative model will generate wrong content. This phenomenon is
called content leakage (Hu et al., 2020). In an extreme case, a model can
learn to copy the reference sample to the output; despite its perfect training
loss, it is useless because it always generates wrong content in practice.

This paper proposes a simple but effective technique to deal with the
training-inference mismatch when we learn controllable auto-regressive models
in an unsupervised-style manner. As shown in Fig. 1d, we train the model under
the non-parallel setting, _i.e_., we pair arbitrary content A with an
arbitrary sample B from the training dataset. Instead of directly using sample
B as style (in which case we have no ground-truth output), we jointly learn a
style transformation function, which estimates the style difference between A
and B and transforms the style of sample B to the style of A. The generative
model then takes content A and the transformation output (that contains the
style of A and the unrelated content information from B) to reconstruct sample
A. The proposed method enables us to use sample A as the ground truth while
learning in the non-parallel setting—the intended usage during inference.
Additionally, our method provides a systematic way to interpolate between the
style of two samples by scaling the estimated style difference between two
reference samples. We call the method style equalization. Note that for style
equalization to work, the style transformation and difference estimator need
to be carefully designed, such that no content information from content A can
be transferred through sample B. We defer the discussion to Sec. 4.

We apply the proposed method on two signal domains, speech and online-
handwriting, and evaluate the performance carefully via quantitative
evaluation (by computing content error rates) and conducting qualitative user
studies. Experimental results show that by tackling training-inference
mismatch with the proposed style equalization, we are able to learn strong
unsupervised controllable sequence generative models that have competitive
performance even when compared to existing methods that utilize style
supervision like speaker labels. On LibriTTS, style equalization achieves
close style replication (3.5 real oracle vs. 3.5 proposed in style opinion
score) and content reproduction errors (6.6% real oracle vs. 9.5% proposed) to
real samples.

##  2 Related Work

Controllable generative sequence models are not new in the literature;
however, the majority of these methods utilize style supervision, whereas the
paper focuses on developing an unsupervised-style method. Table 5 provides an
overview of the related works.

#### Unsupervised-style methods.

Unsupervised methods extract style information directly from samples, _i.e_.,
without any style labels or pretrained style embeddings. Existing unsupervised
methods train models under the parallel setting, as shown in Fig. 1c. To
prevent content leakage, most existing methods introduce a bottleneck on the
capacity of the style encoder by representing style as a single (time-
invariant) vector and limiting its dimension (Wang et al., 2018; Hsu et al.,
2018; Hu et al., 2020; Ma et al., 2018). Wang et al. (2018) propose Global
Style Token (GST), which represents a style vector as a linear combination of
a learned dictionary (called style tokens) shared across the dataset. The
number of style tokens (the implicit dimension of the style vector) is
carefully controlled to prevent content leakage. As we will see in Sec. 3, the
bottleneck not only reduces the amount of content information contained in the
style vector but also sacrifices style information.

Alternative loss formulations have also been proposed to limit content
information contained in the style representation. Hu et al. (2020) minimize
the mutual information between the style vector and the content sequence but
requires a pretrained content encoder and adversarial learning, which makes
training their model difficult. Hsu et al. (2018) approximate the posterior
distribution of the style vector using a mixture of Gaussian distributions
with a small number of mixtures. Ma et al. (2018) utilize a discriminator
conditioned on both the generated output and the content (similar to a content
recognizer). Akuzawa et al. (2018) anneal the Kullback-Leibler divergence to
control the amount of information contained in style. Henter et al. (2018)
utilize phoneme segmentation (McAuliffe et al., 2017) to avoid learning the
alignment between content ${\bf c}$ and output ${\bf x}$.

Priming is a technique that is introduced to control the style of auto-
regressive generative sequence models (Graves, 2013; Aksan et al., 2018).
Since the hidden state of a Recurrent Neural Network (RNN) contain all
information about current generation, including style, we can initialize the
RNN by pre-rolling the reference sample through the RNN. Utilizing priming
requires the content of the reference style. For example, Aksan et al. (2018)
learn a character recognizer and use it during inference. Moreover, since the
hidden state contains residual content from the reference example, it often
generates unexpected artifacts at the beginning of the sequence, as will be
seen in Sec. 5.

#### Supervised-style methods.

Many existing controllable generative models utilize style supervision, either
directly by passing attribute labels as inputs or implicitly by grouping
training data with their attribute labels. In the following, we briefly
introduce various supervised controllable sequence models. While using style
supervision avoids training-inference mismatch, it limits the style control on
a few sparsely-defined attribute classes. For instance, given a speech audio,
we can recognize the spoken texts, the accent, or even the speaker, but
provided solely with these attribute labels, it is impossible to exactly
reconstruct the original speech audio. The sparsely-defined attributes are
insufficient to capture the entire style information.

User identifications or their embeddings have been used to learn multi-speaker
text-to-speech models (Jia et al., 2018; Gibiansky et al., 2017; Kameoka et
al., 2020; Donahue et al., 2020; Chen et al., 2021; Dhariwal et al., 2020;
Valle et al., 2020; Kim et al., 2020; Hayashi et al., 2020; Sun et al., 2020b;
Skerry-Ryan et al., 2018; Sun et al., 2020a), voice conversion models (Qian et
al., 2019; Lee et al., 2021) and handwriting models (Kotani et al., 2020;
Bhunia et al., 2021; Kang et al., 2020; Davis et al., 2020). In addition to
user identifications, predefined features like pitch, phoneme duration,
loudness, and timbre have also been used by existing methods (Ren et al.,
2020; Qian et al., 2020; Dhariwal et al., 2020; Valle et al., 2020). Instead
of using speaker labels as input, Kameoka et al. (2018); Kaneko & Kameoka
(2018); Kaneko et al. (2019a, b) group training samples by their speaker
labels and apply adversarial learning to learn voice conversion models that
change speaker voices while keeping the content of the input.

#### Image methods.

Controllable generative models have also been developed for images (Härkönen
et al., 2020; Esser et al., 2019; Singh et al., 2019; Lample et al., 2017;
Karras et al., 2020; Brock et al., 2019; Collins et al., 2020; Shen et al.,
2020; Esser et al., 2020; Goetschalckx et al., 2019; Pavllo et al., 2020;
Zhang et al., 2018; Chan et al., 2021; Kwon & Ye, 2021; Kazemi et al., 2019),
which control the object class, pose, lighting, _etc_., of an image. Many
image style transform methods have also been developed (Isola et al., 2017;
Zhu et al., 2017; Gatys et al., 2016; Kotovenko et al., 2019). However, there
is a fundamental difference between image and sequence problems. In image
generative models, we do not need to learn the content-output alignment. The
content is usually defined globally as an image class or as pixel labels,
_e.g_., segmentation map. In contrast, our content is given as text, the
output is mel-spectrogram of a waveform, and the content and output have
different lengths. To utilize the input content sequence, generative sequence
models need to align the content and the output sequences and translate text
to the output signal modality. The complication exacerbates the training-
inference mismatch for sequence methods, since copying the style input is
easier than utilizing the input content.

##  3 Controllable Generative Sequence Models

We focus on learning controllable auto-regressive generative models,
$p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$, where ${\bf
x}=[x_{1},\dots,x_{T}]$ is the output sequence, ${\bf c}$ is the content
sequence, and ${\bf z}=[z_{1},\dots,z_{T}\,|\,z_{t}\in{\mathbb{R}}^{\ell}]$ is
the reference style information. Note that, in our model, style is also
represented as a sequence that changes over time. Under the style-unsupervised
setting, we are given a dataset $\mathcal{X}=\\{({\bf x}^{i},{\bf
c}^{i}),i\in\\{1\dots n\\}\\}$ that contains the ground-truth output sequence
${\bf x}^{i}$ and the corresponding content ${\bf c}^{i}$, but we do not have
supervision on ${\bf z}$. Therefore, we treat $z_{t}$ as a latent variable
with a learnable prior distribution $p(z_{t}|{\bf x}_{1..{t{-}1}},{\bf c})$
and optimize the log-likelihood of ${\bf x}$ conditioned on ${\bf c}$,
$\operatorname*{\mathbb{E}}_{({\bf x},{\bf c})}\log p({\bf x}|{\bf c})$.

Our model is a variational RNN (Chung et al., 2015), and we maximize a
variational lower bound of the likelihood:

$\displaystyle\operatorname*{\mathbb{E}}_{({\bf x},{\bf c})}\ \log p({\bf
x}|{\bf c})=\operatorname*{\mathbb{E}}_{({\bf x},{\bf c})}\sum_{t=1}^{T}\log
p(x_{t}|{\bf x}_{1..{t{-}1}},{\bf c})$
$\displaystyle=\operatorname*{\mathbb{E}}_{({\bf x},{\bf
c})}\sum_{t=1}^{T}\log\operatorname*{\mathbb{E}}_{z_{t}\sim p(z_{t}|{\bf
x}_{1..{t{-}1}},{\bf c})}p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$
$\displaystyle\geq\operatorname*{\mathbb{E}}_{({\bf x},{\bf
c})}\sum_{t=1}^{T}\operatorname*{\mathbb{E}}_{z_{t}\sim q(z_{t}|{\bf x},{\bf
c})}\log p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$
$\displaystyle\qquad\qquad-\mathcal{D}_{KL}\left(q(z_{t}|{\bf x},{\bf
c})\,||\,p(z_{t}|{\bf x}_{1..{t{-}1}},{\bf c})\right),$ (1)

where $\mathcal{D}_{KL}$ represents the Kullback-Leibler (KL)-divergence. In
eq. (1), we use the chain rule to expand $p({\bf x}|{\bf c})$ into
$p(x_{1}|{\bf c})\,p(x_{2}|x_{1},{\bf c})\cdots p(x_{T}|{\bf
x}_{1..T{-}1},{\bf c})$, introduce the variational approximation $q(z_{t}|{\bf
x},{\bf c})$ of the posterior distribution $p(z_{t}|{\bf x},{\bf c})$ for all
$t$, and apply Jensen’s inequality. Note that since $q$ is a variation
approximation of the posterior distribution, it can be conditioned on any
variable. As we will see in Sec. 4, the proposed style equalization
manipulates the input to $q$ such that we condition $q$ on a style-transformed
unrelated sample ${\bf x}^{\prime}$ instead of ${\bf x}$.

Fig. 2a shows an overview of the network used in the paper — the input content
${\bf c}$ is processed by the content attention, the style encoder (shown in
green) models $q(z_{t}|{\bf x}^{r},{\bf c})$, and the decoder (shown in gray)
models $p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$ using output from the
style encoder and the content attention. Note that during inference, ${\bf
x}^{r}$ is the reference example that we replicate the style of, and it
generally contains an unrelated content, _i.e_., ${\bf c}^{r}\neq{\bf c}$.
During training, the ground-truth sample ${\bf x}$ is used as the reference
style to optimize eq. (1), _i.e_., ${\bf x}^{r}={\bf x}$ and ${\bf c}^{r}={\bf
c}$, which is different from inference. Therefore a generative model can learn
to copy the style input to the output (and ignore the content input), leading
to incorrect generation during inference. This phenomenon can be remedied by
limiting the capacity of the style encoder, _e.g_., by decreasing the
dimensionality of the style representation. However, to achieve the lower
bound of eq. (1) and hence a higher generation quality, we need the style
encoder to contain enough capacity such that $q(z_{t}|{\bf x},{\bf c})\equiv
p(z_{t}|{\bf x},{\bf c})$ for all $t$. In this paper, we provide an
alternative training procedure that bypasses this trade-off, allowing us to
use a powerful style encoder (shown in Fig. 2b) with a high-dimensional style
representation.

Figure 2: Model used in the paper. (a) an overview of the entire model, which
includes a style encoder (in green), content attention, and decoder (in gray).
Note that the input content ${\bf c}$ can be the output of a content-embedding
network (used in speech synthesis) or one-hot encoding of characters (used in
handwriting synthesis). $a_{t}$ is the output of the content attention at time
$t$, which is a linear combination of the elements in ${\bf c}$. (b) the
proposed style encoder without style equalization. (c) the proposed style
encoder with the style equalization. $\phi$ computes the vector $\delta$ that
encodes the style difference between ${\bf x}^{\prime}$ and ${\bf x}$.
$\mathcal{M}$ applies this transformation to ${\bf x}^{\prime}$ to match the
style of ${\bf x}$.

##  4 Style Equalization

Let $({\bf x},{\bf c})$ and $({\bf x}^{\prime},{\bf c}^{\prime})$ be two
samples from the training set. To match the inference setting, we should train
the model using ${\bf c}$ as content and ${\bf x}^{\prime}$ as style input.
However, neither ${\bf x}$ nor ${\bf x}^{\prime}$ can be used as the ground-
truth output sequence. If we use ${\bf x}$ as the ground-truth output sequence
but ${\bf x}^{\prime}$ as style input, the generative model will learn to
ignore the style encoder since ${\bf x}^{\prime}$ contains unrelated style
information. In other words, the variational approximation $q({\bf z}|{\bf
x}^{\prime},{\bf c})$ is a poor approximation to the true posterior $p({\bf
z}|{\bf x},{\bf c})$. Alternatively, if we use ${\bf x}^{\prime}$ as the
ground-truth output sequence, the content given in ${\bf c}$ will be ignored.

We introduce a learnable style transformation function $\mathcal{M}({\bf
x}^{\prime},\delta)$ to transform the style of ${\bf x}^{\prime}$ by the
amount specified by a style difference vector $\delta\in{\mathbb{R}}^{k}$. We
first estimate $\delta$ with a learnable function $\phi$, _i.e_.
$\delta=\phi({\bf x}^{\prime},{\bf x})$; we then transform ${\bf x}^{\prime}$
with $\mathcal{M}({\bf x}^{\prime},\delta)$. By jointly optimizing
$\mathcal{M}$, $\phi$, and the rest of the generative model using eq. (1)
(with the input to $q$ modified as described), the model learns to transfer
style information from ${\bf x}$ through $\mathcal{M}$ to maximize the log-
likelihood of the ground-truth ${\bf x}$. In other words, we approximate the
posterior distribution $p({\bf z}|{\bf x},{\bf c})$ with $q({\bf
z}|\mathcal{M}({\bf x}^{\prime},\phi({\bf x}^{\prime},{\bf x})),{\bf c})$,
which is a better approximation than $q({\bf z}|{\bf x}^{\prime},{\bf c})$. We
call this method style equalization.

Note that, for style equalization to be successful, $\mathcal{M}\circ\phi$
should not transfer any content-related information (_e.g_., copy the entire
sequence) from ${\bf x}$ but only its style information so that the decoder
will utilize the transferred style and will rely on provided content input to
generate the output. Therefore the design of $\mathcal{M}$ is critical.

#### Design of $\mathcal{M}$ and $\phi$.

An important observation we use in the design of $\mathcal{M}$ and $\phi$ is
that content information (_i.e_., sequence of phonemes or characters) is
strongly time-dependent whereas the style can be reasonably well approximated
by a time-independent representation (_e.g_., voice characteristics of a
speaker and microphone response, _etc_.). By designing $\phi$ such that no
time-dependent information is stored in $\delta$, we can satisfy that content-
related information is not leaked but still transfer the time-independent
style information. Thus, we utilize average pooling over the time dimension in
$\phi$, and to improve the time-invariant property of the convolutional
network in our style encoder, we use convolutional filters without padding and
with low-pass filtering before down-sampling (Zhang, 2019).

As shown in Fig. 2c, to estimate the style difference vector $\delta$ between
two sequences ${\bf x}$ and ${\bf x}^{\prime}$, we first compute their style
features ${\bf f}$ and ${\bf f}^{\prime}$ using a convolutional network. Note
that ${\bf f}$ and ${\bf f}^{\prime}$ are $s$-dimensional feature sequences
with different lengths. We define

$\displaystyle\phi({\bf x}^{\prime},{\bf x})=\mbox{avg}(A\,{\bf
f})-\mbox{avg}(A\,{\bf f}^{\prime})\mbox{ and }$
$\displaystyle\mathcal{M}({\bf x}^{\prime},\phi({\bf x}^{\prime},{\bf
x}))={\bf f}^{\prime}+A^{\top}\phi({\bf x}^{\prime},{\bf x}),$ (2)

where avg represents taking mean across time, $A\in{\mathbb{R}}^{k\times s}$
is a learnable linear transform, and ${\bf f}^{\prime}+A^{\top}\phi({\bf
x}^{\prime},{\bf x})$ means that the vector $A^{\top}\phi({\bf
x}^{\prime},{\bf x}){\in}{\mathbb{R}}^{s}$ is added to each time step of ${\bf
f}^{\prime}$. Intuitively, the design assumes that the style information lies
on a $k$-dimensional subspace, and we equalize the style between ${\bf
x}^{\prime}$ and ${\bf x}$ by minimizing their differences in the subspace. It
also satisfies the identity property by construction — $\phi({\bf x}^{r},{\bf
x}^{r})=0$ and $\mathcal{M}({\bf x}^{r},0)={\bf f}^{r}$ — which enables style
interpolation (see below) and allows us to treat style equalization as a
training procedure and remove it from the model during inference (as shown in
Fig. 2b). Note that making a convolutional network entirely time-invariant is
an important ongoing research problem (Karras et al., 2021); please see Sec. H
for the discussions on our limitations.

#### Interpolation between two styles.

Once learned, $\mathcal{M}$ and $\phi$ can be used to interpolate style during
inference. Given two style references, ${\bf x}^{s}$ and ${\bf x}^{t}$, we
interpolate between them with $\mathcal{M}({\bf x}^{s},\alpha\,\phi({\bf
x}^{s},{\bf x}^{t}))$, where a scalar $\alpha\,{\in}\,{\mathbb{R}}$ controls
the interpolation. By changing $\alpha$, we traverse a one-dimensional
manifold that starts from the original style (with $\alpha=0$, since
$\mathcal{M}({\bf x}^{s},\phi({\bf x}^{s},{\bf x}^{s}))={\bf f}^{s}$) and ends
at the target style (with $\alpha=1$). Note that, unlike existing generative
models that support style interpolation in post-processing, $\mathcal{M}$ and
$\phi$ are trained to transform style by design.

##  5 Experiments

To demonstrate the generality of the proposed method, we train and evaluate it
on two signal domains, speech and handwriting, with the same model
architecture design. In the following, we introduce the model architecture
used in the experiments, the baselines, the metrics, and the results. More
details are provided in Sec. C.

###  5.1 Model Architecture

Our model is auto-regressive and composed of (i) a decoder that is modeling
$p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$, (ii) a content attention
module, (iii) a style encoder that is modeling $q(z_{t}|\cdot,{\bf c})$, and
(iv) a network that models the prior distribution of $z_{t}$. Fig. 2a shows an
overview of the model. The backbone of the model, namely the content attention
and the decoder, uses a standard architecture that was proposed by Graves
(2013) for handwriting synthesis and later extended to speech in variations of
Tacotron (Shen et al., 2018; Wang et al., 2018). The variational approximation
$q(z_{t}|\cdot)$ and the prior distribution are modeled as multivariate
Gaussian distributions with a diagonal covariance matrix.

Our style encoder is composed of a convolutional network and a multi-head
attention layer, as shown in Fig. 2b. The convolutional network extracts the
style feature sequence ${\bf f}^{r}$ from the reference style input ${\bf
x}^{r}$. We use multi-head attention to extract relevant style information at
every time step from ${\bf f}^{r}$ with the query computed from the hidden
state of the LSTM, $h_{t}$, which contains information about past generations,
and the currently focused content $a_{t}$. Thus, while $\delta$ contains only
time-invariant information, our style representation is a time-varying
sequence. The intuition is if the model utters a particular phoneme, it should
be able to find the information in the style reference and mimic it.

For style equalization, we insert $\mathcal{M}$ and $\phi$ into the style
encoder, as shown in Fig. 2c. Since style equalization is only able to
transfer time-independent information, when we utilize this procedure, the
network will not be able to learn time-dependent style. To enable learning
time-dependent style information during training, half of the batches, we use
${\bf x}^{\prime}={\bf x}$, which means that the difference vector $\delta=0$,
hence the decoder directly uses the ground-truth style information which
contains time-dependent style information. We analyze the effect of style
attention and its ability to represent time-varying style information in Sec.
5.4.

###  5.2 Speech Synthesis

We train and evaluate the proposed method on two multi-speaker speech
datasets. VCTK dataset (Yamagishi et al., 2019) contains 110 speakers and 44
hours of speech, and LibriTTS dataset (Zen et al., 2019) contains 2,311
speakers and 555 hours of speech in the training set.

#### Baselines.

We compare the proposed method with Global Style Tokens (GST-n) (Wang et al.,
2018) with various numbers of tokens n. For completeness, we also compare with
Tacotron 2 (Shen et al., 2018) (even though it does not have style control),
Tacotron-S and GST-nS. Tacotron-S / GST-nS are Tacotron / GST-n with style
supervision — a pretrained speaker embedding (Snyder et al., 2018) that was
trained on the VoxCeleb dataset, which contains $2,000$ hours of speech from
$7,000$ speakers (Chung et al., 2018). We use ESPnet-TTS (Hayashi et al.,
2020), a widely used implementation of the baselines and follow their training
recipe. They achieve similar performance as those listed in the original
papers. With no publicly available implementation, we cannot compare with (Hsu
et al., 2018) and (Ma et al., 2018). All methods are trained on the same
dataset, while Tacotron-S and GST-nS have additional style supervision from
VoxCeleb dataset. All methods output $80$-dimensional mel-spectrograms with
the sampling rate equal to 22,050 Hz and the window size equal to 1,024, which
are converted to waveforms using a pretrained WaveGlow vocoder (Prenger et
al., 2019). The content input is represented as phonemes, following (Shen et
al., 2018).

#### Metrics.

We measure the content generation errors as Word Error Rate (WER), using a
pretrained speech recognition model, ESPnet (kamo naoyuki, 2021). To evaluate
the style replication accuracy of the methods, we use a speaker classification
network (Deng et al., 2019) (see Sec. D) and measure the style similarity
between reference and output generations. We report cos-sim, which is the
cosine similarity between the reference example and its corresponding output
generation, and sRank, which is the rank of the reference speaker out of all
speakers based on their cosine similarities. We report the empirical mean and
standard deviation of these metrics on 100 pairs of reference and synthetic
samples.

We also report the style opinion score following the protocol used by Zhao et
al. (2020). To evaluate the style similarity between a generated output and a
style reference, users were given pairs of reference and synthesized audio,
and asked if “the two samples could have been produced by the same speaker in
a similar environmental condition”, and asked to score with “ 4 (Absolutely
same)”, “3 (Likely same)”, “2 (Likely different)”, “1 (Absolutely different)”.
We synthesized 100 samples using each method with the same style example and
target content. A total of $15$ users participated in the study, and we
collected $630$ responses.

We also provide an oracle (a pseudo upper-bound) where we select a different
real speech sample from the same speaker from the dataset, and evaluate style
similarity and content error. This provides a good calibration for our
evaluation metrics and opinion studies.

Table 1: Quantitative results on VCTK dataset. The reference style inputs are
seen (randomly selected from the training set). WER measures content accuracy;
cosine-similarity (cos-sim) and sRank measure style similarity.

Method Parallel text Nonparallel text WER (%) cos-sim $\uparrow$ sRank
$\downarrow$ WER (%) cos-sim $\uparrow$ sRank $\downarrow$ Tacotron $16.0\pm
1.7$ $0.05\pm 0.13$ $53.1\pm 29.1$ $16.4\pm 1.2$ $0.05\pm 0.12$ $53.9\pm 27.8$
Tacotron-S $13.6\pm 0.7$ $0.24\pm 0.18$ $16.4\pm 20.9$ $16.3\pm 0.4$ $0.22\pm
0.18$ $18.0\pm 21.9$ GST-16 $18.6\pm 0.9$ $0.23\pm 0.15$ $21.4\pm 21.9$
$18.5\pm 1.1$ $0.23\pm 0.16$ $21.1\pm 22.4$ GST-64 $16.9\pm 0.5$ $0.23\pm
0.17$ $24.4\pm 23.1$ $27.5\pm 0.4$ $0.22\pm 0.16$ $25.2\pm 24.4$ GST-16S
$8.3\pm 0.1$ $0.34\pm 0.18$ $10.8\pm 15.2$ $17.7\pm 0.8$ $0.31\pm 0.17$
$13.0\pm 20.0$ GST-64S $14.1\pm 0.3$ $0.33\pm 0.18$ $11.4\pm 16.3$ $24.7\pm
1.0$ $0.32\pm 0.18$ $12.7\pm 18.1$ Proposed $\mathbf{7.4\pm 0.2}$
$\mathbf{0.73\pm 0.12}$ $\mathbf{1.5\pm 2.1}$ $\mathbf{9.5\pm 0.4}$
$\mathbf{0.64\pm 0.14}$ $\mathbf{1.9\pm 4.2}$ Oracle $6.6\pm 0.0$ $1.0\pm 0.0$
$1.0\pm 0.0$ $6.6\pm 0.0$ $0.57\pm 0.16$ $1.6\pm 4.1$

Table 2: Quantitative results on LibriTTS-all-960 dataset.

Method Seen speakers, parallel text Seen speakers, nonparallel text Unseen
speakers, nonparallel text WER (%) cos-sim $\uparrow$ sRank $\downarrow$ WER
(%) cos-sim $\uparrow$ sRank $\downarrow$ WER (%) cos-sim $\uparrow$ sRank
$\downarrow$ Tacotron $64.4\pm 4.1$ $0.00\pm 0.10$ $1218\pm 671$ $52.2\pm 1.7$
$0.01\pm 0.10$ $1140\pm 651$ $52.2\pm 0.6$ $0.00\pm 0.10$ $847\pm 563$
Tacotron-S $14.9\pm 0.0$ $0.23\pm 0.22$ $430\pm 584$ $18.7\pm 0.2$ $0.24\pm
0.22$ $370\pm 562$ $13.5\pm 0.0$ $0.16\pm 0.16$ $289\pm 436$ GST-64 $38.2\pm
2.2$ $0.12\pm 0.19$ $706\pm 701$ $33.3\pm 3.4$ $0.12\pm 0.19$ $700\pm 739$
$30.4\pm 2.2$ $0.09\pm 0.16$ $535\pm 610$ GST-192 $19.3\pm 0.3$ $0.10\pm 0.17$
$786\pm 725$ $17.8\pm 0.6$ $0.09\pm 0.16$ $823\pm 719$ $18.0\pm 0.7$ $0.07\pm
0.14$ $587\pm 582$ GST-64S $19.7\pm 0.7$ $0.39\pm 0.23$ $150\pm 305$ $20.4\pm
1.6$ $0.40\pm 0.23$ $143\pm 334$ $16.5\pm 0.2$ $0.28\pm 0.17$ $121\pm 259$
GST-192S $13.8\pm 0.7$ $0.39\pm 0.23$ $137\pm 309$ $15.4\pm 1.1$ $0.41\pm
0.22$ $126\pm 317$ $13.4\pm 0.2$ $0.29\pm 0.18$ $139\pm 316$ Proposed
$\mathbf{6.2\pm 0.5}$ $\mathbf{0.82\pm 0.14}$ $\mathbf{1.7\pm 4.1}$
$\mathbf{9.4\pm 0.3}$ $\mathbf{0.78\pm 0.14}$ $\mathbf{1.8\pm 6.0}$
$\mathbf{7.6\pm 0.9}$ $\mathbf{0.57\pm 0.15}$ $\mathbf{7.4\pm 42.6}$ Oracle
$6.5\pm 0.0$ $1.0\pm 0.0$ $1.0\pm 0.0$ $6.5\pm 0.0$ $0.85\pm 0.06$ $1.0\pm
0.0$ $6.5\pm 0.0$ $0.50\pm 0.23$ $3.6\pm 24.9$

Table 3: Style opinion scores of speech synthesizers.

VCTK, seen speakers LibriTTS, seen speakers LibriTTS, unseen speakers GST-64
GST-16S Proposed Oracle GST-192 GST-192S Proposed Oracle GST-192 GST-192S
Proposed Oracle $2.1\pm 1.0$ $3.3\pm 0.9$ $3.8\pm 0.4$ $3.8\pm 0.4$ $1.4\pm
0.6$ $2.8\pm 1.0$ $\mathbf{3.6\pm 0.6}$ $3.5\pm 0.9$ $1.2\pm 0.5$ $2.6\pm 0.9$
$\mathbf{3.5\pm 0.7}$ $3.5\pm 0.9$

#### Results.

Table 1 shows the results on VCTK dataset. Let us first look at the parallel-
text setting. Without any style control, Tacotron achieves a low cosine
similarity. Utilizing GST and speaker embedding (Tacotron-S and GST-nS)
improve cosine similarities. With the proposed style equalization and the
style encoder, our model achieves the highest similarity and lowest sRank.

When comparing the WERs between the parallel and non-parallel settings, we can
see the adversarial effect of content leakage — the models with a high-
capacity style encoder (GST-64, GST-16S, and GST-64S) produce a much higher
word-error rate in the non-parallel setting than in the parallel setting,
while the small capacity GST-16 is largely unaffected. In comparison, our
model is less affected by the parallel / nonparallel settings. The
supplemental website shows the generated audios.

Table 2 shows the results on LibriTTS dataset. As can be seen from the
results, the large variety (_e.g_., more accents, higher background noise,
difficult microphone effects) makes it a more difficult dataset than VCTK, as
reflected by the higher WERs of Tacotron than those in VCTK. The proposed
method, on the other hand, is able to learn from the noisy data and mimic the
wide variety of styles in the dataset. In addition, since the training set
contains more than $2,000$ speakers, the proposed method learns to generalize
and mimic the voices of unseen speakers in the validation set, as demonstrated
by the high cosine similarity in the right-most column of Table 2. The
supplemental website shows the generated audios, including those by a model
using the same architecture as ours but trained without style equalization.
While the model performs well in the parallel setting, it fails to generate
the correct content in the nonparallel setting, and we can clearly hear the
content from the nonparallel style input leaked into the generated output. In
comparison, utilizing style equalization during training significantly
improves the problem. Please also see the supplemental website for results
with style sampled from the prior distribution or interpolated between two
examples.

Table 3 shows style opinion scores on the two datasets. The proposed method
achieves high scores, similar to those of the oracle.

Finally, it is worth noting that by directly mimicking the reference style
example, our models achieve higher or comparable cos-sim and style opinion
scores in Table 1-3 than the oracle, which is a random sample in the dataset
from the same speaker. These results indicate the effectiveness of mimicking
the style of the given reference example over utilizing speaker embedding,
which learns a generic voice of a speaker.

###  5.3 Online-Handwriting Synthesis

Online-handwriting synthesis aims to generate sequences of pen movements on a
writing surface. A handwriting sample is represented as a sequence of
$(x,y,p)$ triplets, where $x$ and $y$ are the coordinates of the pen on the
surface, and $p$ is a binary variable indicating whether the pen touches the
surface over time. We apply the proposed method to a subset of a proprietary
dataset collected for research and development. The subset consists of 600k
online handwriting samples that were written by 1,500 people in English,
French, German, Italian, Spanish, and Portuguese.

Figure 3: Handwriting generation results and evaluation. The reference style
examples are shown in black, and the outputs are shown in blue.

#### Baselines and metrics.

While there exist many handwriting synthesizers, most of them requires writer
labels (Kotani et al., 2020; Bhunia et al., 2021; Kang et al., 2020) or
character segmentation (Davis et al., 2020; Aksan et al., 2018; Kotani et al.,
2020) (Table 5). We compare with the method proposed by Graves (2013) that
uses priming for style encoding; since the proposed style equalization is a
training mechanism, we also compare with an ablation of our model trained
without style equalization. Note that all these models use the same decoder
and content attention. Similar to speech, we measure content generation error
as Character Error rate (CER) with a pretrained handwriting recognizer, and we
conduct style-opinion-score study on 12 users and collected 320 responses. We
also provide an oracle where we select a different real handwriting sample
from the same writer from the dataset and compute the metrics.

#### Results.

Fig. 3 shows synthesized handwriting from each of the methods on unseen style
examples. As can be seen, while Graves (2013) with priming is an effective
method to replicate the reference style, it often outputs artifacts at the
beginning due to the residual information in the hidden state. The model
without style equalization produces high-quality replication in the parallel
setting; however, it suffers severely under the nonparallel setting and
produces wrong content. By training with the proposed style equalization, the
CER in the nonparallel setting improved significantly. We demonstrate style
interpolate in Fig. 3b and results with styles sampled from the learned prior
distribution in Fig. 3c.

Note that due to privacy reasons, the handwriting reference examples shown in
the paper and the supplemental website are synthetic. They are close
reproductions of unseen real styles using a generative model with a different
architecture. The generations shown here are very similar when real samples
are used as style input. All evaluations reported in Fig. 3d use real unseen
style examples.

(a) 100%, ${\bf c}^{r}={\bf c}$

(b) 50%, ${\bf c}^{r}={\bf c}$

(c) 50%, ${\bf c}^{r}\neq{\bf c}$

(d) 50%, ${\bf c}^{r}{=}{\bf c}$, ${\bf x}^{r}$ transformed

Figure 4: Style attention weights of two models trained on LibriTTS during
inference on unseen styles. (a) style equalization always applied during
training (100%) and (b,c,d) style equalization applied to 50% of the batches.
All share the same input content ${\bf c}$, (a,b,d) share the same input
style, but (d) transforms the input style to that of (c).

###  5.4 Analysis of Style Attention

Finally, we analyze how our model utilizes the time-dependent style
information contained in the reference example ${\bf x}^{r}$ by examining the
attention weights of the style attention module. As discussed in Sec. 4, while
our style representation is a time-dependent sequence, style equalization can
only transfer time-independent global style information from the style
reference. Therefore, when we apply style equalization to all the samples
during training, we learn a time-independent representation. This can be seen
from Fig. 4a where the attention weights become constant over time during
inference.

We encourage our style encoder to learn the time-dependent aspects of style by
applying style equalization only to 50% of the training samples. As can be
seen from the time-varying attention weights in Fig. 4b-d, this training
procedure enables the style encoder to utilize style information more
efficiently by focusing time instances with similar signal and context. For
example, when the style reference contains the target content, _i.e_., ${\bf
c}^{r}={\bf c}$ (Fig. 4b), the attention weights form a block-diagonal
pattern, indicating the model focuses to the time instances of the style
signal that match the current content and context. When ${\bf c}^{r}\neq{\bf
c}$ (Fig. 4c), the weights are still well localized over time to gather pieces
of time-dependent style information from matching signal in ${\bf x}^{r}$.
Last, we transform the parallel style (Fig. 4b) to that of a nonparallel one
(Fig. 4c) during inference in Fig. 4d. Comparing Fig. 4b & d, the diagonal
shape disappears, indicating the style transform reduces the source content
information.

The flexibility to apply/remove style equalization (and hence the
representation bottleneck) during training is one of the main differences
compared to (Wang et al., 2018; Hsu et al., 2018), which always apply the
style bottleneck.

##  6 Conclusion

This paper proposes a simple but effective training strategy, style
equalization, to mitigate the training-inference mismatch and learn a
generative sequence model where style and content can be controlled separately
without utilizing any style supervision. We demonstrate 1) replication of
styles from a single style reference, 2) interpolation between two reference
styles, and 3) generating new styles. Experiments on both speech and
handwriting domains show the effectiveness of the proposed method in
mitigating the training-inference mismatch, which enables our models to obtain
high-quality synthesis results.

## References

  * Aksan et al. (2018) Aksan, E., Pece, F., and Hilliges, O.  Deepwriting: Making digital ink editable via deep generative modeling.  In _Conference on Human Factors in Computing Systems (CHI)_ , pp. 1–14, 2018.
  * Akuzawa et al. (2018) Akuzawa, K., Iwasawa, Y., and Matsuo, Y.  Expressive speech synthesis via modeling expressions with variational autoencoder.  In _Interspeech_ , 2018.
  * Alegre et al. (2013) Alegre, F., Amehraye, A., and Evans, N.  Spoofing countermeasures to protect automatic speaker verification from voice conversion.  In _2013 IEEE International Conference on Acoustics, Speech and Signal Processing_ , pp. 3068–3072. IEEE, 2013.
  * Bhunia et al. (2021) Bhunia, A. K., Khan, S., Cholakkal, H., Anwer, R. M., Khan, F. S., and Shah, M.  Handwriting transformers.  _arXiv preprint arXiv:2104.03964_ , 2021.
  * Brock et al. (2019) Brock, A., Donahue, J., and Simonyan, K.  Large scale GAN training for high fidelity natural image synthesis.  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Brundage et al. (2018) Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al.  The malicious use of artificial intelligence: Forecasting, prevention, and mitigation.  _arXiv preprint arXiv:1802.07228_ , 2018.
  * Chan et al. (2021) Chan, E. R., Monteiro, M., Kellnhofer, P., Wu, J., and Wetzstein, G.  pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5799–5809, 2021.
  * Chen et al. (2021) Chen, M., Tan, X., Li, B., Liu, Y., Qin, T., sheng zhao, and Liu, T.-Y.  AdaSpeech: Adaptive text to speech for custom voice.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Chen et al. (2020) Chen, T., Kumar, A., Nagarsheth, P., Sivaraman, G., and Khoury, E.  Generalization of audio deepfake detection.  In _Proc. Odyssey 2020 The Speaker and Language Recognition Workshop_ , pp. 132–137, 2020.
  * Chung et al. (2015) Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., and Bengio, Y.  A recurrent latent variable model for sequential data.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2015\.
  * Chung et al. (2018) Chung, J. S., Nagrani, A., and Zisserman, A.  VoxCeleb2: Deep speaker recognition.  _Proc. Interspeech_ , pp. 1086–1090, 2018.
  * Collins et al. (2020) Collins, E., Bala, R., Price, B., and Susstrunk, S.  Editing in style: Uncovering the local semantics of GANs.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Davis et al. (2020) Davis, B., Tensmeyer, C., Price, B., Wigington, C., Morse, B., and Jain, R.  Text and style conditioned GAN for generation of offline handwriting lines.  In _British Machine Vision Conference (BMVC)_ , 2020.
  * Deng et al. (2019) Deng, J., Guo, J., Xue, N., and Zafeiriou, S.  Arcface: Additive angular margin loss for deep face recognition.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Dhariwal et al. (2020) Dhariwal, P., Jun, H., Payne, C., Kim, J. W., Radford, A., and Sutskever, I.  Jukebox: A generative model for music.  _arXiv preprint arXiv:2005.00341_ , 2020.
  * Dolhansky et al. (2020) Dolhansky, B., Bitton, J., Pflaum, B., Lu, J., Howes, R., Wang, M., and Canton Ferrer, C.  The deepfake detection challenge dataset.  _arXiv e-prints_ , pp. arXiv–2006, 2020.
  * Donahue et al. (2020) Donahue, J., Dieleman, S., Bińkowski, M., Elsen, E., and Simonyan, K.  End-to-end adversarial text-to-speech.  _arXiv preprint arXiv:2006.03575_ , 2020.
  * Esser et al. (2019) Esser, P., Haux, J., and Ommer, B.  Unsupervised robust disentangling of latent characteristics for image synthesis.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * Esser et al. (2020) Esser, P., Rombach, R., and Ommer, B.  A disentangling invertible interpretation network for explaining latent representations.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Evans et al. (2013) Evans, N. W., Kinnunen, T., and Yamagishi, J.  Spoofing and countermeasures for automatic speaker verification.  In _Interspeech_ , pp. 925–929, 2013.
  * Gatys et al. (2016) Gatys, L., Ecker, A., and Bethge, M.  A neural algorithm of artistic style.  _Journal of Vision_ , 16(12):326–326, 2016.
  * Gibiansky et al. (2017) Gibiansky, A., Arik, S., Diamos, G., Miller, J., Peng, K., Ping, W., Raiman, J., and Zhou, Y.  Deep voice 2: Multi-speaker neural text-to-speech.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 2966–2974, 2017.
  * Goetschalckx et al. (2019) Goetschalckx, L., Andonian, A., Oliva, A., and Isola, P.  GANalyze: Toward visual definitions of cognitive image properties.  In _IEEE International Conference on Computer Vision (ICCV)_ , 2019.
  * Graves (2013) Graves, A.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_ , 2013.
  * Güera & Delp (2018) Güera, D. and Delp, E. J.  Deepfake video detection using recurrent neural networks.  In _2018 15th IEEE international conference on advanced video and signal based surveillance (AVSS)_ , pp. 1–6. IEEE, 2018.
  * Härkönen et al. (2020) Härkönen, E., Hertzmann, A., Lehtinen, J., and Paris, S.  GANSpace: Discovering interpretable GAN controls.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Hayashi et al. (2020) Hayashi, T., Yamamoto, R., Inoue, K., Yoshimura, T., Watanabe, S., Toda, T., Takeda, K., Zhang, Y., and Tan, X.  Espnet-TTS: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , 2020.
  * Hendrycks & Gimpel (2016) Hendrycks, D. and Gimpel, K.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.
  * Henter et al. (2018) Henter, G. E., Lorenzo-Trueba, J., Wang, X., and Yamagishi, J.  Deep encoder-decoder models for unsupervised learning of controllable speech synthesis.  _arXiv preprint arXiv:1807.11470_ , 2018.
  * Hsu et al. (2018) Hsu, W.-N., Zhang, Y., Weiss, R. J., Zen, H., Wu, Y., Wang, Y., Cao, Y., Jia, Y., Chen, Z., Shen, J., et al.  Hierarchical generative modeling for controllable speech synthesis.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Hu et al. (2020) Hu, T.-Y., Shrivastava, A., Tuzel, O., and Dhir, C.  Unsupervised style and content separation by minimizing mutual information for speech synthesis.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , 2020.
  * Hua et al. (2016) Hua, G., Huang, J., Shi, Y. Q., Goh, J., and Thing, V. L.  Twenty years of digital audio watermarking — a comprehensive review.  _Signal processing_ , 128:222–242, 2016.
  * Hutchinson (1989) Hutchinson, M. F.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.  _Communications in Statistics-Simulation and Computation_ , 18(3):1059–1076, 1989.
  * Islam et al. (2019) Islam, M. A., Jia, S., and Bruce, N. D.  How much position information do convolutional neural networks encode?  In _International Conference on Learning Representations (ICLR)_ , 2019.
  * Isola et al. (2017) Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A.  Image-to-image translation with conditional adversarial networks.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1125–1134, 2017.
  * Jia et al. (2018) Jia, Y., Zhang, Y., Weiss, R. J., Wang, Q., Shen, J., Ren, F., Chen, Z., Nguyen, P., Pang, R., Moreno, I. L., et al.  Transfer learning from speaker verification to multispeaker text-to-speech synthesis.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , pp. 4485–4495, 2018.
  * Kameoka et al. (2018) Kameoka, H., Kaneko, T., Tanaka, K., and Hojo, N.  StarGAN-VC: Non-parallel many-to-many voice conversion using star generative adversarial networks.  In _Spoken Language Technology Workshop (SLT)_ , pp. 266–273. IEEE, 2018.
  * Kameoka et al. (2020) Kameoka, H., Huang, W.-C., Tanaka, K., Kaneko, T., Hojo, N., and Toda, T.  Many-to-many voice transformer network.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2020.
  * kamo naoyuki (2021) kamo naoyuki.  ESPnet2 pretrained model, kamo-naoyuki/librispeech_asr_train_asr_conformer6_n_fft512_hop_length256_raw_en_bpe5000_scheduler_confwarmup_steps40000_optim_conflr0.0025_sp_valid.acc.ave,fs=16k,lang=en, March 2021.  URL https://doi.org/10.5281/zenodo.4604066.
  * Kaneko & Kameoka (2018) Kaneko, T. and Kameoka, H.  CycleGAN-VC: Non-parallel voice conversion using cycle-consistent adversarial networks.  In _European Signal Processing Conference (EUSIPCO)_ , pp. 2100–2104. IEEE, 2018.
  * Kaneko et al. (2019a) Kaneko, T., Kameoka, H., Tanaka, K., and Hojo, N.  CycleGAN-VC2: Improved cycleGAN-based non-parallel voice conversion.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 6820–6824. IEEE, 2019a.
  * Kaneko et al. (2019b) Kaneko, T., Kameoka, H., Tanaka, K., and Hojo, N.  StarGAN-VC2: Rethinking conditional methods for starGAN-based voice conversion.  In _INTERSPEECH_ , 2019b.
  * Kang et al. (2020) Kang, L., Riba, P., Wang, Y., Rusiñol, M., Fornés, A., and Villegas, M.  Ganwriting: Content-conditioned generation of styled handwritten word images.  In _European Conference on Computer Vision (ECCV)_ , pp. 273–289. Springer, 2020.
  * Karras et al. (2020) Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T.  Analyzing and improving the image quality of StyleGAN.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Karras et al. (2021) Karras, T., Aittala, M., Laine, S., Härkönen, E., Hellsten, J., Lehtinen, J., and Aila, T.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 34, 2021.
  * Kazemi et al. (2019) Kazemi, H., Iranmanesh, S. M., and Nasrabadi, N.  Style and content disentanglement in generative adversarial networks.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 848–856, 2019.
  * Kim et al. (2020) Kim, J., Kim, S., Kong, J., and Yoon, S.  Glow-TTS: A generative flow for text-to-speech via monotonic alignment search.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 33, 2020.
  * Kingma & Ba (2015) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations (ICLR)_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In _International Conference on Learning Representations (ICLR)_ , 2014.
  * Kinnunen et al. (2020) Kinnunen, T., Delgado, H., Evans, N., Lee, K. A., Vestman, V., Nautsch, A., Todisco, M., Wang, X., Sahidullah, M., Yamagishi, J., et al.  Tandem assessment of spoofing countermeasures and automatic speaker verification: Fundamentals.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 28:2195–2210, 2020.
  * Kotani et al. (2020) Kotani, A., Tellex, S., and Tompkin, J.  Generating handwriting via decoupled style descriptors.  In _European Conference on Computer Vision (ECCV)_ , pp. 764–780. Springer, 2020.
  * Kotovenko et al. (2019) Kotovenko, D., Sanakoyeu, A., Lang, S., and Ommer, B.  Content and style disentanglement for artistic style transfer.  In _IEEE International Conference on Computer Vision (ICCV)_ , pp. 4422–4431, 2019.
  * Kwon & Ye (2021) Kwon, G. and Ye, J. C.  Diagonal attention and style-based gan for content-style disentanglement in image generation and translation.  In _IEEE International Conference on Computer Vision (ICCV)_ , pp. 13980–13989, 2021.
  * Lample et al. (2017) Lample, G., Zeghidour, N., Usunier, N., Bordes, A., DENOYER, L., and Ranzato, M. A.  Fader networks:manipulating images by sliding attributes.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Lee et al. (2021) Lee, S.-H., Kim, J.-H., Chung, H., and Lee, S.-W.  Voicemixer: Adversarial voice style mixup.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 34, 2021.
  * Lyu (2020) Lyu, S.  Deepfake detection: Current challenges and next steps.  In _2020 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)_, pp. 1–6. IEEE, 2020.
  * Ma et al. (2018) Ma, S., Mcduff, D., and Song, Y.  Neural TTS stylization with adversarial and collaborative games.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * McAuliffe et al. (2017) McAuliffe, M., Socolof, M., Mihuc, S., Wagner, M., and Sonderegger, M.  Montreal forced aligner: Trainable text-speech alignment using kaldi.  In _Interspeech_ , volume 2017, pp. 498–502, 2017.
  * Meng et al. (2021) Meng, C., Song, J., Song, Y., Zhao, S., and Ermon, S.  Improved autoregressive modeling with distribution smoothing.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Pavllo et al. (2020) Pavllo, D., Lucchi, A., and Hofmann, T.  Controlling style and semantics in weakly-supervised image generation.  In _European Conference on Computer Vision (ECCV)_ , pp. 482–499. Springer, 2020.
  * Ping et al. (2018) Ping, W., Peng, K., Gibiansky, A., Arik, S. Ö., Kannan, A., Narang, S., Raiman, J., and Miller, J.  Deep voice 3: Scaling text-to-speech with convolutional sequence learning.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Prenger et al. (2019) Prenger, R., Valle, R., and Catanzaro, B.  Waveglow: A flow-based generative network for speech synthesis.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 3617–3621. IEEE, 2019.
  * Qian et al. (2019) Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M.  AutoVC: Zero-shot voice style transfer with only autoencoder loss.  In _International Conference on Machine Learning (ICML)_ , pp. 5210–5219. PMLR, 2019.
  * Qian et al. (2020) Qian, K., Zhang, Y., Chang, S., Hasegawa-Johnson, M., and Cox, D.  Unsupervised speech decomposition via triple information bottleneck.  In _International Conference on Machine Learning (ICML)_ , pp. 7836–7846. PMLR, 2020.
  * Ramachandran et al. (2018) Ramachandran, P., Zoph, B., and Le, Q.  Searching for activation functions.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Ren et al. (2020) Ren, Y., Hu, C., Tan, X., Qin, T., Zhao, S., Zhao, Z., and Liu, T.-Y.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  _arXiv preprint arXiv:2006.04558_ , 2020.
  * Shen et al. (2018) Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al.  Natural TTS synthesis by conditioning waveNet on mel spectrogram predictions.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 4779–4783, 2018.
  * Shen et al. (2020) Shen, Y., Gu, J., Tang, X., and Zhou, B.  Interpreting the latent space of GANs for semantic face editing.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Singh et al. (2019) Singh, K. K., Ojha, U., and Lee, Y. J.  FineGAN: Unsupervised hierarchical disentanglement for fine-grained object generation and discovery.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Skerry-Ryan et al. (2018) Skerry-Ryan, R., Battenberg, E., Xiao, Y., Wang, Y., Stanton, D., Shor, J., Weiss, R., Clark, R., and Saurous, R. A.  Towards end-to-end prosody transfer for expressive speech synthesis with tacotron.  In _International Conference on Machine Learning (ICML)_ , pp. 4693–4702, 2018.
  * Snyder et al. (2018) Snyder, D., Garcia-Romero, D., Sell, G., Povey, D., and Khudanpur, S.  X-vectors: Robust dnn embeddings for speaker recognition.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 5329–5333. IEEE, 2018.
  * Sun et al. (2020a) Sun, G., Zhang, Y., Weiss, R. J., Cao, Y., Zen, H., Rosenberg, A., Ramabhadran, B., and Wu, Y.  Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and autoregressive prosody prior.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 6699–6703, 2020a.
  * Sun et al. (2020b) Sun, G., Zhang, Y., Weiss, R. J., Cao, Y., Zen, H., and Wu, Y.  Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 6264–6268. IEEE, 2020b.
  * Valle et al. (2020) Valle, R., Li, J., Prenger, R., and Catanzaro, B.  Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens.  In _IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP)_ , pp. 6189–6193. IEEE, 2020.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I.  Attention is all you need.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), _Advances in Neural Information Processing Systems (NeurIPS)_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Wang et al. (2018) Wang, Y., Stanton, D., Zhang, Y., Ryan, R.-S., Battenberg, E., Shor, J., Xiao, Y., Jia, Y., Ren, F., and Saurous, R. A.  Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis.  In _International Conference on Machine Learning (ICML)_ , pp. 5180–5189. PMLR, 2018.
  * Wu et al. (2017) Wu, Z., Yamagishi, J., Kinnunen, T., Hanilcci, C., Sahidullah, M., Sizov, A., Evans, N., Todisco, M., and Delgado, H.  Asvspoof: the automatic speaker verification spoofing and countermeasures challenge.  _IEEE Journal of Selected Topics in Signal Processing_ , 11(4):588–604, 2017.
  * Yamagishi et al. (2019) Yamagishi, J., Veaux, C., and MacDonald, K.  CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92), [sound].  _University of Edinburgh. The Centre for Speech Technology Research (CSTR). https://doi.org/10.7488/ds/2645._ , 2019.
  * Zen et al. (2019) Zen, H., Dang, V., Clark, R., Zhang, Y., Weiss, R. J., Jia, Y., Chen, Z., and Wu, Y.  LibriTTS: A corpus derived from librispeech for text-to-speech.  _arXiv preprint arXiv:1904.02882_ , 2019.
  * Zhang (2019) Zhang, R.  Making convolutional networks shift-invariant again.  In _International Conference on Machine Learning (ICML)_ , pp. 7324–7334. PMLR, 2019.
  * Zhang et al. (2018) Zhang, Y., Zhang, Y., and Cai, W.  Separating style and content for generalized style transfer.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 8447–8455, 2018.
  * Zhao et al. (2020) Zhao, Y., Huang, W.-C., Tian, X., Yamagishi, J., Das, R. K., Kinnunen, T., Ling, Z., and Toda, T.  Voice conversion challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion.  _arXiv preprint arXiv:2008.12527_ , 2020.
  * Zhu et al. (2017) Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _IEEE International Conference on Computer Vision (ICCV)_ , pp. 2223–2232, 2017.

##  Appendix A Broader Impact

The technology we develop in the paper, like other artificial intelligence
technologies, potentially has both positive and negative impacts to our
society (Brundage et al., 2018). One potential risk associated with all
generative models is creating fake digital content. If deployed irresponsibly,
speech and handwriting synthesis could facilitate deceptive interactions,
including mimicking a person’s voice or handwriting in automatic speared
phishing attacks, gaining security access, or directing public opinions.
Examples of responsible deployments of the technology include (but not limited
to):

  * •

A system-level authentication every time the technology is used to register
style or generate output.

  * •

An encryption system to protect registered style.

  * •

A watermarking system (for both speech and handwriting) such that the
generations can be easily identified by human or detection systems (Hua et
al., 2016).

Beyond system-level security measures, technologies to identify fake digital
content have also been rapidly developed (Chen et al., 2020; Lyu, 2020; Güera
& Delp, 2018; Dolhansky et al., 2020; Alegre et al., 2013; Wu et al., 2017;
Evans et al., 2013; Kinnunen et al., 2020). Despite the potential negative
societal impacts, we believe that the technology will have a larger positive
impact, such as enabling new accessibility capabilities (_e.g_., helping mute
people speak in their voices and paralyzed people write in their handwriting)
and better human-computer interaction (_e.g_., by improving downstream speech
and handwriting recognizers).

##  Appendix B Details of the Training Procedure

Training with style equalization is straightforward — we jointly optimize all
the model parameters by maximizing the log-likelihood lower bound as described
in eq. (1) of the main paper. We found the following optional steps to be
useful to improve training and the quality of the generation.

  * •

During training, we found it helpful to add a small amount of Gaussian noise
to the ground-truth $x_{t{-}1}$ that is fed back to the bottom LSTM.
Intuitively, we are simulating the noise caused by sampling the output
distribution during inference at training time. We found that it makes the
inference more stable. A similar method is proposed by Meng et al. (2021) to
learn auto-regressive models.

  * •

To encourage the basis $A$ that is used by $\phi$ in eq. (2) to capture a wide
variety of styles, we maintain $A$ as an orthonormal basis. We normalize each
column $a_{i}$ in $A$ to be unit norm in the architecture and minimize
$|a_{i}^{\top}a_{j}|^{2}$ for all $i$ and $j\neq i$. The minimization is
conducted by minimizing the trace of $(A^{\top}A)^{2}$, which is estimated
efficiently using Hutchinson’s trace estimator (Hutchinson, 1989) with 100
random samples from a standard Normal distribution. The estimated value is
used as a regularization with loss weight equal to 1.

Overall, we optimize the following objective function

$\max_{\theta,A}\operatorname*{\mathbb{E}}_{\begin{subarray}{c}({\bf x},{\bf
c})\sim\mathcal{X}\\\ ({\bf x}^{\prime},{\bf c}^{\prime})\sim\mathcal{X}\\\
{\bf n}\sim\mathcal{N}(0,\sigma^{2})\end{subarray}}\
\sum_{t=1}^{M}\operatorname*{\mathbb{E}}_{z_{t}\sim
q_{\theta}(z_{t}|\mathcal{M}_{\theta}({\bf x}^{\prime},\phi_{\theta}({\bf
x}^{\prime},{\bf x}),{\bf x}_{1..{t{-}1}},{\bf c})}\log
p_{\theta}(x_{t}|z_{t},{\bf x}_{1..{t{-}1}}+{\bf n}_{1..{t{-}1}},{\bf c})\\\
-\mathcal{D}_{KL}\left(q_{\theta}(z_{t}|\mathcal{M}_{\theta}({\bf
x}^{\prime},\phi_{\theta}({\bf x}^{\prime},{\bf x}),{\bf x}_{1..{t{-}1}},{\bf
c})\,||\,p_{\theta}(z_{t}|{\bf x}_{1..{t{-}1}},{\bf
c})\right)-\tr((A^{\top}A)^{2}),$ (3)

where $\theta$ represents all network parameters (except $A$). We use the
reparameterization trick that is commonly used in variational autoencoders
(Kingma & Welling, 2014) with one sample to estimate the inner expectation in
eq. (3), and we use ADAM (Kingma & Ba, 2015) with $\beta_{1}=0.9$,
$\beta_{2}=0.98$, and a learning rate schedule used by Vaswani et al. (2017)
with a warm-up period of $4,000$ iterations to optimize the objective
function. The learning rate increases rapidly within the warm-up period to
$10^{-4}$ and decreases slowly after then.

Figure 5: Overview of the model. (a) shows an overview of the entire model
without style equalization (used during inference since $\delta=\phi({\bf
x}^{r},{\bf x}^{r})=0$). It includes a style encoder (in green), a content
attention, a decoder (in gray), and a LSTM at the bottom. Note that the input
content ${\bf c}$ can be the output of a content-embedding network (used in
speech synthesis) or one-hot encoding of characters (used in handwriting
synthesis). $a_{t}$ is the output of the content attention at time $t$, which
is a linear combination of the elements in ${\bf c}$. (b) shows an overview of
the entire model with style equalization (used during training or during
interpolation). $\phi$ computes the vector $\delta$ that encodes the amount of
style transformation between ${\bf x}^{\prime}$ and ${\bf x}$. $\mathcal{M}$
applies this transformation to ${\bf x}^{\prime}$ to match the style of ${\bf
x}$. Please see Table 4 for details about individual blocks.  Table 4: Block
formulation used in Fig. 5

Block name Architecture conv blocks blur $\rightarrow$ conv(3, $f_{1}$, 2, 0)
$\rightarrow$ Swish $\rightarrow$ dropout($0.1$) $\rightarrow$ blur
$\rightarrow$ conv(3, $f_{2}$, 2, 0) $\rightarrow$ Swish $\rightarrow$
dropout($0.1$) $\rightarrow$ blur $\rightarrow$ conv(3, $f_{3}$, 2, 0)
$\rightarrow$ Swish $\rightarrow$ dropout($0.1$) $\rightarrow$ blur
$\rightarrow$ conv(3, $f_{4}$, 2, 0) $\rightarrow$ Swish $\rightarrow$
dropout($0.1$)
$(f_{1},f_{2},f_{3},f_{4})=\begin{cases}(32,64,128,256),&\text{for
handwriting}\\\ (256,384,512,512),&\text{for speech}\end{cases}$ multihead
attention number of heads = 4 $\text{dimension of query, key,
value}=\begin{cases}128,&\text{for handwriting}\\\ 64,&\text{for VCTK}\\\
192,&\text{for LibriTTS}\end{cases}$ bottom LSTM number of layers = 1
$\text{dimension}=\begin{cases}512,&\text{for handwriting}\\\ 2048,&\text{for
speech}\end{cases}$ top LSTM number of layers = 2
$\text{dimension}=\begin{cases}512,&\text{for handwriting}\\\ 2048,&\text{for
speech}\end{cases}$ content attention number of Gaussian windows = 10 See
(Graves, 2013) for the exact formulation. content encoder (speech-only)
conv(5, 256, 1, 2) $\rightarrow$ Swish $\rightarrow$ conv(5, 256, 1, 2)
$\rightarrow$ Swish $\rightarrow$ conv(5, 256, 1, 2) $\rightarrow$ Swish
$\rightarrow$ bidirectional LSTM (dim=256) blur: 1D low-pass filtering with
kernel $[1\ 3\ 3\ 1]$ conv($k,f,s,p$): 1D convolution with kernel size $k$,
feature dimension $f$, stride $s$, and padding $p$ Swish: Swish nonlinearity
(Ramachandran et al., 2018; Hendrycks & Gimpel, 2016) dropout($p$): dropout
with probability $p$

##  Appendix C Model Architecture Details

In this section, we provide more details about the model architecture used in
the paper. Fig. 5 shows an overview of the model, and Table 4 lists the
individual block formulations used in Fig. 5. We use the same architecture for
both handwriting and speech synthesis, except for the hyper-parameters, which
we list at the end of the section.

The model can be separated into a backbone and a style encoder. The backbone
is composed of a decoder, content attention, and an LSTM at the bottom. It is
a standard model architecture and has been used and extended in many works of
handwriting and speech synthesis (Graves, 2013; Shen et al., 2018; Wang et
al., 2018; Hsu et al., 2018).

The bottom LSTM is one-layer, and it accumulates information from the past
outputs ${\bf x}_{1..{t{-}1}}$ and the previously attended content
$a_{1},\dots,a_{t{-}1}$ into its hidden state $h_{t}$. The content attention,
which is proposed by Graves (2013), utilizes moving Gaussian windows to
calculate attention weights for the content. The focused content $a_{t}$ at
time $t$ is a linear combination of the elements in ${\bf c}$ based on the
attention weights. The decoder at the top is a two-layer LSTM that takes all
available information, including $h_{t}$, $z_{t}$, and $a_{t}$, and outputs
the parameters of $p(x_{t}|z_{t},{\bf x}_{1..{t{-}1}},{\bf c})$. As mentioned
in Sec. B, we add a small Gaussian noise to $x_{t{-}1}$ that is passed to the
bottom LSTM.

Our style encoder is composed of a 4-layer convolutional network and a multi-
head attention layer. Given a style reference input ${\bf x}^{r}$, the
convolutional network extracts the feature sequence ${\bf f}^{r}$. We apply
low-pass filtering before every sub-sampling (Zhang, 2019) to avoid aliasing
caused by sub-sampling in the convolutional network. To maintain time-
invariant of the convolutional network, we do not add any padding (Islam et
al., 2019). Given the past outputs $h_{t}$ and the currently focused content
$c_{t}$, we use multi-head attention to extract relevant information from
${\bf f}^{r}$. The query vector is computed from $h_{t}$ and $c_{t}$ using a
linear layer, and the key and the value vectors are the individual feature
vectors contained in the sequence ${\bf f}^{r}$ without positional encoding.
The intuition is that if the model plans to write a specific character or
utter a specific word, it should find the information in the style reference
and mimic it. The variational approximation $q(z_{t}|\cdot)$ is a multivariate
Gaussian distribution with a diagonal covariance matrix. The prior
distribution $p(z_{t}|{\bf x}_{1..{t{-}1}},{\bf c})$ is also modeled as a
multivariate Gaussian distribution with a diagonal covariance matrix, and we a
two-layer feed-forward network to compute its means and standard deviations
from $h_{t}$ and $c_{t}$. When the style equalization is used, $\mathcal{M}$
and $\phi$ are inserted into the style encoder, as shown in Fig. 5(b).

Now we summarize the hyper-parameters used for handwriting and speech. Please
also see Table 4.

  * •

For handwriting, the dimension of all LSTMs are $512$. The final linear layer
outputs a 122-dimensional vector, which is used to parameterize the output
distribution. The output distribution includes a mixture of $20$ bivariate
Gaussian distributions that model the pen movement, a Bernoulli distribution
for pen lifting, and a Bernoulli distribution for sequence stops. The
posterior and the prior Gaussian distributions are $256$-dimensional. The
convolutional network in the style encoder has four layers; all of them use
kernel size 3, stride 2, and no padding. Their feature dimensions are
$3\rightarrow 32\rightarrow 64\rightarrow 128\rightarrow 256$. We use dropout
with a dropping rate equal to 0.1 after each nonlinearity in the convolutional
network. The multihead attention has 4 heads, the dimension of the query, key,
and value vectors are all 256. The dimension of $\delta$ (_i.e_., $k$) is
$128$. The input content ${\bf c}$ is represented as 200-dimensional one-hot
vectors of the input characters. The standard deviation of the Gaussian noise
is $0.1$, and during inference, we reduce the standard deviation of the output
distribution to $0.9$ of the original one. The model is trained for 100 epochs
on a machine with 8 A100 GPUs, and the training took 36 hours.

  * •

For speech, all LSTMs have the same $2048$ dimension. The final linear layer
outputs a 484-dimensional vector, which is used to parameterize the output
distribution. The output distribution includes a mixture of three
80-dimensional Gaussian distributions with diagonal covariance that models the
mel-spectrogram and a Bernoulli distribution for sequence stops. The
convolutional network in the style encoder has four layers; all of them use
kernel size 3, stride 2, no padding. Their feature dimensions are
$80\rightarrow 256\rightarrow 384\rightarrow 512\rightarrow 512$. We use
dropout with a dropping rate equal to 0.1 after each nonlinearity in the
convolutional network. The multihead attention has 4 heads, the dimension of
the query, key, and value vectors are all 256.

The input sentence is represented as phonemes, which contain 148 symbols. We
follow the pre-processing used by Shen et al. (2018) for phoneme and mel-
spectrogram extraction. We also follow Shen et al. (2018) and use a
bidirectional LSTM and a convolutional network to encode the dependencies
between phonemes. The architecture is the same as that used by Shen et al.
(2018). The posterior and prior Gaussian distributions are $512$-dimensional,
and the dimension of $\delta$ (_i.e_., $k$) is $64$ for VCTK dataset and $192$
for LibriTTS dataset. The standard deviation of the added noise is $0.2$, and
during inference, we reduce the standard deviation of the output distribution
to $0.74$ of the original one. The VCTK model is trained for 70 epochs on a
machine with 8 A100 GPUs, and the training took 12 hours; the LibriTTS model
is trained for 25 epochs on a machine with 8 A100 GPUs, and the training took
3 days. During inference, the LibriTTS model generates $\sim$380 mel-
spectrogram frames (4.4 seconds) per second on a single A100 GPU.

##  Appendix D Style Classifier Network

As we mentioned in Sec. 5.2, we train a speaker classifier using the objective
function proposed by Deng et al. (2019). Using the features extracted by the
speaker classifier, we measure the style similarity between two waveforms
using the cosine similarity between the features. The style classifier
comprises a convolutional network, an LSTM, and a linear layer that transforms
the last hidden state into the feature that we use to compute the cosine
similarity. The input to the convolutional network is the 80-dimensional
phoneme, which is extracted using the same procedure as the one used by Shen
et al. (2018). The convolutional network has four layers; all of them use
kernel size 3, stride 2, valid padding, and swish non-linearity (Ramachandran
et al., 2018). Their feature dimensions are $80\rightarrow 256\rightarrow
384\rightarrow 512\rightarrow 512$. We use dropout with a dropping rate equal
to 0.1 after each non-linearity in the convolutional network. The LSTM has one
layer, and its dimension is 512. We split the training set of LibriTTS-all-960
into the training, validation, and test sets by a ratio of 85%, 7.5%, 7.5%,
respectively. We use the same learning rate schedule and optimizer mentioned
above to train the classifier. The classifier achieves 96.5% validation
accuracy.

##  Appendix E Overview of Related Works

We provide a high-level summary of various controllable sequence generative
models in Table 5. In the table, we compare the methods on their needs of :
(1) user identification or pretrained embedding, (2) phoneme or character
segmentation, (3) content recognizer or pretrained encoder, (4) their training
loss and procedure, and (5) the applications shown in the papers. As can be
seen and to the best of our knowledge, while there exist many controllable
sequence generative models, our proposed method is the first that does not
require user ID, segmentation, pretrained recognizer and proven to be
applicable on both speech and handwriting domains.

Table 5: Overview of controllable sequence models. The table provides a high-
level overview of various controllable sequence models. For details, please
see individual references.

Method No user ID or embedding needed No segmentation needed No recognizer
needed Training method Domain (Kim et al., 2020) ❍ ⚫ ⚫ log-likelihood speech
(Chen et al., 2021) ❍ ❍ ⚫ log-likelihood speech (Donahue et al., 2020) ❍ ⚫ ⚫
adversarial speech (Donahue et al., 2020) ❍ ⚫ ⚫ adversarial speech (Kameoka et
al., 2020) ❍ ⚫ ⚫ log-likelihood speech (Gibiansky et al., 2017) ❍ ◗
(pretrained) ⚫ log-likelihood speech (Jia et al., 2018) ❍ ⚫ ⚫ log-likelihood
speech (Skerry-Ryan et al., 2018) ❍ ⚫ ⚫ log-likelihood speech (Sun et al.,
2020a) ❍ ⚫ ⚫ log-likelihood speech (Lee et al., 2021) ❍ ⚫ ⚫ adversarial speech
(Kaneko et al., 2019b) ❍ ⚫ ⚫ adversarial speech (Kaneko et al., 2019a) ❍
(group data) ⚫ ⚫ adversarial speech (Kaneko & Kameoka, 2018) ❍ (group data) ⚫
⚫ adversarial speech (Kameoka et al., 2018) ❍ ⚫ ⚫ adversarial speech (Davis et
al., 2020) ⚫ ◗ (pretrained) ❍ adversarial + log-likelihood handwriting (Kang
et al., 2020) ❍ ⚫ ❍ adversarial + log-likelihood handwriting (Bhunia et al.,
2021) ❍ ⚫ ❍ adversarial + log-likelihood handwriting (Kotani et al., 2020) ❍ ◗
(pretrained) ⚫ log-likelihood handwriting (Hsu et al., 2018) ◗ (not on
LibriTTS) ⚫ ⚫ log-likelihood speech (Hu et al., 2020) ⚫ ⚫ ◗ (pretrained
content encoder) adversarial + log-likelihood speech (Aksan et al., 2018) ⚫ ❍
❍ log-likelihood handwriting (Akuzawa et al., 2018) ⚫ ⚫ ⚫ log-likelihood + KL-
annealing speech (Henter et al., 2018) ⚫ ❍ ⚫ log-likelihood speech (Sun et
al., 2020b) ❍ ❍ ⚫ log-likelihood speech (Ma et al., 2018) ⚫ ⚫ ⚫ adversarial
speech (Graves, 2013) ⚫ ⚫ ⚫ log-likelihood handwriting GST (Wang et al., 2018)
⚫ ⚫ ⚫ log-likelihood speech Proposed style equalization ⚫ ⚫ ⚫ log-likelihood
speech, handwriting

##  Appendix F Additional Ablation Study: Does ${\bf x}^{\prime}$ Need to be a
Valid Sample?

During training, the proposed style equalization randomly selects a sample
from the training dataset as ${\bf x}^{\prime}$, which is unrelated to the
ground-truth ${\bf x}$. One interesting question can be raised naturally:
“Since ${\bf x}^{\prime}$ is unrelated to ${\bf x}$, do we really need it to
be a valid sample?”. Theoretically, since our design of $\mathcal{M}$ and
$\phi$ in eq. (2) prevents content leakage and transfers time-independent
ground-truth style information from ${\bf x}$ through ${\bf z}$, as long as
${\bf x}^{\prime}$ does not contain the content information about ${\bf x}$,
the learned model should be able to control style and content separately
during inference. To verify the hypothesis, we conduct the following ablation
studies:

  1. 1.

${\bf x}^{\prime}$ is a fixed vector: We initialize ${\bf x}^{\prime}$ as a
random vector but fixed it during training and inference.

  2. 2.

${\bf x}^{\prime}$ is a random noise: We randomly sample ${\bf x}^{\prime}$
from the standard Gaussian distribution.

As can be seen from Table 6, both methods are able to produce high-quality
models that can control content and style (low WER and sRank and high cos-
sim). They also achieve higher cos-sim than GST-nS, which utilizes additional
speaker information. Nevertheless, the proposed model (_i.e_., using real
samples as ${\bf x}^{\prime}$) is able to utilize both time-independent and
time-dependent style information of ${\bf x}$ (see discussion in Sec. 5.4),
and thus, our model still outperforms the two models.

The following properties also make the proposed method more favorable than the
other two cases:

  * •

During inference, the two models still need to run $\mathcal{M}$ and $\phi$.
In comparison, our usage of ${\bf x}’$ allows us to remove $\mathcal{M}$ and
$\phi$ when mimicking a reference example.

  * •

Interpolation between two reference examples becomes non-trivial. By design,
using a valid sample as ${\bf x}’$ enables the model to learn to transform the
style from one real sample to another. In contrast, the other two models only
learn to transfer style to a random noise (or a fixed vector).

Table 6: Ablation study results on the role of ${\bf x}^{\prime}$. All models
are trained on LibriTTS-all-960 dataset.

Method Seen speakers, parallel text Seen speakers, nonparallel text WER (%)
cos-sim $\uparrow$ sRank $\downarrow$ WER (%) cos-sim $\uparrow$ sRank
$\downarrow$ ${\bf x}^{\prime}$ is a fixed vector $8.0\pm 0.2$ $0.69\pm 0.22$
$14\pm 78$ $7.1\pm 0.1$ $0.70\pm 0.20$ $12\pm 95$ ${\bf x}^{\prime}$ is random
noise $8.4\pm 0.0$ $0.72\pm 0.17$ $4.7\pm 32$ $\mathbf{6.7\pm 0.2}$ $0.72\pm
0.16$ $5.1\pm 56$ Proposed (${\bf x}^{\prime}$ is a real sample)
$\mathbf{6.2\pm 0.5}$ $0.82\pm 0.14$ $\mathbf{1.7\pm 4.1}$ $9.4\pm 0.3$
$0.78\pm 0.14$ $\mathbf{1.8\pm 6.0}$ Method Unseen speakers, parallel text
Unseen speakers, nonparallel text WER (%) cos-sim $\uparrow$ sRank
$\downarrow$ WER (%) cos-sim $\uparrow$ sRank $\downarrow$ ${\bf x}^{\prime}$
is a fixed vector $9.2\pm 0.3$ $0.53\pm 0.18$ $22\pm 97$ $\mathbf{6.2\pm 0.1}$
$0.53\pm 0.18$ $23\pm 103$ ${\bf x}^{\prime}$ is random noise $8.9\pm 0.0$
$0.49\pm 0.16$ $14\pm 55$ $6.3\pm 0.2$ $0.48\pm 0.16$ $13\pm 56$ Proposed
(${\bf x}^{\prime}$ is a real sample) $\mathbf{6.8\pm 0.1}$ $\mathbf{0.63\pm
0.15}$ $\mathbf{9.0\pm 63}$ $7.6\pm 0.9$ $0.57\pm 0.15$ $\mathbf{7.4\pm 43}$

##  Appendix G Utilizing Time-dependent Style Information

In the section, we try to answer the question: “does our model utilize time-
dependent style information?” When a model fully utilizes time-dependent style
information, the generated output should be a reconstruction of the reference
example in the parallel setting. On the other hand, if a model utilizes only
time-independent style information, its outputs in the parallel setting can
still be different from the reference example, _i.e_., the same sentence
spoken by the same user can be very different in different contexts. The
difference in style replication quality provides the foundation to the
following evaluation.

We first evaluate the capability of our model to utilize time-dependent style
information by comparing the results between parallel (where all the time-
dependent style information is available) and non-parallel (where little time-
dependent and mostly time-independent style is available) settings. As can be
seen in Table 1, Table 2 and Fig. 3d, our results (WER, CER, and cos-sim) in
the parallel setting are better than those in the non-parallel setting. The
results show that the model is capable of utilizing the time-dependent
information in the reference example.

Next, we verify this capability by visualizing the style attention weights in
the three scenarios in Fig. 4b and c. In the parallel setting (Fig. 4b), the
model strongly attends to the correct time instance, whereas in the non-
parallel setting (Fig. 4c), the model attends to localized pieces in the
reference example. Additionally, we invite the readers to qualitatively
evaluate the capability via the handwriting/speech generation results provided
in the supplemental website. As can be seen, the reproduction is
“qualitatively closer“ in the parallel setting and in the nonparallel setting
when a character appears in the style reference within a similar context,
_e.g_., in Fig. 6b, the character ‘A’ and ‘a’ in the first non-parallel text
example and ‘p’ in the second example. To make the observation, in Fig. 6b, we
provide a nonparallel example that has overlapping content as the reference
style. As can be seen, the generated result is qualitatively similar to the
reference example at the overlapped regions even when they are separated by
unrelated text.

The ablation study in Table 6 compares our model (which is trained to utilize
time-dependent style information via applying style equalization to 50% of the
batches) with two models that always transform style inputs (and thus can only
use time-independent style information). While all models in the comparison
use the same style encoder, our model, with its capability to utilize time-
dependent style information, achieves the highest cosine similarities among
all three models.

Finally, we compare the same model trained with our method (which applies
style equalization to 50% of the batches) and entirely without style
equalization in Fig. 3d. While both models can utilize time-dependent
information, the model trained without style equalization suffers from content
leakage (as shown by the high CER in the nonparallel text setting). In
comparison, the proposed model is less effected by the settings.

In summary, the proposed method (utilizing the style attention module and
applying style equalization on half of the batches) enables the model to
utilize time-dependent style information while avoiding catastrophic content
leakage.

Figure 6: Parallel and nonparallel text generation results. In the figure, the
reference examples are shown in black, and the generated results are shown in
blue.

##  Appendix H Limitations

Similar to most unsupervised methods, there is no guarantee that the proposed
style equalization can entirely prevent content leakage, _i.e_. no ground-
truth content information contained in the transformed output. The proposed
method rely on the time-invariant bottleneck to prevent the ground-truth
content information leaking through the transformed style. However, our
convolutional network, which is used down-sample the input style examples, has
a receptive field equal to 76 mel-spectrogram frames (about 1 second) /
handwriting strokes (3 characters) in our models. Within the receptive field,
the time-dependent information is retained. In other words, within the
receptive field of the convolutional network, our assumption that we only
transfer time-invariant information is violated. While we utilize low-pass
filtering before any down-sampling and avoid padding as suggested by Zhang
(2019), we do not deal with the aliasing due to the nonlinearity functions (as
in (Karras et al., 2021)). Thereby, our style transformation is not entirely
time-invariant and thus the transformed output can contain time-dependent
information exploitable by the model. When the model overfits the training
data, our style transform can still leak content information that can
eventually be utilized by the decoder. We observe that this phenomenon happens
after the model overfits the training set, and by examining the validation
loss and the KL-divergence between the prior and the posterior distributions,
we can avoid it in practice. As shown by our ablation studies comparing
with/without style equalization (Fig. 3a for handwriting and the supplemental
webpage for speech), the proposed method effectively reduces content leakage.

##  Appendix I More Speech and Handwriting Generation Results

On our website (https://apple.github.io/ml-style-equalization), we show an
extensive list of speech and handwriting samples generated by the proposed and
the baseline methods. Note that it may take a while for the speech results to
load, and if the audio players do not contain the play button, please increase
the size of the browser window. To remove the effect of the vocoder when
comparing synthesized speech samples with real speech samples, all real speech
samples (including those in the style opinion score evaluations) are converted
to mel-spectrogram and reconstructed back to waveform using the same vocoder
that is used by the generative models, _i.e_., waveglow (Prenger et al.,
2019).

For speech synthesis, the webpage contains

  * •

a video showcasing the generation of speech with various styles and content

  * •

nonparallel-text generation with seen speakers from LibriTTS-all-960

  * •

nonparallel-text generation with unseen speakers from LibriTTS-all-960

  * •

an ablation study that compares training with and without style equalization

  * •

interpolation between two unseen style reference speech

  * •

generated speech with random styles sampled from the learned prior

  * •

nonparallel-text generation with seen speakers from VCTK

  * •

parallel-text generation with seen speakers from VCTK

For handwriting synthesis, the webpage contains

  * •

a video showcasing the online generation of handwriting with various styles
and content

  * •

nonparallel-text generation with unseen style

  * •

parallel-text generation with unseen style

  * •

generated handwriting with random styles sampled from the learned prior

  * •

interpolation between two unseen style reference handwriting