# Recent Advancements in Self-Supervised Paradigms for Visual Feature
Representation

Mrinal Anand
Department of Computer Science
Indian Institute of Technology, Gandhinagar
<EMAIL_ADDRESS>
Aditya Garg
Department of Computer Science
Indian Institute of Technology, Gandhinagar
<EMAIL_ADDRESS>

###### Abstract

We witnessed a massive growth in the supervised learning paradigm in the past
decade. Supervised learning requires a large amount of labeled data to reach
state-of-the-art performance. However, labeling the samples requires a lot of
human annotation. To avoid the cost of labeling data, self-supervised methods
were proposed to make use of largely available unlabeled data. This study
conducts a comprehensive and insightful survey and analysis of recent
developments in self-supervised paradigm for feature representation. In this
paper, we investigate the factors affecting the usefulness of self-supervision
under different settings. We present some of the key insights concerning two
different approaches in self-supervision, generative and contrastive methods.
We also investigate the limitations of supervised adversarial training and how
self-supervision can help overcome those limitations. We then move on to
discuss the limitations and challenges in effectively using self-supervision
for visual tasks. Finally, we highlight some open problems and point out
future research directions.

##  1 Introduction

Supervised learning is a popular choice to solve many machine learning
problems. A large impact has been observed in the field of computer vision
under the supervised setting, i.e. learning from labeled data. Deep Neural
Networks have shown remarkable results when used on problems such as image
classification [26, 47], object detection [4], image generation [12] and
semantic segmentation [10, 29]. Under supervised settings, a model is trained
on the human-annotated labels. Supervised learning encompasses models where
for every corresponding input, we are given a target label, and the model is
then trained to learn that label.

These remarkable performances of supervised learning algorithms result from
the large number of human annotations needed to label the sophisticated
datasets. However, collecting and manual labeling of large datasets are
expensive and time-consuming. ImageNet [32] has around 14 Million images, and
each image might fall in more than one out of 22000 classes. This example
showcases the expense incurred to collect large, curated datasets. This has
motivated the researchers to explore the domain of unsupervised learning and
to learn from the vast amount of unlabeled data available.

Unsupervised learning refers to those algorithms that use the input data to
infer patterns without any explicit supervision. Self-supervised learning is a
subset of unsupervised learning where labels are generated automatically. In
this work, we study the recent advancements in self-supervised learning that
have been proposed to learn robust features representation from the unlabelled
data. It is also seen as a way to use a large amount of untapped data
available on the internet. This approach has been around for some time, but
the recent advancements have shown remarkable results that are comparable or
better to those shown by supervised approaches.

The idea for self-supervision comes from contextual embeddings used in natural
language processing. The idea here is to map every word to a feature vector
such that a word can be predicted if the context(a few words before or after)
is given. In essence, context prediction is just a pretext for the network to
learn effective word embeddings. A similar method is used in a self-supervised
setting where tasks are performed under the pretext of learning
representations. Through this work, we aim to showcase a trend that has been
followed in the recent advancements in the field of self-supervised learning
and give insights that could be helpful for machine learning practitioners.

Outline of the paper: The entire paper is organized as follows. Section 2
narrates the different pretext tasks used in self-supervised learning
methodology. Section 3 describes the generative and contrastive learning
paradigms. Section 4 presents the important factors affecting the learned
representation through pre-training. Section 5 explores the adversarial
robustness in the context of self-supervision. Finally, Section 6 discusses
the limitations in the current methodology and present the future directions
for the self-supervised paradigm.

##  2 Pretext Tasks for Self-Supervision

A pretext task is a proxy task for the network to solve with the goal of
learning feature representations that can be easily transferable to specific
downstream tasks. It is ensured that pretext tasks are as hard as possible,
and also that original data is in no way destroyed. Downstream tasks are the
primary vision tasks for which the self-supervised model is being trained. The
performance of the model on these tasks determines the success of the model.
Downstream tasks can be anything ranging from object detection, object
classification, or image segmentation. Popular augmentations that are being
used extensively in self-supervised models are shown in Figure 1. Some of the
popular pretext tasks that are used in self-supervised learning:

  1. 1.

Exemplar CNN [19] \- N patches are sampled from every image from different
locations in the image and scaled randomly. Patches should be picked from
parts where the gradient is thicker as these parts usually carry the different
objects. Every patch is augmented in a variety of different ways, and the
pretext task would be to differentiate between patches of different images.
The feature representation learned performs well in object classification and
descriptor matching.

  2. 2.

Rotation [20] \- An image is rotated by a multiple of 90°. Then the model is
trained to classify the rotation into four classes, namely [0, 90, 180, 270].
This is also an effective way to force the model to learn high-level features.
They demonstrate that rotation transformation that was applied to an image
will require the model to understand the concept of the location of an object
in the image, its type, and its pose.

  3. 3.

Relative Patches [15] \- This task extracts multiple patches from an image and
asks for the relation between them. One patch is randomly sampled, and another
patch is sampled from the eight neighboring patches. Now the relative position
is asked between them. To prevent low level features from slipping, additional
noise is added. This task has been found to be useful for both object
detection and visual data mining. This has also been adapted into a Jigsaw
puzzle game [39] task for the model to learn. This jigsaw puzzle has been seen
to be useful for object detection and classification.

  4. 4.

Colorization [56] \- A model is trained to appropriately color a grayscale
image. It is surprisingly useful as a pre-training task for diverse downstream
tasks like object classification, detection, and segmentation. However,
evaluating the performance requires a manual human intervention which can be
very time-consuming.

Figure 1: Different augmentation techniques used as pretext tasks ( image
source [11] ).

##  3 Two Paradigms for Self-Supervision

In this section, we will discuss the two main approaches in statistical
modeling. The first one being generative modeling and other one is
discriminative approach. In the generative approach models the conditional
probability of the input given the target while the discriminative models the
conditional probability of the target given the input.

###  3.1 Generative Learning

Majority of the work in generative image learning involves training the model
to reconstruct the original data in great detail. Their goal this way is to
also learn semantically meaningful representation. These generative modeling
techniques are very powerful and have been found to represent the data very
well.

Autoregressive Models: An effective line of approach in generative learning is
to model it as a product of conditional distributions. Effectively, we then
have to predict the future instances given the past instances. Recent models
that have shown some levels of success are PixelCNN [51] and PixelRNN [40].
These models are known to be computationally expensive because they try to
model the extremely complex relationship between pixels.

Autoencoding Models: Traditional autoencoding models encode the input and then
decode it to give an output, which is as similar to the input as possible. The
model is forced to learn a function that encodes the input into a
representation, and then this representation is later decoded back to the
original image. The task here is to learn a function that effectively captures
the semantic features in the data. This function is then used in downstream
tasks. Autoencoders cannot handle complex images easily because it tries to
capture as much information as possible but not all the relevant information.
This causes the decoded image to be degraded. There also presents the risk of
overfitting when this kind of autoencoding model is used for feature
representation.

Denoising Autoencoder [53] was introduced to rectify the overfitting in
traditional autoencoding models and also to improve the robustness of the
model. It introduces some random noise in the original image and trains the
traditional autoencoder to reproduce the original image. An interesting fact
was noticed during experiments for this model. As the noise is increased, the
model tends to learn more about larger structures that span across multiple
input dimensions and thus capture more important features. This makes it a
good pre-training task. The problem with this kind of noise addition is that
it still is pretty low level and does not force the model to learn a thorough
semantic understanding of the image.

Context Encoder [42] was proposed to learn a more semantic meaningful
representation. Instead of adding a low-level noise, we remove a patch from an
image and force the model to fill the patch and output an image similar to the
original image. The paper provides interesting intuition regarding the use of
loss in their model. They demonstrate that simple L2 loss between the two
images results in the model learning to paint a blurry picture. They
hypothesize that this might be because it is ’safer’ for the model to predict
the mean of the distribution. They show that having both adversarial loss and
L2 loss makes the inpainting more realistic. Zhang et al. [56] demonstrated
that context encoder does not perform well on large-scale semantic
representation learning benchmarks. This might be because the loss function is
not able to capture inpainting quality properly, or it could be an easy task
that can be solved without much semantically meaningful information and just
by using the local information. Though, it has been seen that context, encoder
works nicely for the object detection task.

Split Brain Autoencoder [57] is another kind of autoencoder where the color
channels of an image are randomly divided into two mutually exclusive subsets,
and then the network is divided into two disjoint sub-networks where each
subnetwork is then trained to predict the other subset of data. It has been
seen to have an excellent performance for image classification.

Variational Autoencoders [30] has spawned many variations, and these
variations have been successful in modeling data. It works differently from
the other variants of autoencoder because it maps the data into a distribution
rather than a vector representation. So, instead of encoding the data into a
vector, we encode it into distribution and then use that distribution to
decode the original input data. Some popular variants are VQ-VAE [52] and VQ-
VAE2 [44]. VQ-VAE has shown that it can be used to model the long-term
dependencies, and thus many important features can also be captured.

Generative Adversarial Models: Popularly known as GANs [21] have come up as a
powerful tool to learn models that stem from very complex distribution. It
consists of two different networks, the generator, and the discriminator. The
generator synthesize image in order to fool the discriminator in believing
that the synthesized image is real. Radford et al. [43] has shown that GANs
are able to capture powerful semantic information, which can be very useful in
self-supervised pretraining. To use these features for representation
learning, Donahue et al. [17] came up with a novel Generative architecture
BiGAN. Along with the normal discriminator and generator, this model also
includes another network called the encoder. This encoder turns the real
images into latent representations. The task here is then similar to that of
an autoencoder where the generator and the encoder have to be inverse of each
other. BiGAN has been shown to be effective in learning visual
representations. This can be classified as a self-supervised method because
the discriminator learns to assign labels such as ’fake’ or ’real’ to images
and thus self-supervised learning is involved in all kinds of GANs.

BigBiGAN [18], a modification to BiGANs, replaces the BiGAN generator with
BigGAN [7]’s generator. BigGAN is a more modern GAN architecture that has
proved more capable of capturing useful semantic features.

###  3.2 Contrastive Learning

Contrastive learning has come up recently as an alternative to generative
modelling. The objective here is to have the model learn to compare between
similar and dissimilar examples. The objective function used here is the Noise
Contrastive Estimation (NCE) [24]. The NCE objective is formulated as follows:

$\mathop{\mathbb{E}[-\log\frac{f(x,x^{+})}{f(x,x^{+})+\sum_{k=1}^{K}f(x,x_{k}^{-})}]}$
(1)

Formulation of the NCE loss function (1). $x^{+}$ is the augmented version of
x, and $x_{k}^{-}$ are dissimilar to x. K is the number of negative samples we
want to compare our sample x to.

In the recent past, we have witnessed a large amount of models that uses this
contrastive loss in different settings. Models that are based on maximizing
mutual information have been around for a long time in unsupervised learning.
Mutual information (MI) [5, 3, 6] is a measure of the dependence between two
signals. [28] defines mutual information between two variables A and B as the
decrease in uncertainty of A if we have prior knowledge about B. The task is
modeled around learning a representation while also maximizing this MI between
both the inputs and representation. MI has always been difficult to compute,
but the recent breakthrough in techniques to efficiently approximate MI has
made these models possible.

Recent work like Contrastive Predictive Coding [41] have moved away with
maximizing MI [27] between the input and the learned representations but
instead have tried to maximize the lower bound on MI [5]. They theoretically
show how the lower bound on MI increases as the Contrastive Loss decreases. In
CPC for visual tasks, the role of the pretext task is to predict the
representations of a patch somewhere below another patch from other patches.
The task here is similar to the one performed in an autoregressive model.

Figure 2: Constrastive Instance Learning. X is an image from the dataset. T is
the set of image transformations. $\theta$ is the parameters for the encoder.
At the end we compare with the negative samples ( image source [9] ).

The early success of models that maximize the mutual information was
challenged by Tschannen et al. [50], where they argued with credible evidence
that maximizing mutual independence was not the main reason for the success of
the early models. They also empirically show that MI maximization and
representation quality are not co-related. They believe that the success lies
in the encoder architecture and negative sampling.

Instance level discrimination [54] is a pretext task which forms the basis of
the many successful recent models. Each image is encoded into a feature
representation, and then they are spread across the feature space so that
similar samples are not as far as dissimilar samples. The distance metric
would be the L2 distance between the representations. Based on CPC and
Instance Discrimination, another model called Contrastive MultiView Coding
(CMC) [48] was proposed. Instead of using spatial prediction as the pretext
task, this model learns to create representations by comparing against
multiple augmentations of an image. As the number of views increases, the
performance becomes better but at the cost of computational power. Similar to
CMC, Augmented Multiscale Deep InfoMax (AMDIM) [2] was proposed. The major
difference between AMDIM and CMC is that AMDIM maps two augmented versions of
the original image to as well as their intermediate representation to the
similar space while CMC works on matching image channels in a contrastive
fashion.

It has been observed that contrastive learning seems to benefit from a large
number of negative samples. This observation, combined with the instance level
discrimination, inspired MoCo or Momentum Contrast [25]. It has a queue that
contains the recently encoded batches, which are used as the negative samples.
The samples in the queue are known as keys, and the query is the sample that
is encoded and then compared to the keys. An important information to note
about MoCo is that it does not run backpropagation on all the samples in the
queue. Instead, it backpropagates on the query encoder, and a momentum update
is run on the parameters of the key encoder. It performs instance-level
discrimination but brings in more negative samples, and so is a self-
supervised learning technique.

Pretext Invariant Representation Learning or PIRL [36] tries to learn
representations that are invariant to the augmentations applied on them. It
transforms an image and then encourages the representations of the augmented
image and original image to be invariant. The pretext task here is to learn
that augmented representations of the same image are invariant. SimCLR [11]
was introduced later, which trained the model in an end to end manner. It is
similar in theory to PIRL, but it generalizes the approach and adds a non-
linear projection on top of the representation layer to boost the model’s
accuracy. Batches of larger size, which considerably affect the computation
cost, have to be used but are necessary for the model’s success. A generalized
version of these models is Figure 2.

The SimCLR shed light on the importance of the choice of data augmentation
operations in the success of a model. They highlight the effectiveness of the
composition of random cropping and random color distortion augmentations in
their model. They have performed further analysis to show that this non-linear
projection layer actually helps maintain important information that otherwise
would have been removed by the contrastive loss. These studies helped MoCo,
and they weeded out the deficiencies in their model. MoCo v2 [13] is the
enhanced version of MoCo and utilizes some of the techniques from SimCLR to
provide a model that is computationally less expensive and also gives better
performance than SimCLR.

Clustering-based methods such as DeepCluster [8] uses a simple method such as
k-means and an encoder to learn representations for the visual features.
DeepCluster uses k-means algorithm, where it first assigns every feature
vector to its nearest center. The parameters of the network are then optimized
on the basis of the assigned labels. These two steps are then performed
repeatedly.

Local Aggregation [58], an improvement over DeepCluster, scraps the formation
of hard clusters, and it forms soft clusters, which has a much lower
computational cost compared to DeepCluster. It optimizes a soft clustering
metric which is computationally more efficient than the cross-entropy function
that needs to be recalculated at every step. These changes have led to better
performances on different benchmarks and also better computational efficiency.

SwAV [9] is a recent model that proposes the use of clustering to learn
effective representations. The normal contrastive model uses pairwise
comparisons between data, but this model here discriminates between groups of
images with similar features. Unlike contrastive methods, this model works
well even with small batches. Thus it is computationally very inexpensive. The
paper also empirically demonstrates the potential of cluster-based approaches
for self-supervised learning. It modifies DeepCluster [8] to give results
superior to even SimClR [11]. Their own model SwAV gives results on par with
DeepCluster v2, but SwAV has its own advantages. Not only is it an online
model, but it also scales very nicely. It is very suitable for large datasets.

Table 1 compares the accuracy of some recent models on the ImageNet [32]
dataset. As shown in the table some of contrastive methods like SwAV and MoCo
v2 achieves near the supervised accuracy.

Method | Accuracy
---|---
Supervised |  76.5%
Local Aggregation | 58.8%
MoCo v2. | 71.1%
MoCo | 60.6%
PIRL | 63.6%
CPC v2. | 63.8%
SimCLR | 70.0%
BigBiGAN | 56.6%
SwAV | 75.3%
Table 1: Table to compare Top 1 accuracy on ImageNet [32] dataset between
popular self-supervised models. All these results are for models using
ResNet-50 [26] architecture with 24 Million Parameters. Note that we use
frozen weights from the models to measure performance.

##  4 Self-Supervised Pretraining

Pretraining has shown a significant performance boost on visual tasks [16].
Pretraining a model with large datasets like ImageNet [32] and MS COCO [35]
helps the model to converge faster. Moreover, it also helps in transferring
the learned representation across different visual tasks. The aim of self-
supervised pretraining is to learn a robust feature representation from
unlabelled data, such that fewer and fewer labeled samples are required to
train a model to its maximum capacity. Generally, deep models largely benefit
from pretraining when labeled data are scarce. However, practical usage of
self-supervised pretraining depends on several other aspects also, Newell et
al. [38] presents four different factors that affect the performance of
pretraining-

  * •

Data: The feature learned through self-supervision directly depends on the
complexity of the dataset. The complexity of the dataset is linked to the
diversity of the samples and the number of labeled samples to saturate the
performance on a task.

  * •

Model: The maximum accuracy achieved depends upon the backbone architecture of
the model. Model having large parameters generally have high expressivity
power. Hence it can better generalize on unseen examples.

  * •

Pretext Task and Proxy loss: The pre-text task has a strong influence on the
type features learned, and therefore the choice of pre-text task can affect
the performance of the downstream task. Usually, the pre-text task like
Rotation [20] and Jigsaw Puzzle [39] learns global representation, and task
like Colorization [56] learn more semantic/boundary level representation. The
choice between contrastive or pixel-wise loss also affects the type of
representation learned through self-supervision.

  * •

Downstream Task: The different downstream tasks require different kinds of
supervised signals and can largely benefit if a suitable pre-text task is used
for training. Broadly any downstream task can be either a semantic heavy or
geometric heavy task. Another significant distinction between these vision
tasks is the requirement of dense prediction along with the image space, i.e.,
per-pixel wise prediction or single prediction per sample. Different pre-text
tasks have a different affects on the performance of the model. However, self-
supervised learning cannot be used for all downstream tasks since it relies on
automatic label generation.

Now we highlight some of the major findings related to the choice of the
correct pre-text task for a given downstream task and other associated
insights -

Number of Labeled Samples: Self-supervised pretraining has a significant
impact on the performance when the number of labeled samples is small. The
gain in the performance gets negligible as the number of labeled samples
increases. This observation is unaffected by the choice of pretext task or the
choice of downstream task.

Affect on Downstream Task: It has been observed that CMC based pretraining
performs better than other pretext tasks like AMDIM, Rotation, and VAEs for
object detection and pose estimation. On the contrary, rotation and AMDIM
perform better on low-level tasks like segmentation and depth estimation [38].
One of the possible reason could be CMC applies a contrastive loss to global
features while AMDIM and Rotation motivates to match similar low-level
features to similar space.

Model Selection: Popular choices of model for pre-training are ResNet [26],
VGG [45] and AlexNet [33]. The choice of model largely depends on the type of
downstream task. For capturing more fine-grained information, VGG based model
having a small kernel size and small stride is preferred over AlexNet based
model, which have relatively large kernels and stride. Using large kernels
imposes too many parameters in the model to tune, and a large convolution
stride can ignore some fine-grained features in the lower layers.

Affect of freezing the weights: Freezing the weights restricts the overall
expressive power of the deep neural networks. It has been empirically studied
that performance suffers from freezing the layer weights [38], but on the
contrary, it is advisable to freeze the model layers if the number of labeled
samples is very low; otherwise, it can lead to overfitting.

##  5 Adversarial Robustness through Self-Supervision

Adversarial Attacks are the perturbed inputs especially designed to fool
neural network [22], and Adversarial Training has shown great results in
providing Adversarial robustness to a deep model. However, performing
supervised adversarial training in model space, i.e., optimizing the model
parameters, has several drawbacks on its own.

  * •

Accuracy Drop: Model training using adversarial loss is generally accompanied
by a significant drop in accuracy.

  * •

Label Leakage: Label Leakage happens when the performance of the model is
better on adversarial images rather than clean images[34]. This happens
because the true label information leaks on the adversarial examples.

  * •

Task Dependency: A model trained adversarially for object detection cannot
provide the same adversarial robustness to classification models.

A different line of research focuses on processing the input as an adversarial
defense [23, 55]. Guo et al. [23] studies bit-depth reduction, JPEG
compression, total variance minimization, and image quilting as the
perturbation in the input space. Athalye et al. [1] showed that these defense
suffers from obfuscated gradients and can easily be broken under white-box
settings.

Naseer et al. [37] proposed a Neural Representation Purifier that works on
input space. The proposed technique is formulated as a min-max game that
learns an optimal input processing function that enhances robustness in a
self-supervised setting. Figure 3 depicts the whole pipeline in which a
purifier network generates a clean image from a given perturbed image and a
self-supervised perturbation (SSP) network that generates the perturbation
using deep perceptual features of the input image. At the time of inference,
we simply add this adversarially trained purifier network before the target
model, as shown in Figure 3.

Figure 3: Neural Representation Purifier Network that generates noisy free
images from perturbed input images ( image source [37] ).

We want to highlight a few points for the readers that using adversarial
training in a self-supervised manner by default eliminates the problem of
label leakage as no true labels are involved while training. Moreover, working
in input space rather than model space removes the problem of accuracy drop
and task dependency as original model parameters remain unaffected.

##  6 Discussion and Future Directions

Through diverse experimentation, Kolesnikov et al. [31] showed that neither
the architectures are coherent with the pretext task nor the self-supervised
pretext task is consistent with the architectures in terms of performance. One
possible future direction is to develop such a pretext task that is
practically invariant to the downstream task, or in other words, it learns
domain invariant representation that can easily be transferred to different
vision tasks.

There is no concept of transfer across datasets in visual feature learning.
Models like BERT [14] have been trained on large text corpora, and these
models can be used for many other applications on different datasets after
fine-tuning and are shown to give excellent results.

As demonstrated by Song et al. [46], alternative measurements of information
have to be looked for as Mutual Information estimation has been found to be
challenging, and there is no satisfactory method that optimizes Mutual
information. Mutual Information Estimators that are being used have lots of
limitations and often lead to exponentially growing variance.

All recent state-of-the-art contrastive paradigms use a similar technique,
which is to push representations of different views of the same image closer
together and for different images as far as possible. The view mentioned here
is the same one mentioned for Contrastive Multiview Coding [48]. Tian et al.
[49] have presented a paper that shows how critical view selection is. They
discuss how important it is to select views that have a balanced invariance,
that captures just enough information to label the images. It is also
discussed how each downstream task needs a different set of optimal views.
They also present a model that outperforms many of the state of the art
models. This shows that there is still a distance to go in terms of selecting
optimal views for downstream tasks.

There is a lot of ground to cover in regards to the theoretical understanding
of the state of the art models. We still do not have a very clear reason for
the success of Mutual Information based models [50].

Self supervised learning algorithms have shown promising results in vision so
far and it could be the next revolution in the field of visual feature
learning.

##  7 Conclusion

We discussed compelling results that have been achieved by the recent models
in the self-supervised domain. These models have bridged the gap between
unsupervised and supervised learning considerably. In this work, we first
discussed the diverse pretext task used for self-supervision and investigated
how different pretext tasks correlate to different downstream tasks. We then
discussed some of the important works which follow the two different
methodologies in self-supervised learning, i.e., generative and contrastive
learning. The section also gives insights related to their methodologies. We
further present the factor involved in pretraining a model under self-
supervised settings and investigate some of the competing choices that machine
learning practitioner encounters while using self-supervision. We then briefly
touch upon the challenges in supervised adversarial learning and how self-
supervision helps tackle those problems. Finally, we end with a discussion on
limitations in current methodology and point out some future research
directions.

## References

  * [1] A. Athalye, Nicholas Carlini, and D. Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  ArXiv, abs/1802.00420, 2018.
  * [2] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views, 2019.
  * [3] Suzanna Becker.  Mutual information maximization: models of cortical self-organization.  Network: Computation in neural systems, 7(1):7–31, 1996.
  * [4] Sara Beery, Guanhang Wu, Vivek Rathod, Ronny Votel, and Jonathan Huang.  Context r-cnn: Long term temporal context for per-camera object detection, 2020.
  * [5] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm.  Mine: mutual information neural estimation.  arXiv preprint arXiv:1801.04062, 2018.
  * [6] Anthony J Bell and Terrence J Sejnowski.  An information-maximization approach to blind separation and blind deconvolution.  Neural computation, 7(6):1129–1159, 1995.
  * [7] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [8] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze.  Deep clustering for unsupervised learning of visual features.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149, 2018.
  * [9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [10] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation, 2018.
  * [11] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [12] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 2172–2180. Curran Associates, Inc., 2016.
  * [13] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [14] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [15] Carl Doersch, Abhinav Gupta, and Alexei A Efros.  Unsupervised visual representation learning by context prediction.  In Proceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015.
  * [16] Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell.  Decaf: A deep convolutional activation feature for generic visual recognition, 2013.
  * [17] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.  Adversarial feature learning.  arXiv preprint arXiv:1605.09782, 2016.
  * [18] Jeff Donahue and Karen Simonyan.  Large scale adversarial representation learning.  In Advances in Neural Information Processing Systems, pages 10542–10552, 2019.
  * [19] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox.  Discriminative unsupervised feature learning with exemplar convolutional neural networks.  IEEE transactions on pattern analysis and machine intelligence, 38(9):1734–1747, 2015.
  * [20] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  arXiv preprint arXiv:1803.07728, 2018.
  * [21] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [22] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  CoRR, abs/1412.6572, 2015.
  * [23] Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten.  Countering adversarial images using input transformations.  In International Conference on Learning Representations, 2018.
  * [24] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 297–304, 2010.
  * [25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * [27] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670, 2018.
  * [28] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization, 2019.
  * [29] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam.  Searching for mobilenetv3, 2019.
  * [30] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [31] Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer.  Revisiting self-supervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [32] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012.
  * [33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097–1105. Curran Associates, Inc., 2012.
  * [34] A. Kurakin, Ian J. Goodfellow, and S. Bengio.  Adversarial machine learning at scale.  ArXiv, abs/1611.01236, 2017.
  * [35] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft coco: Common objects in context, 2015.
  * [36] Ishan Misra and Laurens van der Maaten.  Self-supervised learning of pretext-invariant representations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6707–6717, 2020.
  * [37] M. Naseer, S. Khan, M. Hayat, F. S. Khan, and F. Porikli.  A self-supervised approach for adversarial robustness.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 259–268, 2020.
  * [38] A. Newell and J. Deng.  How useful is self-supervised pretraining for visual tasks?  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7343–7352, 2020.
  * [39] Mehdi Noroozi and Paolo Favaro.  Unsupervised learning of visual representations by solving jigsaw puzzles.  In European Conference on Computer Vision, pages 69–84. Springer, 2016.
  * [40] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks.  arXiv preprint arXiv:1601.06759, 2016.
  * [41] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [42] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544, 2016.
  * [43] Alec Radford, Luke Metz, and Soumith Chintala.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434, 2015.
  * [44] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  In Advances in Neural Information Processing Systems, pages 14866–14876, 2019.
  * [45] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [46] Jiaming Song and Stefano Ermon.  Understanding the limitations of variational mutual information estimators.  arXiv preprint arXiv:1910.06222, 2019.
  * [47] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alex Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning, 2016.
  * [48] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [49] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning.  arXiv preprint arXiv:2005.10243, 2020.
  * [50] Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic.  On mutual information maximization for representation learning.  arXiv preprint arXiv:1907.13625, 2019.
  * [51] Aaron Van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al.  Conditional image generation with pixelcnn decoders.  In Advances in neural information processing systems, pages 4790–4798, 2016.
  * [52] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  In Advances in Neural Information Processing Systems, pages 6306–6315, 2017.
  * [53] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In Proceedings of the 25th international conference on Machine learning, pages 1096–1103, 2008.
  * [54] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733–3742, 2018.
  * [55] Yi Zeng, Han Qiu, Gerard Memmi, and Meikang Qiu.  A data augmentation-based defense method against adversarial attacks in neural networks, 2020.
  * [56] Richard Zhang, Phillip Isola, and Alexei A Efros.  Colorful image colorization.  In European conference on computer vision, pages 649–666. Springer, 2016.
  * [57] Richard Zhang, Phillip Isola, and Alexei A Efros.  Split-brain autoencoders: Unsupervised learning by cross-channel prediction.  In CVPR, 2017.
  * [58] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins.  Local aggregation for unsupervised learning of visual embeddings.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6002–6012, 2019.


# Robust Disentangled Variational Speech Representation Learning for Zero-shot
Voice Conversion

###### Abstract

Traditional studies on voice conversion (VC) have made progress with parallel
training data and known speakers. Good voice conversion quality is obtained by
exploring better alignment modules or expressive mapping functions. In this
study, we investigate zero-shot VC from a novel perspective of self-supervised
disentangled speech representation learning. Specifically, we achieve the
disentanglement by balancing the information flow between global speaker
representation and time-varying content representation in a sequential
variational autoencoder (VAE). A zero-shot voice conversion is performed by
feeding an arbitrary speaker embedding and content embeddings to the VAE
decoder. Besides that, an on-the-fly data augmentation training strategy is
applied to make the learned representation noise invariant. On TIMIT and VCTK
datasets, we achieve state-of-the-art performance on both objective
evaluation, i.e., speaker verification (SV) on speaker embedding and content
embedding, and subjective evaluation, i.e., voice naturalness and similarity,
and remains to be robust even with noisy source/target utterances.

Index Terms—  Self-supervised Disentangled representation learning, zero-shot
style transfer, voice conversion, variational autoencoder

##  1 Introduction

Voice Conversion (VC) seeks to automatically convert the non-linguistic
information of a source speaker to a target speaker, while keeping the
linguistic content unchanged. The non-linguistic information may include
timbre (i.e., speaker identity), emotion, accent or rhythm, to name a few. Due
to its potential for applications in privacy protection, security and
entertainment industry etc. [1, 2, 3], VC has received long-term research
interest.

We can categorise current VC systems into two methodologies. The first one
employs a conversion model to map source acoustic features to target acoustic
features [1, 4, 5] . For the conventional VC approaches with parallel training
data, acoustic features are first extracted from the source and target
utterances. Then, the acoustic features are aligned frame-wise with an
alignment module [6]. Studies have shown that the alignment step can be
bypassed through using sequence-to-sequence models for direct source-target
acoustic mapping with better VC performance [7]. For direct mapping VC with
nonparallel training data, progress has been made with generative adversarial
networks (GAN) based many-to-many VC systems [8, 9]. Although widely
investigated, the direct mapping method assumes that the speaker of source-
target VC pair is pre-known, which limits the application of such models in
the real world. To relax this constraint, the second methodology constructs VC
based on explicitly learned speaking style and content representations. Among
these approaches, phonetic posteriorgrams (PPGs) is widely used as the speaker
independent content representations [10, 11], and speaker embeddings extracted
from a pre-trained speaker verification model are often assumed to carry
timbre information [12]. They have been successfully applied to tasks such as
many-to-many VC or any-to-many VC. For zero-shot VC, both AUTOVC and AdaIN-VC
construct encoder-decoder frameworks [13, 14]. The encoder compress the
speaking style and the content information into the latent embedding, and the
decoder generate a voice sample by combining a speaking style embedding and a
content embedding. To achieve a better VC performance, these models require
positive pair of utterances (i.e., two utterances come from the same speaker)
during training, and the systems have to rely on pre-trained speaker models.

In this study, we focus on the problem of speaker identity conversion. We
extend VAE as the backbone framework for learning disentangled content
representation and speaking style representation, where balanced content and
style information flow is achieved in the VAE training [15]. We show that the
vanilla VAE [15, 16] loss can be extended to force strong disentanglement
between speaker and content components, which is intuitively explained from
three levels. In addition, we explore to make the learned representation
robust against background noise/music and interfering speaker etc. An on-the-
fly data augmentation is introduced as the inductive bias to the VAE training,
with this training strategy, we arrive at a denoising disentangled sequential
VAE (D-DSVAE), where low quality speech input is allowed to test for VC. With
all these contributions, our proposed system achieves the state-of-the-art VC
performance and improved robustness.

##  2 Proposed Methods

We start with introducing some notations. Denote a speech segment variable
$X=[x_{1},x_{2},...,x_{T}]$, which is the STFT or Mel-Spectrogram in our
implementation. Denote $Z_{S}\in\mathbb{R}^{d_{S}}$ as speaker style latent
representation and
$Z_{C}=[z_{c1},z_{c2},...,z_{c_{T}}]\in\mathbb{R}^{d_{C}\times T}$ as speech
content latent representations. Denote $\theta$ as model parameters. The
proposed VC system adopts the modified form of DSVAE [16] as our backbone,
which is shown in Fig.1. The model takes $X$ as input, which is first passed
into a shared encoder $E_{share}$ and it gives
$W=[w_{1},w_{2},...,w_{T}]\in\mathbb{R}^{d_{W}\times T}$. Here $d_{S}$,
$d_{C}$ and $d_{W}$ are positive integers. Then a speaker encoder
$E_{speaker}$ and a content encoder $E_{content}$ take $W$ as input and model
the posterior distribution $q_{\theta}(Z_{S}|W)=q_{\theta}(Z_{S}|X)$ and
$q_{\theta}(Z_{C}|W)=q_{\theta}(Z_{C}|X)$ respectively. $Z_{S}$ and $Z_{C}$
are then obtained via sampling from $q(Z_{S}|X)$ and $q(Z_{C}|X)$. The
expectation is that $Z_{S}$ only encodes speaker information and $Z_{C}$ only
encodes content information. During the generation stage, the concatenation of
$Z_{S}$ and $Z_{C}$ are passed into a shared decoder $D_{share}$ to generate
the spectrogram $\hat{X}$, i.e. $\hat{X}=D(Z_{S},Z_{C})$. A vocoder is then
applied to convert $\hat{X}$ to waveform. When performing voice conversion
from source $X_{1}$ to target $X_{2}$, and the converted speech is
$\hat{X}_{12}=D(Z_{S2},Z_{C1})$, where $Z_{S2}$ and $Z_{C1}$ are sampled from
$q(Z_{S}|X_{2})$ and $q(Z_{C}|X_{1})$ respectively, as shown in Fig.1. In the
following sections, we first highlight the probabilistic graphic models in
Sec.2.1. Then, the objective function is introduced in Sec.2.2, then we
discuss its validity via proposing three sufficient conditions for achieving
disentanglement between $Z_{S}$ and $Z_{C}$.

Fig. 1: A flow diagram of proposed VC system.

###  2.1 Disentanglement-Aware Probabilistic Graphical Models

The frequently used method to achieve disentanglement between two components
is to let these two components be probabilistically independent with each
other. Following such intuition, we factorize both the prior distributions and
posterior distributions of $Z_{S}$ and $Z_{C}$ by following independence
assumption, which is consistent with [16, 17]. We denote $Z=[Z_{S},Z_{C}]$ as
a joint latent representation.

##### Prior

The joint prior distribution is factorised as follows:

$p_{\theta}(Z)=p(Z_{S})p_{\theta}(Z_{C})=p(Z_{S})\prod_{t=1}^{T}P_{\theta}(z_{ct}|z_{<t})$
(1)

where $p(Z_{S})$ is a standard normal distribution and $p_{\theta}(Z_{C})$ is
modeled by an autoregressive LSTM.

##### Posterior

The joint posterior distribution is factorised as follows:

$\begin{split}q_{\theta}(Z|X)&=q_{\theta}(Z_{S},Z_{C}|W)=q_{\theta}(Z_{S}|W)q_{\theta}(Z_{C}|W)\end{split}$
(2)

Here, $q_{\theta}(Z_{S}|W)$ and $q_{\theta}(Z_{C}|W)$ are modeled by two
independent LSTMs. Implementation Detailes can be found in 3.3.

###  2.2 Loss Objectives

We first provide the loss objective as shown in Eq. 3, which is simply the
vanilla VAE [15] loss. After that, we will explain how it is related to
disentanglement and how the disentanglement is achieved.

$\begin{split}&\mathcal{L}=\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(X|Z_{S},Z_{C})}[-log(p_{\theta}(X|Z_{S},Z_{C}))]+\\\
&\mathbb{E}_{p(X)}[\alpha kl(p(Z_{S})||q_{\theta}(Z_{S}|X))+\beta
kl(p_{\theta}(Z_{C})||q_{\theta}(Z_{C}|X))]\end{split}$ (3)

where $kl(.,.)$ denotes KL divergence between two distributions, $\alpha$ and
$\beta$ are two balancing factors. The loss objective does not directly
enforce the disentanglement between $Z_{S}$ and $Z_{C}$, however, we will
intuitively explain in the following sub-sections how the disentanglement
could be achieved from three levels.

####  2.2.1 Variational Mutual Information and KL Vanishing in VAE

In the original form of VAE [15], the mutual information
$MI(X,Z)=kl(p(X,Z)||p(X)p(Z))$ has its variational form
$\hat{MI}_{\theta}(X,Z)=kl(q_{\theta}(X,Z)||p(X)p(Z))=\mathbb{E}_{p(X)}[kl(q_{\theta}(Z|X)||p(Z))]$,
which we call the variational mutual information. The objective of VAE can be
reformulated as:

$\mathcal{L}_{VAE}(\theta)=\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(Z|X)}[-log(p_{\theta}(X|Z))]+\hat{MI}_{\theta}(X,Z)$
(4)

The first term in RHS of Eq. 4 is the reconstruction loss. It is observable
that minimizing both two terms are typically not achievable considering that
lower reconstruction loss means higher variational mutual information between
$X$ and $Z$, by assuming that we already achieve a good variational estimator,
i.e. $q_{\theta}(Z|X)\approx p(Z|X)$. While there are exceptions when the
decoder is designed to have powerful self-supervised generation capacity [18],
these cases are not taken into consideration since the decoder is not such
powerful in our scenario. Thus, VAE aims to reach a balance between the
reconstruction loss and variational mutual information. It follows that the
variational mutual information is lower bounded:

$\inf_{\theta^{*}}\hat{MI}_{\theta^{*}}(X,Z)\geq A(X,Z)>0$ (5)

where $\theta^{*}$ denotes a local optimal solution, $A(X,Z)$ is denotes the
bounded variational mutual information, which is similar to $I_{C}$ in Eq. 2
in [19]. The above formula guarantees that KL vanishing [18] will not happen
during VAE training, which is the foundation of disentanglement.

####  2.2.2 Information Flow Between Multiple Latent Variables in VAE

We argue that balanced information flow between $Z_{S}$ and $Z_{C}$ leads to
disentanglement. Based on the Eq. 4, the loss objective proposed in Eq. 3 can
be formulated as:

$\begin{split}\mathcal{L}(\theta)&=\mathbb{E}_{p(X)}\mathbb{E}_{q_{\theta}(X|Z_{S},Z_{C})}[-log(p_{\theta}(X|Z_{S},Z_{C}))]\\\
&+\alpha\hat{MI}_{\theta}(X,Z_{S})+\beta\hat{MI}_{\theta}(X,Z_{C})\end{split}$
(6)

where the factorised form of [16] and [17] is a special case when
$\alpha=\beta=1$. Following Sec. 2.2.1, the summation of information encoded
by $Z_{S}$ and information encoded by $Z_{C}$ is sufficient enough to
reconstruct the input speech, that is:

$\inf_{\theta^{*}}\\{\alpha\hat{MI}_{\theta^{*}}(X,Z_{S})+\beta\hat{MI}_{\theta^{*}}(X,Z_{C})\\}\geq
A(X,Z)>0$ (7)

Intuitively, by assuming convergence, when $\frac{\beta}{\alpha}$ is extremely
large, the gradient of two variational mutual information terms will be
dominated by $\hat{MI}(X,Z_{C})$, which results in the vanishing of
$KL(q_{\theta}(Z_{C}|X)||p(Z_{C}))$ so that the information encoded by the
latent variable will be dominated by $Z_{S}$. Similar conclusion could be
derived for the other side. The phenomenon of such imbalance is also observed
in [20]. Theoretical expression of such gradient dominance is that, there
exist $(\alpha_{1},\beta_{1}),(\alpha_{2},\beta_{2})$ such that:

$\alpha_{1}\hat{MI}_{\theta^{*}}(X,Z_{S})+\beta_{1}\hat{MI}_{\theta^{*}}(X,Z_{C})=\alpha_{1}\hat{MI}_{\theta^{*}}(X,Z_{S})$
(8)
$\alpha_{2}\hat{MI}_{\theta^{*}}(X,Z_{S})+\beta_{2}\hat{MI}_{\theta^{*}}(X,Z_{C})=\beta_{2}\hat{MI}_{\theta^{*}}(X,Z_{C})$
(9)

We experimentally observe that the variational mutual information loss is
independent of reconstruction loss, i.e.
$\alpha_{1}\hat{MI}_{\theta^{*}}(X,Z_{S})\approx\beta_{2}\hat{MI}_{\theta^{*}}(X,Z_{C})\approx
A(X,Z)$. In that sense, there is a double-sided information flow between
$Z_{S}$ and $Z_{C}$. By choosing a proper $\frac{\beta^{*}}{\alpha*}$, it
would be possible that there is no overlap between the information encoded by
$Z_{S}$ and $Z_{C}$, i.e. $\hat{MI}_{\theta^{*}}(Z_{S},Z_{C})=0$. Eq. 8 and
Eq. 9 will be experimentally justified in Sec. 3.

####  2.2.3 Time-invariant and Time-Variant Disentanglement

We argue that applying average pooling techniques over the time dimension of a
sequence of speech features leads to further disentanglement. According to
Sec. 2.2.2, even if $\hat{MI}(Z_{S},Z_{C})=0$, it is still undetermined which
latent variable encodes speaker information or content information. Inspired
by the 1D instancenorm method proposed in [14], we attempt to keep the speaker
information by only taking the average pooling on speech features to derive
the posterior distribution $q(Z_{S}|X)$. By choosing $\alpha=\alpha*$ and
$\beta=\beta*$ as indicated in the second sufficient condition, $Z_{S}$ will
encode time-invariant information while $Z_{C}$ will encode time-variant
information. Other normalizations techniques like instancenorm2D also help a
lot to obtain better disentanglement performance, which will be discussed in
Sec. 3.

##  3 Experiments

###  3.1 Dataset and Data Preprocessing

##### TIMIT

We follow the official train/test split: 462 speakers for training and 24
speakers for testing [21]. For speaker verification (SV) task, we choose all
possible trials from test set, which gives 18336 trials. 200 dimensional STFT
features are extracted from a raw waveform with 25ms/10ms framing
configuration. During training, the length of segment is fixed to 20 frames.

##### VCTK

For VCTK corpus [22], 90% of the speakers are randomly selected for training
and the remaining 10% as for testing. For the evaluation of disentanglement
performance in the SV task, we generate 36900 trials (22950 nontarget trials
and 14040 target trials) from test set. We extract melspectrogram as features
with a framing configuration of 64ms/16ms. The feature dimension is set to 80.
We select a segment of 100 frames for the VAE training.

###  3.2 Model Architectures

##### Encoder, Decoder, Vocoder

There are two models designed for TIMIT and VCTK respectively. The encoder is
composed of a shared encoder, speaker encoder and content encoder. (i) For
TIMIT, the shared encoder is a 2-layer MLP with hidden size of 256. The
content encoder is 2-layer BiLSTM with hidden size of 512, followed by a RNN
layer with hidden size of 512. Then a 2-layer MLP of hidden size (512,64) is
applied to model $q_{\theta}(Z_{C}|X)$. The speaker encoder is almost the same
with content encoder except that there is an average pooling layer after RNN,
and a 2-layer MLP is then applied to model $q_{\theta}(Z_{S}|X)$. The decoder
is 1-layer MLP, followed by 2-layer BiLSTM which is followed by 2-layer MLP,
where the hidden size is 256. Griffin-lim algorithm [23] is applied as
vocoder. (ii) For VCTK, modified from [13], the shared encoder is composed of
three convolutional layers with 512 channels. Each convolutional layer is
followed by a linear layer with dimension 512 and an Instancenorm2D layer
[24]. Also modified from [13], decoder includes a prenet with 512 channels and
a postnet, which is a BiLSTM with hidden size of 512, followd by three
convolutional layers with 512 channels, followed by a BiLSTM with hidden size
of 512 and two separate linear layers to project the hidden dimension to 80 to
model $p_{\theta}(X|Z_{S},Z_{C})$. A wavenet [25] pretrained on VCTK is used
as Vocoder. Note that vocoder is only used for inference.

##### Prior Distribution

The prior distribution includes $p(Z_{S})$ and $p_{\theta}(Z_{C})$. $p(Z_{S})$
is modeled by a Normal distribution $N(0,\mathbb{I}_{d_{S}})$, where there are
no parameters.
$p_{\theta}(Z_{C})=\prod_{t=1}^{T}P_{\theta}(Z_{Ct}|Z_{C\tau<t})$ is modeled
by an auto-regressive LSTM with hidden size of 512. Each hidden cell is
followed by two one-layer MLPs with hidden size 512 to model
$P_{\theta}(Z_{Ct}|Z_{C\tau<t})$, from which $Z_{Ct}$ is sampled and passed
into the next LSTM cell.

###  3.3 Implementation Details

In the following, we first specify the hyper-parameters used for training, and
then introduce evaluation metrics and training details. Next, the noise
invariant method is introduced. Lastly, we detail the inference processes
which include speaker verification, voice conversion.

##### Hyper-parameters

Optimizer is adam with initial learning rate of 5e-4. Learning rate is decayed
every 5 epochs with a factor of 0.95. Weight decay is 1e-4. Batchsize is 256.
Both speaker dimension($d_{S}$) and content dimension($d_{C}$) are set to 64.
We performed grid search for $\alpha$ and $\beta$ and set $\alpha=1,\beta=20$
for TIMIT part and $\alpha=0.01,\beta=10$ for VCTK part. The embedding size
$d_{S}=d_{C}=64$.

##### Evaluation Metrics

(i) EER (Equal Error Rate) is used as the evaluation metrics for
disentanglement. Lower EER on speaker embeddings and higher EER on content
embeddings typically (would be clarified in the discussion section) indicate
better disentanglement, as observed in [26, 16, 17]. (ii) We conduct a mean
opinion score test to evaluate our proposed approach. Two different set are
provided for MOS test. For both seen to seen VC and unseen to unseen VC, we
select 6 speakers (3 females and 3 males), each speaker with one utterance. So
30 test cases are included in the set. The listener needs to give a score for
each sample in a test case according to the criterion: 1 = Bad; 2 = Poor; 3 =
Fair; 4 = Good; 5 = Excellent. The final score for each model is calculated by
averaging the collected results.

##### Noise Invariant training

In the training of normal VAE, the VAE encoder takes clean acoustic feature
$X$ as the input, and produce embedding $Z_{S}$ and $Z_{C}$. The decoder tries
to reconstruct $X$ with the feeding information $Z_{S}$ and $Z_{C}$. To make
the learned embedding $Z_{S}$ and $Z_{C}$ robust against background noise, we
make a simple yet effective change in the training process. In the new data
flow, a data augmentation module is applied to $X$, denoted as $X^{aug}$. The
VAE encoder produces ${Z_{S}}^{aug}$ and ${Z_{C}}^{aug}$, which is the
augmented version of $Z_{S}$ and $Z_{C}$. The decoding process
$D({Z_{S}}^{aug},{Z_{C}}^{aug})$ reconstructs acoustic feature
$\hat{X}^{aug}$, and we maximize the likelihood of $\hat{X}^{aug}$ and clean
reference $X$. By introduce this inductive bias, we expect the VAE framework
not only disentangle global and local information, but also perform denoising
at the latent space. In this study, clean utterance is augmented by MUSAN
dataset [27] with a balanced “noise”, “music” and “babble” distribution.

##### Inference Experiments

Since the inference tasks are performed on the utterance-level, we also start
with making some notations. Denote an utterance
$U=[{X}^{(1)},{X}^{(2)},...,{X}^{(K)}]$, where ${X}^{(k)}$ is the kth segment.
Denote ${Z_{S}}^{(k)}$ and ${Z_{C}}^{(k)}$ as the corresponding latent speaker
and content variables for $X^{(k)}$. Here both ${Z_{S}}^{(k)}$ and
${Z_{C}}^{(k)}$ can be either the mean or a sample from
$q({Z_{S}}^{(k)}|X^{(k)})$ and $q({Z_{C}}^{(k)}|X^{(k)})$ as we observed no
significant difference made by these two. We implement two fundamental
experiments: speaker verification and voice conversion.

(i) Speaker Verification. We apply the same method with [16, 17] to derive the
speaker and content embedding: $\mu_{S}(U)=(\Sigma_{k=1}^{K}{Z_{S}}^{(k)})/K$
and $\mu_{C}(U)=(\Sigma_{k=1}^{K}(\Sigma_{t=1}^{T}{z_{c}}_{t}^{(k)}/T))/K$.
For TIMIT trials, we vary the value of $\frac{\beta}{\alpha}$ to observe the
EER, and results are in Table 1. We implement t-SNE [28] visualization of
speaker and content embeddings for VCTK, as shown in Fig. 2. For speaker
embedding, the cluster pattern is clear, while for content embedding, the
evenly distributed scatters demonstrates that only limited (if not zero)
speaker information is leaked in the content embedding.

(ii) Voice Conversion. Denote $U_{ij}$ as the converted utterance which takes
$U_{i}$ as source and $U_{j}$ as target. We first obtain $\mu_{S}(U_{j})$ as
defined before. Second, pass ${X_{i}}^{(k)}$ into the model and we will have
${Z_{C}}_{i}^{(k)}$. in generation stage,
$D(\mu_{S}(U_{j}),{Z_{C}}_{i}^{(k)})$ gives $\hat{X_{i}}^{(k)}$; then
$U_{ij}=[\hat{X_{i}}^{(1)},\hat{X_{i}}^{(2)},...,\hat{X_{i}}^{(K)}]$. We vary
the value of $\frac{\beta}{\alpha}$ to observe the results of voice
conversion. Since we do not observe big difference between VCTK and TIMIT
parts, we just present the results for TIMIT in Fig. 3 to illustrate the
double-sided information flow as proposed in the second sufficient condition.

Fig. 2: Visualization of Speaker and Gender Clustering on VCTK

###  3.4 Results and discussions

##### Disentanglement

As observed in Fig.3, only half of the utterance is converted to the target
speaker for both two source utterances when $\frac{\beta}{\alpha}=1$, which
indicates that $Z_{S}$ does not encode the whole speaker information and
$Z_{C}$ encodes part of the speaker information. When $\frac{\beta}{\alpha}$
is increased to 10 or 20, the speaker EER becomes lower and content EER
becomes higher, which indicates better disentanglement. Fig.3 also illustrates
this point since $\frac{\beta}{\alpha}=10$ or $20$ successfully achieves voice
conversion. When $\frac{\beta}{\alpha}=100$, Fig. 3 indicates that instead of
performing voice conversion, it actually does utterance swapping, which means
that $Z_{S}$ encodes almost everything and $Z_{C}$ encodes almost nothing.
When this case happens, it is still reasonable that speaker EER is low and
content EER is high, as observed in Table 1. To this end, we can derive
another conclusion that good eer is actually the necessary but not sufficient
condition for good disentanglement for two reasons. First,
$\frac{\beta}{\alpha}=1$ and $\frac{\beta}{\alpha}=100$ give similar EER,
however, there is no disentanglement for the latter case. Second,
$\frac{\beta}{\alpha}=10$ gives better EER, however, $\frac{\beta}{\alpha}=20$
gives better disentanglement if we take a closer look at middle part of the
spectrogram in male to female conversion in Fig. 3. The better way to look at
disentanglement is to look at both EER and information flow. Fig.3 shows that
as $\frac{\beta}{\alpha}$ is increasing, $Z_{S}$ encodes more information and
$Z_{C}$ encodes less information, which is consistent with the assumption in
our proposed second sufficient condition. For VCTK trails, the best SV EER of
$Z_{S}$ is 2.3%, while the EER of $Z_{S}$ and $Z_{C}$ w.r.t. the best VC
performance are 4.6% and 44.5%. The difference indicates that balancing the
information flow for disentanglement in VC remains a challenging problem.

Fig. 3: Visualization of Double-sided Information Flow. Four models are
trained with four different $\frac{\beta}{\alpha}$. Given a pair of male and
female voice, we perform double-sided voice conversion. For each row, the
first figure is the reconstructed spectrogram. The remaining four figures are
the converted spectrograms. For the purpose of presentation, we just use fixed
length of segment for voice conversion Table 1: EER (%) for TIMIT test trials
on varying $\frac{\beta}{\alpha}$.

.  $\frac{\beta}{\alpha}$ 1 10 20 100 DSVAE[16] $\mu_{S}$ 5.40 3.25 4.16 5.01
4.94 $\mu_{C}$ 31.09 38.83 37.16 38.79 17.49

##### Voice conversion111Samples of voice conversion can be found at
https://jlian2.github.io/Robust-Voice-Style-Transfer

Table 2 shows the MOS results of different models. For fair comparison, we
also generate VC samples from AUTOVC and AdaIN-VC, where we use their
pretrained models with our generated VC pairs. As illustrated in the table,
our proposed system outperforms AUTOVC and AdaIN-VC with a large margin in
both naturalness and similarity. To test the effectiveness of noise invariant
training, we conduct an additional MOS test. For test the noise invariant VC
model, we apply background noise with signal to noise ratio range of 3-10 dB.
As indicated in the MOS test, we only see marginal performance degradation
compared with clean input, which shows the effectiveness of noise invariant
training.

Table 2:  The results of the MOS (95% CI) test on different models.

| seen to seen | unseen to unseen
---|---|---
model | naturalness | similarity | naturalness | similarity
AUTOVC [13] |  2.65$\pm$0.12 |  2.86$\pm$0.09 |  2.47$\pm$0.10 |  2.76$\pm$0.08
AdaIN-VC [14] |  2.98$\pm$0.09 |  3.06$\pm$0.07 |  2.72$\pm$0.11 |  2.96$\pm$0.09
Ours |  3.40$\pm$0.07 |  3.56$\pm$0.06 |  3.22$\pm$0.09 |  3.54$\pm$0.07
Ours(noisy) |  3.23$\pm$0.09 |  3.43$\pm$0.07 |  3.12$\pm$0.08 |  3.47$\pm$0.08

##  4 Conclusion

In this study, we proposed a novel zero-shot voice conversion system. By
decomposing speech into speaker representation $Z_{C}$ and content
representation $Z_{S}$ with a sequential VAE framework, the conversion can be
as simple as: a) swapping the speaker representation; b) feeding source
content embedding and target speaker embedding to the decoder to generate the
acoustic features; c) convert the acoustic feature to the waveform with a
vocoder. We improved the DSVAE framework by analyzing the information flow
global between speaker representation and local content representation. Noise
invariant training was investigated, which enabled the VC system to handle low
quality speech input. For both SV and VC tasks, we achieved the state-of-the-
art system performance on TIMIT and VCTK. We believe this study serves as a
preliminary research work, and can be beneficial to other domains, such as
speech recognition, text to speech studies.

## References

  * [1] Berrak Sisman, Junichi Yamagishi, Simon King, and Haizhou Li,  “An overview of voice conversion and its challenges: From statistical modeling to deep learning,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
  * [2] Fahimeh Bahmaninezhad, Chunlei Zhang, and John HL Hansen,  “Convolutional neural network based speaker de-identification.,”  in ISCA Odyssey, 2018.
  * [3] Liqiang Zhang, Chengzhu Yu, Heng Lu, Chao Weng, Chunlei Zhang, Yusong Wu, Xiang Xie, Zijin Li, and Dong Yu,  “Durian-sc: Duration informed attention network based singing voice conversion system,”  arXiv preprint arXiv:2008.03009, 2020.
  * [4] Yannis Stylianou, Olivier Cappé, and Eric Moulines,  “Continuous probabilistic transform for voice conversion,”  IEEE Transactions on speech and audio processing, vol. 6, no. 2, pp. 131–142, 1998.
  * [5] Tomoki Toda, Alan W Black, and Keiichi Tokuda,  “Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,”  IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235, 2007.
  * [6] Donald J Berndt and James Clifford,  “Using dynamic time warping to find patterns in time series,”  in KDD workshop, 1994, pp. 359–370.
  * [7] Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, and Li-Rong Dai,  “Sequence-to-sequence acoustic modeling for voice conversion,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 3, pp. 631–644, 2019.
  * [8] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo,  “Stargan-vc: Non-parallel many-to-many voice conversion using star generative adversarial networks,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 266–273.
  * [9] Takuhiro Kaneko and Hirokazu Kameoka,  “Cyclegan-vc: Non-parallel voice conversion using cycle-consistent adversarial networks,”  in 2018 26th European Signal Processing Conference (EUSIPCO). IEEE, 2018, pp. 2100–2104.
  * [10] Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen Meng,  “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,”  in 2016 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2016, pp. 1–6.
  * [11] Haohan Guo, Heng Lu, Na Hu, Chunlei Zhang, Shan Yang, Lei Xie, Dan Su, and Dong Yu,  “Phonetic posteriorgrams based many-to-many singing voice conversion via adversarial training,”  arXiv preprint arXiv:2012.01837, 2020.
  * [12] Mingyang Zhang, Yi Zhou, Li Zhao, and Haizhou Li,  “Transfer learning from speech synthesis to voice conversion with non-parallel training data,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 1290–1302, 2021.
  * [13] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson,  “Autovc: Zero-shot voice style transfer with only autoencoder loss,”  in International Conference on Machine Learning. PMLR, 2019, pp. 5210–5219.
  * [14] Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee,  “One-shot voice conversion by separating speaker and content representations with instance normalization,”  arXiv preprint arXiv:1904.05742, 2019.
  * [15] Diederik P Kingma and Max Welling,  “Auto-encoding variational bayes,” 2014.
  * [16] Yingzhen Li and Stephan Mandt,  “Disentangled sequential autoencoder,”  arXiv preprint arXiv:1803.02991, 2018.
  * [17] Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf,  “S3vae: Self-supervised sequential vae for representation disentanglement and data generation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6538–6547.
  * [18] Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick,  “Lagging inference networks and posterior collapse in variational autoencoders,”  arXiv preprint arXiv:1901.05534, 2019.
  * [19] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy,  “Deep variational information bottleneck,”  arXiv preprint arXiv:1612.00410, 2016.
  * [20] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn,  “Gradient surgery for multi-task learning,”  arXiv preprint arXiv:2001.06782, 2020.
  * [21] John S Garofolo,  “Timit acoustic phonetic continuous speech corpus,”  Linguistic Data Consortium, 1993, 1993.
  * [22] Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald,  “Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit,”  2017\.
  * [23] Daniel Griffin and Jae Lim,  “Signal estimation from modified short-time fourier transform,”  IEEE Transactions on acoustics, speech, and signal processing, vol. 32, no. 2, pp. 236–243, 1984.
  * [24] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky,  “Instance normalization: The missing ingredient for fast stylization,”  arXiv preprint arXiv:1607.08022, 2016.
  * [25] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu,  “Wavenet: A generative model for raw audio,”  arXiv preprint arXiv:1609.03499, 2016.
  * [26] Wei-Ning Hsu, Yu Zhang, and James Glass,  “Unsupervised learning of disentangled and interpretable representations from sequential data,”  arXiv preprint arXiv:1709.07902, 2017.
  * [27] David Snyder, Guoguo Chen, and Daniel Povey,  “Musan: A music, speech, and noise corpus,”  arXiv preprint arXiv:1510.08484, 2015.
  * [28] Laurens Van der Maaten and Geoffrey Hinton,  “Visualizing data using t-sne.,”  Journal of machine learning research, vol. 9, no. 11, 2008.