# A Closer Look At Feature Space Data Augmentation For Few-Shot
Intent Classification

Varun Kumar, Hadrien Glaude, Cyprien de Lichy, William Campbell
Amazon Alexa
Cambridge, MA, USA
<EMAIL_ADDRESS>

###### Abstract

New conversation topics and functionalities are constantly being added to
conversational AI agents like Amazon Alexa and Apple Siri. As data collection
and annotation is not scalable and is often costly, only a handful of examples
for the new functionalities are available, which results in poor
generalization performance. We formulate it as a Few-Shot Integration (FSI)
problem where a few examples are used to introduce a new intent. In this
paper, we study six feature space data augmentation methods to improve
classification performance in FSI setting in combination with both supervised
and unsupervised representation learning methods such as BERT. Through
realistic experiments on two public conversational datasets, SNIPS, and the
Facebook Dialog corpus, we show that data augmentation in feature space
provides an effective way to improve intent classification performance in few-
shot setting beyond traditional transfer learning approaches. In particular,
we show that (a) upsampling in latent space is a competitive baseline for
feature space augmentation (b) adding the difference between two examples to a
new example is a simple yet effective data augmentation method.

##  1 Introduction

Virtual artificial assistants with natural language understanding (NLU)
support a variety of functionalities. Throughout the lifespan of the deployed
NLU systems, new functionalities with new categories, are regularly
introduced. While techniques such as active learning Peshterliev et al.
(2018), semi-supervised learning Cho et al. (2019b) are used to improve the
performance of existing functionalities, performance for new functionalities
suffers from the data scarcity problem.

Recently, Few-Shot Learning has been explored to address the problem of
generalizing from a few examples per category. While it has been extensively
studied Koch et al. (2015); Snell et al. (2017); Vinyals et al. (2016) for
image recognition, a little attention has been paid to improve NLU performance
in the low-data regime. Moreover, researchers have been mostly working on the
unrealistic setting that considers tasks with few categories unseen during
(pre)training, each with only a few examples, and introduces new categories
during test time. We argue that a more realistic setting is Few-Shot
Integration (FSI) where new categories with limited training data are
introduced into an existing system with mature categories. FSI is well aligned
with the goal of lifelong learning of conversational agents and measures the
performance in a real-life system setting when only a few examples of a new
class are added to the existing data from the old classes. To address the poor
generalization in data scare scenarios, several pre-training methods such as
ELMo Peters et al. (2018), Generative pre-trained Transformer Radford et al.
(2018), BERT Devlin et al. (2018), have been proposed which are trained on a
large amount of unannotated text data. Such pre-trained models can be fine-
tuned on a particular NLP task and have shown to greatly improve
generalization. However, in FSI setting where only a handful of examples are
provided, building accurate NLU model is still a challenging task.

In this paper, we focus on Feature space Data Augmentation (FDA) methods to
improve the classification performance of the categories with limited data. We
study six widely different feature space data augmentation methods: 1)
upsampling in the feature space Upsample, 2) random perturbation Perturb, 3)
extrapolation Devries and Taylor (2017) Extra, 4) conditional variational
auto-encoder (CVAE) Kingma and Welling (2013) CVAE, 5) delta encoder that have
been especially designed to work in the few-shot learning setting Schwartz et
al. (2018) Delta, 6) linear delta which is a linear version of the delta
encoder Linear. While Upsample, Perturb, Extra and Linear doesn’t require any
training beyond hyper-parameter optimization, Delta and CVAE are trained deep
neural network generators.

We compare these six FDA techniques on two open datasets for Intent
Classification (IC) : SNIPS Coucke et al. (2018) and Facebook Dialog corpus
Gupta et al. (2018). We show that BERT combined with Linear data augmentation
provides an effective method to bootstrap accurate intent classifiers with
limited training data. We make the following contributions:

  1. 1.

We propose the FSI evaluation, a relaxation of the few-shot learning setting
that aims to better model the requirement of modern NLU systems. We provide a
comprehensive evaluation of FSI for text classification and show that Upsample
and Perturb are simple yet efficient baselines that are often neglected in
few-shot learning evaluations.

  2. 2.

We provide an in-depth analysis of various FDA methods. We show that complex
methods such as Delta and CVAE do not always improve over simple methods like
Linear, and the performance heavily depends on the feature extractor.

  3. 3.

Finally, we provide guidance on when and how to apply FDA for FSI. We show
that FDA consistently provides gains on top of the unsupervised pre-training
methods such as BERT in FSI setting.

##  2 Related work

Few-shot learning has been studied extensively in the computer vision domain.
In particular, several metric learning based methods Koch et al. (2015);
Vinyals et al. (2016); Snell et al. (2017); Rippel et al. (2015) has been
proposed for few-shot classification where a model first learns an embedding
space and then a simple metric is used to classify instances of new categories
via proximity to the few labeled training examples embedded in the same space.
In addition to metric-learning, several meta-learning based approaches Ravi
and Larochelle (2016); Li et al. (2017); Finn et al. (2017) have been proposed
for few-shot classification on unseen classes.

Recently, Few-Shot Learning on text data has been explored using metric
learning Yu et al. (2018); Jiang et al. (2018). In Yu et al. (2018), authors
propose to learn a weighted combination of metrics obtained from meta-training
tasks for a newly seen few-shot task. Similarly, in Cheng et al. (2019),
authors propose to use meta-metric-learning to learn task-specific metric that
can handle imbalanced datasets.

Generative models are also widely used to improve classification performance
by data augmentation. For example, generative models are used for data
augmentation in image classification Mehrotra and Dukkipati (2017); Antoniou
et al. (2018); Zhang et al. (2018), text classification Gupta (2019), anomaly
detection Lim et al. (2018). Data augmentation through deformation of an image
has been known to be very effective for image recognition. More advanced
approaches rely on Auto-Encoders (AEs) or Generative Adversarial Networks
(GANs). For example, in Mehrotra and Dukkipati (2017) the authors combine
metric-learning with data augmentation using GANs for few-shot learning.
However, classical generative approaches require a significant amount of
training data to be able to generate good enough examples that will improve
classification accuracy. To overcome this challenge, (Hariharan and Girshick,
2017) proposed to augment the training data in the feature space. This both
eases the generation problem and enforces generation of discriminative
examples. In addition, the authors propose to transfer deformations from base
classes to new classes, which allows circumventing the data scarcity problem
for new classes. Finally, in Schwartz et al. (2018), authors used an
Autoencoder to encode transformations between pairs of examples of the same
class and apply them to an example of the new class.

Generative models are a good candidate for FSI tasks, as one can just combine
the generated data for new classes with the old classes training data
Hariharan and Girshick (2017); Wang et al. (2018). For text classification,
several text generation based data augmentation techniques have also been
explored Hou et al. (2018); Zhao et al. (2019); Guu et al. (2018); Yoo et al.
(2018); Cho et al. (2019a). However, generating discrete sequences, e.g. text,
is known to be quite difficult and requires lots of training data. That is
why, in this paper, we focus on generative models, which augment data in
latent(feature) space to solve a few-shot integration problem for text
classification.

##  3 Data Augmentation in Feature Space

(a) Learning the feature space

(b) Learning with augmented data

Figure 1: Data augmentation in the feature space

Feature space data Augmentation (FDA) is an effective method to improve
classification performance on different ML tasks Chawla et al. (2002);
Hariharan and Girshick (2017); Devries and Taylor (2017); Guo et al. (2019).
As shown on Figure 1, FDA techniques usually work by first learning a data
representation or feature extractor, and then generating new data for the low
resource class in the feature space. After generating data, a classifier is
trained with real and augmented data.

For IC, we finetune a pre-trained English BERT-Base uncased model
111https://github.com/huggingface/pytorch-transformers to build our feature
extractor. The BERT model has $12$ layers, $768$ hidden states, and $12$
heads. We use the pooled representation of the hidden state of the first
special token ([CLS]) as the sentence representation. A dropout probability of
$0.1$ is applied to the sentence representation before passing it to the
1-layer Softmax classifier. BERT Encoder and MLP classifier are fine-tuned
using cross-entropy loss for IC task. Adam Kingma and Ba (2014) is used for
optimization with an initial learning rate of $5e-5$.

For data augmentation, we apply six different FDA methods, described below, to
generate new examples in the feature space. Finally, we train a 1- layer
Softmax classifier as in the feature learning phase.

###  3.1 Upsampling

The simplest method to augment training data for underrepresented categories
is to duplicate the existing training data. Upsampling is a well studied
technique to handle the class imbalance problem Estabrooks et al. (2004). We
show that for intents with limited labeled data, upsampling the existing data
in latent space consistently improves model performance, and thus is a good
baseline method for FDA techniques. We call this method Upsample.

###  3.2 Random Perturbation

Adding random noise to the existing training data is another simple yet
effective data augmentation technique. Random perturbation data augmentation
has been previously used to improve the performance of classification models
as well as for sequence generation models. For example, Kurata et al. (2016)
applied additive and multiplicative perturbation to improve the text
generation for data augmentation. In our experiments, we apply both additive
and multiplicative perturbation to the existing training data. We sample noise
from a uniform distribution [-1.0, 1.0]. We use Perturb to refer to this
method.

###  3.3 Conditional VAE

Conditional Variational Autoencoder (CVAE) is an extension of Variational
Autoencoder (VAE) Kingma and Welling (2013) which can be used to generate
examples for a given category. All components of the model are conditioned on
the category. First, we train CVAE on the sentence representations and then
generate new examples by sampling from the latent distribution. The encoder
and decoder sub-networks are implemented as multi-layer perceptrons with a
single hidden layer of $2048$ units, where each layer is followed by a
hyperbolic tangent activation. The encoder output $Z$ is $128$-dimensional.
Mean Square Error (MSE) loss function is used for reconstruction. All models
are trained with Adam optimizer with the learning rate set to $10-3$.

###  3.4 Linear Delta

A simple method to generate new examples is to first learn the difference
between a pair of examples, and then add this difference to another example.
In this case, we first compute the difference $X_{i}-X_{j}$ between two
examples from the same class and then add it to a third example $X_{k}$ also
from the same class as shown in (1). We use Linear to refer to this method.

$\hat{X}=(X_{i}-X_{j})+X_{k}$ (1)

###  3.5 Extrapolation

In Devries and Taylor (2017), authors proposed to use extrapolation to
synthesize new examples for a given class. They demonstrated that
extrapolating between samples in feature space can be used to augment
datasets. In extrapolation, a new example, $\hat{X}$ is generated according to
(2). In our experiments, we use $\lambda=0.5$. We call this method Extra.

$\hat{X}=(X_{i}-X_{j})*\lambda+X_{i}$ (2)

###  3.6 Delta-Encoder

Delta-Encoder Schwartz et al. (2018) extends the idea of learning differences
between two examples using an autoencoder-based model. It first extracts
transferable intra-class deformations (deltas) between same-class pairs of
training examples, then applies them to a few examples of a new class to
synthesize samples from that class. Authors show that Delta-Encoder can learn
transferable deformations from different source classes which can be used to
generate examples for unseen classes. While the authors used Delta-Encoder to
generate examples for unseen classes, in our experiments, for FSI, we also use
the examples from the target class to the train both the feature extractor and
the Delta-Encoder along with all other examples. Then we generate new examples
for the target category using trained delta encoder. For data generation, we
try two different approaches to select a source sentence pair.

  1. 1.

DeltaR: Sample a pair of sentences ($X_{i}$, $X_{j}$) from a randomly selected
class. DeltaR applies deltas from multiple source categories to synthesize new
examples.

  2. 2.

DeltaS: Sample a pair of sentences ($X_{i}$, $X_{j}$) from the target
category. DeltaS applies deltas from the same target category.

The encoder and decoder sub-networks are implemented as multi-layer
perceptrons with a single hidden layer of $512$ units, where each layer is
followed by a leaky ReLU activation ($max(x,0.2*x)$). The encoder output $Z$
is $16$-dimensional. L1 loss is used as reconstruction loss. Adam optimizer is
used with a learning rate of $10-3$. A high dropout with a $50$% rate is
applied to all layers, to avoid the model memorizing examples.

##  4 Experiment

###  4.1 Datasets

We evaluate different FDA techniques on two public benchmark datasets, SNIPS
Coucke et al. (2018), and Facebook Dialog corpus (FBDialog) Gupta et al.
(2018). For SNIPS dataset, we use train, dev and test split provided by Goo et
al. (2018) 222https://github.com/MiuLab/SlotGated-SLU.

SNIPS dataset contains $7$ intents which are collected from the Snips personal
voice assistant. The training, development and test sets contain $13,084$,
$700$ and $700$ utterances, respectively. FBDialog has utterances that are
focused on navigation, events, and navigation to events. FBDialog dataset also
contains utterances with multiple intents as the root node. For our
experiment, we exclude such utterances by removing utterances with COMBINED
intent root node. This leads to $31,218$ training, $4,455$ development and
$9,019$ testset utterances. Note that while SNIPS is a balanced dataset,
FBDialog dataset is highly imbalanced with a maximum $8,860$ and a minimum of
$4$ training examples per intent.

###  4.2 Simulating Few-Shot Integration

In virtual assistants, often a new intent development starts with very limited
training data. To simulate the integration of a new intent, we randomly sample
$k$ seed training examples from the new intent, referred to as target intent,
and keep all the data from other intents. We also remove the target intent
data from the development set. We train the feature extractor on the resulting
training data, and then generate $100$, $512$ examples using different
augmentation methods for the target intent. To account for random fluctuations
in the results, we repeat this process $10$ times for a given target intent
and report the average accuracy with the standard deviation. In all
experiments, models are evaluated on the full test set.

Size | Method | SNIPS | FBDialog
---|---|---|---
No Augmentation | 98.14 (0.42) | 94.99 (0.18)
5% | Upsample | 98.14 (0.47) | 95.01 (0.16)
Perturb |  98.26 (0.40) | 94.98 (0.19)
Linear | 98.14 (0.45) |  95.02 (0.21)
Extra | 98.14 (0.45) |  95.02 (0.20)
CVAE | 98.14 (0.45) | 94.98 (0.24)
DeltaR | 98.23 (0.46) | 95.00 (0.22)
DeltaS |  98.26 (0.42) | 95.00 (0.20)
10% | Upsample | 98.14 (0.47) | 94.94 (0.18)
Perturb | 98.23 (0.41) | 94.98 (0.24)
Linear | 98.09 (0.50) |  95.02 (0.18)
Extra | 98.11 (0.49) | 95.01 (0.19)
CVAE | 98.20 (0.42) | 94.99 (0.26)
DeltaR |  98.26 (0.42) | 94.99 (0.21)
DeltaS | 98.23 (0.42) | 94.97 (0.22)
20% | Upsample | 98.14 (0.45) | 95.02 (0.12)
Perturb | 98.14 (0.44) | 94.99 (0.20)
Linear | 98.17 (0.43) | 95.05 (0.23)
Extra | 98.14 (0.45) | 95.07 (0.11)
CVAE | 98.11 (0.44) | 94.98 (0.23)
DeltaR |  98.26 (0.40) |  95.08 (0.19)
DeltaS | 98.20 (0.46) | 95.04 (0.22)

Table 1: IC accuracy on SNIPS and Facebook dataset with all training data,
reported as mean (SD).

##  5 Results and Discussion

###  5.1 FDA For Data-Rich Classification

For both datasets, we generate $5\%$, $10\%$, and $20\%$ examples using
different FDA methods. Then, we train a classifier using both generated as
well as real data. Table 1 shows that augmenting data in feature space
provides only minor improvements in classification accuracy. In particular, on
SNIPS dataset, Pertub and DeltaR improve accuracy from $98.14$ to $98.26$. On
FBDialog dataset, DeltaR provides a minor gain, $95.02$ to $95.08$ over
upsample baseline.

###  5.2 Impact Of The Number Of Seed Examples

To understand the impact of the number of seed examples, we vary it to $5$,
$10$, $15$, $20$, $25$, and $30$ for SNIPS’s AddToPlaylist. For each
experiment, we generate $100$ examples using different FDA methods. Figure 2
shows that as the number of seed examples increases, the accuracy of the model
goes up. We also observe that for a few seed examples $5$ \- $15$, Linear
outperforms other FSA methods. Finally, gains are less significant after $30$
seed examples.

Figure 2: IC accuracy on SNIPS’s AddToPlaylist intent with varying number of
seed examples. $100$ examples are generated using different FDA techniques. As
indicated by the accuracy trend, increasing the seed examples leads to better
performance.

###  5.3 Few-Shot Integration

We simulate FSI IC for all $7$ intents of SNIPS dataset. For FBDialog dataset,
we run simulations on the six largest intents, viz. GetDirections,
GetDistance, GetEstimatedArrival, GetEstimatedDuration, GetInfoTraffic, and
GetEvent. Since, BERT generalizes well with just $30$ examples, to compare the
effectiveness of different FDA methods, we use $10$ seed examples in FSI
simulations. For each intent, we select $k=10$ seed training examples and use
all training data for other intents.

Table 2 shows average accuracy for all intents’ FSI simulations. Results on
individual intent’s FSI simulations can be found in Appendix’s Table 5 and
Table 6. On both datasets, all FDA methods improve classification accuracy
over no augmentation baseline. Also, Upsample provides huge gains over no
augmentation baseline. Additionally, on both datasets, with $512$ augmented
examples, Linear and DeltaS works better than Perturb and Upsample.

# | Method | SNIPS | FBDialog
---|---|---|---
No Augmentation | 87.46(2.87) | 81.29(0.11)
100 | Upsample | 94.26(1.66) |  84.34(1.84)
Perturb | 94.18(1.74) | 84.04(1.95)
CVAE | 94.10(1.83) | 84.10(1.94)
Linear |  94.36(1.69) | 84.31(1.9)
Extra | 94.30(1.68) | 84.13(1.83)
| DeltaR | 91.32(3.12) | 81.97(0.76)
| DeltaS | 94.28(1.92) | 83.50(1.92)
512 | Upsample | 95.68(0.86) | 89.03(0.99)
Perturb | 95.65(0.92) | 89.02(0.99)
CVAE | 95.46(1.03) | 88.71(1.09)
Linear |  95.87(0.87) |  89.30(1.03)
Extra | 95.82(0.89) | 89.21(0.99)
| DeltaR | 95.33(1.56) | 87.28(1.46)
| DeltaS |  95.88(1.04) | 89.15(1.12)

Table 2: Average IC accuracy for all intents’ FSI simulations on SNIPS and
FBDialog dataset. For each simulation, $k=10$ seed examples are used for
target intent. Scores are reported as mean (SD). Refer to Appendix’s Table 5
and Table 6 for individual intents’ results.

###  5.4 Upsampling: Text Space vs Latent Space

# | Method | Overall Mean
---|---|---
No Augmentation | 94.38(1.23)
100 | Upsample | 94.53(1.12)
Perturb | 94.52(1.18)
CVAE | 94.53(1.18)
Linear | 94.53(1.12)
Extra | 94.53(1.13)
| DeltaR |  94.62(1.16)
| DeltaS | 94.57(1.14)
512 | Upsample | 94.67(1.11)
Perturb | 94.68(1.14)
CVAE | 94.73(1.11)
Linear | 94.67(1.11)
Extra | 94.67(1.11)
| DeltaR |  94.88(1.12)
| DeltaS | 94.74(1.12)
Table 3: IC accuracy on SNIPS dataset in the FSI setting, reported as mean
(SD). The $10$ seed examples are upsampled to $100$ to train the feature
extractor. Refer to Appendix’s Table 7 for individual intents’ results.

In this section, we explore how upsampling in text space impacts performances
as it is supposed to both improve the feature extractor and the linear
classifier, compared to Upsample. To investigate whether upsampling in text
space helps FDA, we upsampled the $10$ seed examples to $100$ and repeat the
FSI experiments on all $7$ intents of the SNIPS dataset. Table 3 shows the
mean accuracy of all $7$ intents FSI simulations results for different FDA
techniques. FSI simulations scores for individual intents can be found in
Appendix’s Table 7. We observe that upsampling in text space improves the no
augmentation baseline for all intents. The mean accuracy score improves from
$87.46$ to $94.38$. We also observe that different FDA techniques further
improve model accuracy. Interestingly, upsampling in text space helps DeltaR
the most. Surprisingly, upsampling in latent space provides better performance
than upsampling in the text space. In particular, without upsampling the seed
examples to learn the feature extractor, the best score is $95.88$ for DeltaS,
whereas with text space upsampling the best score decreases to $94.88$. This
decrease in performance is only seen with BERT and not with the Bi-LSTM
feature extractor (see Table 4). We hypothesize that upsampling text data
leads to BERT overfitting the target category which results in less
generalized sentence representations. Overall, we found that augmentation in
the latent space seems to work better with BERT, and is more effective than
text space upsampling.

###  5.5 Effect Of The Pre-trained BERT Encoder

In FSI setting, Fine-Tuned BERT model provides very good generalization
performance. For example, for SNIPS’s RateBookIntent (column Book in Table 5),
it yields $96.81\%$ accuracy. Overall for BERT representations, Linear and
DeltaS augmentation methods provide the best accuracy.

To investigate whether these augmentation improvements can be generalized to
other sentence encoders, we experiment with a Bi-LSTM sentence encoder. For
feature learning, we use a 1-layer Bi-LSTM encoder followed by a single layer
softmax classifier. In our experiments, we use $128$ as hidden units and $300$
dimension Glove embeddings. For SNIPS dataset, we use $10$ examples of
AddToPlaylist intent and for FB Dialog dataset, we use $10$ examples of
GetDirections intent.

Table 4 shows intent accuracy for SNIPS and Facebook datasets. We find that,
unlike BERT, in the FSI setting, the Bi-LSTM encoder provides a lower
accuracy. In contrast to BERT FSI experiments, DeltaS performs worse than the
Upsample and Perturb baselines. The main reason is that Delta-Encoder’s
performance relies on a good feature extractor and with $10$ seed examples,
the Bi-LSTM encoder fails to learn good sentence representations. To improve
representation learning, we upsample $10$ utterances to $100$ and then train
the feature extractor. Upsampling in text space improves the performance of
both delta encoder methods, DeltaS, and DeltaR. Moreover, for both SNIPS’s
AddToPlayList and FBDialog’s GetDirections intent, DeltaR outperforms all
other FDA methods.

Size | Method | SNIPS’s AddToPlaylist | FBDialog’s GetDirections
---|---|---|---
seed examples ($k$) | 10 | $100^{*}$ | 10 | $100^{*}$
No Augmentation | 80.07 (2.08) | 90.17 (1.39) | 87.44 (0.12) | 87.94 (0.32)
100 | Upsample |  88.27 (1.74) | 90.61 (1.52) | 88.01 (0.26) | 88.17 (0.32)
Perturb | 88.03 (1.52) | 90.86 (1.39) | 88.01 (0.32) | 88.25 (0.31)
Linear | 88.14 (1.62) | 91.06 (1.58) | 88.05 (0.25) | 88.26 (0.32)
Extra | 88.09 (1.57) | 90.74 (1.57) |  88.10 (0.29) | 88.20 (0.3)
CVAE |  88.27 (2.08) | 90.90 (1.69) | 88.04 (0.24) | 88.17 (0.32)
DeltaR | 82.23 (2.21) |  91.46 (1.19) | 87.60 (0.23) |  88.75 (0.43)
DeltaS | 84.4 (2.74) | 91.07 (1.44) | 88.02 (0.22) | 88.57 (0.36)
512 | Upsample | 91.41 (1.03) | 91.61 (1.4) | 88.68 (0.49) | 88.40 (0.35)
Perturb |  91.46 (0.99) | 91.73 (1.32) | 88.89 (0.57) | 88.56 (0.39)
Linear | 91.20 (1.28) | 91.41 (1.52) | 88.97 (0.65) | 88.47 (0.33)
Extra | 91.26 (1.22) | 91.57 (1.55) | 88.85 (0.61) | 88.48 (0.37)
CVAE | 91.39 (0.94) | 91.44 (1.2) | 89.02 (0.52) | 88.48 (0.4)
DeltaR | 87.09 (2.75) |  92.97 (1.2)) | 88.61 (0.35) |  89.70 (0.53)
DeltaS | 89.34 (1.48) | 92.00 (1.25) |  89.34 (0.4) | 89.09 (0.51)

Table 4: IC accuracy on SNIPS’s AddToPlaylist and FBDialog’s GetDirections in
the FSI setting, reported as mean (SD). A $1$-layer Bi-LSTM model is used as a
feature extractor. $100^{*}$ represents $10$ seed examples are upsampled to
$100$ to train the feature extractor.

###  5.6 Is Delta-Encoder Effective On Text?

While on few-shot image classification, Delta-Encoder provides excellent
generalization performance Schwartz et al. (2018) on unseen classes, on text
classification, its performance is heavily dependent on the feature extractor.
We observe that in most cases, DeltaR performs worse than DeltaS which
suggests that unlike for few-shot image classification, Delta-Encoder fails to
learn variations which can be applied to a different category. In addition, in
FSI with BERT encoder, DeltaS performance is close to Linear. This indicates
that in the low-data regime, simple subtraction between BERT sentence
representations is a good proxy to learn intra-class variations. Upsampling
data in text space improves Delta-Encoder performance for both BERT and Bi-
LSTM encoders. As shown in Table 3, with upsampling in text space, DeltaR
performs better than any other FDA method.

###  5.7 Qualitative Evaluation

Figure 3: $10$ seed examples

Figure 4: $10$ seed examples are upsampled to $100$

Figure 5: t-SNE visualization of different data augmentation methods for
AddToPlaylist intent. BERT encoder is used to learn sentence representations.

We observe significant accuracy improvements in all FSI experiments for all
FDA methods. Since Upsample and Perturb also provide significant gains, it
seems that most of the gains come from the fact that we are adding more data.
However, in the FSI setting, Linear and DeltaS method consistently perform
better than both Upsample and Perturb, which indicates that these methods
generate more relevant data than just noise, and redundancy. Here, we focus on
visualizing generated examples from Linear, DeltaS and DeltaR methods using
t-SNE.

Figure 5 shows visualizations for SNIPS’s AddToPlaylist generated sentence
representations using different FDA methods. We use $10$ seed examples of
AddToPlaylist and use BERT as sentence encoder. While data generated by Linear
and Extra are close to the real examples, DeltaS and DeltaR generated examples
form two different clusters. Since, Delta-Encoder performance improves when
seed examples are upsampled in text space, we plot sentence examples from
upsampled data.

Figure 5 shows that when $10$ seed examples are upsampled to $100$, DeltaS
cluster moves closer to the seed examples, and while most of the DeltaR
generated data forms a separate cluster, a few of the generated examples are
close to the seed examples. Since, in experiments with upsampled text
examples, DeltaR performs better than other FDA methods, we hypothesize that
DeltaR increases the amount of variability within the dataset by generating
diverse examples which leads to a more robust model.

##  6 Conclusion and Future Work

In this paper, we investigate six FDA methods including Upsample, Perturb,
CVAE, Delta-Encoder, Extra, and Linear to augment training data. We show that
FDA works better when combined with transfer learning and provides an
effective way of bootstrapping an intent classifier for new classes. As
expected, all FDA methods become less effective when the number of seed
examples increases and provides minor gains in the full-data regime. Through
comparing methods on two public datasets, our results show that Linear is a
competitive baseline for FDA in FSI setting, especially when combined with
transfer learning (BERT).

Additionally, we provide empirical evidence that in few-shot integration
setting, feature space augmentation combined with BERT provides better
performance than widely used text space upsampling. Given that pre-trained
language models provide state of the art performance on several NLP tasks, we
find this result to be in particular encouraging, as it shows potential for
applying FDA methods to other NLP tasks.

Our experiments on Delta-Encoder also shows that unlike few-shot image
classification, Delta-Encoder fails to learn transferable intra-class
variations. This result emphasizes that methods providing improvements in
computer vision domain might not produce similar gains on NLP tasks, thus
underlining the need to develop data augmentation methods specific to NLP
tasks.

## References

  * Antoniou et al. (2018) Anthreas Antoniou, Amos Storkey, and Harrison Edwards. 2018.  Data augmentation generative adversarial networks.
  * Chawla et al. (2002) Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002\.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_.
  * Cheng et al. (2019) Yu Cheng, Mo Yu, Xiaoxiao Guo, and Bowen Zhou. 2019.  Few-shot learning with meta metric learners.
  * Cho et al. (2019a) Eunah Cho, He Xie, and William M Campbell. 2019a.  Paraphrase generation for semi-supervised learning in nlu.  In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_.
  * Cho et al. (2019b) Eunah Cho, He Xie, John Lalor, Varun Kumar, and William M Campbell. 2019b.  Efficient semi-supervised learning for natural language understanding by optimizing diversity.  In _IEEE Automatic Speech Recognition and Understanding Workshop_.
  * Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018.  Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces.  _arXiv preprint arXiv:1805.10190_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Devries and Taylor (2017) Terrance Devries and Graham W. Taylor. 2017.  Dataset augmentation in feature space.  _ArXiv_ , abs/1702.05538.
  * Estabrooks et al. (2004) Andrew Estabrooks, Taeho Jo, and Nathalie Japkowicz. 2004.  A multiple resampling method for learning from imbalanced data sets.  _Computational intelligence_.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_. JMLR.org.
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018.  Slot-gated modeling for joint slot filling and intent prediction.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics_.
  * Guo et al. (2019) Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019.  Augmenting data with mixup for sentence classification: An empirical study.  _arXiv preprint arXiv:1905.08941_.
  * Gupta (2019) Rahul Gupta. 2019.  Data augmentation for low resource sentiment analysis using generative adversarial networks.  In _IEEE International Conference on Acoustics, Speech and Signal Processing_.
  * Gupta et al. (2018) Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018.  Semantic parsing for task oriented dialog using hierarchical representations.  _arXiv preprint arXiv:1810.07942_.
  * Guu et al. (2018) Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018.  Generating sentences by editing prototypes.  _Transactions of the Association of Computational Linguistics_.
  * Hariharan and Girshick (2017) Bharath Hariharan and Ross Girshick. 2017.  Low-shot visual recognition by shrinking and hallucinating features.  In _Proceedings of the International Conference on Computer Vision_.
  * Hou et al. (2018) Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018.  Sequence-to-sequence data augmentation for dialogue language understanding.  _arXiv preprint arXiv:1807.01554_.
  * Jiang et al. (2018) Xiang Jiang, Mohammad Havaei, Gabriel Chartrand, Hassan Chouaib, Thomas Vincent, Andrew Jesson, Nicolas Chapados, and Stan Matwin. 2018.  Attentive task-agnostic meta-learning for few-shot text classification.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML Deep Learning Workshop_.
  * Kurata et al. (2016) Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016.  Labeled data generation with encoder-decoder lstm for semantic slot filling.  In _INTERSPEECH_.
  * Li et al. (2017) Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. 2017.  Meta-sgd: Learning to learn quickly for few-shot learning.  _arXiv preprint arXiv:1707.09835_.
  * Lim et al. (2018) Swee Kiat Lim, Yi Loo, Ngoc-Trung Tran, Ngai-Man Cheung, Gemma Roig, and Yuval Elovici. 2018.  Doping: Generative data augmentation for unsupervised anomaly detection with gan.  In _2018 IEEE International Conference on Data Mining_.
  * Mehrotra and Dukkipati (2017) Akshay Mehrotra and Ambedkar Dukkipati. 2017.  Generative adversarial residual pairwise networks for one shot learning.
  * Peshterliev et al. (2018) Stanislav Peshterliev, John Kearney, Abhyuday Jagannatha, Imre Kiss, and Spyros Matsoukas. 2018.  Active learning for new domains in natural language understanding.  _arXiv preprint arXiv:1810.03450_.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Ravi and Larochelle (2016) Sachin Ravi and Hugo Larochelle. 2016.  Optimization as a model for few-shot learning.
  * Rippel et al. (2015) Oren Rippel, Manohar Paluri, Piotr Dollar, and Lubomir Bourdev. 2015.  Metric learning with adaptive density discrimination.  _arXiv preprint arXiv:1511.05939_.
  * Schwartz et al. (2018) Eli Schwartz, Leonid Karlinsky, Joseph Shtok, Sivan Harary, Mattias Marder, Abhishek Kumar, Rogerio Feris, Raja Giryes, and Alex Bronstein. 2018.  Delta-encoder: an effective sample synthesis method for few-shot object recognition.  In _Advances in Neural Information Processing Systems_.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical networks for few-shot learning.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. 2016.  Matching networks for one shot learning.  In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, _Advances in Neural Information Processing Systems_.
  * Wang et al. (2018) Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. 2018.  Low-shot learning from imaginary data.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_.
  * Yoo et al. (2018) Kang Min Yoo, Youhyun Shin, and Sang goo Lee. 2018.  Data augmentation for spoken language understanding via joint variational generation.
  * Yu et al. (2018) Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018.  Diverse few-shot text classification with multiple metrics.  In _Proceedings of the North American Chapter of the Association for Computational Linguistics_.
  * Zhang et al. (2018) Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. 2018\.  Metagan: An adversarial approach to few-shot learning.  In _Advances in Neural Information Processing Systems_.
  * Zhao et al. (2019) Zijian Zhao, Su Zhu, and Kai Yu. 2019.  Data augmentation with atomic templates for spoken language understanding.  _arXiv preprint arXiv:1908.10770_.

##  Appendix A FSI experiment results for all intents

In all tables, individual columns represent FSI results for an intent, and
Overall Mean column, provides average accuracy for all intents’ FSI
simulations.

# | Method | Playlist | Restaurant | Weather | Music | Book | Work | Event | Overall Mean
---|---|---|---|---|---|---|---|---|---
No Augmentation | 82.63(5.11) | 87.86(3.53) | 84.51(1.3) | 88.07(2.37) | 96.81(2.94) | 85.14(1.53) | 87.19(3.31) | 87.46(2.87)
100 | Upsample | 92.24(2.96) | 97.7(0.67) | 96.44(0.75) | 94.57(1.1) | 97.96(0.82) | 89.61(3.01) | 91.26(2.35) | 94.26(1.66)
Perturb |  93.09(2.55) | 97.41(0.92) | 96.07(1.35) | 94.39(1.13) | 97.86(0.93) | 89.36(2.76) | 91.09(2.53) | 94.18(1.74)
CVAE | 92.4(3.66) | 97.47(0.67) | 96.49(1.07) | 94.36(1.26) | 97.71(1.1) | 89.1(2.79) | 91.2(2.22) | 94.1(1.83)
Linear | 92.61(3.02) | 97.74(0.67) | 96.44(0.77) | 94.63(1.18) |  97.97(0.78) |  89.61(3.05) | 91.53(2.34) |  94.36(1.69)
Extra | 92.36(3.0) | 97.74(0.66) | 96.41(0.77) | 94.6(1.18) | 97.97(0.78) | 89.47(3.11) | 91.51(2.3) | 94.3(1.68)
| DeltaR | 87.07(4.67) | 93.57(4.07) | 91.0(4.23) | 94.87(1.28) | 97.66(1.42) | 85.97(2.34) | 89.11(3.84) | 91.32(3.12)
| DeltaS | 92.64(4.49) |  97.76(0.7) | 96.41(1.25) |  94.99(0.92) | 97.83(0.99) | 88.69(2.69) |  91.64(2.36) | 94.28(1.92)
512 | Upsample | 95.3(1.09) | 98.0(0.64) | 97.63(0.34) | 95.57(0.87) | 98.03(0.55) | 92.0(1.49) | 93.26(1.05) | 95.68(0.86)
Perturb | 95.33(1.2) | 97.94(0.6) | 97.6(0.44) | 95.5(0.91) | 97.91(0.55) | 92.03(1.78) | 93.21(0.99) | 95.65(0.92)
CVAE | 95.46(1.12) | 97.89(0.62) | 97.54(0.43) | 95.36(1.02) | 97.93(0.7) | 91.34(2.17) | 92.73(1.19) | 95.46(1.03)
Linear | 95.39(1.1) |  98.0(0.64) | 97.67(0.36) | 95.74(0.89) |  98.04(0.5) |  92.61(1.47) | 93.66(1.13) |  95.87(0.87)
Extra | 95.36(1.17) | 98.0(0.64) | 97.66(0.37) | 95.74(0.88) | 98.04(0.5) | 92.29(1.52) | 93.63(1.17) | 95.82(0.89)
| DeltaR | 95.36(1.74) | 97.81(0.69) | 97.6(0.44) | 95.9(0.97) | 97.74(1.02) | 90.27(3.44) | 92.61(2.64) | 95.33(1.56)
| DeltaS |  95.66(1.18) | 97.96(0.59) |  97.8(0.45) |  95.91(0.88) | 97.91(0.74) | 92.26(2.57) |  93.66(0.86) |  95.88(1.04)

Table 5: IC accuracy on SNIPS dataset in the FSI setting ($k=10$), reported as
mean (SD).

# | Method | Directions | Distance | Arrival | Duration | Traffic | Event | Overall Mean
---|---|---|---|---|---|---|---|---
No Augmentation | 89.61(0.1) | 89.94(0.09) | 90.56(0.12) | 81.74(0.13) | 68.5(0.13) | 67.39(0.11) | 81.29(0.11)
100 | Upsample | 89.89(0.27) | 93.64(0.87) | 92.95(0.57) | 84.28(3.45) | 68.99(0.49) |  76.26(5.41) |  84.34(1.84)
Perturb | 89.82(0.24) | 93.58(0.84) | 92.81(0.55) |  84.81(3.77) |  69.15(0.68) | 74.07(5.6) | 84.04(1.95)
CVAE | 89.91(0.32) | 93.46(0.77) | 92.7(0.67) | 84.45(3.52) | 69.11(0.9) | 74.94(5.49) | 84.1(1.94)
Linear |  89.93(0.24) |  93.65(0.88) |  92.98(0.57) | 84.2(3.44) | 68.96(0.51) | 76.12(5.77) | 84.31(1.9)
Extra | 89.88(0.27) | 93.61(0.89) | 92.96(0.59) | 84.21(3.43) | 68.94(0.46) | 75.18(5.34) | 84.13(1.83)
| DeltaR | 89.64(0.11) | 92.57(1.3) | 90.79(0.37) | 81.72(0.12) | 68.48(0.08) | 68.63(2.59) | 81.97(0.76)
| DeltaS | 89.88(0.34) | 93.68(0.72) | 92.6(0.76) | 83.88(3.2) | 68.93(0.67) | 72.05(5.83) | 83.5(1.92)
512 | Upsample | 91.93(0.48) | 94.58(0.34) | 93.99(0.31) | 92.56(0.72) | 75.84(2.19) | 85.27(1.87) | 89.03(0.99)
Perturb | 91.78(0.49) | 94.58(0.43) | 94.02(0.25) | 92.53(0.87) |  76.0(2.27) | 85.22(1.61) | 89.02(0.99)
CVAE | 91.85(0.52) | 94.57(0.39) | 94.0(0.34) | 92.45(0.92) | 74.91(2.73) | 84.5(1.61) | 88.71(1.09)
Linear | 92.14(0.66) | 94.6(0.35) | 94.05(0.32) |  92.78(0.67) |  76.0(2.49) | 86.22(1.7) |  89.3(1.03)
Extra |  92.11(0.57) | 94.61(0.35) | 94.04(0.29) | 92.72(0.7) | 75.79(2.45) | 85.98(1.58) | 89.21(0.99)
| DeltaR | 90.43(0.55) | 94.54(0.35) | 93.8(0.3) | 86.64(4.38) | 71.68(1.46) |  86.55(1.75) | 87.28(1.46)
| DeltaS | 91.83(0.47) |  94.66(0.4) |  94.08(0.24) | 92.31(1.45) | 75.81(2.1) | 86.23(2.08) | 89.15(1.12)

Table 6: IC accuracy on FBDialog dataset in the FSI setting ($k=10$), reported
as mean (SD).

# | Method | Playlist | Restaurant | Weather | Music | Book | Work | Event | Overall Mean
---|---|---|---|---|---|---|---|---|---
No Augmentation | 96.0(1.69) | 95.39(1.59) | 96.41(1.18) | 93.1(1.38) | 97.79(0.77) | 88.46(1.14) | 93.49(0.87) | 94.38(1.23)
100 | Upsample | 96.0(1.57) | 95.87(1.26) |  96.51(1.04) | 93.19(1.25) | 97.83(0.7) | 88.63(1.21) | 93.7(0.83) | 94.53(1.12)
Perturb | 96.1(1.64) | 95.7(1.23) | 96.43(1.28) | 93.33(1.1) | 97.8(0.77) | 88.56(1.32) | 93.7(0.9) | 94.52(1.18)
CVAE | 96.07(1.46) |  95.91(1.43) | 96.43(1.31) | 93.2(1.15) | 97.83(0.78) | 88.63(1.28) | 93.66(0.86) | 94.53(1.18)
Linear | 96.0(1.57) | 95.89(1.26) |  96.51(1.04) | 93.19(1.25) | 97.83(0.7) | 88.63(1.21) | 93.7(0.83) | 94.53(1.12)
Extra | 96.0(1.57) | 95.84(1.3) |  96.51 (1.04) | 93.19(1.25) | 97.83(0.7) | 88.63(1.21) | 93.7(0.83) | 94.53(1.13)
| DeltaR | 96.09(1.51) | 95.74(1.46) | 96.44(1.29) |  93.56(0.95) |  97.86(0.75) |  88.79(1.25) |  93.86(0.93) |  94.62(1.16)
| DeltaS |  96.11(1.52) | 95.69(1.44) | 96.46(1.29) | 93.44(0.93) |  97.86(0.75) | 88.64(1.18) | 93.76(0.89) | 94.57(1.14)
512 | Upsample | 96.07(1.54) | 96.09(1.2) | 96.6(1.06) | 93.5(1.14) |  97.87(0.69) | 88.73(1.23) | 93.8(0.92) | 94.67(1.11)
Perturb | 96.23(1.6) | 96.17(1.23) | 96.63(1.13) | 93.49(1.03) | 97.84(0.72) | 88.6(1.3) | 93.79(0.98) | 94.68(1.14)
CVAE | 96.14(1.46) | 96.24(1.18) | 96.63(1.06) | 93.6(1.08) | 97.87(0.75) | 88.76(1.29) | 93.87(0.98) | 94.73(1.11)
Linear | 96.07(1.54) | 96.11(1.21) | 96.6(1.06) | 93.49(1.13) |  97.87(0.69) | 88.76(1.25) | 93.8(0.92) | 94.67(1.11)
Extra | 96.07(1.54) | 96.13(1.18) | 96.6(1.06) | 93.5(1.14) |  97.87(0.69) | 88.73(1.25) | 93.8(0.92) | 94.67(1.11)
| DeltaR |  96.29(1.52) |  96.29(1.34) |  96.71(1.1) |  93.87(1.04) | 97.86(0.75) |  89.11(1.22) |  94.03(0.89) |  94.88(1.12)
| DeltaS | 96.19(1.61) | 96.2(1.23) | 96.69(1.07) | 93.61(0.96) | 97.86(0.75) | 88.84(1.28) | 93.83(0.94) | 94.74(1.12)

Table 7: IC accuracy on SNIPS dataset in the FSI setting, reported as mean
(SD). The $10$ seed examples are upsampled to $100$ to train the feature
extractor.


# LatentSwap: An Efficient Latent Code Mapping Framework for Face Swapping

Changho Choi1, Minho Kim211footnotemark: 1, Junhyeok Lee3, Hyoung-Kyu Song4,
Younggeun Kim5, Seungryong Kim1 Equal contribution.Corresponding author.

###### Abstract

We propose LatentSwap, a simple face swapping framework generating a face swap
latent code of a given generator. Utilizing randomly sampled latent codes, our
framework is light and does not require datasets besides employing the pre-
trained models, with the training procedure also being fast and
straightforward. The loss objective consists of only three terms, and can
effectively control the face swap results between source and target images. By
attaching a pre-trained GAN inversion model independent to the model and using
the StyleGAN2 generator, our model produces photorealistic and high-resolution
images comparable to other competitive face swap models. We show that our
framework is applicable to other generators such as StyleNeRF, paving a way to
3D-aware face swapping and is also compatible with other downstream StyleGAN2
generator tasks. The source code and models can be found at
https://github.com/usingcolor/LatentSwap.

## Introduction

Face swapping aims to produce an image that has the identity of the source
image while maintaining the target image attributes (e.g., facial expression,
background, and pose).

While pioneering approaches primarily used 3D priors (Blanz et al. 2004; Lin
et al. 2012; Yang, Li, and Lyu 2019), applying GANs (Goodfellow et al. 2014)
has produced more realistic face swap images (Petrov et al. 2020; Bao et al.
2018; Li et al. 2020; Chen et al. 2020; Wang et al. 2021; Kim, Lee, and Zhang
2022; Zhu et al. 2021; Xu et al. 2022a). These applications focus on
preserving the source identity information by using independent identity
embedders such as ArcFace (Deng et al. 2019), or preserving the shape and
other facial attributes by using 3D face models such as 3DMM (Blanz and Vetter
1999). Albeit producing credible results, these models crucially suffer from
the typical GAN’s slow and unstable training. Furthermore, each model using
different datasets for training makes reproducing relevant results difficult.
To solve this problem, recent works proposed a smooth identity embedder (Kim,
Lee, and Zhang 2022), or attempted to incorporate pre-trained StyleGAN2
(Karras et al. 2020) weights to take advantage of StyleGAN2’s stability and
performance. In a new direction, there also have been attempts to use
diffusion probability models to perform face swapping (Kim et al. 2022).

To utilize the stability and performance of StyleGAN2 generators into face
swapping task, recent works (Zhu et al. 2021; Xu et al. 2022a, c, b) focused
on translating images to relevant latent codes that can be piped into the
StyleGAN2 generator, which typically employ identity encoders inspired by GAN
inversion techniques. While these models achieved some success, they tend to
require additional auxiliary modules such as face segmentation models or
inputs such as face parsing models. Consequently, such networks do not exploit
the full potential of the StyleGAN2 latent space.

In this paper, we propose an efficient face swapping model, where combined
with a pre-trained GAN inversion model can produce face swap images given
arbitrary source and target images. With a simple module dubbed the latent
mixer consisted of $5$ fully-connected layers and a residual connection, our
model benefits from short and stable training. Furthermore, our model only
takes randomly sampled source and target generator latent code pairs for
training, and does not require a dataset besides using the information from
pretrained models. For optimal performance we attached our model with PTI
(Roich et al. 2021) and the StyleGAN2 generator, benefitting from the full
advantages of the StyleGAN2 latent space. We show that controllable generation
between source and target is possible by systematically modifying loss
objectives, and that specific attributes of the face swapping are disentangled
to particular layers of the latent code. Our framework is applicable to other
generators such as StyleNeRF (Gu et al. 2022) and can be used concurrently
with other downstream tasks such as editing specific attributes. The resultant
face swap results are high quality and realistic, comparable to state-of-the-
art models.

Our contribution can be summarized as follows:

  * •

We propose a simple and fast-training framework called the latent mixer to
generate face swap latent codes for a generator, which does not require
additional dataset for the purpose to train our model.

  * •

While fully exploiting the applied generator’s latent space, our module can
generate face swap images for real, wild images by utilizing an arbitrary pre-
trained GAN Inversion network compatible with the generator.

  * •

Our framework allows for controlling generation by tweaking coefficients of
three simple loss functions.

  * •

Our model is applicable to downstream tasks such as editing and to other
generators such as StyleNeRF, opening up possible applications towards 3D face
swapping.

## Related Work

#### GAN Inversion.

GAN Inversion is a technique to map a real image to a generator space latent
code, which can then be further used for generator-related downstream tasks
such as editing (Härkönen et al. 2020; Voynov and Babenko 2020). Naturally,
after the advent of StyleGAN2, many GAN Inversion frameworks mapping real
images onto the FFHQ domain space have been developed.

The first approaches utilized optimization, where Image2StyleGAN (Abdal, Qin,
and Wonka 2019) and Image2StyleGAN++ (Abdal, Qin, and Wonka 2020) directly
minimized the reconstruction distance from optimizing the latent codes, which
subsequently led to the introduction of the $\mathcal{W+}$ space as well.
These approaches achieve acceptable quality inversion results but in general
take long times to generate such reconstructed images.

Later on, learning-based methods (Tov et al. 2021a; Richardson et al. 2021;
Alaluf, Patashnik, and Cohen-Or 2021; Alaluf et al. 2022) incorporated encoder
networks to map the input image into latent space which can speed up the
training and inference procedures significantly. However, compared to
optimization-based methods it does not directly manipulate the latent code,
hence lacks details and often requires a further optimization process. For
example, PTI (Roich et al. 2021) optimizes the generator parameters after
using e4e (Tov et al. 2021a) to find the $\mathcal{W+}$ latent space latent
code which leads to better visual quality.

#### Latent Space Manipulation.

Latent space editing uses a learned manifold of GAN (Goodfellow et al. 2014)
to control the image attributes such as age, pose, and expression.

Latent space editing methods can employ supervision, namely labeled datasets
or pre-trained classifiers, to find the directions on the latent space which
corresponds to the attributes (Shen et al. 2020; Abdal et al. 2021; Patashnik
et al. 2021). InterFaceGAN (Shen et al. 2020) disentangled the latent space
employing linear transformations, while StyleFlow (Abdal et al. 2021) adopted
a normalizing flow model. StyleClip (Patashnik et al. 2021) uses CLIP (Radford
et al. 2021) guided loss to edit via a text prompt. To the contrary,
unsupervised methods search the directions first and analyze which attributes
are associated with them (Härkönen et al. 2020; Shen and Zhou 2021; Voynov and
Babenko 2020; Wang and Ponce 2021). Examples include GANSpace (Härkönen et al.
2020) which performed PCA on latent codes, SeFa (Shen and Zhou 2021) which
used eigenvector decomposition, and Voynov et al.(Voynov and Babenko 2020)
which utilized mutual information.

## Methodology

[width=0.80]Figures/teaser_ffhq_testset

Figure 1: Face swapping results by LatentSwap. The model, given a source and a
target image, replaces the identity of the target image (each row) to the
identity of the source image (each column) while maintaining the attributes of
the target images such as background and lighting. More results can be found
in supplementary material.

Given a generator trained on a face dataset, we attach a simple yet effective
module to combine source and target latent codes to perform face swapping.
Unlike previous methodologies (Dee 2021; Bao et al. 2018; Chen et al. 2020; Li
et al. 2020; Wang et al. 2021; Xu et al. 2022a; Zhu et al. 2021; Kim, Lee, and
Zhang 2022), we do not require any dataset for training. During training, the
source/target latent codes are randomly sampled, and face swapping is
performed using only these two latent codes as input. For inference, we apply
a pre-trained GAN inversion model to map the real images onto the generator
latent space. Besides the pre-trained inversion model, the generator, and the
identity embedder (Kim, Lee, and Zhang 2022) just to compute the training
loss, no other pre-trained models have been used. We use the StyleGAN2 (Karras
et al. 2020) generator and work on its latent space, due to its versatility
and good generation quality.

Notations. On an arbitrary space $\mathcal{A}$, we denote the source and
target latent codes and the swapped latent code formed by the aforementioned
source and target respectively as $l_{\text{s}}^{\mathcal{A}}$,
$l_{\text{t}}^{\mathcal{A}}$, and $l^{\mathcal{A}}_{\text{swap}}$. The latent
mixer $\mathcal{F}^{\mathcal{A}}$ generates the swapped latent code
$l_{\text{swap}}^{\mathcal{A}}=\mathcal{F}^{\mathcal{A}}(l_{\text{s}}^{\mathcal{A}}$,
$l_{\text{t}}^{\mathcal{A}})$. The generator $G$ produces the corresponding
image $I$ given any $18\times 512$ dimensional vector:
$I=G(l),\,l\in\mathbb{R}^{18\times 512}$. We drop the superscript
$\mathcal{W+}$ for swapped latent codes produced from $\mathcal{W+}$ latent
space, unless distinctions are required.

### Latent Mixer Architecture

The goal of the latent mixer is to allow the swapped code to sufficiently
capture the source’s identity without letting it deviate significantly from
the target’s attributes with small computational resources.

Each latent mixer consists of $5$ fully-connected layers with Swish
(Ramachandran, Zoph, and Le 2017) activation. The latent mixer takes the
source and the target latent codes as input. The two latent codes are then
concatenated, and it goes through $5$ fully-connected layers in sequence. This
output is then added to the target latent code to produce the final swap
latent code. Each latent mixer operates on each of the 18 layers of the latent
codes on $\mathcal{W+}$ space. We add the target latent code to the output of
the last fully-connected layer (He et al. 2016) to generate the final swapped
latent code $l_{\text{swap}}$. It is possible to control the residual
connection coefficient, i.e., how much
$l_{\text{t}_{\text{i}}}^{\mathcal{W+}}$ to add: see Appendix 1 for details.
Detailed structure of the latent mixer is illustrated in Figure 2.

[width=0.98]Figures/mix_generator_pdf

Figure 2: Overall training scheme of the LatentSwap model. We sample source
and target latent codes randomly from a normal distribution, which are mapped
onto $\mathcal{W}$ and subsequently copied 18-fold as a vector on
$\mathcal{W+}$ space. The $\mathcal{W+}$ space vector is piped into the latent
mixer (whose detailed structure is in the black dotted box). The swapped
latent codes are then fed into the generator (the StyleGAN2 pre-trained
weights) at different layers to generate the final face-swapped image. The
gradient can still flow through the generator, but it remains freezed
throughout training.

While we can generate quality face swapped images by only using $2$ or $3$
layers, we note that the optimal performance between training speed and
performance is at $5$ fc layers. We further discuss this in supplementary
information by plotting the loss function during training and the performance
of the network for $2$, $3$, $5$, and $7$ layers, which shows that quality
improvement decreases as layers increase, while the increasing network size
burdens the training speed.

### Training Procedure

First, we sample the latent code for the source and target from a standard
normal distribution:
$l_{\text{s}}^{\mathcal{Z}},l_{\text{t}}^{\mathcal{Z}}\sim\mathcal{Z}=\mathcal{N}(\bm{0},\bm{I})$,
where
$l_{\text{s}}^{\mathcal{Z}},l_{\text{t}}^{\mathcal{Z}}\in\mathbb{R}^{512}$. We
transform this to $\mathcal{W}$ space via 8 pre-trained fc layers, as used in
StyleGAN2, which are also $512$-dimensional:
$l_{\text{s}}^{\mathcal{W}},l_{\text{t}}^{\mathcal{W}}\in\mathcal{W}\subset\mathbb{R}^{512}$.
This latent code is then duplicated $18$ times, resulting in a latent code on
the $\mathcal{W+}$ space consisting of $18$ layers, with each layer consisting
of an identical $512$-dimensional vector: a $\mathcal{W+}$ space latent code
therefore has dimension $18\times 512$. We note that the general
$\mathcal{W+}$ space is larger, which is a set of any concatenation of 18
arbitrary latent codes on the $\mathcal{W}$ space (Abdal, Qin, and Wonka
2019). Note for inference, PTI directly produces the $\mathcal{W}$-space
latent codes.

Each layer of the W+ space is followed by a latent mixer, which produces a
layer-wise swapped latent code. We then collect these layer-wise codes from
all the $18$ latent mixers and concatenate them to obtain the full $18\times
512$-dimensional swapped face latent code $l^{\mathcal{W+}}_{\text{swap}}$.

After obtaining the face swap latent code, we calculate the training loss and
backpropagate the gradients through the model $G$. We only update the weights
in the latent mixer using the optimizer while keeping the StyleGAN2 weights
constant. The gradient flows back to each of the latent mixers at different
points of the generator, achieving layer-wise distinction that corresponds to
different resolutions of the image (Karras, Laine, and Aila 2019). Figure 2
denotes this mechanism graphically.

### Loss Functions

We use three simple loss functions to train our model, consisting of the
identity loss $\mathcal{L}_{\text{id}}$, latent penalty loss
$\mathcal{L}_{\text{lp}}$, and the shape loss $\mathcal{L}_{\text{s}}$ added
together:

$\mathcal{L}=\mathcal{L}_{\text{id}}+\lambda\mathcal{L}_{\text{lp}}+\mu\mathcal{L}_{\text{s}}.$
(1)

where we set $\lambda=10^{2}$ as the default value and fix $\mu=0.1$, which is
set empirically so that the shape loss term will be similar size to the other
loss terms, indicating that all three loss terms will contribute during
training.

Identity Loss. The identity loss $\mathcal{L}_{\text{id}}$ (ID loss) is
defined following the typical conventions of general face swapping models (Li
et al. 2020; Chen et al. 2020; Zhu et al. 2020; Wang et al. 2021; Xu et al.
2021; Kim, Lee, and Zhang 2022) as the cosine distance between the identity
embeddings of the source and swapped images. Therefore, the ID loss helps the
swapped face to share the same identity as the source. This can be written as:

$\mathcal{L}_{\text{id}}=1-\cos(\textrm{ID}(I_{\text{s}}),\textrm{ID}(I_{\text{swap}})).$
(2)

where $\textrm{ID}:\mathbb{R}^{3\times H\times W}\rightarrow\mathbb{R}^{512}$
denotes the identity embedder and $I_{\text{s}}$, $I_{\text{swap}}$ denotes
the source and swapped images. This is described schematically in Figure 3.

[width=0.47]Figures/id_loss

Figure 3: Schematic description of the ID loss. The source and swapped images
are mapped to the Smooth-Swap (Kim, Lee, and Zhang 2022) identity embedding
space. The loss objective is to minimize the cosine distance between the
identity embeddings.

We use the identity embedder from Smooth-Swap, which has a smooth embedding
space leading to more stable and fast training 111A detailed discussion on the
performance of the Smooth-Swap embedder compared to ArcFace is provided on the
original Smooth-Swap paper.. We implemented the identity embedder ourselves
since the official implementation is not released.

Latent Penalty Loss. The equally important aspect in face swapping tasks
alongside identity preservation is to preserve target attributes. Provided
that the mappings from one latent space to another is continuous on the
StyleGAN domain space, the ideal face swap and the target latent codes should
be close in the latent space, especially considering that all attributes of
the target should ideally be preserved after face swapping (Patashnik et al.
2021). This does not necessarily mean that the $\mathcal{F}^{\mathcal{W+}}$
map will be able to generate a latent code which corresponds to the swapped
face (see results for $\mathcal{Z}$ in Figure 5 and Appendix 2.).

We define the latent penalty loss $\mathcal{L}_{lp}$ at a given latent space
as the mean squared error (MSE) distance on the $\mathcal{W+}$ space between
the target and swapped latent codes:

$\mathcal{L}_{\text{lp}}=\frac{1}{\mathrm{dim}(l_{\text{t}}^{\mathcal{W+}})}\lVert
l_{\text{t}}^{\mathcal{W+}}-l^{\mathcal{W+}}_{\text{swap}}\rVert_{2}^{2}.$ (3)

Shape Loss. Shape loss utilizes 2D landmark keypoints as geometric supervision
to accurately guide the model to constrain the face shape. Following previous
models (Wang et al. 2021), we extract the 3DMM (Blanz and Vetter 1999)
coefficients for the source and target images, and construct the fused 3DMM
coefficient by concatenating the source image’s identity and the target
image’s expression and posture to generate the fused image facial landmarks.

We define the shape loss to be the $L_{1}$ distance between the 3D face
landmarks of the fused facial landmark $q_{\text{fuse}}$ and the 3D face
landmarks of the swapped image $q_{\text{swap}}$, which could be written as

$\mathcal{L}_{\text{s}}=\lVert q_{\text{swap}}-q_{\text{fuse}}\rVert_{1}.$ (4)

### Incorporating Real Images

The LatentSwap framework operates on the generator latent space. Therefore, we
require a GAN Inversion network to map the real image to the latent space. For
this, we use PTI (Roich et al. 2021) on optimization mode to map the real
image to the $\mathcal{W}$ space latent code. While it is possible to directly
map onto $\mathcal{W+}$ space using e4e (Tov et al. 2021b) as the encoder
network, we note that during the training process we opted to copy the
$\mathcal{W}$ space latent codes to create a vector on $\mathcal{W+}$ space,
and directly mapping onto $\mathcal{W+}$ space breaks this symmetry. After PTI
generates the corresponding $\mathcal{W}$ space latent codes for the source
and target images, we then optimize the StyleGAN2 weight using the generated
$\mathcal{W}$ space source and target latent codes. The optimized weight is
then used to generate the final face swap image.

## Experiments

### Implementation Details

We use the FFHQ pre-trained StyleGAN2 (Karras et al. 2020) weights as our
generator, which produces $1024\times 1024$ images. We first analyze the
results when the latent mixer takes $\mathcal{W+}$ space vectors.
Subsequently, we study the effects of latent spaces, namely
$\mathcal{Z}/\mathcal{W}$, and $\lambda$. We then perform layer-wise analysis
by applying swapped latent codes at specific resolutions and apply editing
techniques.

The batch size is 64 source-target pairs, of which there is a $1/8$
probability that the pair consists of the same latent codes for each source-
target pair222Not to be interpreted as $8$ pairs within a batch of $64$ are
selected to have the same source and target.. We follow the procedure outlined
in Smooth-Swap(Kim, Lee, and Zhang 2022) to train the identity embedder, which
is used to calculate the ID loss $\mathcal{L}_{\text{id}}$. We train for
$200$k steps with the AdamW (Loshchilov and Hutter 2017) optimizer and a
learning rate of $lr=0.0001$ on $2$ NVIDIA A$100$ $80$GB GPUs on a DGX
Station. For our default settings using $18$ latent mixers, training takes
approximately $77$ hours. The average inference time is 98s, with most of the
time spent on the additional optimization steps; going through the fc layers
only takes 0.2s. See Table 2 for number of parameter comparison.

### Experimental Results

Space | $\lambda$ | ID$\downarrow$ | Expression$\downarrow$ | Pose$\downarrow$ | FID$\downarrow$
---|---|---|---|---|---
$\mathcal{Z}$ | $10^{0\phantom{-}}$ | 0.4690 | 0.15800 |  052.000 |  05.68
$\mathcal{Z}$ | $10^{1\phantom{-}}$ | 0.5100 | 0.08280 |  018.200 |  04.05
$\mathcal{Z}$ | $10^{2\phantom{-}}$ | 0.5790 | 0.00775 |  001.360 |  03.03
$\mathcal{W}$ | $10^{0\phantom{-}}$ | 0.2870 | 0.15900 |  037.600 | 37.50
$\mathcal{W}$ | $10^{1\phantom{-}}$ | 0.3550 | 0.09190 |  012.900 |  08.57
$\mathcal{W}$ | $10^{2\phantom{-}}$ | 0.5290 | 0.01450 |  001.160 |  02.97
$\mathcal{W+}$ | $\phantom{1}0^{\phantom{-0}}$ | 0.4690 | 0.19100 | 190.000 | 171.000
$\mathcal{W+}$ | $10^{-2}$ | 0.0492 | 0.24200 | 288.000 |  05.53
$\mathcal{W+}$ | $10^{-1}$ | 0.0563 | 0.23900 | 275.000 |  04.96
$\mathcal{W+}$ | $10^{0\phantom{-}}$ | 0.2150 | 0.18700 |  068.300 | 30.80
$\mathcal{W+}$ | $10^{1\phantom{-}}$ | 0.3150 | 0.15900 |  023.700 | 19.70
$\mathcal{W+}$ | $10^{2\phantom{-}}$ | 0.3840 | 0.08470 |  007.550 |  06.24
$\mathcal{W+}$ | $10^{3\phantom{-}}$ | 0.5410 | 0.01080 |  00.802 |  02.95

Table 1: Metric performance of LatentSwap, for latent codes sampled from
various latent spaces and with various $\lambda$ coefficients. The result for
the default configuration, taking $\mathcal{W+}$ space latent codes and
training with $\lambda=10^{2}$, has been boldfaced.

Overall, when the latent space is fixed to $\mathcal{W+}$, the metrics show a
clear trend that swapped images are closer to the source at lower $\lambda$,
and closer to the target at higher $\lambda$. We see a similar trend for
latent mixer operating on $\mathcal{W}$ latent codes, but not for the
$\mathcal{Z}$ counterpart. For the custom dataset, Table 1 shows the results
for various sampling spaces and loss function coefficients. We compare with
different face swapping models in Table 2, with real image metrics presented
in supplementary information.

width=center  Model00 ID$\uparrow$ Expression$\downarrow$ Pose$\downarrow$
Params$\downarrow$ DeepFakes 00 $88.39$ $0.1705$ $13.38$ Unknown FaceShifter
00 $90.68$ $0.1223$ $7.65$ $250$M SimSwap 00 $89.73$ $0.0879$ $5.82$ $120$M
HifiFace 00 $98.48$ NA $7.89$ $244$M MegaFS 00 $90.83$ $0.1348$ $7.92$ $338$M
RAFSwap 00 $96.70$ $0.1312$ $7.59$ Unknown InfoSwap 00 99.67 $0.1427$ $9.07$
$251$M Ours00 $93.36$ 0.0673 4.06 $\mathbf{87}$M

Table 2: Metric comparisons of our model on the FaceForensics++ dataset. Our
model achieves comparable performances to other face swapping models for ID
and better performance for all other metrics.

We follow the convention from previous face swapping studies (Zhu et al. 2020;
Xu et al. 2022a). The ID metric is the accuracy of a classification among
swapped faces and all original FF++ faces. The Expression and Pose are the
mean square error between the expression features/pose information between
target and swapped images.

The anomalously low performance of the $\lambda=0$ establish that
$\mathcal{L}_{\text{id}}$ by itself is insufficient to constrain our model to
the suitable latent space, hence that $\mathcal{L}_{\text{lp}}$ is needed.

Qualitative Results. Our method generates realistic and high resolution face
swaps. This shows that the pre-trained PTI model (Roich et al. 2021) works
well with our framework and has mapped the arbitrary image onto the
$\mathcal{W}$ space.

Source |  Target |  Deepfakes |  FaceSwap |  FaceShifter |  MegaFS |  HifiFace |  InfoSwap |  Ours
---|---|---|---|---|---|---|---|---

[width=0.98]Figures/ff++_qual

Figure 4: Face swapping results of LatentSwap compared to other face swapping
models on FF++. We used the code and checkpoints from their official
implementation. Our model performs source identity and target attribute
preservation well compared to other models.

In general, the face shape of the swapped image closely resembles the target
and does not get blurry when the face shape differs between source and target.
Hairstyle, on the other hand, tends to follow the source image. Interestingly,
this tendency has also been visible in Smooth-Swap, which may indicate that
this phenomenon is linked to the smooth identity embedder that both of the
models use.

These descriptions are more poignant when compared to conventional models such
as SimSwap (Chen et al. 2020) and MegaFS (Zhu et al. 2021). SimSwap gives an
acceptable face swapping result, especially regarding background and lighting.
However, it rarely changes the skin tone and is constrained to a $224\times
224$ image size. MegaFS performs qualitatively worse. This is primarily due to
the segmentation model changing only specific parts of the target image,
leading to unnatural background and being vulnerable to occlusion, alongside
showing inexact lighting and face shape.

### Analysis and Ablation Study

Latent Space. In this section, we apply the latent mixer to source/target
latent codes on $\mathcal{Z}$ and $\mathcal{W}$, and compare them with
$\mathcal{W+}$. The results are shown in Figure 5.

Source |  Target |  $\mathcal{Z}$ Mixer |  $\mathcal{W}$ Mixer |  $\mathcal{W}+$ Mixer
---|---|---|---|---
Figure 5: Performance of the latent mixer applied to latent codes from different latent spaces. We used $\lambda=10^{1}$ for $\mathcal{Z}$ and $\mathcal{W}$ space results, whereas for the $\mathcal{W}+$ space we used $\lambda=10^{2}$. $\mathcal{W}+$ space results keep more target attributes compared to other latent spaces while maintaining source identity, which is not preserved for $\mathcal{Z}$ space results. Source |  Target |  $\lambda=0$ |  $\lambda=10^{\text{-}2}$ |  $\lambda=10^{\text{-}1}$ |  $\lambda=10^{0}$ |  $\lambda=10^{1}$ |  $\lambda=10^{2}$ |  $\lambda=10^{3}$
---|---|---|---|---|---|---|---|---
Figure 6: Comparison of face swapping results by various $\lambda$. When
$\lambda=0$, the model trains only with the ID loss and outputs unrealistic
images. As $\lambda$ is increased, we see a gradual shift of the face swap
from source to target.

Table 1 shows that taking inputs from $\mathcal{W+}$ and $\mathcal{W}$ space
gives good and comparable results at their best-performing $\lambda$ values.
However, due to $\mathcal{W}$ space being smaller than $\mathcal{W+}$ space,
the images lack details, and with some target attributes such as hair and
background distorted. Taking inputs from $\mathcal{Z}$ fails to find a
suitable latent code corresponding to the swapped face throughout all
$\lambda$ as seen in the high ID metric values for all $\mathcal{Z}$ space
results in Table 1.

A possible explanation comes from that the sample space of $\mathcal{Z}$
distribution is $\mathbb{R}^{512}$. Therefore, the latent mixer output, being
in $\mathbb{R}^{512}$, also belongs to the sample space of $\mathcal{Z}$ and
the latent mixer output can never escape the sample space of $\mathcal{Z}$. In
contrast, $\mathcal{W}$ and $\mathcal{W+}$ are subspaces embedded in
$\mathbb{R}^{512}$. Therefore, it is possible for the latent mixer to output
latent codes outside $\mathcal{W}$ or $\mathcal{W+}$ space, where it is more
likely for the latent code for the swapped face to exist.

Latent Penalty Loss. We analyze the performance of various $\lambda$ values,
while maintaining other parameters, which is presented in Figure 6. We confirm
visually that the result resembles the source image for small $\lambda$, and
the target image for large $\lambda$, with a smooth transition in between.
Intriguingly, the source identity is swapped at higher $\lambda$ values
compared to other attributes such as background and lighting. When
$\lambda=0$, the images are unrealistic and the model is clearly sampling
outside the intended image domain. Figure 7 shows that (unused)
$\mathcal{L}_{\text{lp}}$ diverges when $\lambda=0$.

Figure 7: $\mathcal{L}_{\text{id}}$ (top panel) and $\mathcal{L}_{\text{lp}}$
(bottom panel) by training step, for various $\lambda$. The inset is the
$\log-\log$ plot of number of steps and $\mathcal{L}_{\text{lp}}$. The losses
are window averaged with a window width of $250$ steps.

This behaviour stems from $\mathcal{L}_{\text{lp}}$ providing a competing
objective to $\mathcal{L}_{\text{id}}$, and training reaching equilibrium when
the two loss terms are similar in magnitude. Therefore, $\lambda$ effectively
limits the deviation of the swapped latent code from the target to
$\mathcal{L}_{\text{lp}}\sim\frac{1}{\lambda}\>\mathcal{L}_{\text{id}}$, which
can be seen in the bottom panel of Figure 7.

For $\lambda=0$ we note an analogue between the behavior of
$\mathcal{L}_{\text{lp}}$ and diffusion. After initial stabilization, the
swapped latent code performs a random walk, corresponding to the classical
diffusion formula $\mathcal{L}_{\text{lp}}\sim\textrm{step}$ at intermediate
steps. Interestingly, at larger steps we observe superdiffusion
$\mathcal{L}_{\text{lp}}\sim\textrm{step}^{\nu},\>\nu>1$ While further work is
needed to exactly account for this dynamical phase transition, this point
could potentially be a indicator of the limit where StyleGAN2 latent space is
well-behaved.

Layer-wise Analysis. In StyleGAN (Karras, Laine, and Aila 2019; Karras et al.
2020, 2021), each of the 18 layers of the $\mathcal{W+}$ latent code is linked
to a specific resolution, and hence to different semantics (Karras, Laine, and
Aila 2019).

We classify the $18$ layers of the $\mathcal{W+}$ space by resolution: coarse
($4^{2}$ – $8^{2}$), middle ($16^{2}$ – $32^{2}$), fine$1$ ($64^{2}$ –
$128^{2}$), and fine$2$ ($256^{2}$ – $1024^{2}$), and investigate the face
swap results when only specific layers use the latent code of
$l_{\text{swap}}$ while using the baseline $l_{\text{t}}$ for others. Figure
10 shows the results.

Results show that specific attributes are associated with particular
resolutions in the $\mathcal{W+}$ space, and displays a direct correlation
with the size of the receptive field of the resolutions. More specifically,
coarse resolutions control attributes such as glasses or face structure.
Middle resolutions are the most important in swapping as they contain the most
information regarding the source identity. Fine1 resolutions embed information
on smaller characteristic length attributes such as source skin, hair, and eye
colors. We find that the fine$2$ resolutions, in general did not contain
information on distinguishable attributes.

Wild Images. Finally, we discuss our framework on wild images instead of human
face specific datasets to test the robustness of our model on extreme cases,
shown in Figure 8.

Source |  Target |  Swap |  Source |  Target |  Swap
---|---|---|---|---|---
Figure 8: Face swapping results on wild images. Each source-target pairs are
extracted from the generic photo rather than a human face dataset.

After carefully tuning the appropriate loss objectives, the results
demonstrate that the model is capable of producing convincing face swap
results on grayscale images, or in multiple-color faces and images with
complex backgrounds.

[width=0.47]Figures/stylenerf

Figure 9: 3D-Aware face swapping results via StyleNeRF. Images were generated
for multiple different camera parameters. Face swap is maintained throughout
different angles displaying good quality overall.

### LatentSwap for 3D-Aware Generator

We apply the LatentSwap framework to the StyleNeRF (Gu et al. 2022) generator
instead of StyleGAN2 generator to demonstrate that our framework can be
applied to generate effective swapped images in a 3D-Aware multiview context.

We maintain the same training and sampling process to create the $\mathcal{W}$
space latent codes for the StyleNeRF generator, with the only difference is
mapping it to the $21\times 512$ dimensional latent code compatible with the
StyleNeRF generator (unlike StyleGAN2 $\mathcal{W+}$ space’s $18\times 512$).
The swapped results are shown in Figure 9

Our simple framework applies to the StyleNeRF generator, and we achieve good
quality face swap for all different orientations of the resultant face images.
This shows that our framework is applicable to a 3D-based model and opens up a
wide array of opportunities to explore face swapping further in this direction
in future works.

### Application: Latent Space Manipulation

We discuss briefly on editing specific attributes (e.g. pose, facial
expression) by just using the latent code without the need of further inputs,
which is possible as our model do not requiring additional modules. We apply
InterFaceGAN (Shen et al. 2020) as the image editing module. From Figure 11 we
can verify that InterFaceGAN is able to edit various specific attributes such
as age, smile, or pose of the swapped faces qualitatively well.

Using only latent codes for editing implies any method editing images on the
StyleGAN2 domain will be applicable to our result. We give the result for
GANSpace (Härkönen et al. 2020) in Appendix 3 as an additional useful example.

Source |  Target |  Swap |  Coarse Only |  Middle Only |  Fine1 Only |  Fine2 Only
---|---|---|---|---|---|---
Figure 10: Resolution-wise analysis on face swapping results. For each case, we use the swapped latent code only at the respective resolution noted, while using target latent code otherwise.  Source |  Target |  Swap |  Age+ |  Age- |  Smile+ |  Smile- |  Pose+ |  Pose-
---|---|---|---|---|---|---|---|---
Figure 11: Editing results on the face swapped images for various attributes.
Using InterfaceGAN, the resultant images successfully edits age, smile, and
pose, without losing the source identity.

## Conclusions

In this paper, we propose LatentSwap, a simple fully-connected layer based
model for face swapping in the StyleGAN (Karras, Laine, and Aila 2019; Karras
et al. 2020, 2021) domain. Our model needs only randomly sampled latent codes
for training and can perform good quality face swapping on arbitrary images
given a pre-trained inversion network and a pre-trained generator. Our
approach enjoys fast and stable training, benefiting from the stability of the
pre-trained models and the simplicity of the framework. The face swapping
outputs are realistic and have high resolution. In addition, our results are
controllable between source and target images by altering the coefficient
between the loss terms. We analyzed the role of each spatial resolutions of
the swapped image latent code, and demonstrated that our framework is
applicable to other generators such as StyleNeRF (Gu et al. 2022) and can work
in conjunction with other StyleGAN2 (Karras et al. 2020) downstream tasks.

## References

  * Dee (2021) 2021\.  DeepFakes (https://github.com/deepfakes/faceswap).
  * Abdal, Qin, and Wonka (2019) Abdal, R.; Qin, Y.; and Wonka, P. 2019.  Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?  In _IEEE/CVF International Conference on Computer Vision (ICCV)_ , 4431–4440.
  * Abdal, Qin, and Wonka (2020) Abdal, R.; Qin, Y.; and Wonka, P. 2020.  Image2StyleGAN++: How to Edit the Embedded Images?  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Abdal et al. (2021) Abdal, R.; Zhu, P.; Mitra, N. J.; and Wonka, P. 2021.  StyleFlow: Attribute-Conditioned Exploration of StyleGAN-Generated Images Using Conditional Continuous Normalizing Flows.  _ACM Trans. Graph._ , 40(3).
  * Alaluf, Patashnik, and Cohen-Or (2021) Alaluf, Y.; Patashnik, O.; and Cohen-Or, D. 2021.  Restyle: A residual-based stylegan encoder via iterative refinement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 6711–6720.
  * Alaluf et al. (2022) Alaluf, Y.; Tov, O.; Mokady, R.; Gal, R.; and Bermano, A. 2022.  HyperStyle: StyleGAN Inversion With HyperNetworks for Real Image Editing.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 18511–18521.
  * Bao et al. (2018) Bao, J.; Chen, D.; Wen, F.; Li, H.; and Hua, G. 2018.  Towards Open-Set Identity Preserving Face Synthesis.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6713–6722.
  * Blanz et al. (2004) Blanz, V.; Scherbaum, K.; Vetter, T.; and Seidel, H.-P. 2004.  Exchanging faces in images.  In _Computer Graphics Forum_ , volume 23, 3, 669–676. Wiley Online Library.
  * Blanz and Vetter (1999) Blanz, V.; and Vetter, T. 1999.  A morphable model for the synthesis of 3D faces.  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , 187–194.
  * Chen et al. (2020) Chen, R.; Chen, X.; Ni, B.; and Ge, Y. 2020.  _SimSwap: An Efficient Framework For High Fidelity Face Swapping_ , 2003–2011.  New York, NY, USA: Association for Computing Machinery.  ISBN 9781450379885.
  * Deng et al. (2019) Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019.  Arcface: Additive angular margin loss for deep face recognition.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4690–4699.
  * Goodfellow et al. (2014) Goodfellow, I. J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative Adversarial Networks.  _Advances in Neural Information Processing Systems_ , 27.
  * Gu et al. (2022) Gu, J.; Liu, L.; Wang, P.; and Theobalt, C. 2022.  StyleNeRF: A Style-based 3D Aware Generator for High-resolution Image Synthesis.  In _International Conference on Learning Representations (ICLR)_.
  * Härkönen et al. (2020) Härkönen, E.; Hertzmann, A.; Lehtinen, J.; and Paris, S. 2020.  GANSpace: Discovering Interpretable GAN Controls.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 9841–9850. Curran Associates, Inc.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778.
  * Karras et al. (2021) Karras, T.; Aittala, M.; Laine, S.; Härkönen, E.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2021.  Alias-Free Generative Adversarial Networks.  In Ranzato, M.; Beygelzimer, A.; Dauphin, Y.; Liang, P.; and Vaughan, J. W., eds., _Advances in Neural Information Processing Systems_ , volume 34, 852–863. Curran Associates, Inc.
  * Karras, Laine, and Aila (2019) Karras, T.; Laine, S.; and Aila, T. 2019.  A Style-Based Generator Architecture for Generative Adversarial Networks.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Karras et al. (2020) Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020\.  Analyzing and Improving the Image Quality of StyleGAN.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kim, Lee, and Zhang (2022) Kim, J.; Lee, J.; and Zhang, B.-T. 2022.  Smooth-Swap: A Simple Enhancement for Face-Swapping With Smoothness.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 10779–10788.
  * Kim et al. (2022) Kim, K.; Kim, Y.; Cho, S.; Seo, J.; Nam, J.; Lee, K.; Kim, S.; and Lee, K. 2022\.  DiffFace: Diffusion-based Face Swapping with Facial Guidance.  _Arxiv_.
  * Li et al. (2020) Li, L.; Bao, J.; Yang, H.; Chen, D.; and Wen, F. 2020.  Advancing High Fidelity Identity Swapping for Forgery Detection.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Lin et al. (2012) Lin, Y.; Wang, S.; Lin, Q.; and Tang, F. 2012.  Face swapping under large pose variations: A 3D model based approach.  In _2012 IEEE International Conference on Multimedia and Expo_ , 333–338. IEEE.
  * Loshchilov and Hutter (2017) Loshchilov, I.; and Hutter, F. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_.
  * Patashnik et al. (2021) Patashnik, O.; Wu, Z.; Shechtman, E.; Cohen-Or, D.; and Lischinski, D. 2021.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2085–2094.
  * Petrov et al. (2020) Petrov, I.; Gao, D.; Chervoniy, N.; Liu, K.; Marangonda, S.; Umé, C.; Facenheim, C. S.; RP, L.; Jiang, J.; Zhang, S.; Wu, P.; Zhou, B.; and Zhang, W. 2020.  DeepFaceLab: Integrated, flexible and extensible face-swapping framework.  Arxiv preprint.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 8748–8763. PMLR.
  * Ramachandran, Zoph, and Le (2017) Ramachandran, P.; Zoph, B.; and Le, Q. V. 2017.  Searching for activation functions.  _arXiv preprint arXiv:1710.05941_.
  * Richardson et al. (2021) Richardson, E.; Alaluf, Y.; Patashnik, O.; Nitzan, Y.; Azar, Y.; Shapiro, S.; and Cohen-Or, D. 2021.  Encoding in style: a stylegan encoder for image-to-image translation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2287–2296.
  * Roich et al. (2021) Roich, D.; Mokady, R.; Bermano, A. H.; and Cohen-Or, D. 2021.  Pivotal Tuning for Latent-based Editing of Real Images.  _ACM Trans. Graph._
  * Shen et al. (2020) Shen, Y.; Gu, J.; Tang, X.; and Zhou, B. 2020.  Interpreting the Latent Space of GANs for Semantic Face Editing.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Shen and Zhou (2021) Shen, Y.; and Zhou, B. 2021.  Closed-form factorization of latent semantics in gans.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1532–1540.
  * Tov et al. (2021a) Tov, O.; Alaluf, Y.; Nitzan, Y.; Patashnik, O.; and Cohen-Or, D. 2021a.  Designing an encoder for stylegan image manipulation.  _ACM Transactions on Graphics (TOG)_ , 40(4): 1–14.
  * Tov et al. (2021b) Tov, O.; Alaluf, Y.; Nitzan, Y.; Patashnik, O.; and Cohen-Or, D. 2021b.  Designing an Encoder for StyleGAN Image Manipulation.  _ACM Trans. Graph._ , 40(4).
  * Voynov and Babenko (2020) Voynov, A.; and Babenko, A. 2020.  Unsupervised discovery of interpretable directions in the gan latent space.  In _International Conference on Machine Learning_ , 9786–9796. PMLR.
  * Wang and Ponce (2021) Wang, B.; and Ponce, C. R. 2021.  A Geometric Analysis of Deep Generative Image Models and Its Applications.  In _International Conference on Learning Representations_.
  * Wang et al. (2021) Wang, Y.; Chen, X.; Zhu, J.; Chu, W.; Tai, Y.; Wang, C.; Li, J.; Wu, Y.; Huang, F.; and Ji, R. 2021.  HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping.  In Zhou, Z.-H., ed., _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , 1136–1142. International Joint Conferences on Artificial Intelligence Organization.  Main Track.
  * Xu et al. (2022a) Xu, C.; Zhang, J.; Hua, M.; He, Q.; Yi, Z.; and Liu, Y. 2022a.  Region-Aware Face Swapping.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7632–7641.
  * Xu et al. (2022b) Xu, Y.; Deng, B.; Wang, J.; Jing, Y.; Pan, J.; and He, S. 2022b.  High-resolution face swapping via latent semantics disentanglement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7642–7651.
  * Xu et al. (2021) Xu, Z.; Yu, X.; Hong, Z.; Zhu, Z.; Han, J.; Liu, J.; Ding, E.; and Bai, X. 2021\.  FaceController: Controllable Attribute Editing for Face in the Wild.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(4): 3083–3091.
  * Xu et al. (2022c) Xu, Z.; Zhou, H.; Hong, Z.; Liu, Z.; Liu, J.; Guo, Z.; Han, J.; Liu, J.; Ding, E.; and Wang, J. 2022c.  StyleSwap: Style-Based Generator Empowers Robust Face Swapping.  In _Proceedings of the European Conference on Computer Vision (ECCV)_.
  * Yang, Li, and Lyu (2019) Yang, X.; Li, Y.; and Lyu, S. 2019.  Exposing deep fakes using inconsistent head poses.  In _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 8261–8265. IEEE.
  * Zhu et al. (2020) Zhu, J.; Shen, Y.; Zhao, D.; and Zhou, B. 2020.  In-domain GAN Inversion for Real Image Editing.  In _Proceedings of European Conference on Computer Vision (ECCV)_.
  * Zhu et al. (2021) Zhu, Y.; Li, Q.; Wang, J.; Xu, C.-Z.; and Sun, Z. 2021.  One Shot Face Swapping on Megapixels.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 4834–4844.