# Restricted Black-box Adversarial Attack
Against DeepFake Face Swapping

Junhao Dong, Yuan Wang, Jianhuang Lai, and Xiaohua Xie The authors are with
the School of Computer Science and Engineering, Sun Yat-sen University,
Guangzhou 510006, China, also with the Key Laboratory of Machine Intelligence
and Advanced Computing, Ministry of Education, Guangzhou 510006, China, and
also with the Guangdong Key Laboratory of Information Security Technology,
Guangzhou 510006, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>xiexiaoh6@
mail.sysu.edu.cn). (Corresponding author: Xiaohua Xie.)

###### Abstract

DeepFake face swapping presents a significant threat to online security and
social media, which can replace the source face in an arbitrary photo/video
with the target face of an entirely different person. In order to prevent this
fraud, some researchers have begun to study the adversarial methods against
DeepFake or face manipulation. However, existing works focus on the white-box
setting or the black-box setting driven by abundant queries, which severely
limits the practical application of these methods. To tackle this problem, we
introduce a practical adversarial attack that does not require any queries to
the facial image forgery model. Our method is built on a substitute model
persuing for face reconstruction and then transfers adversarial examples from
the substitute model directly to inaccessible black-box DeepFake models.
Specially, we propose the Transferable Cycle Adversary Generative Adversarial
Network (TCA-GAN) to construct the adversarial perturbation for disrupting
unknown DeepFake systems. We also present a novel post-regularization module
for enhancing the transferability of generated adversarial examples. To
comprehensively measure the effectiveness of our approaches, we construct a
challenging benchmark of DeepFake adversarial attacks for future development.
Extensive experiments impressively show that the proposed adversarial attack
method makes the visual quality of DeepFake face images plummet so that they
are easier to be detected by humans and algorithms. Moreover, we demonstrate
that the proposed algorithm can be generalized to offer face image protection
against various face translation methods.

###### Index Terms:

DeepFake, black-box, adversarial attack, substitute model.

##  I Introduction

DeepFake, a portmanteau of “deep learning” and “fake”, has drawn broad
attention in recent years. This burgeoning technique can replace the source
face in an existing image or a video with the target face of an arbitrary
identity. However, the abuse of this intriguing technology may result in an
underlying hazard to social media and online security. For instance, it can be
maliciously used to blackmail individuals or to bypass the authentication
mechanism [1].

Although these subtle DeepFake artifacts confuse human vision, several forgery
detection methods can precisely distinguish these manipulated images [2, 3,
4]. However, there exists a time delay between the publishing of forged images
and their corresponding detection results, which may damage personal
reputation by impersonating the victim on social media platforms. Another way
to defend against this malicious face manipulation is to disrupt the
generation stage of DeepFake face swapping, i.e., to distort generated face
swapping images. This straightforward method eliminates the complicated
forgery detection, which can tackle these DeepFake issues from the root
thoroughly.

Figure 1: Illustration of the taxonomy of adversarial attacks against the
target Deep Neural Network (DNN). “Training and inference Backward Propagation
(BP)” are corresponding to the accessibility to backward gradients of the
target model, which are related to training and inference of our adversary
generative model (threat model). “Query” indicates the availability of
inference outputs of the target model.

Deep Neural Networks (DNNs) have made spectacular progress in various computer
vision tasks [5, 6, 7]. Nonetheless, DNNs are especially vulnerable to some
tailored examples synthesized by attaching imperceptible perturbations to
original images. These visually unconscious examples with the added
perturbation are also regarded as adversarial examples [8]. Adversarial
examples can drastically mislead the inference of DNNs to output wrong (or
even specific) results, which poses a significant challenge to current deep
learning applications [9, 10, 11]. A plausible reason for the emergence of the
adversarial example is the linear behavior in high dimensional feature spaces
of DNNs [12]. Thus, researching on adversarial examples helps explain the
learned feature of DNNs and shows a path to attack the improper usage of DNN
mechanisms like DeepFake. Significantly, the attack scenarios are various
according to the accessible information of the target model and its output
results, as shown in Fig. 1. The white-box adversarial attack has the complete
information of the target model. Nevertheless, the semi-whitebox setting [13]
focuses on training a DNN to generate adversarial examples, which do not need
the backward gradients at the inference stage. However, these two types of
attacks are impractical in defending against unknown tampering. The black-box
adversarial attack can only gain the DNN output. In contrast, the restricted
black-box adversarial attack [14] is built on an entirely black box with the
inaccessible output of the target DNN model, which means that we can not
conduct even a single query to the black-box model. Therefore, this attack is
more applicable to real-world scenarios. To achieve general applications, we
consider the restricted black-box adversarial attack, which can defend the
immutability of our photos from DeepFake face swapping or other face editing
systems.

In this paper, we consider constructing a substitute model to simulate the
process of face reconstruction, which is achieved by a deep autoencoder
network. The main reason is that we consider DeepFake face swapping, face
style manipulation, etc. as approximations of face reconstruction. Then, we
conduct adversarial attack towards the accessible substitute model and then
transfer adversarial examples to other unknown face manipulation models.
Specifically, we innovatively build the Transferable Cycle Adversary
Generative Adversarial Network (TCA-GAN) to generate adversarial perturbations
against unknown DeepFake face swapping systems. The training procedure is
mainly based on adding and removing the adversarial perturbation to simulate
both the adversarial attack and defense. In the application stage, TCA-GAN can
generate the adversarial perturbation against the unknown face swapping model
from a single face image. The generalizable adversarial example can be
obtained by adding the adversarial perturbation to the input legitimate face
image, which can disrupt the DeepFake face swapping or even other face
manipulation models. For better generalization results, we also apply a novel
post-regularization on the generated adversarial example to enhance its
generalizability. We show an illustrated example of DeepFake face swapping
with respect to the original image and the adversarial example in Fig. 2.

Figure 2: Comparison of DeepFake face swapping between the legitimate and the
adversarial example. DeepFake aims to replace the source face in the original
image with the fake face of the target identity. The adversarial example can
be easily obtained by adding a visually undetectable adversarial perturbation
(magnified for visibility) to the original image. Note that this tailored
adversarial example is visually indistinguishable from the original example in
the human vision, but can result in a strong disruption to the output face-
swapped image.

To comprehensively validate our proposed method, we build a benchmark of
DeepFake adversarial attacks for future research. Moreover, We evaluate the
face swapping results on both referenced and non-referenced image quality
assessments. Extensive experiments impressively show that the proposed method
can degrade the visual quality of DeepFake face images effectively so that
they are easier to be identified by human eyes and DeepFake detectors. We also
demonstrate that the proposed method can be well generalized to offer
protection against several face translation methods with nearly no cost.

Overall, our main contributions can be summarized as follow:

  * •

To the best of our knowledge, we are the first to successfully construct a
practical restricted black-box attack against DeepFake face swapping.
Specifically, we build the face reconstruction autoencoder as the substitute
model for producing adversarial examples that can be directly transferred to
unknown face manipulation models.

  * •

We design TCA-GAN to generate powerful adversarial examples against DeepFake
systems. Moreover, the post-regularization module is proposed for better
transferability. Extensive experiments show that our method can be generalized
to various face attribute editing models with nearly no cost.

  * •

We construct strong benchmarks on disrupting DeepFake systems when simulating
a real-world scenario of arbitrary face swapping. In addition to disrupting
the visual quality of generated fake images, we demonstrate that the
disruption can further enhance the performance of several image-level DeepFake
detection methods.

##  II Related works

###  II-A Automatical Face Swapping

Face swapping consists of replacing the face of a person with another face of
a different identity. This tricky technique was first conducted in [15], which
applies a 3D morphable model fitting algorithm on both exchanging faces.
Bitouk et al. [16] presented a fully automatic face swapping by replacing the
source face with the candidate face image from a face gallery. Nevertheless,
the target identity of this face swapping is uncontrollable and time-
consuming. Korshunova et al. [17] regarded the face swapping problem as style
transfer to accomplish designated face swapping, which is conducted by a deep
convolutional neural network with efficient preprocessing and postprocessing.
Especially, Bao et al. [18] disentangled the identities and attributes of face
images and proposed an identity preserving GAN-based network for open-set face
synthesizing. Modern deep learning-based face swapping methods eliminate
several fussy face editing steps and synthesize photo-realistic face images
[19, 20, 21]. The most popular technique of these is DeepFake [1], which
induces a plethora of fake public scandals. However, numerous researches have
been explored on DeepFake detection. We consider a straightforward approach to
disturbing DeepFake face swapping.

Figure 3: The flow chart of our proposed method. Due to the accessibility to
the substitute model in the training stage, we train TCA-GAN to produce the
adversarial perturbation against the substitute model according to the input
face image. In the application stage, we append a post-processing module to
regularize the generated adversarial example for better transferability. The
application stage simulates a real-world scenario of attacking DeepFake, while
we can not obtain the face-swapped output or any details of DeepFake.

###  II-B Adversarial attacks against classifiers

As demonstrated by [8], the reason for adversarial examples is the
discontinuities of DNNs. Alternatively, Goodfellow et al. [12] argued that
adversarial examples are results from the linearity in high dimensional space
in DNNs and presented the Fast Gradient Sign Method (FGSM). The iterative
strategy is also exerted into FGSM to build more powerful adversarial attacks
[22, 23]. Moosavi-Dezfooli et al. [24] extend adversarial attack to an
universal (image-agnostic) scenario, i.e., making wrong prediction for each
input image with high confidence. Especially, Su et al. [25] considered an
extremely confined scenario and proposed a one-pixel adversarial perturbation
generative method based on differential evolution. In addition, Croce et al.
[26] introduced a Fast Adaptive Boundary (FAB) attack to find minimal
adversarial perturbation in distinct norm constraints. Particularly, Xiao et
al. [13] proposed AdvGAN to generate more effective adversarial examples
against the classification with GANs. Our method also focus on the same GAN-
based adversary generation. However, we extend it to attacking generative
models in the restricted black-box setting via constructing a robust cycle-
consistent GAN. Zhang et al. [27] studied the optimization on the manifold of
the classification boundary and introduced boundary projection to obtain low
adversarial perturabtion. Zhong et al. [28] proposed a novel surrogate model
to enhance the transferability of adversarial attacks against face recognition
models. Likewise, our goal is to enhance the transferability of adversarial
examples, while we mainly focus on adversarial examples against DeepFake face
swapping to protect in the restricted black-box scenario. While our focus is
not limited to adversarial attacks against malicious face swapping to protect
the security of personal photos, it yields supportive generalization
performance on attacking several face manipulation models as well.

###  II-C Adversarial attacks against generative models

Although adversarial attacks towards classification models have achieved
satisfactory performance, few studies have focused on adversarial attacks
against generative models. Tabacof et al. [29] first investigated adversarial
attacks for autoencoders with latent representation. They indicated that
generative models do not have a definite decision boundary as classification
models and thus are more difficult to disrupt. Kos et al. [11] attached an
auxiliary classifier to the encoder of the target generative models to obtain
classification-based adversarial attacks. Recently, applying adversarial
examples to disturb image-translation models receives heavy investigations
[30, 31, 32]. Particularly, Dong et al. [33] demonstrate impressive results on
disrupting DeepFake face swapping in white-box circumstances. In contrast, we
explore the restricted black-box scenario. Sun et al. [34] described a white-
box adversarial attack to disturb the facial landmark extraction, thus can
induce the misalignment on DeepFake face swapped images. Their method mainly
focuses on the preprocessing of DeepFake face swapping, while we concentrate
on a universal way to disturb the process of face swapping. Ruiz et al. [31]
first construct a multi-query based adversarial attack against image-
translation models in the black-box scenario. Nevertheless, their black-box
scenario is conducted through numerous queries, which is impractical to defend
the malicious face swapping in real-world conditions. In comparison, our
method considers a more practical restricted black-box setting, with no need
for queries to the target face swapping model.

##  III Method

###  III-A Preliminaries

####  III-A1 Brief review of DeepFake

Original DeepFake consists of one shared encoder and two separate decoders to
two face swapping identities respectively. The shared encoder converts the
input face image into a latent representation. Then the decoder transforms the
latent representation to a face image, whose identity is according to the
selected decoder. In the training stage, DeepFake conducts face reconstruction
with the decoder that is consistent with the input identity. For face
swapping, DeepFake decodes the latent variable with the other decoder to
obtain face swapping images and vice versa. Note that these input face images
have already been trained for robust face swapping. Hence, it is also
challenging to disrupt DeepFake face swapping. In this paper, we mainly focus
on attacking this original type of DeepFake face swapping mechanism.

####  III-A2 Notation

In this paper, we intend to build generalizable adversarial examples against
DeepFake face swapping in the black-box scenario. We denote the adversarial
example as $x^{adv}=x+r$, where $x$ is the legitimate input face image, and
$r$ is the appended adversarial perturbation. Let $DF:X\rightarrow Y$ stand
for the inaccessible DeepFake face swapping model, which maps the input face
image $x\in X$ to a face-swapped output $DF\left(x\right)\in Y$. Note that $X$
and $Y$ are corresponding to the domain of source images and face-swapped
target images, respectively. The substitute model
$S\left(\cdot\right)=S_{d}\left(S_{e}\left(\cdot\right)\right)$ is composed of
a downsampling encoder $S_{e}\left(\cdot\right)$ and an upsampling decoder
$S_{d}\left(\cdot\right)$.

Figure 4: Illustration of the training mechanism of TCA-GAN, including two
generators and two domain discriminators. These two generators are utilized to
produce and remove the adversarial perturbation. The corresponding
discriminators differentiate both legitimate examples and adversarial
examples.

Our goal is to find a transferable adversarial perturbation to disrupt the
target DeepFake model simultaneously with no accessing to the DeepFake model.
More pertinently, we expect to enhance the difference between face-swapped
images generated by the legitimate example and the adversarial example
respectively. We first formulate the pursued adversarial perturbation as
follow:

$\displaystyle\max\limits_{r}\quad
F\left[DF\left(x\right),DF\left(x+r\right)\right]$ (1)
$\displaystyle\begin{array}[]{r@{\quad}r<EMAIL_ADDRESS>\end{array}$

where $F$ is the metric to measure the distance between the face-swapped
images, $\epsilon$ is the infinity norm bound to restrict the adversarial
perturbation.

###  III-B TCA-GAN against the substitute model

As aforementioned, query-based black-box adversarial attack is limited by
abundant query times to the black-box model during inference stage, which need
to have the access to the target black-box model. However, we can not conduct
multiple query to black-box models or even access the output of them in some
real-world scenarios. Consequently, we consider a restricted black-box
adversarial attack with no access to the target DeepFake model. According to
the transferability of adversarial examples [12], we distinctly build a
substitute model for generalizable adversary generation. More specifically, we
obtain adversarial examples from the substitute model and expect to transfer
them to the unknown DeepFake face swapping model. Here, the adversarial
example is composed of a legitimate image and the adversarial perturbation
that is generated by TCA-GAN with the prior of legitimate image. Then we apply
a post-regularization on the adversarial example to enhance the
transferability. The flow chart of our adversarial attack is shown in Fig. 3.
On the whole, we acquire the adversarial example against the substitute model
through TCA-GAN. Afterward, we can transfer it to the target DeepFake face
swapping model.

Motivated by [35], we extend the substitute model to the generative model to
simulate the black-box DeepFake model, which is much more harder then the
classification model that has an absolute classification boundary. Considering
that DeepFake and other face manipulation models can be virtually regarded as
rebuilding the objective face, we employ a DNN autoencoder accomplishing face
reconstruction to build the substitute model. Then the adversarial example
disrupting the substitute model can also be transferred to perturb face
manipulation models such as DeepFake. The optimization procedure can be
formalized as bellow:

$\displaystyle\mathcal{L}_{recons}=$
$\displaystyle\left\|S\left(x\right)-x\right\|_{1}+\left\|S\left(\hat{x}\right)-\hat{x}\right\|_{1}$
(2)

where $\hat{x}$ is a slightly wrapped version of the input face image $x$.
This wrapping operation comprises randomly rotation, scaling and shifting. The
reason we take the wrapped face image into consideration is that the
substitute model should possess a certain extent of robustness towards some
subtle perturbations. Therefore, the adversarial example that produced against
the substitute model can present a better transferring performance on the
target model.

After getting the trained substitute model, we then construct TCA-GAN to
generate the adversarial perturbation to fool this model. TCA-GAN is a cyclic
structure, which mainly consists of two generative modules and two
corresponding domain discriminators. These two cycle-consistent adversary
generators separately produce and remove the adversarial perturbation. Note
that the generated adversarial perturbation can be further added to the
original image to get the adversarial example. Respectively, two domain
discriminators and are utilized to distinguish the legitimate example $x$ and
the adversarial example $x^{adv}$. The mechanism of TCA-GAN is presented in
Fig. 4.

The forward adversary generation part is to produce an adversarial
perturbation to combine with the input face image, disrupting the
reconstruction of the substitute model. The generated adversarial examples not
only disrupt the ultimate reconstruction of the substitute model but also
perturb the latent variable in the substitute model. This intuition is based
on the observation that adversarial perturbations against DNNs are
transferable [36]. The similarity between the components of DNNs incur similar
feature representations between different DNNs. Hence, the obtained
adversarial example is prone to be transferred to attack the unaccessible
DeepFake model. The loss function of disturbing the reconstruction can be
written as below:

$\displaystyle\mathcal{L}_{disr}=$
$\displaystyle\exp\left\\{-\left\|S_{e}\left(x\right)-S_{e}\left(G_{P}\left(x\right)+x\right)\right\|_{1}\right\\}$
(3) $\displaystyle+$
$\displaystyle\exp\left\\{-\left\|S\left(x\right)-S\left(G_{P}\left(x\right)+x\right)\right\|_{1}\right\\}$

where $G_{P}\left(\cdot\right)$ is the generator to produce the transferable
adversarial perturbation with the prior of an legitimate face image. Eq. 3 can
also be considered as maximizing the reconstruction results between the
adversarial example and the legitimate example. Oppositely, the reversed
adversary removal part is to get rid of the appended adversarial perturbation
from an adversarial example, i.e., to extract the adversarial perturbation
from the input example. Here, we establish a cycle-consistent structure with
the two-way adding or removing the adversarial perturbation. More formally, we
opt to optimize:

$\mathcal{L}_{cyc}=\left\|G_{R}\left(x+G_{P}\left(x\right)\right)-G_{P}\left(x\right)\right\|_{1}$
(4)

where $G_{R}\left(\cdot\right)$ is the generator to remove the adversarial
perturbation from an input adversarial example. The intuition behind this
mechanism is that the adversarial perturbation generated to append and remove
should be as closer as possible. Hence, it can bring richer supervision on
generating puissant adversarial perturbations. This cycle-consistent structure
can further enhance the generalizability performance of the adversarial
examples.

To balance generators relatively, we construct two domain discriminators to
distinguish adversarial examples and legitimate examples. The adversarial loss
function of our GAN structure can be formalized as follow:

$\displaystyle\mathcal{L}_{adv}$ $\displaystyle=D_{L}(x_{adv})-D_{L}(x)$ (5)
$\displaystyle+D_{A}(x_{adv}-G_{R}(x_{adv}))-D_{A}(x_{adv})$

where $D_{L}(\cdot)$ denotes the discriminator of the domain on legitimate
examples and $D_{A}(\cdot)$ represents the discriminator of the domain on
adversarial examples. For brevity, $x_{adv}=x+G_{P}(x)$ is the adversarial
example created by the adversary generator. Consequently, the adversarial
reaction will further enhance the performance of generators until Nash
equilibrium. Overall, the total objective of TCA-GAN can be formalized as
below:

$\mathcal{L}=\mathcal{L}_{adv}+\lambda_{cyc}\mathcal{L}_{cyc}+\lambda_{disr}\mathcal{L}_{disr}$
(6)

where $\lambda_{cyc}$ and $\lambda_{disr}$ manage the relative significance of
the objective function. We conduct the GAN mechanism via solving the min-max
game, in which we minimize the objective function when training the generator
and maximize it when training the discriminator.

###  III-C Post-regularization

In order to further enhance the transferability of our adversarial examples,
we append a post-regularization to weaken their specificity on the substitute
model. Inspired by the observation from [37, 33] that the more specific
adversarial examples targeting against the deep learning model, the less
generalizable they become. We then slightly shift the attention of generated
adversarial examples away from the substitute model, which means that we aim
at obtaining a second-best adversarial example towards the substitute model
for better generalization.

One effective way to regularize the off-the-shelf adversarial example is to
make a distillation. As aforementioned, we disrupt the substitute model by
increasing the discrepancy of latent variables between the legitimate example
and the adversarial example. We first initialize the new adversarial example
by appending random noise to e the non-smooth vicinity of the original
example. Then we guide this new adversarial example to approximate the
adversarial example generated by TCA-GAN. The corresponding optimization can
be written as follow:

$\displaystyle\max\limits_{x_{radv}}\quad[S_{e}(x_{radv})-S_{e}(S(x))]\circ[S_{e}(x_{adv})-S_{e}(S(x))]$
(7)
$\displaystyle\begin{array}[]{r@{\quad}r<EMAIL_ADDRESS>
where $x_{radv}$ represents the regularized adversarial example and $\circ$ is
the hadamard product. Note that the consequent term in Eq. 7 is served as a
weight on increasing latent discrepancy during the optimization. As
aforementioned, the post-regularization can be regarded as a distillation from
the adversarial example generated from TCA-GAN. Due to the consistency before
and after the reconstruction of the autoencoder, we adopt to maximize the
feature-level distance between regularized adversarial examples and
reconstructed examples. The reason why we choose to involve reconstructed
examples but not original examples is that reconstructed examples can be
viewed as an augmentation of original examples. Thus we can obtain more
generalizable adversarial examples through attacking against both original
example and their augmented versions simultaneously. This optimization can
also induce the regularized adversarial example to be more robust to small
transformation. Furthermore, the regularized adversarial example is optimized
to keep the same high latent discrepancy. To simplify this optimization
procedure, we adopt an iterative gradient ascent strategy on the regularized
adversarial example. Afterward, we project the regularized adversarial example
to a restriction to limit its magnitude. The overall procedure is shown in
Algorithm 1, where $Clip_{\epsilon}$ and $Clip_{image\\_range}$ represents the
clipping in the $\epsilon$-ball and the image range, respectively. Note that
the randomization in the initial stage of adversarial examples can further
prevent the label leaking and gradient masking problem.

Algorithm 1 Post-regularization

0: Adversary generator of TCA-GAN $G_{P}$;Original face image $x$;Substitute
model $S$;Adversarial perturbation bound $\epsilon$;Iterative number
$N$;Iterative step size $\alpha$;

0: Regularized adversarial perturbation $x_{radv}$;

1: $x_{adv}=x+G_{P}(x)$;

2: Randomly initialize $x_{radv}$ from the neighborhood of $x_{adv}$;

3: $x_{rec}=S(x)$

4: $W=S_{e}(x_{adv})-S_{e}(x_{rec})$

5: for $k=1,\ldots,N$ do

6: $L=[(S_{e}(x_{radv})-S_{e}(x_{rec}))\circ W]\ \big{/}\
\left\|x\right\|_{F}$;

7: $r=Clip_{\epsilon}(x_{radv}-x+\alpha sign(\nabla_{x_{radv}}L))$;

8: $x_{radv}=Clip_{image\\_range}(x+r)$

9: end for

10: return $x_{radv}$.

##  IV Experiments

In this section, we first introduce the proposed benchmarks on disrupting
DeepFake face swapping and the measurement of our experimental results. Next,
we make a comparison with other transferrable adversarial attack methods and
conduct an ablation study on the component modules. To further explore the
effectiveness of our method, we verify that the perturbed face-swapped images
can facilitate DeepFake detection. Comprehensively, we also apply our method
on several face manipulation models to validate the generalization
performance.

###  IV-A Experimental Setup

####  IV-A1 Dataset

Following [33], we construct a larger DeepFake face swapping dataset of 6274
images, including 40 man identities and 38 woman identities. To support future
development of DeepFake adversarial attacks, we also train the corresponding
DeepFake face swapping models on the proposed dataset. Note that in order to
simulate the real-world application scenario, the database to train both the
substitute model and TCA-GAN is disjoint with the database on disrupting
DeepFake face swapping. Moreover, CelebA [38] dataset is utilized for
verifying the transfer-based adversarial attack on other face manipulation
models.

####  IV-A2 Metrics

To comprehensively evaluate our proposed method, we conduct both referenced
and non-referenced image quality assessments on the face-swapped images.
Structure SIMilarity (SSIM) [39] index and Feature SIMilarity (FSIM) [40] are
measured on both pairs of input face images and face swapped images in terms
of legitimate examples and adversarial examples. Besides, Blind/Referenceless
Image Spatial QUality Evaluator (BRISQUE) [41] is also utilized to assess the
image quality. A smaller BRISQUE score represents better visual quality.

Method | Source | Face swapping
---|---|---
SSIM$\uparrow$ | FSIM$\uparrow$ | SSIM$\downarrow$ | FSIM$\downarrow$
FGSM [12] | 0.953 | 0.975 | 0.811 | 0.917
PGD [23] | 0.947 | 0.969 | 0.788 | 0.899
DIM [42] | 0.952 | 0.973 | 0.756 | 0.881
TIM [43] | 0.95 | 0.978 | 0.76 | 0.883
ILA [37] | 0.943 | 0.967 | 0.752 | 0.879
APGD [44] | 0.939 | 0.965 | 0.75 | 0.874
ours | 0.951 | 0.974 | 0.731 | 0.873

TABLE I:  Reference image quality assessment on disrupting DeepFake face
swapping in the black-box setting. The best performance is marked in bold.

####  IV-A3 Implementation Details

For the purpose of simulating the practical adversarial attack scenario, the
generated adversarial examples will be further resized to the original size.
Consequently, they will be resized and randomly transformed before entering
the target DeepFake model. In this paper, the image value is normalized to
$\left[0,1\right]$. To obtain the visually undetectable adversarial
perturbation, we restrict the bound $\epsilon$ to be $0.03$. We run the post-
regularization for $10$ iterations with a $0.006$ iterative step size.

###  IV-B Experimental Results

To begin with, we present several face swapping examples in Fig. 5 depending
on the input of legitimate images and adversarial images. We perform a
sequence of experiments to provide empirical evidence of our method on
attacking against DeepFake face swapping. Note that all experiments in this
section are conducted in the restricted black-box scenario, i.e., the target
DeepFake model is inaccessible until the completion of adversary generation.
Moreover, the produced adversarial examples are directly fed in the target
DeepFake face swapping models. The comparison with other adversarial attack
methods is reported in Table I.

(a)

(b)

(c)

(d)

Figure 5: Examples of both original and disrupted face swapping in the
restricted black-box scenario. DeepFake model generates original face-swapped
images (b) and disrupted face-swapped images(d) with input of legitimate
examples (a) and adversarial examples (c), respectively.

The disruption against DeepFake face swapping is evaluated through the
similarity between face-swapped images that are generated from legitimate
examples and adversarial examples, respectively. Due to the infinite norm
restriction on the adversarial perturbation, there remains only a tiny gap in
the visual distances between legitimate examples and adversarial examples
during the comparison. According to the table, we can observe that our method
presents an effective perturbation on DeepFake face swapping while preserving
the adversarial perturbation visually imperceptible. Moreover, we evaluate the
non-referenced image quality assessment BRISQUE as shown in Table II. Note
that our goal is to obtain inferior face-swapped images that are induced by
adversarial examples while keeping the image quality of input adversarial
examples high. Both the referenced and non-referenced image quality
assessments demonstrate that our methods can effectively give rise to a facial
distortion on DeepFake face-swapped images.

Method | Source images | Face-swapped images
---|---|---
Original | Adversarial | Original | Adversarial
FGSM [12] | 23.58 | 22.93 | 35.68 | 38.33
PGD [23] | 25.28 | 40.74
DIM [42] | 24.17 | 44.82
TIM [43] | 24.22 | 44.17
ILA [37] | 27.04 | 44.63
APGD [44] | 24.12 | 44.26
ours |  | 23.97 |  | 47.13

TABLE II: Non-reference image quality assessment BRISQUE on related images.
The best performance is marked in bold.

Cyclic type | SSIM$\downarrow$ | FSIM$\downarrow$ | BRISQUE$\uparrow$
---|---|---|---
w/o cycle-consistency | 0.752 | 0.878 | 45.43
w/ bidirectional cycle-consistency | 0.738 | 0.876 | 46.07
w/ unidirectional cycle-consistency | 0.731 | 0.873 | 47.13

TABLE III:  Ablation study on the cyle-consistency type in TCA-GAN. The best
performance is marked in bold.

(a) Original face translation

(b) Disrupted face translation

Figure 6: Exemplary results for StarGAN with the input of legitimate face
images and adversarial face images. The generated adversarial example is
applicable to arbitrary style translation (Hair color, Gender, and Aged).

(a) Original face translation

(b) Disrupted face translation

Figure 7: Exemplary results for AttGAN with the input of legitimate face
images and adversarial face images. The generated adversarial example is
applicable to arbitrary style translation (Hair color, Gender, and Glasses).

###  IV-C Ablation study

In this section, we analyze the effectiveness of each component module in our
proposed method. On account of only possessing the single field of legitimate
examples, our cycle-consistent loss considers the unidirectional adversary
circulation. The unidirectional generation maintains the consistency between
legitimate examples and reconstructed images that are generated through two
generators in TCA-GAN, respectively. Furthermore, the bidirectional cycle-
consistent version of TCA-GAN considers the cycle reconstruction of
adversarial examples on the basis of the unidirectional one. Besides, these
adversarial examples against the substitute model are regenerated by PGD [23]
per iteration. The comparison of the different versions of cycle-consistency
is presented in Table III. It is worth mention that we mainly focus on
perturbed face-swapped images. We can observe that TCA-GAN with the
unidirectional cycle-consistency conducts much better than the one with
bidirectional cycle-consistency. The main reason is that the obtained
adversarial examples are excessively biased towards the substitute model,
which is hard to transfer to other DNN models. On the contrary, our goal is to
acquire a sub-optimal adversarial example and then generalize it to other face
manipulation models like DeepFake. As a consequence, the guidance of an
excessively powerful adversarial example may lead to an opposite effect to
adversary transferring.

TABLE IV: Ablation results for component modules. The best result in each
column is bold.

| CC | LVD | PRM | SSIM$\downarrow$ | FSIM$\downarrow$ | BRISQUE$\uparrow$
---|---|---|---|---|---|---
1 | ✓ |  |  | 0.79 | 0.903 | 40.95
2 |  | ✓ |  | 0.783 | 0.896 | 42.11
3 |  |  | ✓ | 0.764 | 0.887 | 43.27
4 | ✓ | ✓ |  | 0.761 | 0.884 | 43.94
5 |  | ✓ | ✓ | 0.752 | 0.878 | 45.43
6 | ✓ |  | ✓ | 0.755 | 0.882 | 45.18
7 | ✓ | ✓ | ✓ | 0.731 | 0.873 | 47.13
CC means the cycle consistency mechanism.
LVD means the latent variable disruption.
PRM means the post-regularization module.

Concerning the main design of our proposed framework, we also discover the
effect of component modules. Concretely, we mainly assess the disruption on
face-swapped images in terms of the cycle-consistency, the latent variable
disruption, and the post-regularization. The conducted ablation results are
reported in Table IV. Note that the latent variable disruption denotes that we
induce a disruption on the latent variable of the substitute model when
training TCA-GAN. As the table presented, the latent variable disruption shows
a beneficial impression on enhancing the transferred adversary disruption.
Moreover, we can derive a 3.1$\%$ SSIM performance enhancement with the latent
variable disruption. The disruption on latent variables of the substitute
model can be regarded as disturbing the facial feature during the face
reconstruction. This also demonstrates that the perturbation towards the
latent representation is more likely to generalize to other DNN models.
Remarkably, the post-regularization presents a 3.9$\%$ SSIM performance
boosting result. This post-processing can further help to distill the
generated adversarial examples from TCA-GAN, meanwhile keeping a long
discrepancy on the latent variables in terms of legitimate examples and
regularized adversarial examples. Likewise, the result supports that the sub-
optimal adversarial example has a more robust generalization performance than
the optimal adversarial example. The proposed modules and the integration of
them are demonstrated to be valid to disrupt the DeepFake face swapping
mechanism. In summary, there is a large room for performance improvement on
our proposed DeepFake face swapping dataset.

TABLE V:  The accuracy ($\%$) of several DeepFake detection methods on both
original and distorted face-swapped images.

Method | Original accuracy | Distorted accuracy
---|---|---
Xception [2] | 67.1% | 71.4%
MesoNet [3] | 71.9% | 73.3%
Capsule-forensics [4] | 60.5% | 65.4%
EfficientNetB4Att[45] | 69.8% | 73.6%
CNNDetection[46] | 78.3% | 79.7%

###  IV-D Enhancement on DeepFake detection

Concurrently, DeepFake detection is also served as a solid mechanism to defend
against DeepFake face swapping. Meanwhile, we explore several DeepFake
detection methods to verify the detection facilitation of our methods. To be
specific, we compare the detection success rate of original face-swapped
images and disrupted face-swapped images. The comparison results of both
original and distorted face-swapped image on our proposed dataset is presented
in Table V. The table demonstrates that the distortion on face-swapped images
can enhance the DeepFake detection performance. Moreover, it also shows the
effectiveness of our DeepFake disruption method. The main reason behind the
facilitation is that these DeepFake detection methods are focused on the
imperceptible abnormal area of the input face images, while adversarial
examples can enlarge the corresponding distortion of output face-swapped
images.

Note that under the disturbance of cross domain DeepFake examples, these
DeepFake detection methods present a slightly poor performance on our DeepFake
face swapping dataset. This also demonstrates the significance of our method
to prevent malicious face swapping from the generation stage. Apart from
distorting face-swapped images, our DeepFake disruption method can also be
served as assistance for DeepFake detection.

Method | SSIM$\downarrow$ | FSIM$\downarrow$ | BRISQUE$\uparrow$
---|---|---|---
StarGAN [47] | 0.591 | 0.806 | 49.51
GANImation [48] | 0.796 | 0.898 | 42.27
SaGAN [49] | 0.864 | 0.919 | 39.23
AttGAN [50] | 0.841 | 0.908 | 43.76

TABLE VI:  Quantitative results of image quality assessment on the disrupted
face style translation images.

###  IV-E Generalization to face translation methods

In order to investigate the generalizability ability of our method on other
face manipulation models, we transfer our adversarial attacks against several
face image translation models in the restricted black-box scenario. With
respect to StarGAN [47] and AttGAN [50], the corresponding face image
translation results of both legitimate examples and adversarial examples are
shown in Fig. 6 and Fig. 7. It is clear that adversarial examples are nearly
the same in human vision, yet they can strongly distort the face translation
with respect different attributes. Note that different perturbed image
translation results are all induced by the same input of adversarial example.
In other words, the generated adversarial perturbation is image-agnostic
against these face style translation models, thus can be adapted to arbitrary
face images.

To further quantity the visual distortion caused by restricted adversarial
examples, we present image quality assessments on the disrupted face
translation results as shown in Table VI. Note that the face style translation
is conducted on the CelebA [38] dataset. Moreover, we follow the same setting
as the DeepFake disruption in the restricted black-box scenario. In
consideration of the fixation of the adversarial perturbation upper bound, we
mainly focus on the disturbance on face style-translated images. The
generalization to face style translation models also represents that our
proposed method can be served as an effective way to defend from face
manipulation.

##  V Conclusion

In this paper, we extensively explore the practical DeepFake defense scenario
with a restricted black-box adversarial attack. We design the TCA-GAN method
to generate transferrable adversarial perturbation via attacking a substitute
model. Afterward, we adapt a novel post-regularization on the synthesized
adversarial example to further enhance the generalization ability, which can
induce intense disruption on DeepFake face swapping. For ease of subsequent
researches on DeepFake disruption, we construct a DeepFake face swapping
benchmark, including the face swapping models. To comprehensively evaluate the
disruption induced by our adversarial examples, we conduct experiments on both
referenced and non-referenced image quality assessments. Notably, we also show
that our method can enhance the performance of DeepFake detection, which is
more beneficial to defend against malicious face swapping. Subsequently, the
extension to other face manipulation methods can further demonstrate the
generalization performance of our method. We hope our work can shed light on
protecting personal photos from unauthorized face manipulation in the real-
world scenario.

## References

  * [1] P. Korshunov and S. Marcel, “Deepfakes: a new threat to face recognition? assessment and detection,” _arXiv preprint arXiv:1812.08685_ , 2018.
  * [2] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1–11.
  * [3] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in _2018 IEEE International Workshop on Information Forensics and Security (WIFS)_. IEEE, 2018, pp. 1–7.
  * [4] H. H. Nguyen, J. Yamagishi, and I. Echizen, “Capsule-forensics: Using capsule networks to detect forged images and videos,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 2307–2311.
  * [5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [7] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NIPS_ , 2014, pp. 2672–2680.
  * [8] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _ICLR_ , 2014\.
  * [9] Z. Che, A. Borji, G. Zhai, S. Ling, J. Li, Y. Tian, G. Guo, and P. Le Callet, “Adversarial attack against deep saliency models powered by non-redundant priors,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1973–1988, 2021.
  * [10] L. Gao, Z. Huang, J. Song, Y. Yang, and H. T. Shen, “Push pull: Transferable adversarial examples with attentive attack,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2021.
  * [11] J. Kos, I. Fischer, and D. Song, “Adversarial examples for generative models,” in _SPW_ , 2018, pp. 36–42.
  * [12] I. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” in _ICLR_ , 2015.
  * [13] C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song, “Generating adversarial examples with adversarial networks,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , 2018, pp. 3905–3911.
  * [14] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _Proceedings of the 10th ACM workshop on artificial intelligence and security_ , 2017, pp. 15–26.
  * [15] V. Blanz, K. Scherbaum, T. Vetter, and H.-P. Seidel, “Exchanging faces in images,” in _Computer Graphics Forum_ , 2004, pp. 669–676.
  * [16] D. Bitouk, N. Kumar, S. Dhillon, P. Belhumeur, and S. K. Nayar, “Face swapping: automatically replacing faces in photographs,” in _ACM SIGGRAPH_ , 2008, pp. 1–8.
  * [17] I. Korshunova, W. Shi, J. Dambre, and L. Theis, “Fast face-swap using convolutional neural networks,” in _ICCV_ , 2017, pp. 3697–3705.
  * [18] J. Bao, D. Chen, F. Wen, H. Li, and G. Hua, “Towards open-set identity preserving face synthesis,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 6713–6722.
  * [19] L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Faceshifter: Towards high fidelity and occlusion aware face swapping,” _arXiv preprint arXiv:1912.13457_ , 2019.
  * [20] Y. Nirkin, Y. Keller, and T. Hassner, “Fsgan: Subject agnostic face swapping and reenactment,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 7184–7193.
  * [21] R. Chen, X. Chen, B. Ni, and Y. Ge, “Simswap: An efficient framework for high fidelity face swapping,” in _Proceedings of the 28th ACM International Conference on Multimedia_ , 2020, pp. 2003–2011.
  * [22] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” _arXiv preprint arXiv:1607.02533_ , 2016.
  * [23] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [24] S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, “Universal adversarial perturbations,” in _CVPR_ , 2017, pp. 1765–1773.
  * [25] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” _IEEE Transactions on Evolutionary Computation_ , vol. 23, no. 5, pp. 828–841, 2019.
  * [26] F. Croce and M. Hein, “Minimally distorted adversarial examples with a fast adaptive boundary attack,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 2196–2205.
  * [27] H. Zhang, Y. Avrithis, T. Furon, and L. Amsaleg, “Walking on the edge: Fast, low-distortion adversarial examples,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 701–713, 2021.
  * [28] Y. Zhong and W. Deng, “Towards transferable adversarial attack against deep face recognition,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 1452–1466, 2021.
  * [29] P. Tabacof, J. Tavares, and E. Valle, “Adversarial images for variational autoencoders,” _arXiv preprint arXiv:1612.00155_ , 2016.
  * [30] N. Ruiz and S. Sclaroff, “Disrupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems,” _arXiv preprint arXiv:2003.01279_ , 2020.
  * [31] N. Ruiz, S. A. Bargal, and S. Sclaroff, “Protecting against image translation deepfakes by leaking universal perturbations from black-box neural networks,” _arXiv preprint arXiv:2006.06493_ , 2020.
  * [32] Z. Fang, Y. Yang, J. Lin, and R. Zhan, “Adversarial attacks for multi target image translation networks,” in _2020 IEEE International Conference on Progress in Informatics and Computing (PIC)_. IEEE, 2020, pp. 179–184.
  * [33] J. Dong and X. Xie, “Visually maintained image disturbance against deepfake face swapping,” in _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , 2021.
  * [34] P. Sun, Y. Li, H. Qi, and S. Lyu, “Landmark breaker: Obstructing deepfake by disturbing landmark extraction,” in _2020 IEEE International Workshop on Information Forensics and Security (WIFS)_. IEEE, 2020, pp. 1–6.
  * [35] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _Proceedings of the 2017 ACM on Asia conference on computer and communications security_ , 2017, pp. 506–519.
  * [36] N. Inkawhich, W. Wen, H. H. Li, and Y. Chen, “Feature space perturbations yield more transferable adversarial examples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7066–7074.
  * [37] Q. Huang, I. Katsman, H. He, Z. Gu, S. Belongie, and S.-N. Lim, “Enhancing adversarial example transferability with an intermediate level attack,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 4733–4742.
  * [38] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 3730–3738.
  * [39] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [40] L. Zhang, L. Zhang, X. Mou, and D. Zhang, “Fsim: A feature similarity index for image quality assessment,” _IEEE transactions on Image Processing_ , vol. 20, no. 8, pp. 2378–2386, 2011.
  * [41] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” _IEEE Transactions on image processing_ , vol. 21, no. 12, pp. 4695–4708, 2012.
  * [42] C. Xie, Z. Zhang, Y. Zhou, S. Bai, J. Wang, Z. Ren, and A. L. Yuille, “Improving transferability of adversarial examples with input diversity,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2730–2739.
  * [43] Y. Dong, T. Pang, H. Su, and J. Zhu, “Evading defenses to transferable adversarial examples by translation-invariant attacks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4312–4321.
  * [44] F. Croce and M. Hein, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 2206–2216.
  * [45] N. Bonettini, E. D. Cannas, S. Mandelli, L. Bondi, P. Bestagini, and S. Tubaro, “Video face manipulation detection through ensemble of cnns,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 5012–5019.
  * [46] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot… for now,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 8695–8704.
  * [47] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8789–8797.
  * [48] A. Pumarola, A. Agudo, A. M. Martinez, A. Sanfeliu, and F. Moreno-Noguer, “Ganimation: Anatomically-aware facial animation from a single image,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 818–833.
  * [49] G. Zhang, M. Kan, S. Shan, and X. Chen, “Generative adversarial network with spatial attention for face attribute editing,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 417–432.
  * [50] Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen, “Attgan: Facial attribute editing by only changing what you want,” _IEEE Transactions on Image Processing_ , vol. 28, no. 11, pp. 5464–5478, 2019.