# One-Shot Face Video Re-enactment using
Hybrid Latent Spaces of StyleGAN2

Trevine Oorloff and Yaser Yacoob
University of Maryland, College Park, MD 20742, USA
<EMAIL_ADDRESS>
###### Abstract

While recent research has progressively overcome the low-resolution constraint
of one-shot face video re-enactment with the help of StyleGAN’s high-fidelity
portrait generation, these approaches rely on at least one of the following:
explicit 2D/3D priors, optical flow based warping as motion descriptors, off-
the-shelf encoders, _etc_., which constrain their performance (_e.g_.,
inconsistent predictions, inability to capture fine facial details and
accessories, poor generalization, artifacts). We propose an end-to-end
framework for simultaneously supporting face attribute edits, facial motions
and deformations, and facial identity control for video generation. It employs
a hybrid latent-space that encodes a given frame into a pair of latents:
Identity latent, $\mathcal{W}_{ID}$, and Facial deformation latent,
$\mathcal{S}_{F}$, that respectively reside in the $W+$ and $SS$ spaces of
StyleGAN2. Thereby, incorporating the impressive editability-distortion trade-
off of $W+$ and the high disentanglement properties of $SS$. These hybrid
latents employ the StyleGAN2 generator to achieve high-fidelity face video re-
enactment at $1024^{2}$. Furthermore, the model supports the generation of
realistic re-enactment videos with other latent-based semantic edits (_e.g_.,
beard, age, make-up, _etc_.). Qualitative and quantitative analyses performed
against state-of-the-art methods demonstrate the superiority of the proposed
approach. The project page is located at
https://trevineoorloff.github.io/FaceVideoReenactment_HybridLatents.io/.

Figure 1:  The proposed end-to-end framework facilitates one-shot face video
re-enactment at 10242, purely based on StyleGAN2’s latent spaces without
explicit priors. It also enables temporally consistent latent-based attribute
edits such as age, beard, and make-up to be applied on the re-enactments.
Observe similar head-pose and expression in re-enacted frames from Top to
Bottom in a column.

##  1 Introduction

One-shot face video re-enactment refers to the process of generating a video
by animating the identity of a portrait image (source frame) mimicking the
facial deformations and head pose of a driving video. The increasing interest
in virtual reality has stimulated video re-enactment due to its wide range of
applications (_e.g_., digital avatars, animated movies, telepresence).

The task of one-shot face re-enactment is challenging since it requires
extraction of (1) identity and 3D facial structure of the given 2D source
frame and (2) motion information from the driving frames, to facilitate
realistic animations, despite the unavailability of paired data. Most common
approaches include the use of 2D landmarks [37, 27, 50, 18], 3D parameterized
models [49, 19, 11, 30], or latents [44, 8, 51, 46] to capture the underlying
facial structure and/or motion. Employing strict facial-structure priors may
support rigorous control of the facial structure and motion, but these priors
suffer from lack of generalizability (for different face geometries),
inability to capture fine/complex facial deformations (_e.g_. wrinkles,
internal mouth details such as tongue and teeth), inability to handle
accessories such as eyeglasses, and inconsistencies in prediction which hinder
their performance. In addition to latent-based models, research such as [35,
42] proposed models that alleviate the dependency on pre-defined priors by
predicting 2D/3D keypoints employing additional networks in an unsupervised
manner. Even though such models improved generalization, they are limited to
producing low resolution videos (mostly $256^{2}$, but some $512^{2}$).

StyleGAN’s [22] ability to produce high-resolution ($1024^{2}$) photo-
realistic faces, richness and semantic interpretability of its latent spaces
[28, 47, 13, 33], and the improvements in inversion techniques contributed to
improved re-enactment generations [28, 49, 19]. However, both [19] and [49]
use 3D parameterized models to capture the deformations of the facial
attributes and thus share the drawbacks of using a pre-defined structural
prior as discussed previously.

Considering the latent space manipulations in [47, 13, 28, 2] it is evident
that the latent space of a pre-trained StyleGAN has implicit 3D information
embedded within it. We conjecture that the StyleGAN’s latent spaces are not
yet fully exploited for re-enactment and the use of explicit structural
representations is redundant and limits the performance of StyleGAN to the
capacity-limits of such structural priors. While [28] encodes the facial
deformations of the driving sequence directly on the Style-latent space, it
follows an optimization-based approach to deduce the deformation encoding
which is time-consuming similar to other optimization-based approaches which
limits its practicality.

We address the following question: Can we learn a general model to facilitate
face identity, attributes, and motion edits exploiting the latent spaces of
StyleGAN2 without reliance on explicit 2D/3D facial structure models while
improving the performance of generating realistic, high-quality, and
temporally consistent one-shot face videos?

We propose a novel framework that encodes a portrait image as an Identity
latent, $\mathcal{W}_{ID}$, and a Facial deformation latent,
$\mathcal{S}_{F}$, that reside in the pre-defined latent spaces of StyleGAN2.
This encoding not only facilitates high-quality high-resolution ($1024^{2}$)
one-shot face re-enactment (both same and cross-identity) through the
StyleGAN2 generator, but is also capable of generating re-enactment videos
with realistic facial edits (_e.g_., beard, age, make-up) accommodating latent
manipulation techniques such as [33, 13].

Considering the three prominent intermediate latent spaces of StyleGAN2: $W$,
$W+$, and $SS$: the $W$ space suffers from poor inversion; the $W+$ space has
the best trade-off between inversion quality and editability as demonstrated
by StyleGAN inversion and latent manipulation research [36, 4, 31]; and the
StyleSpace, $SS$, is the most disentangled [47]. We combine the $W+$ and $SS$
spaces into a hybrid space, so that the Identity latent, $\mathcal{W}_{ID}$,
and face feature deformation latent, $\mathcal{S}_{F}$, are learned by
encoders that capture the richness embedded in these latent spaces. Thereby,
simultaneously supporting face attribute edits, facial motions and
deformations, and facial identity control. In summary, our key contributions
include:

  * •

A novel framework that enables high-fidelity robust one-shot face re-enactment
(same and cross-identity) video generation at $1024^{2}$, which also
facilitates realistic facial edits such as age, beard, and make-up,

  * •

A novel approach of using a combination of two pre-defined latent spaces ($W+$
and $SS$) of StyleGAN2 to have zero dependencies on explicit structural facial
priors such as 2D/ 3D landmarks or parameterizations,

  * •

A novel “Cyclic Manifold Adjustment” (CMA), that locally adjusts the
StyleGAN2’s manifold to improve the reconstruction of an out-of-domain source
and enable seamless transfer of facial deformations of the driving video to
the source image.

##  2 Related Work

Latent Space Manipulation of StyleGAN: Since the proposal of StyleGAN, there
has been a plethora of research on the semantic interpretability of the
intermediate latent spaces [33, 13, 47, 34]. Improvements in GAN inversion
techniques [1, 36, 4, 32, 40] complemented such research facilitating
realistic edits of real-world images through latent-space manipulations.

Semantic editing using latent spaces has employed supervised [33, 47, 2] and
unsupervised approaches [13, 39, 29]. These methods enable realistic facial
edits such as head-pose, expression, gaze, gender, age, and eyeglasses by
traversing the latent space of a pre-trained StyleGAN.

Figure 2: The pipeline of the proposed framework. The high-level re-enactment
process (Top), the expanded architectures of the encoding (Bottom-Left) and
re-enactment (Bottom-Right) processes are depicted. In encoding, given a
frame, the Encoder, $E$, outputs a pair of latents: Identity latent,
$\mathcal{W}_{ID}$, and Facial-deformation latent, $\mathcal{S}_{F}$. In re-
enactment, $\mathcal{S}_{F}^{D}$ (driving frame) is added to
$\mathcal{W}_{ID}^{S}$ (source), transformed using $A(\cdot)$ to obtain the
animated $SS$ latent, which is used to obtain the re-enacted frame using the
StyleGAN2 Generator, $G$.

Face Video Re-enactment: Face video re-enactment approaches can be categorized
based on the identity/motion representation or the approach used to generate
the animated frames. The common approaches used for identity/motion
representation include facial landmarks [37, 27, 50, 18], 3D facial priors
[19, 11, 30], 2D/3D predictive keypoint methods [35, 42], and intermediate
latent representations [44, 8, 51, 46]. While 3D facial priors address the
main issue of 2D facial priors, _i.e_., unrealistic deformations caused during
significant motion, their performance is limited by the lack of fine visual-
details surrounding face dynamics (wrinkles, tongue and teeth, non-rigid
deformations), inability to represent accessories such as eyeglasses, and
representation of only the inner face region. Further, the use of 2D/3D facial
priors leads to spatio-temporal incoherence stemming from the inconsistencies
of the landmarks/parameters and poor generalization in cases of varying facial
geometries between the driving and source identities. Even though keypoint
predictive networks and latent-based methods have improved generalization,
they have been limited to low-resolution video generation.

The work on re-enactment either follows an optical flow based warping strategy
[35, 42, 10] or a generative approach [44, 16, 8] to animate frames. Even
though warping methods yield results with high resemblance to the in-the-wild
source images due to operating in image space, they could cause unrealistic
deformations in faces, have weaker generalization compared to generative
approaches, and perform poorly in generating facial structures that were not
visible in the source image (_e.g_., filling in teeth in opening of the mouth,
opening eyes, ears when rotating the head).

In contrast to previous approaches, LIA [44] utilizes a latent-space
navigation based generative approach to facilitate re-enactment. However, the
results are limited to $256^{2}$ and require the pose of the source and the
initial driving frame to be of similar pose which limits the practicality.

StyleGAN-based Face Re-enactment: Recent research [19, 49, 7] employed
StyleGAN2 for high-resolution one-shot face video re-enactment due to its
ability to produce realistic portraits at $1024^{2}$ and the rich semantic
properties of its latent spaces. MegaFR [19] encodes the residual deformation
between the source image and a 3D rendering of the animated frame as an
additive $W+$ space offset. The rendering of the animated frame is a
parameterized approximation obtained using a combination of 3DMM parameters of
the source and driving frames. Bounareli _et al_. in [7] map the difference
between the output parameters of a 3D model onto the $W+$ space to obtain the
re-enacted frame’s latent. StyleHEAT [49] uses 3DMM parameters to capture the
facial deformations to generate flow fields which are used to warp an
intermediate feature space of the generator.

While the above methods yield promising results, they employ 3D models to
parameterize the motion, thus, suffer from the limitations of using 3D priors
as explained previously (_i.e_., inconsistencies, lack of fine-grained
details, limited by the capacity of the 3D model, _etc_.). Our proposed model
does not use explicit 2D/ 3D structural models, instead, we exploit the
editability and disentanglement of the latent spaces ($W+$ and $SS$) and its
implicit 3D priors to encode both the identity and the motion within the
StyleGAN2’s pre-defined latent spaces. Importantly, we propose a unified end-
to-end system trained using a self-supervised approach and hence is not
bounded by the limitations of other models (_e.g_., [49] and [19] rely on
inversion models to obtain the source latent, [49, 19, 7] rely on explicit 3D
models). While StyleGAN2 generation inherently suffers from texture-sticking,
StyleHEAT aggravates the issue as their model warps the intermediate feature
space of the StyleGAN2 generator. Moreover, compared to [19, 7] that encode in
the $W+$ space, we employ a hybrid latent space approach using both $W+$ and
$SS$ latent spaces to exploit high inversion-editability and high
disentanglement respectively.

##  3 Methodology

Preliminaries: The StyleGAN2 generator consists of several latent spaces: $Z$,
$W$, $W+$, and $SS$. The vanilla generation process consists of an initial
random noise latent, $z\in Z\sim\mathcal{N}(\mu,\,\sigma^{2})$, which is
mapped to the $w\in W$ latent using a sequence of fully connected networks,
which is then transformed using layer-wise affine transformations, $A(\cdot)$,
to form the StyleSpace, $SS$. While this model uses the same $w\in W$ latent
to each of the transformations in $A(\cdot)$, StyleGAN2 inversion methods such
as [31, 36, 4] use an extension of the $W$ space defined as $W+$, where the
input to each transformation in $A(\cdot)$ is allowed to differ in order to
improve the reconstruction quality. While there are multiple approaches for
forming $W+$, we refer to the $W+$ space in [31] due to its impressive balance
between the distortion-editability trade-off. Further, as evaluated by [47],
the $SS$ has the best disentanglement, completeness, and informativeness
scores among $W$, $W+$, and $SS$ spaces thus making it the best latent-space
for disentangled manipulations.

Overview: As shown in Fig. 2, the proposed framework comprises of two main
networks, an encoder, $E$, and a decoder, $G$: a pre-trained StyleGAN2
generator. The encoder consists of two heads: Identity Encoder, $E_{ID}$, and
Facial Deformation Encoder, $E_{F}$, preceded by a common Feature Extraction
Network, $F$. Given an input frame, the encoder creates two latents,
$\mathcal{W}_{ID}$ and $\mathcal{S}_{F}$, capturing identity and facial
deformations respectively, such that,

$\displaystyle\mathcal{W}_{ID}=E_{ID}(F(I)),\quad\mathcal{S}_{F}=E_{F}(F(I)),$
(1) $\displaystyle I=G\left\\{A(\mathcal{W}_{ID})+\mathcal{S}_{F}\right\\}.$
(2)

While $\mathcal{W}_{ID}$ resides in the entire $W+$ space _i.e_.,
$\mathcal{W}_{ID}\in W+\in\mathbb{R}^{18\times 512}$, $\mathcal{S}_{F}$
resides in the space spanned by the first 10 $SS$ layers _i.e_.,
$\mathcal{S}_{F}\in SS_{1:10}\in\mathbb{R}^{10\times 512}$. This is based on
(1) the observation in [28, 47] that $SS$ latents corresponding to facial
deformations of interest, (_i.e_., pose, mouth, gaze, eyes, eyebrows, and
chin) lie within the first $10$ layers of $SS$ and (2) avoiding the appearance
jitters caused by edits on high-resolution feature layers [48].

In re-enactment, we follow a frame-wise approach, where a single source frame,
$I^{S}$, and each frame, $I^{D}_{t}$, of the driving sequence are projected to
$\\{\mathcal{W}_{ID}^{S},\mathcal{S}_{F}^{S}\\}$ and
$\\{\mathcal{W}_{ID_{t}}^{D},\mathcal{S}_{F_{t}}^{D}\\}$ respectively (using
Eq. 1). Thereafter, the animated frame, $I^{S\rightarrow D}_{t}$ is generated
using $G$, sourcing $\mathcal{W}_{ID}^{S}$ and $\mathcal{S}_{F}^{D}$ latents,
comprising of the source identity and the driving frame’s facial deformations
respectively.

$I^{S\rightarrow
D}_{t}=G\left\\{A(\mathcal{W}_{ID}^{S})+\mathcal{S}_{F_{t}}^{D}\right\\}$ (3)

As seen in Eq. 3, the additive latent $\mathcal{S}_{F_{t}}^{D}$ constitutes a
latent edit performed on $\mathcal{W}_{ID}^{S}$. Thus, it is important for
$\mathcal{W}_{ID}^{S}$ to reside in the well-behaved editable regions of the
latent spaces of StyleGAN2 (_i.e_., $W+$) and accommodate a wide range of face
deformation latent edits imposed by the driving sequence
($\\{\mathcal{S}_{F_{t}}^{D}\\}$) and for $\mathcal{S}_{F_{t}}^{D}$ to reside
in a highly disentangled region (_i.e_. $SS$) of the latent spaces, so that it
minimizes identity leakage and altering the source identity across frames.
This design enables the manipulation of attributes such as age, beard, make-
up, _etc_. through latent space edits proposed by [33, 13].

$I^{S\rightarrow
D}_{edit}=G\left\\{A(\mathcal{W}_{ID}^{S}+\mathcal{W}_{edit})+\mathcal{S}_{F}^{D}\right\\}$
(4)

###  3.1 Architecture

Feature Extraction Network, $F$: We use a ResNet50-SE backbone [14, 17]
extended with a feature pyramid [24] to extract the coarse, medium, and fine
features of each frame similar to [31]. These levels correspond to the levels
of features addressed by each latent layer as in [21].

Identity Encoder, $E_{ID}$: The $E_{ID}$ consists of network blocks similar to
“map2style” in [31] where the feature maps of the corresponding granularity
are gradually reduced to $\mathbb{R}^{512}$ using a fully convolutional
network. The encoder consists of $18$ such blocks each predicting a single
layer (dimension) of $\mathcal{W}_{ID}\in\mathbb{R}^{18\times 512}$.

Facial Deformation Encoder, $E_{F}$: While $E_{F}$ has a similar architecture
to $E_{ID}$, it consists of only $10$ “map2style” blocks as we limit the $SS$
latent edits to only the first 10 layers of $SS$ as explained above.

Decoder, $G$: We use the pre-trained StyleGAN2 generator, which facilitates
the input of $SS$ latents [47], as the decoder to generate the re-enacted
frames from the latents.

###  3.2 Implementation

Due to the unavailability of paired re-enactment datasets, we follow a self-
supervised training approach to learn the weights of the encoder, $E$. During
training, we randomly sample a single source frame, $I^{S}$, and two driving
frames, $I^{D1}$ and $I^{D2}$, one belonging to the same identity as $I^{S}$
and the other from a randomly selected different identity respectively. The
three frames, $I^{S}$, $I^{D1}$, and $I^{D2}$ are sent through the encoder,
$E$, to obtain the corresponding latents,
$\\{\mathcal{W}^{S}_{ID},\mathcal{S}_{F}^{S}\\}$,
$\\{\mathcal{W}^{D1}_{ID},\mathcal{S}_{F}^{D1}\\}$, and
$\\{\mathcal{W}^{D2}_{ID},\mathcal{S}_{F}^{D2}\\}$ respectively (using Eq. 1).
We learn the weights of $E$ by optimizing over the following loss functions:

Reconstruction Losses: The reconstruction losses are two-fold comprising of a
self-reconstruction loss, Eq. 5, and a re-enactment loss, Eq. 6, which measure
the reconstruction of the source frame, $I^{S\rightarrow S}$, and the same-
identity driving frame, $I^{S\rightarrow D1}$, using the source identity
latent, $\mathcal{W}_{ID}^{S}$, and the corresponding facial deformation
latents.

$\displaystyle\mathcal{L}_{self}=\mathcal{L}_{rec}\left\\{I^{S},G\\{A(\mathcal{W}_{ID}^{S})+\mathcal{S}_{F}^{S}\\}\right\\}$
(5)
$\displaystyle\mathcal{L}_{reenact}=\mathcal{L}_{rec}\left\\{I^{D1},G\\{A(\mathcal{W}_{ID}^{S})+\mathcal{S}_{F}^{D1}\\}\right\\}$
(6)
$\displaystyle\mathcal{L}_{rec}=\lambda_{L2}\mathcal{L}_{L2}+\lambda_{LPIPS}\mathcal{L}_{LPIPS}+\lambda_{GV}\mathcal{L}_{GV}$
(7)

where, $\mathcal{L}_{rec}$ is a weighted sum of the MSE loss,
$\mathcal{L}_{L2}$, LPIPS loss [54], $\mathcal{L}_{LPIPS}$, and Gradient
Variance loss [3], $\mathcal{L}_{GV}$, weighed by $\lambda_{L2}$,
$\lambda_{LPIPS}$, and $\lambda_{GV}$ respectively.

Identity Loss: The identity loss is computed using,

$\displaystyle\mathcal{L}_{id}$
$\displaystyle=\\{1-\left\langle\phi(I^{S}),\phi(I^{S_{ID}})\right\rangle\\}$
(8) $\displaystyle+\\{1-\left\langle\phi(I^{S}),\phi(I^{S\rightarrow
D2})\right\rangle\\}$ (9)

where, the cosine similarity ($\langle\cdot,\cdot\rangle$) of the ArcFace[9]
feature space is measured between the pair of images.
$I^{S_{ID}}=G\\{A(\mathcal{W}_{ID}^{S})\\}$ is the source identity image and
$I^{S\rightarrow D2}=G\\{A(\mathcal{W}_{ID}^{S})+\mathcal{S}_{F}^{D2}\\}$
denotes the re-enactment of the source representing the facial deformations of
$I^{D2}$. Eq. 8 ensures the identity of the source is captured within
$\mathcal{W}_{ID}^{S}$ and also prevents the optimization from converging to
the trivial solution of Eqs. 5 and 6, which is $\mathcal{W}_{ID}^{S}=0$.
Further, Eq. 9 makes sure that puppeteering preserves identity _i.e_.,
minimizes the identity information leakage to $\mathcal{S}_{F}$.

Identity Latent Consistency Loss: For additional control over puppeteering, to
encourage consistent identity latents irrespective of head-pose and facial
attributes of the source, we obtain the $\mathcal{W}_{ID}$ of $I^{S\rightarrow
D2}$ by passing it through $E$ and compute the following loss over the latent
space.

$\mathcal{L}_{w\\_id}=\lVert{W}_{ID}^{S}-{W}_{ID}^{S\rightarrow D2}\rVert_{2}$
(10)

Regularization Loss: Additional regulatory losses are used to reduce the
variance within the $\mathcal{W}_{ID}^{S}$ [36] and to control the facial-
deformation edits, $\mathcal{S}_{F}^{D1}$, to be within the proximity of
$A(\mathcal{W}_{ID}^{S})$ combined in a ratio of $1:\lambda_{S}$.
$\mathcal{W}_{ID}^{S}[i]$ and $\mathcal{S}_{F}^{D1}[j]$ correspond to the
$i^{th}$ and $j^{th}$ dimension of $\mathcal{W}_{ID}^{S}$ and
$\mathcal{S}_{F}^{D1}$ latents respectively.

$\displaystyle\Delta_{i}=\mathcal{W}_{ID}^{S}[i]-\mathcal{W}_{ID}^{S}[1]$ (11)
$\displaystyle\mathcal{L}_{reg}=\sum_{i=1}^{18}\\{\lVert\Delta_{i}\rVert_{2}\\}+\lambda_{S}\sum_{j=1}^{10}\\{\lVert\mathcal{S}_{F}^{D1}[j]\rVert_{2}\\}$
(12)

Feature Reconstruction Loss: Complementary to the reconstruction losses,
feature reconstruction losses are computed using the same loss functions with
the exception of the losses being computed on a dilated masked region
consisting of the mouth, eyes, and eyebrows to increase the emphasis on
capturing facial deformations accurately.

$\displaystyle\mathcal{L}_{f}$
$\displaystyle=\mathcal{L}_{rec}\left\\{M^{S}\odot I^{S},M^{S}\odot
I^{S\rightarrow S}\right\\}$
$\displaystyle+\mathcal{L}_{rec}\left\\{M^{D1}\odot I^{D1},M^{D1}\odot
I^{S\rightarrow D1}\right\\}$ (13)

Additionally, similar to [36] we train a Latent Discriminator, with an
adversarial loss, $\mathcal{L}_{d}$, to encourage the $W_{ID}^{S}$ latents to
be in the well-editable regions of the StyleGAN2 latent space.

Total Loss: The total loss is as follows, where $\lambda_{*}$ represents the
corresponding weights.

$\displaystyle\mathcal{L}=\mathcal{L}_{self}+\mathcal{L}_{reenact}+\lambda_{id}\cdot\mathcal{L}_{id}+\lambda_{w\\_id}\cdot\mathcal{L}_{w\\_id}$
$\displaystyle+\lambda_{d}\cdot\mathcal{L}_{d}+\lambda_{reg}\cdot\mathcal{L}_{reg}+\lambda_{f}\cdot\mathcal{L}_{f}$
(14)

Figure 3:  Cyclic Manifold Adjustment (CMA). For an out-of-domain subject,
$\widetilde{\mathcal{W}}_{ID}^{S}$, $\mathcal{W}_{ID}^{S}$, and
$\\{\mathcal{S}_{F_{t}}\\}$, represent the true identity, identity latent
estimate obtained using $E$, and the sequence of facial deformation latents
obtained from the driving sequence. We locally tweak the StyleGAN2’s manifold
around $\mathcal{W}_{ID}^{S}$, to include the latent space spanned by
$\\{\mathcal{S}_{F_{t}}\\}$ centered around
$\widetilde{\mathcal{W}}_{ID}^{S}$, thus improving the source identity
reconstruction and enabling seamless transfer of facial deformations of the
driving video.  Figure 4:  Qualitative evaluation of same-identity (Top) and
cross-identity (Bottom) re-enactment. Same-Identity Re-enactment: Observe the
lack of sharpness in facial features (_e.g_. teeth, wrinkles, eyes), visual
artifacts around eyes, ears, and mouth, and incorrect facial features in
baseline methods in comparison to our approach. Cross-Identity Re-enactment:
Observe in comparison to the baselines: 4th row: teeth, mouth formation, and
head-pose; 5/6th row: preservation of source identity and lip structure, and
the expression of driving.

###  3.3 Cyclic Manifold Adjustment (CMA)

StyleGAN2 based approaches have a comparatively weaker identity reconstruction
for out-of-domain subjects. While several methods [40, 32] have been proposed
to improve the reconstruction quality of real-world images, none of them could
be directly used for cross-identity re-enactment due to the unavailability of
paired-data. The use of such approaches on the source image alone results in
subpar performance and/or generates visual artifacts when the facial
deformation latents ($\\{\mathcal{S}_{F_{t}}^{D}\\}$) are added.

To improve the identity reconstruction quality of out-of-domain subjects in a
cross-identity re-enactment setting, we propose a novel approach, “Cyclic
Manifold Adjustment”, inspired by PTI [32]. Suppose the true source identity
latent is $\widetilde{\mathcal{W}}_{ID}^{S}$ which is out of StyleGAN2’s
domain. Fine-tuning the StyleGAN2 generator, $G$, using PTI on the source
image, would constrain the local latent space around the source identity,
$\widetilde{\mathcal{W}}_{ID}^{S}$, thus limiting the editability through
facial deformation latents of the driving frames,
$\\{\mathcal{S}_{F_{t}}^{D}\\}$. In contrast, Cyclic Manifold Adjustment
tweaks the latent space manifold around the source identity latent estimate,
$\mathcal{W}_{ID}^{S}$ obtained through $E$, to include the latent space
spanned by the sequence $\\{\mathcal{S}_{F_{t}}^{D}\\}$ centered around
$\widetilde{\mathcal{W}}_{ID}^{S}$ as depicted in Fig. 3. This novel approach
improves the identity reconstruction of the out-of-domain source and enables
seamless transfer of facial deformations of the driving video. We achieve this
by optimizing the cost function,

$\mathcal{L}\\{I^{S}_{cyc},I^{S}\\}+\mathcal{L}\\{I^{D}_{cyc},I^{D}\\}$ (15)

where, $\mathcal{L}$ denotes a combination of LPIPS and L2 losses and
$I^{S}_{cyc}$ and $I^{D}_{cyc}$ are cyclic reconstructions of the source and
driving frames respectively generated as follows.

$\displaystyle\mathcal{W}_{ID\\_{cyc}}^{S},\,\mathcal{S}_{F\\_{cyc}}^{D}=E(I^{S\rightarrow
D})$ (16) $\displaystyle
I^{S}_{cyc}=G\left\\{A(\mathcal{W}_{ID\\_{cyc}}^{S})+\mathcal{S}_{F}^{S}\right\\}$
(17) $\displaystyle
I^{D}_{cyc}=G\left\\{A(\mathcal{W}_{ID}^{D})+\mathcal{S}_{F\\_{cyc}}^{D}\right\\}$
(18)

Method | res. | L1 $\downarrow$ | LPIPS$\downarrow$ |  $\mathcal{L}_{ID}$$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FID$\downarrow$ | FVD$\downarrow$ | FVDM$\downarrow$ |  $\rho_{\scaleto{AU}{3pt}}$$\uparrow$ |  $\rho_{\scaleto{GZ}{3pt}}$$\uparrow$ |  $\rho_{\scaleto{pose}{3pt}}$$\uparrow$
---|---|---|---|---|---|---|---|---|---|---|---|---
FOMM [35] | $256^{2}$ | 2.37 | 0.042 | 0.095 | 32.8 | 0.959 | 22.3 | 102.1 | 19.9 | 0.808 | 0.737 | 0.948
PIRenderer [30] | $256^{2}$ | 3.52 | 0.053 | 0.118 | 29.1 | 0.932 | 28.0 | 145.1 | 26.8 | 0.748 | 0.770 | 0.906
LIA [44] | $256^{2}$ | 2.72 | 0.049 | 0.102 | 31.5 | 0.951 | 26.4 | 105.0 | 19.6 | 0.822 | 0.732 | 0.944
fs-vid2vid [41] | $512^{2}$ | 4.38 | 0.065 | 0.151 | 27.4 | 0.919 | 31.6 | 255.0 | 45.0 | 0.572 | 0.635 | 0.822
StyleHEAT [49] | 10242 | 3.59 | 0.059 | 0.133 | 28.8 | 0.934 | 38.8 | 171.0 | 31.4 | 0.745 | 0.850 | 0.921
Ours | 10242 | 2.28 | 0.027 | 0.097 | 33.1 | 0.967 | 19.3 | 101.0 | 16.6 | 0.829 | 0.872 | 0.952
Ours w/o ID reg | $1024^{2}$ | 2.34 | 0.028 | 0.111 | 32.8 | 0.953 | 20.3 | 115.4 | 20.7 | 0.810 | 0.857 | 0.948
Ours w/o Hybrid | $1024^{2}$ | 2.52 | 0.031 | 0.114 | 31.7 | 0.950 | 20.6 | 116.9 | 23.6 | 0.746 | 0.798 | 0.923

Table 1: Quantitative comparison of one-shot same-identity re-enactment
against baselines. Top: Evaluation results computed over 75 unseen videos
(37.5K total frames) of the HDTF dataset [55]. Our approach yields the best
performance across all metrics except $\mathcal{L}_{ID}$ (comparable with
best) while generating high-resolution re-enactment. Bottom: Ablations
performed for Ours w/o ID reg.: Framework without the Identity regularization
and Ours w/o Hybrid: Framework without the Hybrid latent spaces _i.e_., both
latents in $W+$ space.

Method | FID$\downarrow$ | FVD$\downarrow$ |  $\text{FVD}_{\scaleto{M}{3pt}}$$\downarrow$ |  $\rho_{\scaleto{AU+GZ}{3pt}}$$\uparrow$ |  $\rho_{\scaleto{pose}{3pt}}$$\uparrow$
---|---|---|---|---|---
FOMM [35] | 94.0 | 529.4 | 78.4 | 0.450 | 0.782
PIRenderer [30] | 84.8 | 417.3 | 54.2 | 0.668 | 0.880
LIA [44] | 94.8 | 536.2 | 76.7 | 0.404 | 0.788
fs-vid2vid [41] | 90.6 | 532.7 | 86.7 | 0.493 | 0.745
StyleHEAT [49] | 97.2 | 408.8 | 58.9 | 0.645 | 0.875
Ours | 74.3 | 375.4 | 50.2 | 0.718 | 0.915
Ours w/o ID reg | 85.6 | 399.8 | 53.1 | 0.685 | 0.894
Ours w/o Hybrid | 93.0 | 427.3 | 60.8 | 0.649 | 0.878
Ours w/o CMA | 86.9 | 388.1 | 52.4 | 0.646 | 0.896
Ours – CMA + PTI | 84.8 | 421.2 | 56.8 | 0.627 | 0.890

Table 2: Quantitative evaluation of cross-identity one-shot re-enactment. Top:
Evaluation computed over 75 unseen videos (37.5K frames in total) of the HDTF
dataset [55] and random source frames of different identities. Our approach
yields the best performance across all metrics which is reflective of the
visual results. Bottom: Ablations for Ours w/o ID reg.: Framework without the
Identity regularization; Ours w/o Hybrid: Framework without the Hybrid latent
spaces _i.e_., both latents in $W+$ space; Ours w/o CMA: Framework without
Cyclic Manfold Adjustment (CMA) and Ours – CMA + PTI: Framework replacing
Cyclic Manfold Adjustment (CMA) with PTI[32].

##  4 Experiments and Results

Datasets: We pre-trained the encoder on the CelebV-HQ dataset [56], which
includes diverse high resolutions (min. $512^{2}$) with over 35K videos
involving 15K+ identities. The HDTF dataset [55], which consists of 362 videos
(720p/1080p) with 300+ identities, was used for the fine-tuning stage with an
80-20 non-overlapping train-test split. All the sampled frames were
preprocessed according to [35]. For training 50 samples/video were used and
for evaluation the first 500 samples/video of 75 unseen videos (total 37.5K
frames) were chosen.

Training was performed in two stages consisting of a pre-training stage
followed by a fine-tuning stage. While the former focuses mainly on learning
the features of images, improving generalization, and learning the implicit
prior of the StyleGAN2’s latent space, the latter stage focuses on capturing
the detailed facial deformations and face details.

Method | LPIPS$\downarrow$ | $\mathcal{L}_{ID}\downarrow$ | FID$\downarrow$ | FVD$\downarrow$
---|---|---|---|---
| $\times 10^{-2}$ | $\times 10^{-1}$ | $\times 10^{1}$ | $\times 10^{2}$
FOMM | $7.5\pm 2.1$ | $2.3\pm 1.1$ | $3.2\pm 1.0$ | $2.0\pm 0.7$
PIRenderer | $\underline{5.7\pm 0.4}$ | $\underline{1.2\pm 0.2}$ | $\underline{2.2\pm 0.2}$ | $1.5\pm 0.3$
LIA | $7.8\pm 2.0$ | $2.3\pm 1.0$ | $3.4\pm 1.0$ | $2.0\pm 0.8$
fs-vid2vid | $7.5\pm 1.1$ | $1.8\pm 0.5$ | $3.3\pm 0.6$ | $2.2\pm 0.5$
StyleHEAT | $6.0\pm 0.4$ | $1.4\pm 0.2$ | $3.3\pm 0.5$ | $\underline{1.5\pm 0.2}$
Ours | $\mathbf{2.9\pm 0.2}$ | $\mathbf{1.1\pm 0.1}$ | $\mathbf{1.5\pm 0.2}$ | $\mathbf{0.8\pm 0.1}$

Table 3: Ablation on One-Shot Robustness. We evaluate the robustness of each
model in the task of same-identity re-enactment using 5 driving videos (500
samples/video) and 5 different source frames per driving video (25 source
image-driving video combinations). Our approach yields the least mean and
standard deviation proving its robustness.

Pre-training Stage: The entire Encoder, $E$, was trained during this stage on
the CelebV-HQ dataset for 200K iterations. The Feature Extraction Network,
$F$, and the Identity Encoder, $E_{ID}$, were initialized with the pre-trained
weights of e4e, and the training followed a progressive approach as proposed
in [36], where latent layers are progressively incorporated into the
optimization at regular intervals. The Ranger optimizer (Rectified Adam [25]
\+ Lookahead [52]) was used to train the encoder, while Adam optimizer [23]
was used to train the latent discriminator.

Fine-tuning Stage: Subsequent to the pre-training stage, the network was fine-
tuned for 20K additional iterations on the HDTF training set with the addition
of feature reconstruction losses to capture fine facial attributes. In this
stage, the Feature Extraction Network is frozen and only the two latent-
prediction heads: $E_{ID}$ and $E_{F}$ are fine-tuned at a reduced learning
rate to avoid over-fitting. All hyperparameters remain unchanged, except
$\lambda_{f}$ incorporating feature losses to the loss objective.

Resources: The network was trained on two RTX A6000 GPUs for approximately 100
hours.

Inference Stage: During inference, a driving video and a single source frame
were obtained from the evaluation samples of the HDTF dataset. While the
source frame and the driving video are of the same identity in same-identity
re-enactment, the identities differ in the case of cross-identity re-
enactment. The animated frames were generated as explained in Eqs. 1, 2, 3 and
4. Cyclic manifold adjustment (Sec. 3.3) was used to improve realism and
visual quality. Since a generative architecture is used, our model is capable
of rendering re-enactment videos in real-time ($\sim 30$fps).

###  4.1 Baselines and Metrics

Baselines: We compare our results against a diverse range of state-of-the-art
approaches that are based on: predictive keypoints (FOMM [35]); 3D models
(PIRenderer [30], StyleHEAT[49]), facial landmarks (fs-vid2vid[41]);
intermediate latents (LIA [44]); StyleGAN2-based (StyleHEAT); warping
approaches (FOMM, fs-vid2vid, PIRender, StyleHEAT); and generative approaches
(LIA).

Metrics: We extensively evaluate the results of the proposed framework against
the baselines using (1) reconstruction fidelity: L1 norm pixel loss, Peak
Signal-to-Noise Ratio (PSNR), (2) Identity preservation: Identity loss
($\mathcal{L}_{ID}$ \- computed using [9]), (3) Perceptual quality: LPIPS
[54],SSIM [45], FID [15], (4) Spatio-temporal perceptual quality: FVD [38],
FVDM (FVD over the mouth), (5) Temporal coherence in facial attributes:
$\rho_{\scaleto{AU}{3pt}}$, $\rho_{\scaleto{GZ}{3pt}}$, and
$\rho_{\scaleto{pose}{3pt}}$, the temporal correlation of Action Units
(expressions), gaze, and pose respectively measured using [6].

###  4.2 Evaluation

Quantitative Analysis: The performance of our approach in comparison to the
baselines in the tasks of same-identity re-enactment and cross-identity re-
enactment, are tabulated in Tabs. 1 and 2 respectively. Our approach yields
the best performance across all metrics except $\mathcal{L}_{ID}$ where it
achieves comparable values to the best. A higher $\mathcal{L}_{ID}$ for FOMM
is expected as it operates in the image space as opposed to our generative
approach. However, the performance of our model generating results at
$1024^{2}$, in comparison to the baselines is not fully reflected through the
metrics, as (1) most of the metrics ($\mathcal{L}_{ID}$, FID, FVD, _etc_.) are
computed on downsampled images at a low-resolution which do not reflect the
fine-grained details and (2) the lack of high-resolution ($\geq 1024^{2}$)
data samples.

Qualitative Analysis: The same-identity and cross-identity re-enactment
examples in Fig. 4 demonstrate that our approach yields much improved visual
results compared to the baselines that are prone to visual artifacts,
incorrect facial attributes, and lack of sharpness in scenarios where the
source and driving frames have significant difference in pose and/or
expression, subject is wearing eyeglasses, filling in unseen details (_e.g_.
teeth, ear). Our approach successfully handles these cases while producing
more realistic re-enactment at high-resolution ($1024^{2}$). The versatility
of our model is more pronounced in the cross-identity examples in Fig. 4
(Bottom), where facial deformations of the driving frames are accurately
mimicked in the re-enacted frame while preserving the source identity.
Realistic re-enactments with latent-based attribute edits such as beard, age,
make-up, _etc_. applied to the source image (see [13, 33]) could be generated
as shown in Fig. 1. This is enabled by the design choice of $\mathcal{W}_{ID}$
being in the well-editable $W+$ space.

Based on the visual results, artifacts are most prominent in warping based
approaches (FOMM, fs-vid2vid, PIRenderer, StyleHEAT). Further, all approaches
except StyleHEAT are only capable of low-resolution generation. While
StyleHEAT uses StyleGAN2 to generate re-enactment videos at $1024^{2}$, the
use of a 3D model to capture the facial attributes and warping of the
intermediate feature spaces inhibits its performance. In contrast, our model
exploits implicit priors of the ore-defined latent spaces of StyleGAN2 to
achieve state-of-the-art performance. Please refer to the video examples on
the project page.

Figure 5: Identity Latent Images. Each column consists of the source frame
(Top) and the corresponding Identity latent image (Bottom). Even though the
source images significantly differ in head-pose, expression, and eyes, the
identity latent captured is nearly identical (with neutral pose and
expression) which contributes towards the robustness of our approach.

###  4.3 Ablation Study

Identity Regularization: We evaluate the impact of using the identity-loss
based regularization terms (Eqs. 8 and 10), which are in place to minimize the
identity leakage into the facial deformation latent, $\mathcal{S}_{F}$. The
results are in Tabs. 1 and 2 (Bottom) as Ours w/o ID reg. Considerable
quantitative improvement is seen in cross-identity re-enactment with the
inclusion of identity regularization.

Hybrid Latent Spaces: We propose a hybrid latent approach, where the Identity
latent, $\mathcal{W}_{ID}$, and the Facial deformation latent,
$\mathcal{S}_{F}$, reside in the domains of $W+$ and $SS$ respectively to make
use of the editability and reconstruction of the $W+$ space and the
disentanglement of the $SS$. We validate the use of a hybrid approach against
Ours w/o Hybrid, where both the Identity and Facial deformation latents reside
in the $W+$ space. Based on the results at the bottom of Tabs. 1 and 2, it
could be observed that the scores deteriorate compared to the Hybrid approach,
most likely due to the entanglements within the $W+$ space.

Cyclic Manifold Adjustment (CMA): We evaluate the performance improvement of
using Cyclic Manifold Adjustment in comparison to (1) our framework without
CMA (Tab. 2: Ours w/o CMA) and (2) replacing CMA with PTI[32] (Tab. 2: Ours –
CMA + PTI). While using CMA achieves the best results, it could be observed
that the use of PTI deteriorates the scores compared to Ours w/o CMA. Using
PTI on the source does not guarantee the local manifold around the source to
accommodate the facial deformations of the driving sequence leading to
distorted/ suboptimal results. In contrast, our approach improves the identity
reconstruction of the out-of-domain source and also enables seamless transfer
of facial deformations of the driving video to the source, as we tweak the
generator such that the driving facial deformation edits reside within the
local manifold centered around the source identity latent.

One-shot Robustness: The robustness to different head-poses and facial
attributes of the source frame is evaluated. The performance of same-identity
one-shot re-enactment of 5 driving videos with 5 different source frames per
driving video (25 source image-driving video combinations) are evaluated (see
Tab. 3). We observe that our approach is the most robust with the lowest mean
and standard deviation.

We attribute the robustness to the Identity and Facial deformation
decomposition we employ. As shown in Fig. 5 the network implicitly learns an
identity latent that has a neutral pose and expression irrespective of the
head-pose and facial attributes of the source. While most algorithms attempt
to directly capture the difference between the source and driving frames, in
contrast, we encode the facial deformations relative to an implicitly learned
neutral pose and identity. While LIA follows a similar approach of anchoring
the facial deformations to an implicitly learnt reference, their method
requires the source and the first frame of the driving sequence to have
similar pose and is limited to $256^{2}$.

###  4.4 Limitations

Since we base our model on StyleGAN2, we inherit its limitations of texture
sticking and alignment requirements. Further, handling occlusions and
reconstruction of changing backgrounds are challenging since the StyleGAN
generator is pre-trained for faces. While our model could be adapted to
StyleGAN3[20] to mitigate the issue of texture sticking, the use of StyleGAN2
is preferred due to its latent space being more structured and expressive [5].

###  4.5 Negative Societal Impact

The negative societal impact of our model is similar to that of other DeepFake
algorithms. _i.e_., the impressive performance of the proposed model in one-
shot re-enactment entails the risk of the model being used with malicious
intent. However, the advancements in re-enactment research create social
awareness in the community of such methods and also pave the path for research
on DeepFake detectors [26, 12, 43, 53]. Further, the lack of high resolution
datasets prevents our model from reaching its full potential. Moreover, the
aforementioned limitations of StyleGAN such as texture sticking and
limitations in background reconstruction would provide cues for DeepFake
detectors.

##  5 Conclusion

We propose an end-to-end unified framework to facilitate high-fidelity one-
shot facial video re-enactment at $1024^{2}$, exploiting the implicit priors
in StyleGAN2 latent spaces. The framework reveals the full potential of
StyleGAN2 for face edits, specifically identity, attributes, and facial
deformations in videos. The model is centered around hybrid latent spaces to
encode the identity and facial deformations that exploit the editability,
reconstruction, and disentanglement properties of each latent space, thus
achieving state-of-the-art results both quantitatively and qualitatively.
Moreover, the proposed model is robust to diverse head-poses and expressions
of the source frame. Further, the model facilitates generation of re-enactment
videos with latent-based edits (beard, age, make-up, _etc_.) proposed by
previous research.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  ACM Transactions on Graphics (TOG), 40(3):1–21, 2021.
  * [3] Lusine Abrahamyan, Anh Minh Truong, Wilfried Philips, and Nikos Deligiannis.  Gradient variance loss for structure-enhanced image super-resolution.  In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3219–3223. IEEE, 2022.
  * [4] Yuval Alaluf, Or Patashnik, and Daniel Cohen-Or.  Restyle: A residual-based stylegan encoder via iterative refinement.  arXiv preprint arXiv:2104.02699, 2021.
  * [5] Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani Lischinski, and Daniel Cohen-Or.  Third time’s the charm? image and video editing with stylegan3.  arXiv preprint arXiv:2201.13433, 2022.
  * [6] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency.  Openface 2.0: Facial behavior analysis toolkit.  In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pages 59–66. IEEE, 2018.
  * [7] Stella Bounareli, Vasileios Argyriou, and Georgios Tzimiropoulos.  Finding directions in gan’s latent space for neural face reenactment.  arXiv preprint arXiv:2202.00046, 2022.
  * [8] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky.  Neural head reenactment with latent pose descriptors.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13786–13795, 2020.
  * [9] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  CVPR, 2019.
  * [10] Michail Christos Doukas, Evangelos Ververas, Viktoriia Sharmanska, and Stefanos Zafeiriou.  Free-headgan: Neural talking head synthesis with explicit gaze control.  arXiv preprint arXiv:2208.02210, 2022.
  * [11] Michail Christos Doukas, Stefanos Zafeiriou, and Viktoriia Sharmanska.  Headgan: One-shot neural head synthesis and editing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14398–14407, 2021.
  * [12] Diego Gragnaniello, Davide Cozzolino, Francesco Marra, Giovanni Poggi, and Luisa Verdoliva.  Are gan generated images easy to detect? a critical analysis of the state-of-the-art.  In 2021 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2021.
  * [13] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris.  Ganspace: Discovering interpretable gan controls.  arXiv preprint arXiv:2004.02546, 2020.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [16] Gee-Sern Hsu, Chun-Hung Tsai, and Hung-Yi Wu.  Dual-generator face reenactment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 642–650, 2022.
  * [17] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [18] Po-Hsiang Huang, Fu-En Yang, and Yu-Chiang Frank Wang.  Learning identity-invariant motion representations for cross-id face reenactment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7084–7092, 2020.
  * [19] Wonjun Kang, Geonsu Lee, Hyung Il Koo, and Nam Ik Cho.  One-shot face reenactment on megapixels.  arXiv preprint arXiv:2205.13368, 2022.
  * [20] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  arXiv preprint arXiv:2106.12423, 2021.
  * [21] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [22] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [23] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [24] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [25] Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han.  On the variance of the adaptive learning rate and beyond.  arXiv preprint arXiv:1908.03265, 2019.
  * [26] Francesco Marra, Diego Gragnaniello, Davide Cozzolino, and Luisa Verdoliva.  Detection of gan-generated fake images over social networks.  In 2018 IEEE conference on multimedia information processing and retrieval (MIPR), pages 384–389. IEEE, 2018.
  * [27] Yuval Nirkin, Yosi Keller, and Tal Hassner.  Fsgan: Subject agnostic face swapping and reenactment.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 7184–7193, 2019.
  * [28] Trevine Oorloff and Yaser Yacoob.  Expressive talking head video encoding in stylegan2 latent-space.  arXiv preprint arXiv:2203.14512, 2022.
  * [29] William Peebles, John Peebles, Jun-Yan Zhu, Alexei Efros, and Antonio Torralba.  The hessian penalty: A weak prior for unsupervised disentanglement.  In European Conference on Computer Vision, pages 581–597. Springer, 2020.
  * [30] Yurui Ren, Ge Li, Yuanqi Chen, Thomas H Li, and Shan Liu.  Pirenderer: Controllable portrait image generation via semantic neural rendering.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13759–13768, 2021.
  * [31] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2287–2296, 2021.
  * [32] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  arXiv preprint arXiv:2106.05744, 2021.
  * [33] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [34] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou.  Interpreting the latent space of gans for semantic face editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243–9252, 2020.
  * [35] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  Advances in Neural Information Processing Systems, 32:7137–7147, 2019.
  * [36] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [37] Soumya Tripathy, Juho Kannala, and Esa Rahtu.  Facegan: Facial attribute controllable reenactment gan.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1329–1338, 2021.
  * [38] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  arXiv preprint arXiv:1812.01717, 2018.
  * [39] Andrey Voynov and Artem Babenko.  Unsupervised discovery of interpretable directions in the gan latent space.  In International conference on machine learning, pages 9786–9796. PMLR, 2020.
  * [40] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity gan inversion for image attribute editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11379–11388, 2022.
  * [41] Ting-Chun Wang, Ming-Yu Liu, Andrew Tao, Guilin Liu, Bryan Catanzaro, and Jan Kautz.  Few-shot video-to-video synthesis.  In NeurIPS, 2019.
  * [42] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu.  One-shot free-view neural talking-head synthesis for video conferencing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10039–10049, 2021.
  * [43] Xin Wang, Hui Guo, Shu Hu, Ming-Ching Chang, and Siwei Lyu.  Gan-generated faces detection: A survey and new perspectives.  arXiv preprint arXiv:2202.07145, 2022.
  * [44] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva.  Latent image animator: Learning to animate images via latent space navigation.  arXiv preprint arXiv:2203.09043, 2022.
  * [45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [46] Olivia Wiles, A Koepke, and Andrew Zisserman.  X2face: A network for controlling face generation using images, audio, and pose codes.  In Proceedings of the European conference on computer vision (ECCV), pages 670–686, 2018.
  * [47] Zongze Wu, Dani Lischinski, and Eli Shechtman.  Stylespace analysis: Disentangled controls for stylegan image generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12863–12872, 2021.
  * [48] Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier.  Feature-style encoder for style-based gan inversion.  arXiv e-prints, pages arXiv–2202, 2022.
  * [49] Fei Yin, Yong Zhang, Xiaodong Cun, Mingdeng Cao, Yanbo Fan, Xuan Wang, Qingyan Bai, Baoyuan Wu, Jue Wang, and Yujiu Yang.  Styleheat: One-shot high-resolution editable talking face generation via pretrained stylegan.  arXiv preprint arXiv:2203.04036, 2022.
  * [50] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In European Conference on Computer Vision, pages 524–540. Springer, 2020.
  * [51] Xianfang Zeng, Yusu Pan, Mengmeng Wang, Jiangning Zhang, and Yong Liu.  Realistic face reenactment via self-supervised disentangling of identity and pose.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12757–12764, 2020.
  * [52] Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton.  Lookahead optimizer: k steps forward, 1 step back.  Advances in neural information processing systems, 32, 2019.
  * [53] Mingxu Zhang, Hongxia Wang, Peisong He, Asad Malik, and Hanqing Liu.  Exposing unseen gan-generated image using unsupervised domain adaptation.  Knowledge-Based Systems, 257:109905, 2022.
  * [54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [55] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.  Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3661–3670, 2021.
  * [56] Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change Loy.  CelebV-HQ: A large-scale video facial attributes dataset.  In ECCV, 2022.


SolChecker [54] |  CNCIT 2022 |  ✓ |  |  |  |  |  |  |  ✓ |  | VE6,VE7, VE8, VE11 VE18, VE24 VE30, VE34, VE40
SVChecker [55] |  ICCAIS 2022 |  |  |  |  ✓ |  |  |  |  ✓ |  [56] | VE18, VE25, VE30, VE34, VE38, VE39
SKLEE [57] |  SEFM 2022 |  |  ✓ |  |  |  |  |  |  ✓ |  [58] | VE17, VE25, VE30, VE38, VE39, VE40
Sailfish [59] |  IEEE S&P 2022 |  |  |  |  |  |  ✓ |  |  ✓ |  [60] | VE34, VE38
SciviK [61] |  MVSZ |  |  |  |  |  ✓ |  |  |  ✓ |  |
EthVer [62] |  FC 2021 |  ✓ |  |  |  |  |  |  ✓ |  |  [63] | VE34
SmartPulse [64] |  IEEE S&P 2021 |  |  |  |  |  ✓ |  |  |  ✓ |  [65] |
Pluto [66] |  IEEE TSE 2021 |  |  ✓ |  |  |  |  |  |  ✓ |  [67] | VE25, VE34, VE39
Horus [68] |  IEEE S&P 2021 |  ✓ |  |  |  |  |  ✓ |  ✓ |  |  [69] | VE17,VE19,VE24,VE25,VE30,VE34,
ConFuzzius [31] |  Euro S&P 2021 |  ✓ |  |  ✓ |  |  |  |  |  ✓ |  [70] | VE17,VE19,VE22,VE34,VE25,VE38,VE40
Gas Gauge [71] |  arXiv 2021 |  |  |  ✓ |  |  ✓ |  |  |  ✓ |  [72] | VE28,VE29
Eth2Vec [73] |  BSCI 2021 |  |  |  |  ✓ |  |  |  |  ✓ |  [74] | VE12,VE29,VE34,VE25,VE39
NeuCheck [75] |  SPE 2021 |  |  |  |  |  ✓ |  |  |  ✓ |  [76] | VE32,VE34,VE25,VE40
SmartScan [77] |  ICSE 2021 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE27
SolGuard [78] |  Inf. Sci. 2021 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE19,VE30
Solidifier [79] |  SAC 2021 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE22
ReDetect [80] |  MSN 2021 |  |  ✓ |  |  |  |  |  |  ✓ |  – | VE34
GasChecker [81] |  IEEE TETC 2021 |  |  |  ✓ |  |  |  |  |  ✓ |  – | VE28,VE29
SmarTest [82] |  USENIX 2021 |  |  ✓ |  |  |  |  |  |  ✓ |  [83] | VE17,VE22,VE25,VE30

Frontrunner-Jones [84, 68] |  USENIX 2021 IEEE S&P 2021 |  ✓ |  |  |  |  |  |  ✓ |  |  [85] | VE38
---|---|---|---|---|---|---|---|---|---|---|---
ContractWard [86] |  IEEE TNSE 2021 |  |  |  |  ✓ |  |  |  |  ✓ |  – | VE19,VE30,VE34,VE25,VE38,VE39
ESCORT [87] |  arXiv 2021 |  |  |  |  ✓ |  |  |  |  ✓ |  – | VE22,VE17,VE27,VE34,VE26,VE38
Conkas |  – |  |  ✓ |  |  |  ✓ |  |  ✓ |  ✓ |  [88] | VE25,VE30,VE34,VE38,VE39
VeriSmart [27] |  IEEE S&P 2020 |  ✓ |  |  |  |  |  |  |  ✓ |  [83] | VE17,VE18,VE22,VE25,VE34
VeriSolid [89] |  FC 2019 |  ✓ |  |  |  |  |  |  |  ✓ |  [90] | VE24,VE17,VE19,VE30,VE34
ESBMC [91] |  USENIX 2020 |  |  ✓ |  |  |  |  |  ✓ |  |  [92] |
sFuzz [93] |  ICSE 2020 |  |  |  ✓ |  |  |  |  ✓ |  |  [94] | VE19,VE29,VE30,VE34,VE25,VE39,VE40
GNNSCVD [35] [95] |  IJCAI 2020 |  |  |  |  ✓ |  |  |  |  ✓ |  [96] | VE34,VE39,V40
ContractGuard [97] |  IEEE TSC 2020 |  |  |  |  |  |  ✓ |  ✓ |  |  [98] | VE18,VE19,VE30,VE34,VE25
Ethainter [99] |  PLDI 2020 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE17,VE18,VE19
eThor [100] |  CCS 2020 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE34
SmartSheild [101] |  SANER 2020 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE10,VE30,VE25
VerX [102] |  IEEE S&P 2020 |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  [103] | VE29,VE30
GASOL [104] |  TACAS(2) 2020 |  |  ✓ |  |  |  |  |  ✓ |  |  – | VE28,VE29
RA [105] |  IEEE ICBC 2020 |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  – | VE34
EthPloit [106] |  SANER 2020 |  |  |  ✓ |  |  |  |  |  ✓ |  – | VE17,VE18,VE19,VE30
Harvey [107] |  ESEC/FSE 2020 |  |  |  ✓ |  |  |  |  ✓ |  |  – | VE22
SODA [108] |  NDSS 2020 |  |  |  |  |  |  ✓ |  ✓ |  ✓ |  [109] | VE17,VE18,VE19,VE30,VE34,VE25
Solythesis [110] |  PLDI 2020 |  |  |  |  |  |  ✓ |  |  ✓ |  [111] |
TxSpector [112] |  USENIX 2020 |  |  |  |  |  |  ✓ |  ✓ |  |  [113] | VE20,VE17,VE18,VE27,VE30,VE34
Gastap [114] |  VECoS 2019 |  |  ✓ |  |  |  |  |  ✓ |  |  – | VE28,VE29
ÆGIS [115, 116] |  ASIA CCS 2020 |  |  |  |  |  |  ✓ |  ✓ |  |  [117] | VE17,VE19,VE34
Echidna |  – |  |  |  ✓ |  |  |  |  |  ✓ |  [118] |
Octopus |  – |  |  ✓ |  |  |  ✓ |  |  ✓ |  |  [119] |
Qian et al. [120] |  Access 2020 |  |  |  |  ✓ |  |  |  ✓ |  |  – | VE34
Sereum [121] |  NDSS 2019 |  |  ✓ |  |  |  |  ✓ |  ✓ |  |  [122] | VE34
Annotary [123] |  ESORICS 2019 |  |  ✓ |  |  |  |  |  ✓ |  |  [124] | VE19,VE30,VE36
Mythril |  - |  ✓ |  ✓ |  |  |  |  |  ✓ |  |  [125] | VE29
SolidityCheck [126] |  arXiv 2019 |  ✓ |  |  |  |  |  |  |  |  [127] | VE34,VE25
Slither [128] |  ICSE 2019 |  |  |  |  |  ✓ |  |  |  ✓ |  [129] | VE10,VE20,VE17,VE29
solc-verify [130] |  VSTTE 2019 |  ✓ |  |  |  |  |  |  |  ✓ |  [131] | VE29,VE25
HoneyBadger [132] |  USENIX 2019 |  |  ✓ |  |  |  |  |  ✓ |  |  [133] | VE10,VE20,VE36,VE25
ILF [134] |  CCS 2019 |  |  ✓ |  ✓ |  ✓ |  |  |  ✓ |  ✓ |  [135] | VE17,VE15,VE19,VE20,VE30,VE39
Vultron [136] |  ICSE 2019 |  |  |  ✓ |  |  |  |  |  ✓ |  [137] | VE19,VE29,VE30,VE34,VE25

SIF [138] |  APSEC 2019 |  ✓ |  |  |  |  |  |  ✓ |  ✓ |  [139] |
---|---|---|---|---|---|---|---|---|---|---|---
SolAnalyser [140] |  APSEC 2019 |  ✓ |  |  |  |  |  |  ✓ |  |  [141] | VE18,VE29,VE30,VE25,VE39
sCompile [142] |  ICFEM 2019 |  |  ✓ |  |  |  |  |  ✓ |  |  – | VE24,VE17,VE19,VE29,VE30
FEther [143] |  IEEE Access 7 |  ✓ |  ✓ |  |  |  |  |  |  ✓ |  – |
NPChecker [144] |  OOPSLA 2019 |  |  |  |  |  ✓ |  |  ✓ |  |  – | VE30,VE34,VE38,VE39
SoliAudit [145] |  IOTSMS 2019 |  |  |  ✓ |  ✓ |  |  |  ✓ |  ✓ |  – | VE24,VE18,VE27,VE34,VE25,VE38,VE40
Manticore [146] |  ASE 2019 |  |  ✓ |  |  |  |  |  |  |  [147] |
VeriSol [148] |  ICFEM |  ✓ |  |  |  |  |  |  |  ✓ |  [149] | -
solhint |  - |  |  |  |  |  |  |  |  ✓ |  [150] | -
KEVM [151] |  CSF 2018 |  ✓ |  |  |  |  |  |  ✓ |  |  [152] | VE25
Isabelle/Hol [153] |  CPP 2018 |  ✓ |  |  |  |  |  |  ✓ |  |  [154] | VE25
EtherTrust |  CAV 2018 |  ✓ |  |  |  |  |  |  ✓ |  |  [155] | VE34
SolMet [156] |  WETSEB 2018 |  ✓ |  |  |  |  |  |  ✓ |  |  [157] | VE34
Vandal [158] |  arXiv 2018 |  |  |  |  |  ✓ |  |  ✓ |  |  [159] | VE17,VE18,VE30,VE34
Maian [160] |  ACSAC 2018 |  |  ✓ |  |  |  |  |  ✓ |  |  [161] | VE17,VE20,VE30
Securify v2.0 [162] |  CCS 2018 |  |  ✓ |  |  |  |  |  ✓ |  |  [163] | VE29
teEther [164] |  USENIX 2018 |  |  ✓ |  |  |  |  |  ✓ |  |  [165] | VE17,VE19,VE27,VE30
Osiris [166] |  ACSAC 2018 |  |  ✓ |  |  |  |  |  ✓ |  |  [167] | VE25
SaferSC [168] |  arXiv 2018 |  |  |  |  ✓ |  |  |  ✓ |  |  [169] | VE17,VE20,VE30
ContractFuzzer [170] |  ASE 2018 |  |  |  ✓ |  |  |  |  ✓ |  |  [171] | VE19,VE27,VE29,VE30,VE34,VE39,VE40
F framework [172] |  POST 2018 |  ✓ |  |  |  |  |  |  ✓ |  |  – | VE30,VE34,VE38,VE39,VE40
Zeus [28] |  NDSS 2018 |  ✓ |  |  |  |  |  |  |  ✓ |  – | VE18,VE30,VE34,VE25,VE38
SASC [173] |  NTMS 2018 |  |  ✓ |  |  |  |  |  |  ✓ |  – | VE18,VE26,VE37,VE39
Reguard [174] |  ICSE 2018 |  |  |  ✓ |  |  |  |  |  ✓ |  – | VE34
MadMax [175] |  PACMPL 2018 |  |  |  |  |  ✓ |  |  ✓ |  |  [176] | VE29,VE25
ContractLarva [177] |  RV 2018 |  |  |  |  |  |  |  ✓ |  |  [178] | –
Smartcheck [179] |  WETSB 2018 |  |  |  |  |  |  |  |  ✓ |  [180] | VE18,VE27,VE30,VE34,VE25,VE39,V40
EthIR [181] |  ATVA 2018 |  |  |  |  |  ✓ |  |  ✓ |  |  [182] |
ECFChecker [183] |  POPL 2018 |  |  |  |  |  |  ✓ |  ✓ |  |  [184] |
Oyente [11] |  CCS 2016 |  |  ✓ |  |  |  |  |  ✓ |  |  [185] | VE34,VE26,VE37,VE39
DISTRIBUTION |  |  27.8% |  25.9% |  13.0% |  11.1% |  13.9% |  8.3% |  53.9% |  46.1% |  |