Figure 1.  We present a novel approach to reenact an individual using only a
casual _self-scan_. Using a captured self-scan (Step 1), we train a
personalized image generator, tailored specifically to that individual (Step
2). The personalized generator guarantees the preservation of the individual’s
appearance and facilitates video-driven reenactment. Thanks to the generator’s
semantic latent space, the generated video can be edited and stylized in post-
process (Step 3).

# Facial Reenactment Through a Personalized Generator

Ariel Elazary  Tel-Aviv UniversityIsrael ,  Yotam Nitzan  Tel-Aviv
UniversityIsrael and  Daniel Cohen-Or  Tel-Aviv UniversityIsrael

###### Abstract.

In recent years, the role of image generative models in facial reenactment has
been steadily increasing. Such models are usually subject-agnostic and trained
on domain-wide datasets. The appearance of the reenacted individual is learned
from a single image, and hence, the entire breadth of the individual’s
appearance is not entirely captured, leading these methods to resort to
unfaithful hallucination. Thanks to recent advancements, it is now possible to
train a personalized generative model tailored specifically to a given
individual. In this paper, we propose a novel method for facial reenactment
using a personalized generator. We train the generator using frames from a
short, yet varied, self-scan video captured using a simple commodity camera.
Images synthesized by the personalized generator are guaranteed to preserve
identity. The premise of our work is that the task of reenactment is thus
reduced to accurately mimicking head poses and expressions. To this end, we
locate the desired frames in the latent space of the personalized generator
using carefully designed latent optimization. Through extensive evaluation, we
demonstrate state-of-the-art performance for facial reenactment. Furthermore,
we show that since our reenactment takes place in a semantic latent space, it
can be semantically edited and stylized in post-processing.

Personalization, Facial Reenactment.

††submissionid: 237††journal: TOG††ccs: Computing methodologies Image
manipulation

##  1\. Introduction

Facial reenactment is the task of synthesizing a realistic portrait video of a
target person that replicates the actions of a person appearing in a driving
video. The ability to do so is essential for movie postproduction, augmented
reality (AR), virtual reality (VR) and telepresence applications. The main
challenge of this task is faithfully representing the identity, or appearance,
of the target person, in varying conditions.

For years, works approaching this task took a graphics perspective, explicitly
modeling 3$D$ geometry, usually relying on morphable mesh models (Kim et al.,
2018; Zheng et al., 2022a; Ichim et al., 2015; Thies et al., 2016; Hu et al.,
2017; Thies et al., 2019; Grassal et al., 2022; Lombardi et al., 2018).
Recently, several works have leveraged $2D$ image generative models for the
task (Lin et al., 2022; Wang et al., 2021b; Sun et al., 2022; Abdal et al.,
2022; Zakharov et al., 2019). These methods offer superior image quality than
those based on traditional $3D$ graphics as they inherently support more
facial components including hair, the mouth cavity, and accessories like
glasses. Moreover, a notable advantage of using generative models is that they
support intuitive semantic control, such as aging or adding glasses, through
their latent spaces (Patashnik et al., 2021; Shen et al., 2020; Abdal et al.,
2020) and out-of-domain stylization (Gal et al., 2021; Zhu et al., 2021; Ojha
et al., 2021) (see Fig. 1).

Although methods relying on image generative models have many advantages, they
compromise identity preservation. Previous works have primarily used one or
just a few images (_e.g_., $8$) to learn the appearance of the target person
(Lin et al., 2022; Sun et al., 2022; Wang et al., 2021b; Abdal et al., 2022).
Using one or very few images is usually claimed as an advantage as they are
easy to obtain. However, the full variance and aspects of a person’s
appearance clearly cannot be learned in this setting. To complete missing
information, such methods resort to hallucination based on generic face priors
learned from many different individuals. Realistically, beyond data
acquisition, another factor popularizing this setting was the lack of methods
to train a generator to preserve identity as portrayed in a reasonably sized
personal photo album.

In this work, we leverage recent advancements in generative modelling and
employ an _image generator_ that is _personalized_ – _i.e_. trained
specifically for a given person. As input, we take a _self-scan_ – a short RGB
video that captures the target person in a variety of viewpoints and facial
expressions. Such a self-scan video, albeit short, is sufficient to
personalize a domain-wide (_i.e_., faces) image generator (Karras et al.,
2020) and enables a faithful modelling of the person’s identity. Importantly,
the self-scan is still easy to obtain as it is casually captured using any
standard camera and does not require a high-cost apparatus nor a controlled
environment nor a strict script to follow.

To personalize the generative model, we build upon MyStyle (Nitzan et al.,
2022). Requiring roughly $100-300$ images of an individual, MyStyle fine-tunes
a generic face generator (Karras et al., 2020) to model the individual’s
identity. Importantly, other properties of the generator, such as the image
quality and semantic latent space are preserved. Using the personalized
generator immediately mitigates the challenge of preserving identity.
Reenactment is then reduced to locating a code in the generator’s latent space
that imitates the head pose and expression of the person in a driving frame,
which we perform using latent optimization.

This optimization is nevertheless challenging in itself. Common
representations for pose and expression are often entangled with the identity
information. For example, facial keypoints (Kartynnik et al., 2019) are often
used to track facial motion and specifically used to indicate on the person’s
pose and expression. However, keypoints also indicate the facial geometry and
were shown to encode identity (Nixon, 1985). Due to this fact, previous works
leveraging keypoints were limited to only have a person reenacting themselves
(_i.e_., self-reenactment) (Zakharov et al., 2019). We overcome this
difficulty by using a keypoint-driven representation from which we eliminate
most identity-related information. Concisely, we use a carefully selected
subset of keypoints and represent each with a relative location, with respect
only to other keypoints belonging to the same facial part (_e.g_., left eye).
To capture additional information that is overlooked by the keypoints, such as
the presence of teeth, we directly use the driving frame’s pixels in specific
regions. We optimize the latent code to according to these two terms. Note
that neither term is perfectly agnostic to identity, which might hinder
identity preservation if used in a different setting. However, the
personalized generator is constrained to identity-preserving results and the
optimization can only recover the nearest possible solution.

A key challenge in video generation is maintaining temporal consistency.
Although two consecutive frames are similar, two separate optimization
processes might converge to latent codes that are unnecessarily different. To
mitigate temporal inconsistency, we initialize each optimization process with
a slightly noised version of the latent code produced for the previous frame.
Since optimization is highly susceptible to its initialization, this approach
biases the optimization to converge to a nearby latent and hence, minimizes
unnecessary perturbation between the two consecutive frames.

Through quantitative and qualitative evaluation, we demonstrate that our
method is highly effective at facial reenactment. Specifically, we find that
our method excels at identity preservation, surpassing the performance of
recent state-of-the-art methods, and performs at least on par in terms of
transferring pose and expression.

##  2\. Related Work

###  2.1. Facial Reenactment

Facial Reenactment was traditionally approached from a $3$D-graphics
perspective. Such methods usually start by collecting significant footage of
the target face and reconstructing an explicit $3$D mesh model, using $3$D
Morphable Models (Blanz and Vetter, 1999). At test time, reenactment is
performed by transferring pose and expression from the $3$D model of the
driving video to that of the target 3D face model (Thies et al., 2016, 2015;
Hu et al., 2017; Ichim et al., 2015; Garrido et al., 2014; Cao et al., 2013,
2014; Bouaziz et al., 2013; Blanz et al., 2003; Grassal et al., 2022). A
thorough introduction of facial reconstruction and $3$DMM is provided in
recent surveys (Zollhöfer et al., 2018; Egger et al., 2020). Explicitly
modeling faces in $3$D is challenging and most methods struggle with modelling
hair, mouth cavity and accessories like glasses.

Recently, different approaches were proposed to sidestep these challenges.
First, several works avoided $3$D modelling altogether by warping an image of
the target face (Averbuch-Elor et al., 2017; Siarohin et al., 2019; Drobyshev
et al., 2022). With this approach, faithful changes to pose and expression
possible are relatively limited, often leading to undesired artifacts. A
second approach models faces using other $3$D representations such as neural
implicit representations (Gafni et al., 2021; Zheng et al., 2022a) and point
clouds (Zheng et al., 2022b). Last, and most relevant to our work, several
recent works have incorporated $2$D generative models, mostly GANs (Goodfellow
et al., 2014), to their pipelines. Some of these works have proposed pipelines
that are entirely identity-agnostic. _I.e_., a generative model is trained on
a domain-wide dataset, and a single image of the individual is used as a
starting point for manipulation (Sun et al., 2022; Lin et al., 2022; Wang et
al., 2022, 2021b; Abdal et al., 2022; Hong et al., 2022).

A single image is insufficient to learn the entire breadth of appearances of
an individual, leading such methods to resort to unfaithful hallucination.
Accordingly, other works have incorporated more footage of the individual on
top of generic knowledge of faces. Kim et al. (2018) and the concurrent work
by Wang et al. (2023) classically perform reenactment using $3$DMM but employ
person-specific generative models to transform $3$DMM-based renders to
realistic images. Inevitably, they still partially suffer from limitations
stemming from using $3$D mesh models, such as not modeling the mouth cavity. A
different approach, also taken by this work, is pretraining the model on a
domain-wide dataset and then fine-tuning it for a specific individual.
Zakharov et al. (2019) pretrain on VoxCeleb (Chung et al., 2018), a low-
resolution ($\leq 256$p) video dataset, limiting the eventual quality of their
output. Their method is also limited to only support an individual reenacting
themselves. Cao et al. (2022) pretrain on an unpublished dataset that is
laborious to collect, featuring hundreds of identities, performing scripted
facial expressions while being captured by 90 cameras simultaneously. In
contrast, our domain-wide model is an off-the-shelf StyleGAN2 (Karras et al.,
2020) model, trained on FFHQ (Karras et al., 2019) – a high-resolution
($1024$p) image dataset, that is crawled off the Internet. Our simple
pretraining approach is significantly more accessible, does not compromise
output quality, and could naturally be improved by using future image
datasets.

###  2.2. Generative Prior

In recent years, generative models have immensely improved in countless
foundational tasks such as unconditional synthesis (Karras et al., 2019; Brock
et al., 2018), image-to-image translation (Isola et al., 2017; Zhu et al.,
2017) and text-condition synthesis (Rombach et al., 2022; Ramesh et al.,
2022). Noticeably, unconditional GANs (Goodfellow et al., 2014), and
specifically StyleGAN (Karras et al., 2019, 2020), have drawn attention for
the meaningful representations that emerge in the generator’s latent space.
Such meaningful representations make the generator useful for numerous
downstream applications, beyond simply image synthesis. One line of works find
semantic controls in the generator’s latent space which enable controlled
synthesis and image editing pipelines (Patashnik et al., 2021; Abdal et al.,
2020; Voynov and Babenko, 2020; Shen and Zhou, 2020; Shen et al., 2020;
Härkönen et al., 2020). Other works leverage these pretrained generators as a
prior for downstream tasks ranging from image enhancement (Menon et al., 2020;
Yang et al., 2021; Chan et al., 2021; Richardson et al., 2021; Wang et al.,
2021a; Luo et al., 2020; Pan et al., 2021; Gu et al., 2020), facial
reenactment (Nitzan et al., 2020; Lin et al., 2022; Sun et al., 2022; Abdal et
al., 2022) and even discriminative tasks, such as regression (Xu et al.,
2021).

Methods leveraging unconditional GANs often start with locating a latent code
that corresponds to a desired image. This process, often known as _projection_
, can be carried out using a trained encoder (Richardson et al., 2021; Nitzan
et al., 2020; Tov et al., 2021; Xu et al., 2021; Pidhorskyi et al., 2020) or
using optimization (Abdal et al., 2019; Parmar et al., 2022; Zhu et al., 2020;
Gu et al., 2020; Karras et al., 2020). Encoding is significantly faster than
optimization, but is also less accurate.

Regardless of the projection method, the desired image might not exist within
the output space of the generator, greatly limiting the applicability of this
approach. To overcome this challenge, several works have proposed to slightly
fine-tune the generator in order to improve its expressiveness on specific
inputs (Pan et al., 2021; Roich et al., 2021; Nitzan et al., 2022; Bau et al.,
2020). Within this line of works, Nitzan et al. (2022) recently proposed
MyStyle – a method to fine-tune the generator to faithfully model a specific
person.

In this work, we rely on MyStyle’s personalized generator as prior for
reenactment. Technically, we locate the desired frame by projecting aspects of
the driving frame such as facial keypoints and the eyes and mouth regions.

##  3\. Method

We aim to generate a portrait video in which a target person, appearing in a
self-scan video, is repeating the actions occurring in a driving video. The
self-scan is a short, yet varied, RGB video used to learn the appearance of
the target individual. The simple process of capturing a self-scan is
discussed in subsection 3.1. Our approach is composed of two steps. First,
using the self-scan, we train a personalized image generator, building upon
MyStyle (Nitzan et al., 2022) (subsection 3.2). The personalized generator is
a universal representation of the individual, and can be used in conjunction
to any driving video. Second, we perform latent optimization to locate a
latent code of the personalized generator that faithfully represents the
actions in a driving frame (subsection 3.3).

###  3.1. Self-Scan Capture

To authentically model an individual’s appearance, one must observe them in a
wide variety of settings. For example, the best, or perhaps only, way of
knowing how a person looks like when smiling is to actually see them smiling
beforehand.

We find that a short, single-take, RGB video can include sufficiently varied
head poses and expressions. We call such videos _self-scans_ , as they can be
conveniently captured by the individual themselves. Note that the individual
does not need to be following any script. Instead, the self-scan should
intuitively include the poses and expressions that one desires to be possible
for the generated video. One does not need to exhaust all combinations of
poses and expressions, and performing each once is sufficient. As we
demonstrate later, our method enables disentangling and recomposing poses and
expressions that were not seen jointly in the self-scan.

In practice, for the experiments in this paper, we instructed several
individuals to capture a $20\text{-}30$ seconds, portrait video of themselves
with varying head poses and expressions that could be used in a social
interaction, like conversation.

###  3.2. From a Self-Scan to a Personalized Generator

At the core of our method is an unconditional personalized generator $G$,
mapping a latent code to an image. To train it, we employ the recently
proposed method MyStyle (Nitzan et al., 2022). Using an image dataset of an
individual, MyStyle slightly fine-tunes StyleGAN2 (Karras et al., 2020), a
generic face image generator, to obtain a generator that faithfully preserves
individual’s appearance.

##### Creating the Training Set.

StyleGAN2 and MyStyle are trained on aligned portrait images. We therefore
start with _smoothly_ aligning and cropping all frames in the self-scan
(Tzaban et al., 2022).

MyStyle was demonstrated effective when used with a dataset of
$100\text{-}300$ randomly curated images. A self-scan, however, is composed of
significantly more frames, often over a thousand. Increasing the training set
size slows training and inference times, due to increased dimensionality
(Nitzan et al., 2022). We observe that not all frames are required for
training. Many frames are similar to each other – consecutive frames, for
example, are nearly duplicates. Using two frames that are interchangeable adds
negligible diversity and information to the generator. Intuitively, one of
these frames could be omitted.

Let us denote the set of aligned self-scan crops by $S$. We wish to find a
subset $T\subset S$, where $T$ contains a predetermined number of frames,
marked $N$, and maximizes the subset’s “diversity”. Following previous works
(Ojha et al., 2021; Nitzan et al., 2022), we use the average pairwise LPIPS
(Zhang et al., 2018) distances as a measure of a set’s diversity. Identifying
the optimal subset $T$ is known to be NP-hard but a greedy algorithm was
proven to be quite effective (Ravi et al., 1994). The greedy algorithm
iteratively adds the element of the set $S\setminus T$ whose minimal distance
to any element of current subset $T$ is greatest. See pseudo-code description
in Algorithm 1 in the supplemental. We find that $N=250$ is sufficient.

##### MyStyle training.

Figure 2.  An illustration of MyStyle’s tuning method. A set of $N$ portrait
images of the target individual are projected into StyleGAN’s latent space.
The result is a set of anchors that are the nearest possible neighbors. We
then tune the generator to reconstruct the input images from their
corresponding anchors according to Eq. 1.

Using the filtered self-scan we next train MyStyle (Nitzan et al., 2022). Due
to MyStyle’s central role in training and later on in reenactment, we
concisely repeat the training details.

We start with our training set $T=\\{t_{i}\\}_{i=1}^{N}$ and a StyleGAN2
generator (Karras et al., 2020) pretrained on FFHQ (Karras et al., 2019). Each
image $t_{i}$ is first inverted into StyleGAN’s latent space $\mathcal{W}$,
yielding a latent code $w_{i}$, called anchor. Inversion is performed using an
off-the-shelf inversion encoder, whose architecture follows Richardson et al.
(2021). Now, the pretrained StyleGAN2 generator is fine-tuned using pivotal
tuning (Roich et al., 2021) – _i.e_., to better reconstruct the training
images from their corresponding anchors. Formally, the training loss is given
by

(1)
$\begin{split}\mathcal{L}_{\text{mystyle}}=\sum_{i=1}^{N}\;[\mathcal{L}_{\text{lpips}}(G(w_{i}),t_{i})+\lambda_{\text{pixel}}\norm{G(w_{i})-t_{i}}_{2}],\end{split}$

where $\mathcal{L}_{\text{lpips}}$ is the LPIPS loss (Zhang et al., 2018) and
$\lambda_{\text{pixel}}=10$ is a hyperparameter balancing the two terms.

We illustrate the training process in Fig. 2.

###  3.3. Reenactment by Latent Optimization

Identity preservation, the paramount challenge of reenactment, is guaranteed
from using the personalized generator $G$. Therefore, the task is reduced to
accurately reproducing head poses and facial expressions of the driving
frames. To this end, we employ _latent optimization_ on each driving frame,
directly optimizing a latent code such that the image generated from it would
be suitable. An overview of the process if illustrated in Fig. 4.

####  3.3.1. Latent Space

MyStyle’s point-wise training (Eq. 1) affects only certain regions of the
latent space, making them preserve the individual’s identity (Nitzan et al.,
2022). These regions are modeled well by a slightly dilated convex hull
defined by the anchors $\\{w_{i}\\}_{i=1}^{N}$. To ensure identity
preservation, Nitzan et al. (2022) used the convex hull as the generator’s
latent space, instead of the latent spaces commonly used with StyleGAN, such
as $\mathcal{W}$ and $\mathcal{W}^{+}$ (Abdal et al., 2019). We similarly
adopt and make use of the personalized subspace, dubbed $\mathcal{P}^{+}$, for
latent optimization. A thorough and formal introduction of the latent space is
provided in Appendix B.

####  3.3.2. Loss Function

To guide optimization towards a latent code that replicates the desired head
pose and facial expression, we use $3$D facial keypoints. Some aspects of
expression, like the mouth cavity, are overlooked by keypoints. Therefore, we
also employ a pixel-reconstruction loss directly in specific regions, as will
be soon discussed.

##### Keypoints Loss.

Indicating the locations and shapes of facial parts, facial keypoints hold
rich information regarding head pose and facial expression. However, keypoints
are also affected by the unique appearance of each individual (Nixon, 1985).
Therefore, forcing a target identity to replicate keypoints of another
individual leads to unsatisfactory results, and restricted previous works to
only support self-reenactment (Zakharov et al., 2019). We propose an
alternative guidance, derived from keypoints, that similarly indicates pose
and expression but is significantly more robust to identity.

Specifically, we start from a set of $468$ $3$D keypoints estimated using off-
the-shelf method, MediaPipe (Kartynnik et al., 2019), and make several
modifications, illustrated in Fig. 3.

We first consider only a subset of all keypoints. Most keypoints are essential
for modeling face geometry, but carry very little semantic meaning. Compare,
for example, keypoints on the cheeks or forehead with those on the eyes and
mouth. The eyes and mouth are perceptually more significant and receive
special attention in keypoint estimation (Grishchenko et al., 2020) and
reenactment (Zakharov et al., 2019). Even within the mouth, keypoints
indicating whether the mouth is open or closed are crucial, but those that
indicate the shape of the lips are person-specific and thus irrelevant.
Accordingly, we use a subset of keypoints that constitutes only of – inner
parts of both lips, eyelids and irises.

One can still deduce the distances between the eyes and mouth from our subset
of keypoints. This information is again person-specific and was even used
directly in early face recognition methods (Nixon, 1985). We overcome this
problem by separately normalizing keypoints of the _same_ facial part to have
zero mean. Since all keypoints are now relative, they cannot be used to deduce
pose. We therefore bring back four individual keypoints, indicating the frame
of the head, which are used with their original absolute image coordinates. In
total, we use $66$ keypoints – $16$ for each eye, $5$ for each iris, $20$ for
the mouth and $4$ for the frame of the head.

Finally, the keypoint-based loss function is the sum of distances between
keypoints of the driving and generated images. Formally,

(2)
$\mathcal{L}_{\text{key}}=\sum_{i=1}^{66}\;\norm{\text{keypoints}(I_{d})_{i}-\text{keypoints}(I_{g})_{i}}_{2},$

where keypoints() is the operation extracting our subset of keypoints from the
driving and generated images $I_{d},I_{g}$, respectively.

Figure 3.  Comparison of all keypoints inferred by MediaPipe (Kartynnik et
al., 2019) (left) with our reduced subset (right). The subset chosen conveys
pose and expression while being significantly less sensitive to identity.
Colors other than blue indicate the grouping according to facial parts. The
keypoints of each facial part are normalized separately to have zero mean. The
blue landmarks are used with their absolute, non-normalized, coordinates.

##### Pixel Losses.

Facial keypoints do not model some aspects of expression, most notably – the
mouth cavity, and solely relying on them hinders the fidelity of results (Wang
et al., 2023; Zakharov et al., 2019; Kim et al., 2018; Zheng et al., 2022a).
For example, even with a perfectly transferred lips motion, teeth might
inappropriately appear or disappear from generated frames. To mitigate this
issue, we impose a loss that encourages actual pixel reconstruction between
the mouth region of the driving and generated frames. Formally, this loss
reads as

(3)
$\begin{split}&\mathcal{L}_{\text{mouth}}=\norm{\text{mouth}(I_{d})-\text{mouth}(I_{g})}_{2},\end{split}$

where mouth is the operation of cropping and resizing the mouth region, as
defined by keypoints, from the driving and generated images $I_{d},I_{g}$ (see
Appendix C for details).

At first, this loss might appear counter-productive as it encourages copying
regions from the driving individual to the target individual – harming
identity preservation. However, this is not a concern in our case, since using
a personalized generator guarantees identity preservation. The added pixel
loss can only guide the optimization towards the “nearest” possible solution,
which is necessarily identity preserving.

We find that another region that requires special attention is that of the
irises. Aiming for an expressive reenactment setting, the irises should be
able to convey any expression, such as looking right or down. However,
StyleGAN perpetuates biases inherited from FFHQ, where irises are almost
always looking towards the camera. We find the personalized generator is able
to reconstruct the “iris expressions” seen in the self-scan but struggles to
generalized beyond them.

In practice, the keypoint-driven optimization is able to reproduce the iris at
the right location, at the price of its color – often producing an off-white
iris. We find it possible to recover the correct eye color by adding a
dedicated pixel reconstruction loss on the irises. Formally, the loss reads as

(4) $\begin{split}&\mathcal{L}_{\text{eyes}}=\norm{\text{left-
eye}(I_{c})-\text{left-eye}(I_{g})}_{2}+\\\ &\norm{\text{right-
eye}(I_{c})-\text{right-eye}(I_{g})}_{2},\end{split}$

where left-eye and right-eye are the operations of cropping and resizing these
regions from (details in Appendix C) the generated image $I_{g}$ and an image
serving as reference for the identity $I_{c}$ . The reference image $I_{c}$ is
generated from the center of the personalized subspace.

##### Overall Loss.

The overall reenactment loss reads as

(5)
$\mathcal{L}_{\text{full}}=\mathcal{L}_{\text{key}}+\lambda_{\text{mouth}}\mathcal{L}_{\text{mouth}}+\lambda_{\text{eyes}}\mathcal{L}_{\text{eyes}}+\lambda_{\text{reg}}\mathcal{L}_{\text{reg}},$

where $\mathcal{L}_{\text{reg}}$ is a regularization term adopted from MyStyle
that is partially responsible for optimization being constrained to the latent
space $\mathcal{P}^{+}$, and is thoroughly explained in Appendix B.

Figure 4.  Illustration of our latent optimization method for generating a
reenacted frame. We operate within the personalized latent space
$\mathcal{P}^{+}$ of the generator, ensuring identity preservation and
reducing the task to reproducing pose and expression. To this end, we employ
three losses to compare the generated and driving frames. First, we penalize
differences in a subset of keypoints (Kartynnik et al., 2019), which are
separately normalized (indicated by color) for each facial part to have zero
mean. The subset of normalized keypoints still conveys pose and expression but
is more robust to geometry. Second, we apply an $L_{2}$ reconstruction loss on
pixels in the mouth cavity between driving and generated frame for expression
fidelity. Third, we apply an $L_{2}$ reconstruction loss on pixels in the
irises areas between identity reference, which is generated from the center of
$\mathcal{P}^{+}$, and generated frame for color preservation.

####  3.3.3. Temporal Consistency.

Operating on each frame individually oftentimes leads to temporally incoherent
results. We follow the approach proposed by Tzaban et al. (2022), where
temporal coherency is inherited from the input video by using smooth
operations on individual frames. We take several independent steps.

Similar to the training set, driving frames must also be aligned. The crop and
alignment performed are determined using facial keypoints (Karras et al.,
2019), which we smooth with a Gaussian low-pass filter. Next, each
optimization process is initialized with the result of the previous frame,
with small additive Gaussian noise. Adding noise is necessary to allow
optimization escape the local minima of using the previous latent code again.
We find that this initialization is crucial for biasing the optimization to
consistently converge to nearby latent codes, minimizing unnecessary
perturbation. We finally apply another Gaussian low-pass filter, this time on
the produced latent codes themselves, making the transition between frames
even smoother.

##  4\. Experiments

In the following experiments we use $8$ self-scans of individuals from various
genders, ages and ethnicities. A sample of our results is displayed in Fig. 8,
and additional videos are available in the supplemental. We start by comparing
our method to state-of-the-art facial reenactment methods, qualitatively and
quantitatively in subsection 4.1. In subsection 4.2, we demonstrate that by
performing reenactment through a semantic latent space, we can intuitively
apply semantic editing and stylization in post-process. We then evaluate the
design choices of our method through an ablation study in subsection 4.3, and
the generator’s ability to generalize beyond the pretraining data and even the
self-scan data by novelly composing expressions in subsection 4.4.

###  4.1. Comparison

##### Baselines.

We first compare our method with three state-of-the-art methods. We choose
works representative of several of the main branches of facial reenactment
literature, as described below. Neural Head Avatars (NHA) (Grassal et al.,
2022) receives as input a single video and explicitly models $3$D mesh
surface. Reenactment is performed by transferring parameters between $3$D
morphable models (Li et al., 2017). We note that NHA’s video is
“proffesionally” taken – _i.e_. taken in front of a green screen, with even
lighting, and is also $\times 3$ longer than ours. Latent Image Animator (LIA)
(Wang et al., 2022) is a one-shot method, taking in a single image as input,
and warping it to match driving frames. The warp is determined according to
latent controls learned by a 2D generative model trained for that purpose.
DaGAN (Hong et al., 2022) is a one-shot method, using a dedicated $2$D image-
to-image translation module, translating each driving frame to a reenactment
frame.

##### Comparison Protocol.

Table 1.  Comparison to competing methods, in terms of identity preservation (IDP), identity consistency (IDC), average expression distance (AED), average pose distance (APD), and NIQE (Mittal et al., 2012).  | ID$(\uparrow)$ | APD$\times 100$ $(\downarrow)$ | AED$\times 10$ $(\downarrow)$ | NIQE $(\downarrow)$
---|---|---|---|---
NHA | 0.80 | 0.24 | 2.37 | 19.34
LIA | 0.71 | 0.13 | 1.59 | 45.52
DaGAN | 0.73 | 0.26 | 1.77 | 34.88
Ours | 0.89 | 0.16 | 1.52 | 13.22

Here, we use $4$ of our self-scans videos. We use each video separately as a
self-scan, and also to drive the other self-scans. We use random $5\text{-}10$
seconds sequences from the driving videos and produce a total of $10$ results.
We note that one-shot methods are given the first frontal-pose, neutral-
expression frame in the video of the target individual, and video-based
methods use entire video. NHA also requires an indication of one such frame.

We use multiple quantitative metrics to evaluate the performance of
reenactment. First, We measure identity preservation (ID) with respect to the
training set. To this end, we follow a protocol similar to MyStyle’s, where
for each reenacted frame we compute its average similarity to the top-$5$ most
similar images in the training set and average across all frames. We use
top-$5$ images (instead of MyStyle’s top-$1$), to prevent one-shot methods
from being trivially similar to their reference image. Similarity is measured
in terms of cosine similarity of features extracted from a face recognition
network (Deng et al., 2019a).

Next, we measure pose and expression preservation from the driving video.
Following previous works (Sun et al., 2022; Lin et al., 2022), we measure
Average Pose Distance (APD) and Average Expression Distance (AED). Both are
defined as the mean square error of deep features extracted by a $3$D face
reconstruction model (Deng et al., 2019b). Last, we evaluate the frame’s
quality. Due to the few-shot nature of the problem, popular metrics such as
FID are unsuitable. We instead use NIQE (Mittal et al., 2012), a metric
designed to evaluate image quality without reference.

We report the metrics in Tab. 1, a set of qualitative results in Fig. 9, and
videos in the supplemental. As can be seen, our method outperform all existing
methods on identity and expression preservation, and quality, and performs
competitively with the leader LIA on pose preservation. Specifically, by
comparing generated images to reference self-scan frames, one can observe that
our method excels at preserving identity.

###  4.2. Edits and Stylization in Post-Process

Our reenactment method is built on top of an incredibly semantic image
generator – StyleGAN. We can therefore leverage the myriad of applications and
methods designed for StyleGAN, adding new capabilities on top of reenactment.
In this section we demonstrate two such capabilities, performed in post-
process – semantic editing and stylization. Results for both applications are
presented in Fig. 5.

##### Semantic Editing.

In StyleGAN’s latent space there exist vectors or “directions” that when
traversed upon, affect the generated image in a semantically consistent
manner. For example, one such vector would consistently and gradually add a
smile. The result of our latent optimization method is a set of latent codes,
that turn into the reenacted video once passed through the generator. Here,
instead of generating frames from these latent codes, we first shift them a
constant sized step in the direction of a specific semantic vector. Passing
these shifted latent codes through the generator results in a reenacted video
that was semantically additionally edited. We use three semantic vectors that
are identified using InterFaceGAN (Shen et al., 2020) – smile, pose, and age.

##### Stylization.

Several methods (Gal et al., 2021; Ojha et al., 2021) have been proposed to
fine-tune StyleGAN slightly so that it models different data domains. These
fine-tuned models are known to be aligned with their parent models, _i.e_.
images generated by the two models, from the same latent code, correspond to
each other (Wu et al., 2021). Here, we further fine-tune our personalized
models with StyleGAN-NADA (Gal et al., 2021), a text-driven training method.
We fine-tune with two prompts – “the Joker” and “a sketch”. The latent
optimization still takes place in the personalized generator, but the
resulting latent codes are passed through the generator tuned with StyleGAN-
NADA to produce the final video frames. The result is a reenacted video which
depicts the target individual but they are stylized as the Joker or a sketch.

As can be seen in Fig. 5, the reenacted frames are successfully edited and
stylized. This native post-process capability greatly extends the potential
applications of our method and further encourages the use of semantic
generative models as backbones for reenactment.

Figure 5.  Semantic editing and stylization of the reenacted video performed
in post-process. Before being generated, the reenacted video is represented as
a sequence of codes in StyleGAN’s latent space. The reenacted video can thus
be semantically edited by shifting these latent codes in semantic directions
(Shen et al., 2020) before feeding them to the generator, or stylized by
feeding them to a generator that was fine-tuned for a specific style (Gal et
al., 2021).

###  4.3. Ablation Studies

We next evaluate the contribution of individual aspects of our method through
an ablation study. A sample of results is provided in Fig. 6.

The personalized generator is trained on a subset of self-scan frames that
maximizes diversity. Here, we instead train personalized models on frames that
are evenly spaced across the self-scan, using the same number of frames
$N=250$. We find that these models fail to reproduce relatively uncommon
expressions, such as closing the eyes. The reason becomes immediately clear
when looking at the training set – these expressions, purely by chance, are
entirely not present.

We move on to consider the latent optimization process. First, we use the
naive optimization loss for keypoint-driven reenactment – penalizing
differences in absolute keypoint coordinates. We find that all expressions are
poorly reproduced. This is expected, since these keypoint representations are
entangled with facial geometry and do not trivially correspond across
individuals. Next, we reinstate our keypoint loss $\mathcal{L}_{\text{key}}$,
but still without any pixel loss. While pose and expression are better
preserved, differences from the driving frames are still clear, most notably
in the shape of the lips and the appropriate appearance of teeth in the mouth
cavity. Our final method, adding the pixel loss $\mathcal{L}_{\text{crop}}$
immediately mitigates these issues.

Figure 6.  Ablation study. Evenly spaced self-scan frames miss uncommon
expressions, like closing eyes, prohibiting from reproducing them (_e.g_.,
first row). The naive keypoint-driven reenactment method does not generalize
between different facial geometries and completely fails transferring
expressions. Last, adding the pixel loss $\mathcal{L}_{\text{crop}}$,
generally improves the expression preservation efficacy and specifically
encourages accurate appearance of teeth in the mouth cavity.

###  4.4. Studying the Generator’s Expressiveness

Our reenactment through latent optimization could be considered a retrieval of
the most suitable frame from the personalized generator. It is thus intriguing
to ask – what can and cannot it generate? For this study, we use the
relatively unusual expression of winking. Results are displayed in Fig. 7.

We first look at the generic StyleGAN model trained on FFHQ (Karras et al.,
2019). We apply our latent optimization method, using an inverted image (Roich
et al., 2021) as initialization and a winking driving frame as guidance. This
approach produces results that are not winking and are severely degraded.

We now capture a self-scan that is scripted to intentionally include a single
instances of winking, with a neutral faces expression and frontal pose. From
the self-scan we create two training set, where one is created normally and
the from the other we omit the few winking frames. We train a personalized
model with each of these training sets. Applying our reenactment method with
winking driving frames to the two models, we find that only the model that has
seen a wink in training is able to generate winking frames. We note that the
generated winking frames are varied, composing a wink with other expressions
and poses that were not seen in the single winking training instance.

We deduce two important conclusions. First, including unique expressions, and
possibly other information, extends the model’s expressive power beyond what
is provided by pre-training. On the flip side, an expression that is absent
from the self-scan may not be produced. Second, the model can compose
expressions and poses that were never seen together in training. This allows
self-scans to be casually captured, not requiring a strict script and an
exhaustive coverage of all setting combinations.

Figure 7.  Evaluating different generator’s ability to produce a wink. We
reenact a target individual with two winking driving frames, seen in (a),
through different models (b). The generic StyleGAN model, used in a one-shot
manner, fails to produce winking faces and severely degrades the generated
images in the process. A personalized model whose training set does not
include a wink also fails to a wink. Nevertheless, the optimization being
confined to the generator’s personalized subspace assures that the produced
images are not degraded and faithfully preserve the individual’s appearance.
Last, adding a single instance of winking (c) to the training set, allows the
personalized generator to produce winking images. Note that the generated
images compose a wink with a smile and a head pose that were not seen in
training.  Figure 8.  A sample of qualitative reenactment results produced by
our method (center). Frames from the relevant self-scan are provided on both
sides. Our method successfully reproduces poses and expression from the
driving video while faithfully preserving the target identity. We recommend
zooming in and carefully evaluating identity preservation by carefully
comparing generated and self-scan frames.  Figure 9.  Comparing ours and
state-of-the-art reenactment methods – NHA (Grassal et al., 2022), LIA (Wang
et al., 2022) and DaGAN (Hong et al., 2022). Frames from the self-scans of the
target identity are provided on both sides. One-shot methods – LIA and DaGAN –
take as input only the top image, marked with a black outline. As can be seen,
our results are at least on par at transferring pose and expression and
significantly better at realism and preserving identity. We recommend zooming
in and carefully evaluating identity preservation by carefully comparing
generated and self-scan frames.

##  5\. Discussion and Conclusion

##### Limitations.

We have demonstrated that our method produces state-of-the-art results.
Nevertheless, our method has a few limitations. First, as discussed in
subsections 4.4 and 7, our generator is bounded by the variations appearing in
the self-scan. Driving frames that depict unseen behavior, mostly expressions,
would produce low fidelity results. Second, the optimization process performed
per frame is time-consuming, requiring up to a minute for a single frame.

##### Conclusion.

We have proposed a novel facial reenactment method leveraging a personalized
image generator, trained on a casual self-scan of a single individual.
Identity preservation is guaranteed by the generator being personalized, and
thus we have reduced reenactment to reproducing poses and expressions, solved
using a latent optimization process. Our experiments and analysis demonstrated
that our method outperforms prior work in quality and fidelity, specifically
excelling at identity preservation.

This study highlights the potential of $2$D generative models for reenactment
beyond the one/few-shot setting, effectively bridging a gap between the $2$D
and $3$D streams of works. We hope it will inspire others to pursue this
direction further. A natural first step would be to develop an encoder to
replace latent optimization, substantially accelerating the process.

## Acknowledgments

We are grateful to Or Patashnik for insightful discussions throughout this
work. We also thank Or Patashnik and Omer Tov for proofreading the draft. We
finally thank Michal Elazary, Yogev Nitzan, Daniel Garibi, Zvia Elazary, Inbal
Raz and Noa Rubin for providing us with their self-scans. This research was
supported in part by the Israel Science Foundation (grants no. 2492/20 and
3441/21).

## References

  * (1)
  * Abdal et al. (2019) Rameen Abdal, Yipeng Qin, and Peter Wonka. 2019.  Image2stylegan: How to embed images into the StyleGAN latent space?. In _Proceedings of the IEEE international conference on computer vision_. 4432–4441.
  * Abdal et al. (2020) Rameen Abdal, Peihao Zhu, Niloy Mitra, and Peter Wonka. 2020\.  StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows.  _arXiv preprint arXiv:2008.02401_ (2020).
  * Abdal et al. (2022) Rameen Abdal, Peihao Zhu, Niloy J Mitra, and Peter Wonka. 2022\.  Video2StyleGAN: Disentangling Local and Global Variations in a Video.  _arXiv preprint arXiv:2205.13996_ (2022).
  * Averbuch-Elor et al. (2017) Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen. 2017.  Bringing portraits to life.  _ACM Transactions on Graphics (TOG)_ 36, 6 (2017), 1–13.
  * Bau et al. (2020) David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. 2020.  Semantic photo manipulation with a generative image prior.  _arXiv preprint arXiv:2005.07727_ (2020).
  * Blanz et al. (2003) Volker Blanz, Curzio Basso, Tomaso Poggio, and Thomas Vetter. 2003.  Reanimating faces in images and video. In _Computer graphics forum_ , Vol. 22. Wiley Online Library, 641–650.
  * Blanz and Vetter (1999) Volker Blanz and Thomas Vetter. 1999.  A morphable model for the synthesis of 3D faces. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 187–194.
  * Bouaziz et al. (2013) Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013.  Online modeling for realtime facial animation.  _ACM Transactions on Graphics (ToG)_ 32, 4 (2013), 1–10.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018\.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Cao et al. (2014) Chen Cao, Qiming Hou, and Kun Zhou. 2014.  Displaced dynamic expression regression for real-time facial tracking and animation.  _ACM Transactions on graphics (TOG)_ 33, 4 (2014), 1–10.
  * Cao et al. (2022) Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shun-Suke Saito, Stephen Lombardi, Shih-En Wei, Danielle Belko, Shoou-I Yu, et al. 2022\.  Authentic volumetric avatars from a phone scan.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–19.
  * Cao et al. (2013) Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 2013\.  3D shape regression for real-time facial animation.  _ACM Transactions on Graphics (TOG)_ 32, 4 (2013), 1–10.
  * Chan et al. (2021) Kelvin CK Chan, Xintao Wang, Xiangyu Xu, Jinwei Gu, and Chen Change Loy. 2021.  Glean: Generative latent bank for large-factor image super-resolution. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14245–14254.
  * Chung et al. (2018) J. S. Chung, A. Nagrani, and A. Zisserman. 2018.  VoxCeleb2: Deep Speaker Recognition. In _INTERSPEECH_.
  * Deng et al. (2019a) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019a.  Arcface: Additive angular margin loss for deep face recognition. In _Proc. CVPR_. 4690–4699.
  * Deng et al. (2019b) Yu Deng, Jiaolong Yang, Sicheng Xu, Dong Chen, Yunde Jia, and Xin Tong. 2019b.  Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set. In _IEEE Computer Vision and Pattern Recognition Workshops_.
  * Drobyshev et al. (2022) Nikita Drobyshev, Jenya Chelishev, Taras Khakhulin, Aleksei Ivakhnenko, Victor Lempitsky, and Egor Zakharov. 2022.  Megaportraits: One-shot megapixel neural head avatars.  _arXiv preprint arXiv:2207.07621_ (2022).
  * Egger et al. (2020) Bernhard Egger, William AP Smith, Ayush Tewari, Stefanie Wuhrer, Michael Zollhoefer, Thabo Beeler, Florian Bernard, Timo Bolkart, Adam Kortylewski, Sami Romdhani, et al. 2020\.  3d morphable face models—past, present, and future.  _ACM Transactions on Graphics (TOG)_ 39, 5 (2020), 1–38.
  * Floater (2015) Michael S Floater. 2015\.  Generalized barycentric coordinates and applications.  _Acta Numerica_ 24 (2015), 161–214.
  * Gafni et al. (2021) Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner. 2021.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 8649–8658.
  * Gal et al. (2021) Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. 2021.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ (2021).
  * Garrido et al. (2014) Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormahlen, Patrick Perez, and Christian Theobalt. 2014.  Automatic face reenactment. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4217–4224.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014\.  Generative adversarial nets. In _Advances in neural information processing systems_. 2672–2680.
  * Grassal et al. (2022) Philip-William Grassal, Malte Prinzler, Titus Leistner, Carsten Rother, Matthias Nießner, and Justus Thies. 2022.  Neural head avatars from monocular RGB videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18653–18664.
  * Grishchenko et al. (2020) Ivan Grishchenko, Artsiom Ablavatski, Yury Kartynnik, Karthik Raveendran, and Matthias Grundmann. 2020\.  Attention mesh: High-fidelity face mesh prediction in real-time.  _arXiv preprint arXiv:2006.10962_ (2020).
  * Gu et al. (2020) Jinjin Gu, Yujun Shen, and Bolei Zhou. 2020.  Image processing using multi-code gan prior. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 3012–3021.
  * Härkönen et al. (2020) Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. 2020.  Ganspace: Discovering interpretable gan controls.  _Advances in Neural Information Processing Systems_ 33 (2020), 9841–9850.
  * Hong et al. (2022) Fa-Ting Hong, Longhao Zhang, Li Shen, and Dan Xu. 2022\.  Depth-aware generative adversarial network for talking head video generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3397–3406.
  * Hu et al. (2017) Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017\.  Avatar digitization from a single image for real-time rendering.  _ACM Transactions on Graphics (ToG)_ 36, 6 (2017), 1–14.
  * Ichim et al. (2015) Alexandru Eugen Ichim, Sofien Bouaziz, and Mark Pauly. 2015\.  Dynamic 3D avatar creation from hand-held video input.  _ACM Transactions on Graphics (ToG)_ 34, 4 (2015), 1–14.
  * Isola et al. (2017) Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017.  Image-to-image translation with conditional adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 1125–1134.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proc. CVPR_. 4401–4410.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020\.  Analyzing and improving the image quality of StyleGAN. In _Proc. CVPR_. 8110–8119.
  * Kartynnik et al. (2019) Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann. 2019.  Real-time facial surface geometry from monocular video on mobile GPUs.  _arXiv preprint arXiv:1907.06724_ (2019).
  * Kim et al. (2018) Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt. 2018.  Deep video portraits.  _ACM Transactions on Graphics (TOG)_ 37, 4 (2018), 1–14.
  * Li et al. (2017) Tianye Li, Timo Bolkart, Michael J Black, Hao Li, and Javier Romero. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Trans. Graph._ 36, 6 (2017), 194–1.
  * Lin et al. (2022) Connor Z Lin, David B Lindell, Eric R Chan, and Gordon Wetzstein. 2022.  3d gan inversion for controllable portrait image animation.  _arXiv preprint arXiv:2203.13441_ (2022).
  * Lombardi et al. (2018) Stephen Lombardi, Jason Saragih, Tomas Simon, and Yaser Sheikh. 2018.  Deep appearance models for face rendering.  _ACM Transactions on Graphics (ToG)_ 37, 4 (2018), 1–13.
  * Luo et al. (2020) Xuan Luo, Xuaner Zhang, Paul Yoo, Ricardo Martin-Brualla, Jason Lawrence, and Steven M Seitz. 2020\.  Time-travel rephotography.  _arXiv preprint arXiv:2012.12261_ (2020).
  * Menon et al. (2020) Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin. 2020.  Pulse: Self-supervised photo upsampling via latent space exploration of generative models. In _Proceedings of the ieee/cvf conference on computer vision and pattern recognition_. 2437–2445.
  * Mittal et al. (2012) Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. 2012\.  Making a “completely blind” image quality analyzer.  _IEEE Signal processing letters_ 20, 3 (2012), 209–212.
  * Nitzan et al. (2022) Yotam Nitzan, Kfir Aberman, Qiurui He, Orly Liba, Michal Yarom, Yossi Gandelsman, Inbar Mosseri, Yael Pritch, and Daniel Cohen-Or. 2022.  Mystyle: A personalized generative prior.  _ACM Transactions on Graphics (TOG)_ 41, 6 (2022), 1–10.
  * Nitzan et al. (2020) Yotam Nitzan, Amit Bermano, Yangyan Li, and Daniel Cohen-Or. 2020.  Face Identity Disentanglement via Latent Space Mapping.  _ACM Trans. Graph._ 39, 6, Article 225 (Nov. 2020), 14 pages.  https://doi.org/10.1145/3414685.3417826
  * Nixon (1985) Mark Nixon. 1985\.  Eye spacing measurement for facial recognition. In _Applications of digital image processing VIII_ , Vol. 575. SPIE, 279–285.
  * Ojha et al. (2021) Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang. 2021.  Few-shot Image Generation via Cross-domain Correspondence.  _arXiv preprint arXiv:2104.06820_ (2021).
  * Pan et al. (2021) Xingang Pan, Xiaohang Zhan, Bo Dai, Dahua Lin, Chen Change Loy, and Ping Luo. 2021\.  Exploiting deep generative prior for versatile image restoration and manipulation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ (2021).
  * Parmar et al. (2022) Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, and Krishna Kumar Singh. 2022\.  Spatially-adaptive multilayer selection for gan inversion and editing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11399–11409.
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021.  StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery.  _arXiv preprint arXiv:2103.17249_ (2021).
  * Pidhorskyi et al. (2020) Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto. 2020.  Adversarial Latent Autoencoders. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14104–14113.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Ravi et al. (1994) Sekharipuram S Ravi, Daniel J Rosenkrantz, and Giri Kumar Tayi. 1994.  Heuristic and special case algorithms for dispersion problems.  _Operations Research_ 42, 2 (1994), 299–310.
  * Richardson et al. (2021) Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. 2021.  Encoding in style: a StyleGAN encoder for image-to-image translation. In _Proc. IEEE/CVF CVPR_. 2287–2296.
  * Roich et al. (2021) Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. 2021\.  Pivotal Tuning for Latent-based Editing of Real Images.  _arXiv preprint arXiv:2106.05744_ (2021).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022\.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10684–10695.
  * Shen et al. (2020) Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. 2020\.  InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs.  _arXiv preprint arXiv:2005.09635_ (2020).
  * Shen and Zhou (2020) Yujun Shen and Bolei Zhou. 2020.  Closed-Form Factorization of Latent Semantics in GANs.  _arXiv preprint arXiv:2007.06600_ (2020).
  * Siarohin et al. (2019) Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019\.  First order motion model for image animation.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Sun et al. (2022) Jingxiang Sun, Xuan Wang, Lizhen Wang, Xiaoyu Li, Yong Zhang, Hongwen Zhang, and Yebin Liu. 2022.  Next3D: Generative Neural Texture Rasterization for 3D-Aware Head Avatars.  _arXiv preprint arXiv:2211.11208_ (2022).
  * Thies et al. (2019) Justus Thies, Michael Zollhöfer, and Matthias Nießner. 2019\.  Deferred neural rendering: Image synthesis using neural textures.  _Acm Transactions on Graphics (TOG)_ 38, 4 (2019), 1–12.
  * Thies et al. (2015) Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, and Christian Theobalt. 2015.  Real-time expression transfer for facial reenactment.  _ACM Trans. Graph._ 34, 6 (2015), 183–1.
  * Thies et al. (2016) Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner. 2016\.  Face2face: Real-time face capture and reenactment of rgb videos. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2387–2395.
  * Tov et al. (2021) Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or. 2021.  Designing an Encoder for StyleGAN Image Manipulation.  _arXiv preprint arXiv:2102.02766_ (2021).
  * Tzaban et al. (2022) Rotem Tzaban, Ron Mokady, Rinon Gal, Amit Bermano, and Daniel Cohen-Or. 2022.  Stitch it in time: Gan-based facial editing of real videos. In _SIGGRAPH Asia 2022 Conference Papers_. 1–9.
  * Voynov and Babenko (2020) Andrey Voynov and Artem Babenko. 2020.  Unsupervised Discovery of Interpretable Directions in the GAN Latent Space.  _arXiv preprint arXiv:2002.03754_ (2020).
  * Wang et al. (2023) Lizhen Wang, Xiaochen Zhao, Jingxiang Sun, Yuxiang Zhang, Hongwen Zhang, Tao Yu, and Yebin Liu. 2023.  StyleAvatar: Real-time Photo-realistic Portrait Avatar from a Single Video.  _arXiv preprint arXiv:2305.00942_ (2023).
  * Wang et al. (2021b) Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu. 2021b.  One-shot free-view neural talking-head synthesis for video conferencing. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10039–10049.
  * Wang et al. (2021a) Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. 2021a.  Towards Real-World Blind Face Restoration with Generative Facial Prior. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 9168–9178.
  * Wang et al. (2022) Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. 2022.  Latent image animator: Learning to animate images via latent space navigation.  _arXiv preprint arXiv:2203.09043_ (2022).
  * Wu et al. (2021) Zongze Wu, Yotam Nitzan, Eli Shechtman, and Dani Lischinski. 2021\.  StyleAlign: Analysis and Applications of Aligned StyleGAN Models.  _arXiv preprint arXiv:2110.11323_ (2021).
  * Xu et al. (2021) Yinghao Xu, Yujun Shen, Jiapeng Zhu, Ceyuan Yang, and Bolei Zhou. 2021.  Generative Hierarchical Features from Synthesizing Images. In _CVPR_.
  * Yang et al. (2021) Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. 2021\.  GAN Prior Embedded Network for Blind Face Restoration in the Wild. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 672–681.
  * Zakharov et al. (2019) Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. 2019.  Few-shot adversarial learning of realistic neural talking head models. In _Proceedings of the IEEE/CVF international conference on computer vision_. 9459–9468.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 586–595.
  * Zheng et al. (2022a) Yufeng Zheng, Victoria Fernández Abrevaya, Marcel C Bühler, Xu Chen, Michael J Black, and Otmar Hilliges. 2022a.  Im avatar: Implicit morphable head avatars from videos. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 13545–13555.
  * Zheng et al. (2022b) Yufeng Zheng, Wang Yifan, Gordon Wetzstein, Michael J. Black, and Otmar Hilliges. 2022b.  PointAvatar: Deformable Point-based Head Avatars from Videos.
  * Zhu et al. (2020) Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. 2020\.  In-domain GAN inversion for real image editing.  _arXiv preprint arXiv:2004.00049_ (2020).
  * Zhu et al. (2017) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017.  Unpaired image-to-image translation using cycle-consistent adversarial networks. In _Proc. IEEE ICCV_. 2223–2232.
  * Zhu et al. (2021) Peihao Zhu, Rameen Abdal, John Femiani, and Peter Wonka. 2021\.  Mind the gap: Domain gap control for single shot domain adaptation for generative adversarial networks.  _arXiv preprint arXiv:2110.08398_ (2021).
  * Zollhöfer et al. (2018) Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, and Christian Theobalt. 2018.  State of the art on monocular 3D face reconstruction, tracking, and applications. In _Computer graphics forum_ , Vol. 37. Wiley Online Library, 523–550.

##  Appendix A Supplemental Overview

In Appendix B we elaborate on details related to the latent optimization
process that were kept out of the main paper in sake of clarity. Then,
Appendix C provides technical implementation details.

##  Appendix B Latent Space for Optimization

At the core of our method is a latent optimization process in the latent space
of a personalized generator, trained with MyStyle (Nitzan et al., 2022). To
ensure identity preservation, MyStyle use a novel latent space for all its
downstream application. In this work, we adopt this latent space, without any
modifications. This section thoroughly describes how we use that latent space,
partially repeating discussions appearing in MyStyle for the sake of
completeness.

###  B.1. Native Latent Space, $\mathcal{P}$

Nitzan et al. (2022) observed that only a certain region of the generator’s
latent space has been personalized as result of training (see subsection 3.2).
To capture this region, they propose using a slightly “dilated” convex hull
spanned by the anchors $\\{w_{i}\\}$. Formally, they define

(6)
$\mathcal{P}_{\beta}=\\{\sum_{i}\alpha_{i}w_{i}|\,\sum_{i}\alpha_{i}=1,\forall
i:\alpha_{i}\geq-\beta\\},$

where $\beta$ is a positive, typically small, number defining the amount of
dilation. Note that $\mathcal{P}_{\beta}$ is a subspace of the generators
$\mathcal{W}$ latent space. The hyperparameter $\beta$ is usually omitted from
notation, and we instead denote the space as $\mathcal{P}$. With respect to
$\mathcal{P}$, they choose to represent latent codes in Barycentric
coordinates (Floater, 2015), _i.e_., using the convex coefficients
$\\{\alpha_{i}\\}$. Optimization is thus technically performed on a vector of
these coefficients – $\boldsymbol{\mathbf{\alpha}}$. Changing back to standard
coordinates is trivially done by $\sum_{i}\alpha_{i}w_{i}$.

Several measures are taken to ensure that the optimization process remains
within the bounds of $\mathcal{P}_{\beta}$. First, the vector
$\boldsymbol{\mathbf{\alpha}}$ is passed through a shifted softplus function:

(7)
$\tilde{\boldsymbol{\mathbf{\alpha}}}=\frac{1}{s}log(1+e^{s(\boldsymbol{\mathbf{\alpha}}+\beta)})-\beta,$

where all operations are element-wise, and $s$ is a “sharepness”
hyperparameter we set to $s=100$. This operation yields a new vector
$\tilde{\boldsymbol{\mathbf{\alpha}}}$ whose minimal value is greater than
$-\beta$.

Next, a regularization term is added to the optimization loss to encourage the
sum of coefficients to be $1$. Formally,

(8)
$\mathcal{L}_{\text{sum}}(\tilde{\boldsymbol{\mathbf{\alpha}}})=(\sum_{i}\tilde{\alpha}_{i}-1)^{2}.$

Finally, the optimization result $\boldsymbol{\mathbf{\alpha}}^{*}$ is
explicitly normalized to have a sum of $1$.

###  B.2. Extended Latent Space, $\mathcal{P}^{+}$

First proposed by Abdal et al. (2019), an incredibly popular approach to
increase StyleGAN’s expressivity is using an extended latent space
$\mathcal{W}^{+}$ (Richardson et al., 2021; Menon et al., 2020; Luo et al.,
2020). Concisely, this latent space is composed by using a possibly different
latent code in each layer of the generator. With its greater expressivity, it
was also shown that $\mathcal{W}^{+}$ provides a less faithful prior. This has
led previous works to regularize the extent to which it is exploited, _e.g_.,
by encouraging latent code in different layers to still be similar under
$L_{2}$ norm (Tov et al., 2021).

Nitzan et al. (2022) followed these ideas directly and imported them to
$\mathcal{P}$. First, $\mathcal{P}$ is extended to $\mathcal{P}^{+}$ across
generator’s layers by allowing each layer to add an offset:
$\boldsymbol{\mathbf{\alpha}}_{\beta}^{+}=(\boldsymbol{\mathbf{\alpha}}_{\beta}+\boldsymbol{\mathbf{\Delta}}_{0},...,\boldsymbol{\mathbf{\alpha}}_{\beta}+\boldsymbol{\mathbf{\Delta}}_{N})$
Second, a regularization term is added to the optimization process and prevent
severe deviation across layers. Formally,

(9)
$\mathcal{L}_{\text{delta}}(\Delta)=\sum_{i}\norm{\boldsymbol{\mathbf{\Delta_{i}}}}_{2}.$

###  B.3. Final Optimization loss

The two regularization terms described in Eqs. 9 and 8 were jointly referred
to in the main paper’s Eq. 5 as regularization terms assuring the optimization
is constrained to the personalized latent space. The full optimization loss is
thus given by

(10)
$\mathcal{L}_{\text{full}}=\mathcal{L}_{\text{key}}+\lambda_{\text{mouth}}\mathcal{L}_{\text{mouth}}+\lambda_{\text{eyes}}\mathcal{L}_{\text{eyes}}+\lambda_{\text{delta}}\mathcal{L}_{\text{delta}}+\lambda_{\text{sum}}\mathcal{L}_{\text{sum}}.$

Data: Set $S$, distance function $d$, subset size $N$.

Let $s_{i},s_{j}$ be the elements of set $S$ who have the greatest distance
$d$. $T\leftarrow\\{s_{i},s_{j}\\}$

while  _$|T| <N$_ do

$s^{\prime}\leftarrow\underset{s\in S\setminus T}{\operatorname*{arg\,max}}\
\underset{t\in T}{\min}\ d(s,t)$;

$T\leftarrow T\cup\\{s^{\prime}\\}$;

end while

Return $T$.

ALGORITHM 1 Greedy algorithm to maximize subset’s diversity (Ravi et al.,
1994).

##  Appendix C Additional Technical Details

###  C.1. Self-Scan Frame selection

The formal description of the frame selection algorithm is provided in
Algorithm 1.

###  C.2. Crops of the Mouth and Eyes

Our optimization’s pixel loss $\mathcal{L}_{\text{crop}}$ is calculated on
crops of the mouth and eyes that are automatically extracted from the
generated and driving frames. We next detail how these crops are obtained. We
crop the minimal rectangles that bounds the keypoints from our subset for each
region of interest – the outer parts of the lips and the irises. We then take
center crops of the rectangles from the larger of the two rectangles, on each
axis, to have the rectangles at the same size.

###  C.3. Implementation details

We next specify all hyperparameters. For the latent space, we use $\beta=0.02$
in all experiments. The weights of the different loss terms
$\lambda_{\text{crop}}=1,\lambda_{\text{delta}}=10,\lambda_{\text{sum}}=10$.
The additive Gaussian noise for the last frame result has $\sigma=0.05$. The
Gaussian low-pass filter on the produced latent codes has $\sigma=2$. The
number of self-scan frames in all experiments $N=250$.

Training takes roughly $3.5$ hours on a single $V100$, and optimization
requires up to a minute for a single frame.