#  PEGASUS: Personalized Generative 3D Avatars with Composable Attributes

Hyunsoo Cha Byungjun Kim Hanbyul Joo
Seoul National University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

We present, _PEGASUS_ , a method for constructing personalized generative 3D
face avatars from monocular video sources. As a compositional generative
model, our model enables disentangled controls to selectively alter the facial
attributes (e.g., hair or nose) of the target individual, while preserving the
identity. We present two key approaches to achieve this goal. First, we
present a method to construct a person-specific generative 3D avatar by
building a synthetic video collection of the target identity with varying
facial attributes, where the videos are synthesized by borrowing parts from
diverse individuals from other monocular videos. Through several experiments,
we demonstrate the superior performance of our approach by generating unseen
attributes with high realism. Subsequently, we introduce a zero-shot approach
to achieve the same generative modeling more efficiently by leveraging a
previously constructed personalized generative model.

Figure 1: PEGASUS. We propose a personalized generative 3D face avatars from
monocular video sources.

##  1 Introduction

Building a personalized 3D avatar for representing individuals in virtual
spaces has the potential to bring significant advancements in the field of
AR/VR and applications within the metaverse. Importantly, the method should be
user-friendly to allow casuals to build their avatars without the need for
complex capture systems. It should also offer a high level of realism,
depicting the fine-grained details of an individual’s geometry and appearance,
and, importantly, the avatar should be animatable to mirror the user’s facial
expressions in the virtual space. Yet, the 3D avatar does not need to maintain
the exact replica of the user’s single appearance, as users may prefer to
alter their avatars. This includes modifications such as changing hairstyles,
adding accessories like hats, or even altering facial parts to make the avatar
more aesthetically pleasing look or adopting the appearance of celebrities or
any desired shapes.

Recent technologies make it possible to build high-quality 3D face avatars for
general consumers from monocular video inputs only [20, 71, 70, 3, 19, 17]. By
leveraging parametric morphable face models [39, 4], these technologies
produce realistic animatable human avatars from sparse monocular videos that
capture naturally moving facial images, by canonicalizing observed cues into a
canonical space. However, the previous approaches mainly focus on creating the
exact replica from the input videos, without providing the functionality to
alter the subparts of the avatars, such as hairstyles or nose. As an
alternative direction, generative models in producing realistic faces have
been studied in the 2D field, by producing 2D human faces with diverse
appearance changes and facial expression changes [59, 51, 73]. 3D-aware
generative models leveraging the pre-trained 2D generative models are also
presented for generative 3D face modeling [9, 10]. While this approach shows
realistic faces, they are not fully animatable without having explicit mapping
to the 3D morphable models, and, thus, it is difficult to reenact the facial
expression from the target or viewpoint changes by keeping the identity.

In this work, we present, PEGASUS, a method to build a _personalized
generative 3D avatar_ from monocular video inputs. In contrast to the previous
work [70, 71], our 3D avatar enables compositional controllability, where
users can make alternations for a specific facial component or multiple
components such as hair, nose, or accessories, as shown in Fig. 1, while
preserving the identity of the target person. The control can be performed by
changing the disentangled latent codes defined in a continuous latent space.
Our personalized generative 3D avatar is constructed from the monocular video
of the target individual. Additionally, to learn the possible variations of
each facial component, we also leverage other available monocular videos from
arbitrary individuals, where our personalized generative models can
automatically learn continuous disentangled latent spaces.

However, there exist significant challenges in consolidating the monocular
videos from multiple individuals into a personalized generative model
construction for the target individual. Building a model with many individuals
often results in the failure to preserve the fine-grained appearance details
of the target individual, and, more critically, changing the latent space can
lead to changes in the entire facial appearance, rather than selectively
altering the desired subpart. As a solution, we present an approach by
synthesizing part-swapped videos of the target individual by replacing a
specific facial part with the one from other individuals, as shown in Fig. 4.
Built with diverse part-swapped videos, our generative 3D avatars, PEGASUS,
can preserve high-quality details for the target individuals, while equipped
with the generative power to selectively alter each facial part. While our
generative 3D model already shows satisfactory performance, it involves the
time-consuming process of constructing a set of part-swapped videos. As a more
rapid and efficient solution, we further introduce an approach that achieves
the same objectives through _zero-shot part transfer_ , leveraging previously
constructed personalized generative models. Through several experiments, we
demonstrate the superior performance of our approach when compared to
alternative methods.

Our contribution is summarized as follows: (1) the first method to build
personalized generative 3D avatars from monocular video sources; (2)
Disentangled controllability to selectively alter a subpart or multiple parts
of the 3D faces of the target individual; and (3) the 3D part transfer
approach to efficiently implement personalized generative models without
additional training.

##  2 Related Work

3D Face Reconstruction. Reconstructing realistic 3D faces has been a long-
standing problem in computer vision. To deal with the inherent diversity and
dynamics of human faces, a 3D parametric head model [4, 39, 50, 14, 11, 75]
represents 3D faces with a set of parameters that model the shape, pose, and
expression of the faces. Several methods create photo-realistic 3D facial
avatars from 2D images [17, 19, 70, 71, 61, 2]. For example, PointAvatar [71]
introduces a point-based representation to reconstruct high-frequency details
in facial attributes. Going beyond the reconstruction of a 3D face avatar from
videos, our method extends the previous work [71] to model a person-specific
generative avatar with various composable facial attributes learned from
videos of multiple humans.

Face Editing in 2D/3D. GAN [18]-based approaches have proven the effectiveness
in editing facial images [59, 51, 63, 66, 73, 32, 10, 74, 72, 9]. For
instance, SemanticStyleGAN [51] edits the target faces for various facial
attributes, including hair. EG3D [9] generates the novel avatars from random
noise inputs leveraging a tri-plane generator. Inspired by a tri-plane
generator, several approaches employ the tri-plane generator to generate the
3D avatars [13, 55, 60] from 2D image collections. Unlike EG3D-based
approaches, our methods do not need the GAN-based prior and large-scale
datasets.

Recently, diffusion model [24, 54, 42]-based image editing has seen rapid
advancement. Moreover, several approaches [22, 45, 6] present text-guided face
editing results by leveraging the pretrained Text-to-Image (T2I) diffusion
model. However, facial attributes are difficult to specify with text prompts
in many cases. Following the 2D diffusion emerging, the text-to-image-based
diffusion model generates high-quality 3D avatars from the textural
description [48, 25, 8, 1, 67, 6, 21, 28, 38, 57, 65]. TECA [65] manipulates
the attributes of the 3D avatar in a two-stage generation. These approaches
struggle to manipulate the variation of facial expressions and head poses from
text prompts.

Compositional Modeling for 3D Avatars. Several methods [62, 31, 26, 46, 23]
propose to edit implicit representations. To generate the compositional 3D
avatar, recent methods [37, 33] introduce decoupled representation with
garments or attributes. SCARF and DELTA [15, 16] create the 3D avatar in the
hybrid representation to transfer the garments or hair without additional
optimization. To reconstruct realistic hair strands, recent methods [49, 53,
58] propose to reconstruct high-fidelity 3D hair strands in multi-view
settings. Our approaches do not need separated representations and additional
optimizations to generate compositional avatars. Our person-specific
generative avatars are generated from 2D image sequences without the need for
3D scans.

Figure 2: Method Overview. Our model, PEGASUS, infers the 3D point locations
in the deformed space $\mathbf{x}^{d}$, normal $\mathbf{n}^{d}$, shading
$\mathbf{s}^{d}$, point segment cues $\chi^{d}$, and the albedo
$\mathbf{a}^{d}$ for each queried canonical point $\mathbf{x}^{gc}$,
conditioned by the latent code $\mathbf{z}$. To deform the generic canonical
point cloud $\mathbf{x}^{gc}$ to the deformed point $\mathbf{x}^{d}$, we apply
$\mathbf{x}^{gc}$ to two offsets, $\mathcal{O}^{gc\rightarrow sc}$ and
$\mathcal{O}^{sc\rightarrow fc}$, and blendshapes bases
$\mathcal{E},\mathcal{P}$ and LBS weights $\mathcal{W}$. After multiplying
$\mathbf{s}^{d}$ and $\mathbf{a}^{d}$, we obtain color $\mathbf{c}^{d}$ in
deformed space. We infer the binary segmentation cues $\chi^{d}$ to estimate a
synthesis part. We leverage the RGB, normal, and segment loss to encourage
high-quality appearances and accurate segmentation.

##  3 Preliminaries: PointAvatar [71]

Our approach is based on a monocular 3D avatar creation method, PointAvatar,
by extending the single identity-based method into a personalized generative
model. PointAvatar represents the target avatar via the initial canonical
learnable point representations $P_{c}=\\{x^{c}_{i}\\}_{i=\\{1\cdots N\\}}$,
where $x^{c}_{i}\in\mathbb{R}^{3}$ represents $i$-th learnable point defined
in the canonical space (denoted as the superscript $c$). By estimating the
offset value $\mathcal{O}_{i}^{c\rightarrow fc}$ from a trained MLP, the
canonical points are deformed into the _FLAME-canonical space_ (denoted as
$fc$) as:
$\mathbf{x}^{fc}_{i}=\mathbf{x}^{c}_{i}+\mathcal{O}_{i}^{c\rightarrow fc}$.
Subsequently, the points are deformed into the posed space as leveraging FLAME
model [39]:

$\displaystyle\mathbf{x}^{d}=\text{LBS}(\mathbf{x}^{d-},\mathbf{J}(\psi),\theta,\mathcal{W})$
(1)
$\displaystyle\mathbf{x}^{d-}=\mathbf{x}^{fc}+B_{P}(\bm{\theta};\mathcal{P})+B_{E}(\bm{\psi};\mathcal{E})$
(2)

, where $\mathbf{x}^{d-}$ denote the point after applying the blendshapes and
before applying transformation via LBS. $\theta$, $\psi$, $\beta$ are the
expression, pose, and identity parameters of the FLAME model, respectively,
for animating the avatar, and $\mathcal{E},\mathcal{P},$ and $\mathcal{W}$ are
the expression blendshapes, pose blendshapes, and LBS weights, respectively,
which are estimated by an MLP. The normal of each point $\mathbf{n}_{c}$ is
defined as the SDF, which is the canonical network’s output as follows:
$\mathbf{n}_{c}=\nabla_{\mathbf{x}_{c}}\text{SDF}(\mathbf{x}_{c})$. The normal
of the deformation space $\mathbf{n}_{d}$ is represented by a deformation
network which deforms the canonical point set $P_{c}$ to the deformed point
set $P_{d}=\\{\mathbf{x}_{d}^{i}\\}$. The point deformation is fully
differentiable, so it can define the normal deformation as follows:

$\mathbf{n}_{d}=l\mathbf{n}_{c}\left({\partial\mathbf{x}_{d}\over\partial\mathbf{x}_{c}}\right)^{-1}$
(3)

, where $l$ denotes the normalizing factor, which ensures the output of normal
value should be the unit length. The RGB of a point is represented by
$c_{d}=s_{d}\circ a$, the Hadamard Product of the shading $s_{d}$, and albedo
$a$.

##  4 Our Method

###  4.1 Personalized Generative Avatar Model

Our generative avatar model takes, as inputs, a latent code
$\mathbf{z}\in\mathbb{R}^{(D+1)\times d}$, and FLAME parameters $\beta$,
$\theta$, and $\psi$. The latent code is the concatenation of $D$ different
latent codes $\\{\mathbf{z}^{j}\\}_{j=0...D}$, where each latent code
$\mathbf{z}^{j}\in\mathbb{R}^{d}$ controls the identity of the humans or the
subpart such as hair and nose. We treat that $\mathbf{z}^{0}$ controls the
overall identity variations while changing other codes $\mathbf{z}^{j\neq 0}$
varies only the subparts of the face, preserving the same identity represented
by $\mathbf{z}^{0}$. By changing FLAME parameters, we can animate the avatars
to have varying face poses and expressions. The identity parameter of FLAME
$\beta$ also affects the overall coarse shape of the avatar, and we assume the
parameter is fixed for the same individual with the same $\mathbf{z}^{0}$. By
expending the PointAvatar [71], our avatar model is represented by a set of
_generic_ (or person-agnostic) canonical point
${P}^{gc}=\\{\mathbf{x}_{i}^{gc}\\}_{i=\\{1,\cdots,N\\}}$. To this end, our
model, $\mathcal{M}_{\phi}$, infers the 3D point locations in the deformed
space $\mathbf{x}_{i}^{d}$, normal vector $\mathbf{n}_{i}\in\mathbb{R}^{3}$,
and the albedo color $\mathbf{a}_{i}\in\mathbb{R}^{3}$ for each queried
canonical point as:

$\mathcal{M}_{\phi}(\mathbf{x}_{i}^{gc},\mathbf{z},\bm{\beta},\bm{\theta},\bm{\psi})=\\{\mathbf{x}_{i}^{d},\mathbf{n}^{d},\mathbf{a}_{i}\\},$
(4)

where $\mathbf{x}_{i}^{d}$ represents the 3D point after applying identity and
appearance variations controlled by $\mathbf{z}$, as well as the facial pose
and expression changes by FLAME parameters. Fig. 2 represents an overview of
PEGASUS.

In contrast to the original PointAvatar, which represents a single identity
only, we train a single avatar model to represent multiple face appearances,
where appearance can vary by changing disentangled latent codes $\mathbf{z}$.
We tackle this challenging problem by introducing the generic canonical space,
which is person-agnostic. While our model expresses varying identities
ideally, we observe that it is very difficult to cover very diverse people
with a single implicit model, which often produces blurry avatars, as
demonstrated in our ablations. Yet, we demonstrate that our model can
successfully achieve the goal of a personalized generative avatar model,
allowing face part variation while preserving the same identity. Importantly,
in order to build the personalized generative avatar model, we present a way
to synthesize the dataset of the target individual via part-swapping,
described in Subsec. 4.2.

Multi-staged Canonical Spaces and Point Deformation. While the original
PointAvatar considers two-staged deformation (canonical, FLAME-canonical, and
deformed space), we consider one more stage, resulting in generic canonical
(gc), subject-specific canonical (sc), subject-specific FLAME-canonical (fc),
and deformed space (d). Similar to PointAvatar, we empirically found
introducing generic enables us to avoid bad local minima in training the model
with multiple face appearance.

The generic canonical space and the point locations defined in this space
${P}^{gc}=\\{\mathbf{x}^{gc}_{i}\\}_{i=1...N}$ are shared among all
identities. We first map the points $\mathbf{x}^{gc}_{i}$ from the generic
canonical space into the subject-specific canonical space
${P}^{sc}=\\{\mathbf{x}^{sc}_{i}\\}_{i=1...N}$ by adding point offsets
$\mathcal{O}_{i}^{gc\rightarrow sc}$ that are conditioned by latent code
$\mathbf{z}$. Subsequently, we then map the points in the subject-specific
canonical space into the FLAME-canonical space via another point offset
$\mathcal{O}_{i}^{sc\rightarrow fc}$, similar to the PointAvatars. That is,

$\displaystyle\mathbf{x}^{sc}_{i}=\mathbf{x}^{gc}_{i}+\mathcal{O}_{i}^{gc\rightarrow
sc}$ (5)
$\displaystyle\mathbf{x}^{fc}_{i}=\mathbf{x}^{sc}_{i}+\mathcal{O}_{i}^{sc\rightarrow
fc}$ (6)

, where $\mathcal{O}_{i}^{gc\rightarrow sc}$ and
$\mathcal{O}_{i}^{sc\rightarrow fc}$ are inferred from the learned deformation
MLP model. Intuitively, our subject-specific canonical space is equivalent to
the “canonical space” of PointAvatars, where we introduced one more prior
stage to handle multiple identities.

As in PointAvatars, we use a coordinate-based MLP to infer deformation
offsets, blendshapes, and LBS weights:

$MLP(\mathbf{z},\mathbf{x}_{i}^{gc})=\\{\mathcal{O}_{i}^{gc\rightarrow
sc},\mathcal{O}_{i}^{sc\rightarrow
fc},\mathcal{E},\mathcal{P},\mathcal{W}\\}.$ (7)

The deformed point is then computed as:

$\displaystyle\mathbf{x}^{d-}=\mathbf{x}^{fc}+B_{S}(\bm{\beta};\mathcal{S})+B_{P}(\bm{\theta};\mathcal{P})+B_{E}(\bm{\psi};\mathcal{E})$
(8)
$\displaystyle\mathbf{x}^{d}=\text{LBS}(\mathbf{x}^{d-},\mathbf{J}(\psi),\theta,\mathcal{W})$
(9)

Different from PointAvatar, we leverage the shape blendshapes basis $B_{S}$ of
the FLAME, allowing us to change the coarse shape of the avatar by controlling
the shape parameter of the FLAME, which is useful for building our synthetic
DB to enable better face alignments (Subsec. 4.2).

Figure 3: DB Avatar. Examples with the same FLAME parameters.

(a) Hair

(b) Hat

(c) Nose

(d) Mouth

(e) Eyes

(f) Brows

Figure 4: Part-Swapped Videos of the Target Individual.

Canonical Representations. We use an MLP to infer the SDF value
$\sigma_{i}\in\mathbb{R}$, albedo $\mathbf{a}_{i}\in\mathbb{R}^{3}$, shading
$\mathbf{s}_{i}\in\mathbb{R}^{3}$, point segment cues $\chi_{i}\in[0,1]$ for
the $i$-th point at the subject canonical space $\mathbf{x}_{i}^{sc}$:

$MLP(\mathbf{z},\mathbf{x}_{i}^{sc})=\\{\sigma_{i},\mathbf{a}_{i},\chi_{i}\\}.$
(10)

Note that we consider these cues on the subject canonical space
$\mathbf{x}_{i}^{sc}$, rather than the generic canonical space since we
empirically find inferring it in the generic canonical space suffers from
local minima issue. Similar to PointAvatar, the SDF cues are used to infer
surface normal in the subject canonical space $\mathbf{n}_{sc}$ and the ones
in the deformed space can be computed as in Eq. 3. Note that, different from
PointAvatar, the cues in the canonical space are also conditioned by latent
code $\mathbf{z}$, allowing for varying by controlling latent codes for part
appearance changes. Furthermore, we additionally include the binary
segmentation cues $\chi_{i}$ to estimate a “synthesis” part in the current
identity represented by $\mathbf{z}$, which is used in our _Zero Shot
Transfer_ approach (Sec. 5).

Comparison over PointAvatar. The major difference from the PointAvatar is the
use of latent codes $\mathbf{z}$ to enable the single model can handle varying
appearance changes. For this purpose, we modify the model, including a generic
canonical stage and injecting the $\mathbf{z}$ into the submodules. We also
made several modifications, including (1) the beta controlling part in Eq.
(8), which is important for fitting the subject and face attribute when
generating a synthetic database, (2) inferring the segmentation mask for the
usage in _Zero Shot Transfer_.

###  4.2 Synthetic DB Generation via Part Swapping

We aim to build our personalized generative model to preserve the target human
identity, while allowing changing facial attributes, such as hair, nose, or
wearing a hat. To learn such a model, we need the videos of the target human
with all such variations, which is not available in practice. We present a
solution to synthesize such variations from other video sources by swapping a
face subpart of the target identity person with others. Examples are shown in
Fig. 4. We collect a set of monocular videos, denoted as
$V^{db}=\\{V_{i},p_{i}\\}_{i=1\cdots K}$ from various individuals to model
various types of facial attribute variations, as shown in Fig. 3. For each
video $V_{i}$, we determine the target facial attribute $p_{i}\in\mathbf{P}$
which we want to use for the swapping, where $\mathbf{P}=\\{\text{hair, nose,
hat, eyes, eyebrows, mouth}\\}$.

For each monocular video $V_{i}$ from the facial attribute DB, we build a
personalized avatar $\mathcal{M}_{i}$ using our avatar generation module with
only the single video identity. Here we set the identity latent code
$\mathbf{z}_{0}$ as learnable while setting other parts accordingly. Once
built, the avatar is animatable following the FLAME parameters.

Face Part Swapping. We denote the input video of the target person as
$V^{tp}$. The goal of our face part swapping is to replace the facial
attributes $V^{tp}$ with the one using the person appeared in by $i$-th
attribute video $V_{i}$. Since both videos have different poses, viewpoints,
and facial expressions of different individuals, such replacement is non-
trivial in 2D video space. Our idea is to leverage the animatable avatar model
$M_{i}$ constructed from $V_{i}$ to render the facial attribute aligned into
the target identity’s videos, $V^{tp}$. This can be performed by inputting the
FLAME parameters and camera parameters obtained from $V^{tp}$ into $M_{i}$,
and rendering only the necessary attribute region with blending. To choose the
selected attribute regions specified by the corresponding attribute
$p_{i}\in\mathbf{P}$, we use an off-the-shelf face segmentation model, BiSeNet
[64], to obtain the mask of the desired target attribute $p_{i}$. Then, we can
synthesize the attribute part into the target human videos $V^{tp}$ as
follows:

$\displaystyle I_{\text{i-th
attributes}}=\mathcal{R}(\mathcal{M}_{i}(\bm{\theta}_{\text{tp}},\bm{\beta}_{\text{tp}},\bm{\psi}_{\text{tp}}))$
$\displaystyle I_{\text{target-swapped}}=\bm{1}_{\text{tp}}\cdot
I_{\text{tp}}+\bm{1}_{\text{i-th attributes}}\cdot\mathcal{B}(I_{\text{i-th
attributes}})$

, where $\mathcal{R}$ denotes the rendering function from the avatar $M_{i}$
with the FLAME parameters obtained from $V^{tp}$. $\bm{1}_{\text{tp}}$ is the
segmentation mask to select the target subject regions excluding the attribute
parts, and $\bm{1}_{\text{i-th attributes}}$ is segmentation for the
attributed part of from $M_{i}$’s rendering, respectively. Note that we use
the $\beta_{\text{subject}}$ to make better alignment from $M_{i}$ into the
target identity, which was the motivation for introducing the $\beta$
parameter in building our avatar, different from the original PointAvatar
model. $\mathcal{B}$ denotes the blending function, where we use Poisson
Blending [44] to reduce artifacts. We further perform post-processing to
enhance the quality of part-swapped images using OpenCV’s dilate and erode
function to remove holes. As a special preprocessing for hair-swapping, it is
empirically advantageous to synthesize the target person’s hair into a bald
head before the blending, where we leverage Stable Diffusion [48] with auto-
generated mask images. See Supp. mat. for details.

We denote $\hat{V}^{tp}_{i}$ as the part-swapped videos by $i$-th attribute DB
identity. Examples are shown in Fig. 4. Note that the resulting videos contain
the same target identity with varying attributes via synthesis, which we use
to build our personalized generative models.

###  4.3 Learning for Personalized Generative Model

Figure 5: Zero Shot Transfer. PEGASUS generates high-quality and natural
appearances via zero-shot transfer.

Latent Code Setting. We train our model by using $V^{tp}$ and synthesized
videos $\\{\hat{V}^{tp}_{i}\\}_{i=1\cdots K}$ For each video, we set the
latent code $\mathbf{z}=\\{\mathbf{z}^{p}\\}_{p=0,\cdots,D}$ according to the
attribute types. Specifically, we use the same shared learnable identity
latent code $\mathbf{z}_{0}$ for all videos, given that the videos are for the
same identity. If a video is about the variation of $p$-th attribute category,
where $p\in\mathbf{P}$, we assign a separate learnable latent code for that
part $\mathbf{z}^{p}$, by keeping other latent code parts shared. With this
setup, we allow the model can have the latent codes in a disentangled manner
so that each attribute code part can represent the corresponding facial
subparts.

Loss Function. We follow the PointAvatar [71] to define loss functions. The
total loss is as follows:

$\begin{split}\mathcal{L}&=\lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}}+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}+\lambda_{\text{FLAME}}\mathcal{L}_{\text{FLAME}}+\lambda_{\text{vgg}}\mathcal{L}_{\text{vgg}}\\\
&+\lambda_{\text{normal}}\mathcal{L}_{\text{normal}}+\lambda_{\text{seg}}\mathcal{L}_{\text{seg}}+\lambda_{\mathbf{z}~{}\text{reg}}\mathcal{L}_{\mathbf{z}~{}\text{reg}}\end{split}$
(11)

, where $\mathcal{L}_{\text{rgb}}$, $\mathcal{L}_{\text{mask}}$,
$\mathcal{L}_{\text{FLAME}}$ penalize RGB, mask, FLAME parameter differences
respectively. $\mathcal{L}_{\text{vgg}}$ is based on the VGG feature to
enhance the rendered image quality. Different from previous work [71, 70], we
also include three more losses, $\mathcal{L}_{\text{normal}}$,
$\mathcal{L}_{\text{seg}}$ and $\mathcal{L}_{\mathbf{z}~{}\text{reg}}$. We
adopt the normal loss as follows:
$\mathcal{L}_{\text{normal}}=\|\mathbf{n}-\mathbf{n}^{d}\|$ , and we
empirically find its advantage in producing better-quality avatars. We
generate the pseudo ground truth normal $\mathbf{n}$ from the $V^{tp}$ and the
avatar trained with a single identity of each $V^{db}$. We also include
segmentation loss to predict facial attribute categories $\chi_{i}$ per each
point. See more details in Supp. Mat.

Training Strategy. We train PEGASUS in a coarse-to-fine manner. First,
following PointAvatar, we upsample the number of points and reduce the radii
of the points during the training with the constant period of epochs. Second,
we train our model in the two-step strategy. In the first stage, we train our
model using the target individual $V^{tp}$, which is no part swapped on the
face, with the latent codes $\mathbf{z}$ until the beginning of the training.
In the second stage, we use all of the part-swapped videos $\hat{V}_{i}^{tp}$
until the end of training. Please check the details in Supp. Mat.

##  5 Generative Avatar via Zero-Shot Transfer

We present an alternative method to efficiently achieve the goal of a
personalized generative avatar without producing part-swapped synthesized
videos. Our core idea is based on the assumption that we already have the
previously constructed personalized avatar model for an identity (denoted as
the _source human_), $\mathcal{M}_{\phi}$, with the functionally to control
the face attribute variations. Given the new identity’s video (denoted as the
_target human_), we first train our generative avatar architecture with the
single video of the target human, resulting in $\mathcal{M}_{th}$. Then, we
aim to achieve the same goal of the personalized avatar for the target human,
by fusing the controlled attributed part of $\mathcal{M}_{\phi}$ and the
remaining part $\mathcal{M}_{th}$, which we call a “zero-shot model”.
Specially, given the SMPL parameters and input latent codes inputs, we can
drive both models as:

$\displaystyle\mathcal{M}_{\phi}(\mathbf{x}^{gc},\mathbf{z},\bm{\beta},\bm{\theta},\bm{\psi})=\\{\mathbf{x}_{\phi}^{d},\mathbf{n}_{\phi}^{d},\mathbf{a}_{\phi},\chi_{\phi}\\},$
(12)
$\displaystyle\mathcal{M}_{th}(\mathbf{x}^{gc},\mathbf{z},\bm{\beta},\bm{\theta},\bm{\psi})=\\{\mathbf{x}_{th}^{d},\mathbf{n}_{th}^{d},\mathbf{a}_{th},\chi_{th}\\}.$
(13)

The final version of the avatar is constructed by combining the subsets of
point clouds from both avatars, using the estimated segmentation masks,
$\chi_{\phi}$ and $\chi_{th}$:

$\displaystyle
P_{\text{naive}}=\\{\mathbf{x}_{\phi}^{d,i}\\}_{\chi^{i}_{\phi}=1}\cup\\{\mathbf{x}_{th}^{d,i}\\}_{\chi^{i}_{th}=0}$
(14)

, $\chi^{i}_{\phi}$ and $\chi^{i}_{th}$ are the segmentation masks of the face
attribute we currently try to control via $\mathbf{z}$. Intuitively, from the
source human we transfer all points that belong to the target attribute
$\chi^{i}_{\phi}=1$, and from the target human we only remain the points that
do not belong to the target attribute $\chi^{i}_{th}=0$. While we find this
naive composition is already compelling, we observe that there exists a gap
between the fused parts. To enhance the quality, we further perform an
additional optimization processing to better alignment, with color blending.
See supp. mat. for the post-processing. Examples of our zero-shot modeling are
shown in Fig. 5.

##  6 Experiments

Figure 6: Single Part-Swap Avatar on Hair. Our synthesis method creates a
photo-realistic avatar with a hairstyle that is accurately transferred.

#### Datasets.

As the attribute database, we collect publicly available 109 videos from the
Internet, and build their individual PointAvatar model as shown in Fig. 3. For
the target person $V^{tp}$ used of the personalized generative avatar, we
select the publicly available videos from NerFACE [17], and the individuals
are shown in Fig. 4 and Fig. 6. To reenact the reconstructed avatar into
unseen facial poses and expressions, we extract FLAME parameters using DECA
[14] from our own monocular video with diverse facial orientations and
expressions.

###  6.1 Part-Swapping Comparison with Baselines

Given that we are the first to build a personalized generative model, there is
no direct competitor to compare the full generative functionality. Thus, we
consider a sub-problem of building an animatable 3D avatar by transferring a
facial attitude from another video source. While the resulting output is not a
generative model due to its limitation of producing unseen attributes, one can
use this strategy to alter parts of the face, assuming a large number of
attribute source videos are available. In this evaluation, we only consider
hairstyles as our attribute and consider 5 videos with different hairstyles.
Examples are shown in Fig. 6.

Baselines. We consider possible alternative approaches to building the 3D
avatar of the target individual with the hair from another video.

_DELTA:_ DELTA [16] achieves the transfer of hairstyles from a source to a
target by employing a hybrid approach that combines both explicit and implicit
representations. The major goal of DELTA is aligned to this sub-problem test,
while it does not have generative functionality.

_E4S[41] \+ PointAvatar (E4S+PA):_ E4S [41] employs GAN inversion for the face
swapping. As a way of building a 3D avatar, we first replace the hair of the
target individual in 2D spaces on all image frames via E4S. Then, we apply the
original version of PointAvatar to make it into a 3D avatar model. Note that
the GAN-based method does not guarantee the view consistency on the
synthesized images, resulting in blurry 3D avatar construction.

_Custom Diffusion[36] \+ PointAvatar (CD+PA):_ Similar to the E4S+PA, we can
apply the Custom Diffusion model as a tool to produce hair-changed 2D images
for the target individual, conditioned by the hair-style of other video
source. Then, we apply the original PointAvatar.

_$\text{Ours}_{\text{swap}}$ \+ PointAvatar (Ours${}_{\text{swap}}$ \+ PA):_
We also include a simplified version of ours as a baseline, where we produce
the part-swapped 2D videos (described in Sec. 4.2) for each hairstyle transfer
and apply PointAvatar.

_$\text{Ours}_{\text{person-gen}}$ and $\text{Ours}_{\text{zero-shot}}$ :_ We
show the performance of our generative models using the latent codes
corresponding to the target hairstyles. Note our models can produce not just
various hairs, but all other attribute styles.

Metrics. After we build 3D avatars of the target individual by transferring
the hairstyle from video sources, we apply unseen facial expressions and head
orientation to visualize the avatar in diverse novel poses and render them
into images. For the comparison, we consider both the naturalness or the 3D
avatar and identity preservation of the target individual. We use two metrics,
Fréchet Inception Distance (FID) and Kernel Inception Distance (KID), to
evaluate the naturalness of the rendering of the produced 3D avatar. In
computing FID and KID, we compare the distributions of rendered outputs of the
3D avatars with the background matted FFHQ [29]. To quantify whether the
output 3D avatars preserve the original identity of the target human, we also
include ArcFace [12] metric. Here, we compare the rendering of the edited
version with the rendering of the non-edit PointAvatar with the same unseen
face pose.

Method | Naturalness | Identity
---|---|---
FID${\downarrow}$ |  KID${\downarrow}$ |  ArcFace${\uparrow}$
CD + PA | 181.60 | 0.1367 | 0.6691
E4S + PA | 176.64 | 0.1416 | 0.5701
DELTA | 198.40 | 0.1797 | 0.6732
Ours${}_{\text{swap}}$+PA | 169.54 | 0.1406 | 0.7179
Ours${}_{\text{person-gen}}$ | 190.10 | 0.1696 | 0.6883
Ours${}_{\text{zero shot}}$ | 191.47 | 0.1881 | 0.7792
Table 1: Quantitative Comparison. The synthesis method (upper rows) and full
model of hair category (lower rows).

Results. We show the quantitative comparison in Tab. 1 and we also show
example results in Fig. 6. As shown in the table, the 3D avatar produced by
our face-swap _$Ours_{\text{swap}}+PA$_ achieves the best metrics at FID and
ArcFace, showing better naturalness while keeping the identity of the target
individual. Although the custom diffusion-based output _CD+PA_ shows the
better result in the KID metric, it changes the identity significantly,
resulting in low performance in the ArcFace metric. Our full generative model
_Ours ${}_{\text{person-gen}}$_ also shows convincing performance even though
the model is much more generic and trained to express diverse variations. It
outperforms all other baseline methods in preserving identity while showing
comparable naturalness. Our zero-shot generative model _Ours ${}_{\text{zero-
shot}}$_ shows the best identity-preserving performance because its face part
is identical to the non-edited PA while transferring the hair part from _Ours
${}_{\text{person-gen}}$_.

###  6.2 Evaluating Generative Performance

Method | Naturalness | Identity
---|---|---
FID${\downarrow}$ |  KID${\downarrow}$ |  ArcFace${\uparrow}$
Ours${}_{\text{no synthesis, latent swap}}$ | 231.62 | 0.2630 | 0.6285
Ours${}_{\text{no synthesis, latent interp.}}$ | 240.17 | 0.2482 | 0.4722
Ours${}_{\text{synthesis, latent interp.}}$ | 206.87 | 0.1839 | 0.8127
Table 2: Evaluating Generative Performance. Quantitative comparison by
producing appearance via latent code interpolation.

We also compare the generative performance of our models. As the baseline, we
consider the scenario of using entire videos, including target individual
$V^{tp}$ and face attribute videos $V^{db}$ into the generative model without
our facial part-swap approach. Once trained, we check the unseen appearances
by interpolating the latent codes of two seen samples during training.
However, we consider two ways of interpolation: (1) naive interpolation
between the latent codes $(\mathbf{z}_{A},\mathbf{z}_{B})$ of two original
videos (latent interpolation), (2) via latent code swapping by keeping the
target individual’s latent code $\\{z_{p}\\}_{p\neq i}$ and other sources’
attribute latent code $\\{z_{q}\\}_{q=i}$ (latent swapping),
$\mathbf{z}=\\{z_{p},z_{q}\\}_{p\neq i,q=i,i\in[0,n(\mathbf{P})]}$. For
quantitative evaluation, we use the same metric as Sec. 6.1 to measure the
naturalness and identity preserving. The quantitative result is shown in Tab.
2, and example qualitative results are shown in Fig. 7. The outputs of our
model show compelling performance in producing realistic face part variations
while keeping the identity. As expected, both interpolation strategies of the
baseline models struggle to generate realistic avatars for the interpolated
latent codes.

###  6.3 Ablation Studies and More Results

$\mathcal{O}_{gc}$ | $\mathcal{O}_{sc}$ |  PSNR$\uparrow$ |  SSIM${\uparrow}$ |  LPIPS${\downarrow}$
---|---|---|---|---
$\times$ | $\times$ | 20.92 | 0.9059 | 0.1351
$\times$ | $\checkmark$ | 21.55 | 0.9033 | 0.1292
$\checkmark$ | $\checkmark$ | 21.75 | 0.9059 | 0.1291
Table 3: Ablation Study: offsets. Note that the offsets represent the output
of the deformation network. Multi-staged canonical spaces produce better image
quality.

Multi-Stage Canonical Space. We compare our multi-stage canonical space
framework with the alternative framework with one-stage or two-stage (by
PointAvatar) frameworks. For quantitative comparison on multi-stage canonical
space, we use PSNR, SSIM, and LPIPS [69] metrics. We evaluate them on unseen
test sequences with novel head poses and facial expressions from all of the
synthesized videos in Fig. 4. In Tab. 3, our multi-stage canonical space and
point deformation outperforms the one and two-stage deformation of all
metrics.

Figure 7: Qualitative Comparison. PEGASUS show compelling performance in
producing realistic face part variations.

(a) Without Normal Loss

(b) With Normal Loss

Figure 8: Ablation Study: Normal Loss. The normal loss improves the RGB and
geometry quality of our model.

Normal Loss. We also show the advantage of our normal loss, used for training
the avatar model. In Fig. 8, the result shows that the normal loss improves
the RGB and normal qualities, resulting in more realistic appearances.

More Qualitative Results. We further demonstrate the performance of our
methods by showing the ability to control multiple parts as shown in Fig. 9,
and also by showing more interpolation results as shown in Fig. 10 and our
supp. video.

(a) No Edit

(b) Hair, Nose

(c) Hat, Mouth

(d) Eyes, Nose

Figure 9: Multiple Composition. PEGASUS generates the avatar with multiple
face attributes. Figure 10: Person-specific Interpolation. We interpolate the
attribute latent code $\mathbf{z}^{p},p\in\mathbf{P}$ between two avatars.

##  7 Discussion

We present a method for constructing personalized generative 3D face avatars
from monocular video sources. As a compositional generative model, our model
enables disentangled controls to selectively alter the facial attributes of
the target individual, while preserving the identity. Notably, our
personalized generative model is built exclusively from monocular videos,
without relying on complex multi-view system setups. To achieve the goal, we
first present a method to construct a person-specific generative 3D avatar by
building a synthetic video collection of the target identity with varying
facial attributes, where the videos are synthesized by borrowing parts from
diverse individuals from other monocular videos. we also present a zero-shot
approach to achieve the same generative modeling more efficiently.

Limitation As a limitation, the quality of our personalized avatar still does
not reach the photo-realistic quality, showing noticeable artifacts. Building
a more generative model to include multiple identities in a single model can
be another exciting extension of our model.

Due to the reliance on non-physical-based methods for generating the synthetic
DB, our approach exhibits limitations in achieving physical accuracy,
particularly for the replacement of facial features such as noses, mouths, and
eyes.

## References

  * Aneja et al. [2023] S. Aneja, J. Thies, A. Dai, and M. Nießner.  Clipface: Text-guided editing of textured 3d morphable models.  In _Proc. ACM SIGGRAPH_ , pages 1–11, 2023.
  * Bai et al. [2023] Y. Bai, Y. Fan, X. Wang, Y. Zhang, J. Sun, C. Yuan, and Y. Shan.  High-fidelity facial avatar reconstruction from monocular video with generative priors.  In _Proc. CVPR_ , pages 4541–4551, 2023.
  * Bharadwaj et al. [2023] S. Bharadwaj, Y. Zheng, O. Hilliges, M. J. Black, and V. Fernandez-Abrevaya.  Flare: Fast learning of animatable and relightable mesh avatars.  _arXiv preprint arXiv:2310.17519_ , 2023.
  * Blanz and Vetter [1999] V. Blanz and T. Vetter.  A morphable model for the synthesis of 3d faces.  In _Proc. ACM SIGGRAPH_ , 1999.
  * Bradski [2000] G. Bradski.  The OpenCV Library.  _Dr. Dobb’s Journal of Software Tools_ , 2000.
  * Brooks et al. [2023] T. Brooks, A. Holynski, and A. A. Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proc. CVPR_ , pages 18392–18402, 2023.
  * Bulat and Tzimiropoulos [2017] A. Bulat and G. Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks).  In _International Conference on Computer Vision_ , 2017.
  * Cao et al. [2023] Y. Cao, Y.-P. Cao, K. Han, Y. Shan, and K.-Y. K. Wong.  Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models.  _arXiv preprint arXiv:2304.00916_ , 2023.
  * Chan et al. [2022] E. R. Chan, C. Z. Lin, M. A. Chan, K. Nagano, B. Pan, S. De Mello, O. Gallo, L. J. Guibas, J. Tremblay, S. Khamis, et al.  Efficient geometry-aware 3d generative adversarial networks.  In _Proc. CVPR_ , pages 16123–16133, 2022.
  * Chang et al. [2023] S. Chang, G. Kim, and H. Kim.  Hairnerf: Geometry-aware image synthesis for hairstyle transfer.  In _Proc. ICCV_ , pages 2448–2458, 2023.
  * Daněček et al. [2022] R. Daněček, M. J. Black, and T. Bolkart.  Emoca: Emotion driven monocular face capture and animation.  In _Proc. CVPR_ , pages 20311–20322, 2022.
  * Deng et al. [2019] J. Deng, J. Guo, N. Xue, and S. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In _Proc. CVPR_ , pages 4690–4699, 2019.
  * Dong et al. [2023] Z. Dong, X. Chen, J. Yang, M. J. Black, O. Hilliges, and A. Geiger.  Ag3d: Learning to generate 3d avatars from 2d image collections.  _arXiv preprint arXiv:2305.02312_ , 2023.
  * Feng et al. [2021] Y. Feng, H. Feng, M. J. Black, and T. Bolkart.  Learning an animatable detailed 3d face model from in-the-wild images.  _ACM Transactions on Graphics (ToG)_ , 40(4):1–13, 2021.
  * Feng et al. [2022] Y. Feng, J. Yang, M. Pollefeys, M. J. Black, and T. Bolkart.  Capturing and animation of body and clothing from monocular video.  In _Proc. ACM SIGGRAPH ASIA_ , pages 1–9, 2022.
  * Feng et al. [2023] Y. Feng, W. Liu, T. Bolkart, J. Yang, M. Pollefeys, and M. J. Black.  Learning disentangled avatars with hybrid 3d representations.  _arXiv preprint arXiv:2309.06441_ , 2023.
  * Gafni et al. [2021] G. Gafni, J. Thies, M. Zollhofer, and M. Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  In _Proc. CVPR_ , pages 8649–8658, 2021.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio.  Generative adversarial nets.  _NeurIPS_ , 27, 2014.
  * Grassal et al. [2022] P.-W. Grassal, M. Prinzler, T. Leistner, C. Rother, M. Nießner, and J. Thies.  Neural head avatars from monocular rgb videos.  In _Proc. CVPR_ , pages 18653–18664, 2022.
  * Guo et al. [2023] C. Guo, T. Jiang, X. Chen, J. Song, and O. Hilliges.  Vid2avatar: 3d avatar reconstruction from videos in the wild via self-supervised scene decomposition.  In _Proc. CVPR_ , pages 12858–12868, 2023.
  * Haque et al. [2023] A. Haque, M. Tancik, A. A. Efros, A. Holynski, and A. Kanazawa.  Instruct-nerf2nerf: Editing 3d scenes with instructions.  _arXiv preprint arXiv:2303.12789_ , 2023.
  * Hertz et al. [2022] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2023] H.-I. Ho, L. Xue, J. Song, and O. Hilliges.  Learning locally editable virtual humans.  In _Proc. CVPR_ , pages 21024–21035, 2023.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _NeurIPS_ , 33:6840–6851, 2020.
  * Hong et al. [2022] F. Hong, M. Zhang, L. Pan, Z. Cai, L. Yang, and Z. Liu.  Avatarclip: Zero-shot text-driven generation and animation of 3d avatars.  _arXiv preprint arXiv:2205.08535_ , 2022.
  * Jiang et al. [2022] K. Jiang, S.-Y. Chen, F.-L. Liu, H. Fu, and L. Gao.  Nerffaceediting: Disentangled face editing in neural radiance fields.  In _Proc. ACM SIGGRAPH ASIA_ , pages 1–9, 2022.
  * Johnson et al. [2016] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14_ , pages 694–711. Springer, 2016.
  * Kamata et al. [2023] H. Kamata, Y. Sakuma, A. Hayakawa, M. Ishii, and T. Narihira.  Instruct 3d-to-3d: Text instruction guided 3d-to-3d conversion.  _arXiv preprint arXiv:2303.15780_ , 2023.
  * Karras et al. [2019] T. Karras, S. Laine, and T. Aila.  A style-based generator architecture for generative adversarial networks.  In _Proc. CVPR_ , pages 4401–4410, 2019.
  * Ke et al. [2022] Z. Ke, J. Sun, K. Li, Q. Yan, and R. W. Lau.  Modnet: Real-time trimap-free portrait matting via objective decomposition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 1140–1147, 2022.
  * Kim et al. [2023a] H. Kim, G. Lee, Y. Choi, J.-H. Kim, and J.-Y. Zhu.  3d-aware blending with generative nerfs.  _arXiv preprint arXiv:2302.06608_ , 2023a.
  * Kim et al. [2022] T. Kim, C. Chung, Y. Kim, S. Park, K. Kim, and J. Choo.  Style your hair: Latent optimization for pose-invariant hairstyle transfer via local-style-aware hair alignment.  In _Proc. ECCV_ , pages 188–203. Springer, 2022.
  * Kim et al. [2023b] T. Kim, S. Saito, and H. Joo.  Ncho: Unsupervised learning for neural 3d composition of humans and objects.  _arXiv preprint arXiv:2305.14345_ , 2023b.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023.
  * Kumari et al. [2023] N. Kumari, B. Zhang, R. Zhang, E. Shechtman, and J.-Y. Zhu.  Multi-concept customization of text-to-image diffusion.  In _Proc. CVPR_ , pages 1931–1941, 2023.
  * Li et al. [2023] J. Li, S. Saito, T. Simon, S. Lombardi, H. Li, and J. Saragih.  Megane: Morphable eyeglass and avatar network.  In _Proc. CVPR_ , pages 12769–12779, 2023.
  * Li [2023] S. Li.  Instruct-video2avatar: Video-to-avatar generation with instructions.  _arXiv preprint arXiv:2306.02903_ , 2023.
  * Li et al. [2017] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero.  Learning a model of facial shape and expression from 4d scans.  _ACM Trans. Graph._ , 36(6):194–1, 2017.
  * Liu et al. [2023a] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023a.
  * Liu et al. [2023b] Z. Liu, M. Li, Y. Zhang, C. Wang, Q. Zhang, J. Wang, and Y. Nie.  Fine-grained face swapping via regional gan inversion.  In _Proc. CVPR_ , pages 8578–8587, 2023b.
  * Meng et al. [2021] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Nair and Hinton [2010] V. Nair and G. E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Proceedings of the 27th international conference on machine learning (ICML-10)_ , pages 807–814, 2010.
  * Pérez et al. [2023] P. Pérez, M. Gangnet, and A. Blake.  Poisson image editing.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pages 577–582. 2023.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022\.
  * Ranade et al. [2022] S. Ranade, C. Lassner, K. Li, C. Haene, S.-C. Chen, J.-C. Bazin, and S. Bouaziz.  Ssdnerf: Semantic soft decomposition of neural radiance fields.  _arXiv preprint arXiv:2212.03406_ , 2022.
  * Ravi et al. [2020] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari.  Accelerating 3d deep learning with pytorch3d.  _arXiv:2007.08501_ , 2020.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proc. CVPR_ , pages 10684–10695, 2022.
  * Rosu et al. [2022] R. A. Rosu, S. Saito, Z. Wang, C. Wu, S. Behnke, and G. Nam.  Neural strands: Learning hair geometry and appearance from multi-view images.  In _Proc. ECCV_ , pages 73–89. Springer, 2022.
  * Sanyal et al. [2019] S. Sanyal, T. Bolkart, H. Feng, and M. J. Black.  Learning to regress 3d face shape and expression from an image without 3d supervision.  In _Proc. CVPR_ , pages 7763–7772, 2019.
  * Shi et al. [2022] Y. Shi, X. Yang, Y. Wan, and X. Shen.  Semanticstylegan: Learning compositional generative priors for controllable image synthesis and editing.  In _Proc. CVPR_ , pages 11254–11264, 2022.
  * Simonyan and Zisserman [2014] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Sklyarova et al. [2023] V. Sklyarova, J. Chelishev, A. Dogaru, I. Medvedev, V. Lempitsky, and E. Zakharov.  Neural haircut: Prior-guided strand-based hair reconstruction.  _arXiv preprint arXiv:2306.05872_ , 2023.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Sun et al. [2022] J. Sun, X. Wang, Y. Shi, L. Wang, J. Wang, and Y. Liu.  Ide-3d: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis.  _ACM TOG_ , 41(6):1–10, 2022.
  * Telea [2004] A. Telea.  An image inpainting technique based on the fast marching method.  _Journal of graphics tools_ , 9(1):23–34, 2004\.
  * Wang et al. [2023a] T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al.  Rodin: A generative model for sculpting 3d digital avatars using diffusion.  In _Proc. CVPR_ , pages 4563–4573, 2023a.
  * Wang et al. [2023b] Z. Wang, G. Nam, T. Stuyck, S. Lombardi, C. Cao, J. Saragih, M. Zollhöfer, J. Hodgins, and C. Lassner.  Neuwigs: A neural dynamic model for volumetric hair capture and animation.  In _Proc. CVPR_ , pages 8641–8651, 2023b.
  * Xu et al. [2022] Y. Xu, Y. Yin, L. Jiang, Q. Wu, C. Zheng, C. C. Loy, B. Dai, and W. Wu.  Transeditor: Transformer-based dual-space gan for highly controllable facial editing.  In _Proc. CVPR_ , pages 7683–7692, 2022.
  * Xu et al. [2023a] Y. Xu, Z. Shu, C. Smith, J.-B. Huang, and S. W. Oh.  In-n-out: Face video inversion and editing with volumetric decomposition.  _arXiv preprint arXiv:2302.04871_ , 2023a.
  * Xu et al. [2023b] Y. Xu, H. Zhang, L. Wang, X. Zhao, H. Huang, G. Qi, and Y. Liu.  Latentavatar: Learning latent expression code for expressive neural head avatar.  _arXiv preprint arXiv:2305.01190_ , 2023b.
  * Yenamandra et al. [2021] T. Yenamandra, A. Tewari, F. Bernard, H.-P. Seidel, M. Elgharib, D. Cremers, and C. Theobalt.  i3dmm: Deep implicit 3d morphable model of human heads.  In _Proc. CVPR_ , pages 12803–12813, 2021.
  * Yin et al. [2022] F. Yin, Y. Zhang, X. Cun, M. Cao, Y. Fan, X. Wang, Q. Bai, B. Wu, J. Wang, and Y. Yang.  Styleheat: One-shot high-resolution editable talking face generation via pre-trained stylegan.  In _Proc. ECCV_ , pages 85–101. Springer, 2022.
  * Yu et al. [2018] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In _Proc. ECCV_ , 2018.
  * Zhang et al. [2023a] H. Zhang, Y. Feng, P. Kulits, Y. Wen, J. Thies, and M. J. Black.  Text-guided generation and editing of compositional 3d avatars.  _arXiv preprint arXiv:2309.07125_ , 2023a.
  * Zhang et al. [2022] J. Zhang, A. Siarohin, Y. Liu, H. Tang, N. Sebe, and W. Wang.  Training and tuning generative neural radiance fields for attribute-conditional 3d-aware face generation.  _arXiv preprint arXiv:2208.12550_ , 2022.
  * Zhang et al. [2023b] L. Zhang, Q. Qiu, H. Lin, Q. Zhang, C. Shi, W. Yang, Y. Shi, S. Yang, L. Xu, and J. Yu.  Dreamface: Progressive generation of animatable 3d faces under text guidance.  _arXiv preprint arXiv:2304.03117_ , 2023b.
  * Zhang et al. [2023c] L. Zhang, A. Rao, and M. Agrawala.  Adding conditional control to text-to-image diffusion models, 2023c.
  * Zhang et al. [2018] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proc. CVPR_ , pages 586–595, 2018.
  * Zheng et al. [2022] Y. Zheng, V. F. Abrevaya, M. C. Bühler, X. Chen, M. J. Black, and O. Hilliges.  Im avatar: Implicit morphable head avatars from videos.  In _Proc. CVPR_ , pages 13545–13555, 2022.
  * Zheng et al. [2023] Y. Zheng, W. Yifan, G. Wetzstein, M. J. Black, and O. Hilliges.  Pointavatar: Deformable point-based head avatars from videos.  In _Proc. CVPR_ , pages 21057–21067, 2023.
  * Zhu et al. [2020] J. Zhu, Y. Shen, D. Zhao, and B. Zhou.  In-domain gan inversion for real image editing.  In _Proc. ECCV_ , pages 592–608. Springer, 2020.
  * Zhu et al. [2021] P. Zhu, R. Abdal, J. Femiani, and P. Wonka.  Barbershop: Gan-based image compositing using segmentation masks.  _arXiv preprint arXiv:2106.01505_ , 2021.
  * Zhu et al. [2022] P. Zhu, R. Abdal, J. Femiani, and P. Wonka.  Hairnet: Hairstyle transfer with pose changes.  In _Proc. ECCV_ , pages 651–667. Springer, 2022.
  * Zielonka et al. [2022] W. Zielonka, T. Bolkart, and J. Thies.  Towards metrical reconstruction of human faces.  In _Proc. ECCV_ , pages 250–269. Springer, 2022.
  * zllrunning [2019] zllrunning.  face-parsing.pytorch.  https://github.com/zllrunning/face-parsing.PyTorch, 2019.

##  Appendix A Synthetic DB Generation

In this section, we provide further details of our synthetic DB generation via
part swapping, introduced in Sec. 4.2 of our main manuscript.

#### Hair.

We empirically find that removing the hair of the target subject is necessary
before swapping the hair from the attribute DB. To create a bald head
representation of the target individual, we utilize the Stable Diffusion [48],
employing auto-generated mask images for this purpose. To generate the hair
mask, we utilize an off-the-shelf face parsing network [64, 76]. We dilate the
mask image using a kernel of size 20 from OpenCV [5]. Then, to generate an
image of the target person with a bald head, we employ Stable Diffusion in
conjunction with ControlNet [68]. The prompt to generate the bald head is
”bald, clean skin, smooth bald, small head, albedo”. The negative prompt is
”hair, wrinkles, shadow, light reflection, tattoo, sideburns, facial hair,
cartoonish, abstract interpretations, hat, head coverings”. The examples are
shown in Fig. 11.

#### Other Attributes.

Our goal is to synthesize the shape and appearance of the facial attribute
from the attribute DB into the target individual as seamlessly as possible. To
achieve this, we first render the avatar from an attribute DB into the same
view, shape, and facial expressions as the target frame of the target
individual’s video, as described in Sec. 4.2 in our main manuscript.
Subsequently, we acquire the mask of the rendered facial attribute by
employing a face parsing network [64, 76] and then slightly enlarge it by
applying the dilate function in OpenCV. We also perform the segmentation for
the target individual’s image to acquire the mask of the target facial
attribute by utilizing the face parsing network [64, 76], where the target
facial part is subsequently “removed” via inpainting by employing the Fast
Marching Method [56]. This process can be considered as a similar process of
“bald head synthesis” before integrating the desired facial part from the
attribute source. Finally, we seamlessly integrate the facial attribute from
the attribute avatar into the target individual using Poisson blending [44].
Examples of nose and mouth synthesis employing this technique are illustrated
in Fig. 12.

Figure 11: Stable Diffusion Inpainting. We leverage Stable Diffusion [48] and
ControlNet [68] to remove the target’s hair and make it bald, in order to
synthesize different hair. The automatically generated mask images represent
the area designated for inpainting. Figure 12: Poisson Blending Inpainting. We
use Poisson blending [44] to synthesize the facial attribute and the target’s
face.

#### Tracking and Masking.

To extract FLAME parameters from images, along with their corresponding camera
parameters, we utilize the DECA model [14]. When FLAME parameters are directly
extracted using the DECA model, we notice that the head pose estimation is
noisy and jittery, particularly in the frames where the eyes in the original
images are blinking. To improve the FLAME parameter estimation quality,
following the similar process of PointAvatar [71], we apply an optimization
procedure to align the 2D projection of FLAME’s facial landmarks with the
detection outputs of an off-the-shelf 2D facial landmark detector [7]. This
optimization process is based on the assumption that the quality of the 2D
landmark detection is more precise. We minimize the point-wise distance
between the landmark obtained from FLAME and the 2D facial landmark to
optimize the shape, pose, and camera parameters. Different from PointAvatar’s
approach, instead of using a singular translation vector for each video, we
employ a unique vector for every image frame in scenarios involving in-the-
wild video tracking.

To create the foreground mask image, we leverage an off-the-shelf background
matting network [30] to obtain the portrait mask images from the videos. We
use the face parsing network [64, 76] to obtain part segmentations of the
faces and leverage SegmentAnything Model [35, 40] for segmenting head
accessories.

##  Appendix B Postprocessing of Zero-Shot Transferred Avatar

Figure 13: Zero-Shot Landmarks for Optimization. The red dot represents our
personalized generative model’s $k$-nearest neighbor of 3D Landmarks from
FLAME keypoints, and the blue dot represents the target’s $k$-nearest neighbor
of 3D Landmarks from FLAME keypoints.

(a) Before Optimize.

(b) After Optimize.

(c) Color Blending.

Figure 14: Zero-Shot Optimization Steps. The red box represents the additional
part to fill the empty space. Through zero-shot modeling, we generate an
avatar with a high-quality and reasonable appearance in three stages of post-
processing in Sec. B.

We provide further details of the Eq. (14) in our main manuscript, which is
the process of combining the subsets of point clouds from both avatars. In
short, the zero-shot process is performed via three steps: (1) naive
composition after segmentation by introducing additional point clouds for the
missing region; (2) optimization by aligning facial landmarks for better
alignment; and (3) color blending for the added points for seamless outputs.

#### Obtaining the Additional Part from the Source Human.

We use the estimated segmentation masks of the face attribute $\chi_{\phi}$
and $\chi_{th}$ that can be controlled via latent code $\mathbf{z}$ to select
the _target human_ ’s point cloud except for the facial attribute
$\chi_{th}=0$ and _source human_ ’s point cloud that includes the facial
attribute $\chi_{\phi}=1$:

$\displaystyle
P_{\text{naive}}=\\{\mathbf{x}_{\phi}^{d,i}\\}_{\chi^{i}_{\phi}=1}\cup\\{\mathbf{x}_{th}^{d,i}\\}_{\chi^{i}_{th}=0}$
(15)

When we remove the facial attribute from the _target human_ and bring in the
facial attribute from the _source human_ , it creates an empty space between
the two point clouds. To fill this missing region, as shown in Fig. 14(a), we
bring in additional parts from the _source human_. Formally, this can be
represented as follows:

$\displaystyle P_{\text{naive w/
add}}=P_{\text{naive}}\cup\\{\mathbf{x}_{\phi}^{d,i}\\}_{\chi^{i}_{\phi,\text{add}}=1}$
(16)

To create the additional segmentation mask $\chi_{\phi,~{}\text{add}}^{i}$, we
borrow the knowledge from the FLAME [39] by leveraging $k$-nearest neighbor
$\mathcal{N}$. $\mathcal{N}_{k}(P_{1},P_{2})$ denotes the $k$-nearest
neighbors in $P_{2}$ for each point in $P_{1}$.
$\arg\min\mathcal{N}_{k}(P_{1},P_{2})$ represents the indices of the
$k$-nearest neighbors from points in $P_{1}$ to points in $P_{2}$ [47]. We
omit the subscript $k$ when $k=1$.

Note that $\\{\mathbf{x}_{\phi}^{d,i}\\}_{\chi^{i}_{\phi,\text{add}}=1}$
denotes the additional point clouds from the _source human_ to fill the gaps
between the _source human_ and _target human_ because of the exception of
_target human_ ’s attribute, as shown in the red box of Fig. 14(a). To create
$\chi_{\phi,\text{add}}^{i}$, We exclude the vertices from the FLAME vertices
$\mathbf{x}_{th}^{\text{FLAME}}$ that are not associated with the additional
part by using $\chi_{th}$ and the back of the head part of the FLAME that we
designate. We denote the mask cue for obtaining FLAME corresponding to the
additional part as $\chi_{th,\text{add}}^{\text{FLAME}}$.

$\mathbf{x}_{th,\text{add}}^{\text{FLAME}}=\\{\mathbf{x}_{th}^{\text{FLAME}}\\}_{\chi_{th,\text{add}}^{\text{FLAME}}}$
(17)

We apply $\mathcal{N}$ to $\mathbf{x}_{\phi}^{d}$ and
$\mathbf{x}_{th,\text{add}}^{\text{FLAME}}$ to obtain the nearest neighbor of
_source human_. To create the additional part only, we use $(1-\chi_{\phi})$
except for _source human_ ’s attribute.

$\chi_{\phi,\text{add}}=(1-\chi_{\phi})\circ{\arg\min}\mathcal{N}_{k}(\mathbf{x}_{th,\text{add}}^{\text{FLAME}},\mathbf{x}_{\phi}^{d})$
(18)

, where $\circ$ represents the Hadamard product. We use $k=2000$ to generate
the additional point clouds as described in Fig. 14(a).

#### Optimization Step.

After the naive composition, there is still a gap between the _source human_
’s face attribute and _target human_ ’s other parts because of the
misalignment of the subject-specific FLAME canonical space, as shown in Fig.
14(a). To solve this issue, we apply the optimization process to minimize the
distance between the _source human_ and _target human_. To obtain the landmark
points, we apply the $k$-nearest neighbor function between the landmarks of
deformed FLAME vertices [14] and $\mathbf{x}^{d}$ as follows:

$\mathbf{x}^{\text{landmarks}}=\mathcal{N}(\mathbf{x}^{\text{FLAME
landmarks}},\mathbf{x}^{d})$ (19)

We leverage the distance between the _source human_ ’s 3D landmark points and
the _target human_ ’s 3D landmark points as shown in Fig. 13.

$\mathbf{d}_{1}=\|\mathbf{x}_{th}^{\text{landmarks}}-\mathbf{x}_{\phi}^{\text{landmarks}}\|_{2}^{2}$
(20)

Furthermore, we calculate the squared distances between points in the
additional _source human_ part, denoted as
$\\{\mathbf{x}_{\phi}^{d}\\}_{\chi_{\phi,\text{add}}=1}$, and points in the
_target human_ , represented by $\\{\mathbf{x}_{th}^{d}\\}_{\chi_{th}=0}$,
from the $k$-nearest neighbors. For simplicity, the superscript $i$ is
omitted.

$\mathbf{d}_{2}=\|\mathcal{N}(\\{\mathbf{x}_{\phi}^{d}\\}_{\chi_{\phi,\text{add}}=1},\\{\mathbf{x}_{th}^{d}\\}_{\chi_{th}=0})\|_{2}^{2}$
(21)

We optimize the learnable angle-axis rotation vector $R\in\mathbb{R}^{3}$ and
translation vector $t\in\mathbb{R}^{3}$ to minimize the distance
$\mathbf{d}=\mathbf{d}_{1}+\mathbf{d}_{2}$ by Adam optimizer [34]. Note that
we apply the rotation and translation vector at the subject-specific FLAME-
canonical space.

$\mathbf{x}_{\phi,\text{moved}}^{fc}=R\cdot\\{\mathbf{x}_{\phi}^{fc}\\}+t$
(22)

We obtain the moved _source human_ ’s point cloud
$\mathbf{x}_{\phi,\text{moved}}^{d}$ from
$\mathbf{x}_{\phi,\text{moved}}^{fc}$ by Eq. (8). As a consequence, the
optimized point cloud is represented as follows:

$P_{\text{optim}}=\\{\mathbf{x}_{\phi,\text{moved}}^{d}\\}_{\chi_{\phi}\circ\chi_{\phi,\text{add}}=1}\cup\\{\mathbf{x}_{th}^{d}\\}_{\chi_{th}=0}$
(23)

The optimized rendering result is shown in Fig. 14(b).

#### Blending Step.

To generate a natural rendering of the additional part, denoted as
$\\{\mathbf{x}_{\phi}^{d}\\}_{\chi_{\phi,\text{add}}=1}$, we leverage the
feature information from the _target human_ using the $k$-nearest neighbor.

$\mathbf{c}_{\text{add}}^{d}={\arg\min}\mathcal{N}(\mathbf{x}_{th},\\{\mathbf{x}_{\phi}^{d}\\}_{\chi_{\phi,\text{add}}=1})\circ\mathbf{c}_{th}^{d}$
(24)
$\mathbf{n}_{\text{add}}^{d}={\arg\min}\mathcal{N}(\mathbf{x}_{th},\\{\mathbf{x}_{\phi}^{d}\\}_{\chi_{\phi,\text{add}}=1})\circ\mathbf{n}_{th}^{d}$
(25)

The RGB and normal of the additional part come from the target human, so we
obtain the naturally blended avatar through the zero-shot model. The natural
blended results are shown in Fig. 14(c).

##  Appendix C Loss Functions

###  C.1 Network Architecture

Figure 15: Network Architecture of PEGASUS. In PEGASUS, the latent code
$\mathbf{z}$ serves as a condition for all the MLPs.

#### Configuration.

The total loss for PEGASUS is defined as follows:

$\begin{split}\mathcal{L}&=\lambda_{\text{rgb}}\mathcal{L}_{\text{rgb}}+\lambda_{\text{mask}}\mathcal{L}_{\text{mask}}+\lambda_{\text{FLAME}}\mathcal{L}_{\text{FLAME}}+\lambda_{\text{vgg}}\mathcal{L}_{\text{vgg}}\\\
&+\lambda_{\text{normal}}\mathcal{L}_{\text{normal}}+\lambda_{\text{seg}}\mathcal{L}_{\text{seg}}+\lambda_{\mathbf{z}~{}\text{reg}}\mathcal{L}_{\mathbf{z}~{}\text{reg}}\end{split}$
(26)

We leverage the loss functions from the facial implicit representations from
monocular inputs [71, 70] as follows:

$\displaystyle\mathcal{L}_{\text{rgb}}=\|c-c^{\text{GT}}\|$ (27)
$\displaystyle\mathcal{L}_{\text{mask}}=\|M-M^{\text{GT}}\|$ (28)
$\displaystyle\mathcal{L}_{\text{vgg}}=\|F_{\text{vgg}}(c)-F_{\text{vgg}}(c^{\text{GT}})\|$
(29)
$\begin{split}\mathcal{L}_{\text{FLAME}}={\frac{1}{N}}&\sum_{i=1}^{N}(\lambda_{e}\|{\mathcal{E}}_{i}-{\hat{\mathcal{E}}}_{i}\|_{2}\\\
&+\lambda_{p}\|\mathcal{P}_{i}-{\hat{\mathcal{P}}}_{i}\|_{2}\\\
&+\lambda_{w}\|\mathcal{W}_{i}-{\hat{\mathcal{W}}}_{i}\|_{2})\end{split}$ (30)

Following PointAvatar, $c$ and $c^{GT}$ denote the color of the rendering
images from PEGASUS and ground-truth color. $M$ denotes the mask from PEGASUS
obtained by $\mathbf{m}_{\text{pix}}=\sum_{i}\alpha_{i}\mathbf{T}_{i}$.
$F_{\text{vgg}}(\cdot)$ represent the features of pretrained VGG network [27,
52]. $\mathcal{E},\mathcal{P},~{}\text{and}~{}\mathcal{W}$ are the pseudo
ground truth of the $k$-nearest neighbor vertices of the FLAME [39]. Note that
our method, PEGASUS, does not predict the shape blendshapes basis
$\mathcal{S}$, directly using the $k$-nearest neighbor vertices of the FLAME.

Given ground-truth object mask $M_{\text{seg}}^{GT}$ and the predicted
segmentation cues $\chi^{d}$, the rendered color of the segmented point cloud
represents $\mathcal{R}(\chi^{d}\circ\mathbf{x}^{d})$. The segmentation loss
is defined as:

$\mathcal{L}_{\text{seg}}=\text{BCE}(\mathcal{R}(\chi^{d}\circ\mathbf{x}^{d}),M_{\text{seg}}^{GT})$
(31)

BCE represent the Binary Cross-Entropy loss. $\mathcal{R}$ is the alpha
composition rendering function. We leverage the alpha composition function of
PyTorch3D [47] to render the predicted segmentation cues.

We adopt the normal loss to encourage high-fidelity geometry and texture as
follows:

$\mathcal{L}_{\text{normal}}=\|\mathbf{n}-\mathbf{n}^{d}\|$ (32)

We generate the pseudo ground truth normal $\mathbf{n}$ from the $V^{tp}$ and
the avatar trained with a single identity of each $V^{db}$. We apply the
regularization of latent code to be close to zero.

$\mathcal{L}_{\mathbf{z}~{}\text{reg}}=\|\mathbf{z}\|$ (33)

##  Appendix D Implementation Details

Figure 16: Sinlge Part-Swap Avatar on Hat. Our synthesis method creates high-
quality and properly wearing avatars.

In Fig. 15, we show the network architecture of PEGASUS. Following PointAvatar
[71], we leverage ReLU activation function [43] for shading MLP, and Softplus
activation function for canonical and deformation MLP for every layer. Sig
denotes the sigmoid function in Fig. 15. Different from PointAvatar, we use an
additional layer to output segmentation cues $\chi$ in canonical MLP. Also, we
use two layers of MLP to create subject-specific canonical offset
$\mathcal{O}^{gc\rightarrow sc}$

###  D.1 Training Strategy

We train PEGASUS in two stages. In the first stage, we only use the target
individual from $V^{tp}$ for training. In this way, the initial point cloud is
deformed from a sphere to have a reasonable face shape. In the second stage,
we continue training using all part-swapped videos from $\hat{V}_{i}^{tp}$. We
have empirically find that this two-stage training shows more reliable
training. In all of our experiments, we start the second stage from the 10th
epoch, using 1600 point clouds.

##  Appendix E More Results and Ablation Study

(a) 1-Stage

(b) 2-Stages

(c) 3-Stages

(d) GT

Figure 17: Ablation: Offsets $\mathcal{O}$. Our three-stage canonical space
framework creates reasonable and accurate reconstruction.

In Fig. 16, our synthesis method maintains the identity better than other
baselines and also shows the hat similar to the original while being
appropriately worn by the avatar. In Fig. 17 and Tab. 3, our multi-stage
canonical space and point deformation method outperforms the best metrics and
quality compared to other approaches. Notably, as an example of Fig. 17, the
closest high-quality reconstruction to the Ground Truth (GT) is achieved by
the three-stage approach.

Figure 18: Additional Results of Zero-Shot Transfer. PEGASUS robustly
transfers facial attributes to any target human without the need for
additional training. Figure 19: Zero-Shot Interpolation. With the help of
interpolation-capable segmentation cues by the segmentation network of
canonical MLP, we create interpolation in a zero-shot model.

#### Result on Zero-Shot Transfer.

Fig. 18 presents additional results of zero-shot transfer. PEGASUS robustly
and naturally transfers facial attributes to any target human in the wild.
Fig. 19 demonstrates facial attribute interpolation in zero-shot modeling,
aided by latent code $\mathbf{z}$ interpolation. This shows that segmentation
cues are capable of interpolation by the canonical MLP.