# Few-Shot Head Swapping in the Wild

Changyong Shu1 Hemao Wu2 Hang Zhou1 Jiaming Liu111footnotemark: 1 Zhibin Hong1
Changxing Ding2 Junyu Han1 Jingtuo Liu1 Errui Ding1 Jingdong Wang1
1Department of Computer Vision Technology (VIS), Baidu Inc., 2South China
University of Technology
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>{201830252427<EMAIL_ADDRESS>Corresponding authors.

###### Abstract

The head swapping task aims at flawlessly placing a source head onto a target
body, which is of great importance to various entertainment scenarios. While
face swapping has drawn much attention, the task of head swapping has rarely
been explored, particularly under the few-shot setting. It is inherently
challenging due to its unique needs in head modeling and background blending.
In this paper, we present the Head Swapper (HeSer), which achieves few-shot
head swapping in the wild through two delicately designed modules. Firstly, a
Head2Head Aligner is devised to holistically migrate pose and expression
information from the target to the source head by examining multi-scale
information. Secondly, to tackle the challenges of skin color variations and
head-background mismatches in the swapping procedure, a Head2Scene Blender is
introduced to simultaneously modify facial skin color and fill mismatched gaps
in the background around the head. Particularly, seamless blending is achieved
with the help of a Semantic-Guided Color Reference Creation procedure and a
Blending UNet. Extensive experiments demonstrate that the proposed method
produces superior head swapping results in a variety of scenes111Demo videos
and code are available at https://jmliu88.github.io/HeSer..

##  1 Introduction

Human cognition of identity appearance is profoundly affected by not only
facial structures but also head shapes and hairstyles. Head swapping, the
ability to seamlessly replace the head in a target image with a source one (as
shown in Fig. LABEL:fig:fig1) would be of great importance to a variety of
scenarios such as movie and advertisement composition, virtual humans
creation, and deepfake video detection, etc.

While face swapping has long been a topic of interest [4, 21, 25, 27, 52],
only a few studies have been carried out on the task of head swapping.
DeepFaceLab [27] requires large manual intervention to generate head swapping
results, and StylePoseGAN [29] tends to change the color of body skin and
background in the target image in an undesired manner. Moreover, both methods
fail to address few-shot head swapping, particularly for in-the-wild scenes.

We identify several properties that make few-shot head swapping more
challenging than face swapping: 1) Head swapping requires not only perfect
facial identity and expression modeling, but also capturing the structural
information of a whole head and the non-rigid hair. Thus previous identity
extraction strategies for face swapping [25, 4] cannot be directly applied to
head swapping. 2) There would be a huge _region mismatch between swapped head
edges and backgrounds_ caused by the editing of head shapes and hairstyles.
Such a problem does not exist in the face swapping setting. 3) Moreover,
similar to face swapping, the color difference between source and target skins
needs to be handled carefully.

In this paper, we propose a framework called Head Swapper (HeSer), which
generates high fidelity head swapping results in the wild based on a few
frames. Our key insight is to _positionally and emotionally align the source
head with the target in a unified blender that seamlessly handles both color
and background mismatches._ Two modules, namely the _Head2Head Aligner_ and
the _Head2Scene Blender_ are devised. The Head2Head Aligner is responsible for
finding a latent representation of a whole head as well as the facial details.
It aligns the source head to the same pose and expression as the target image
in a holistic manner. The identity, expression, and pose information are
prominently balanced in a style-based generator by encoding multi-scale local
and global information from both images. Moreover, a subject-specific fine-
tuning procedure could further improve identity preservation and pose
consistency.

To further blend the aligned head into the target scene, we devise a module
named Head2Scene Blender, which provides both 1) color guidance for facial
skins and 2) padding priors for inpainting the gaps on the background around
the head. Thus both the skin color and edge-backgrounds mismatches between
source and target can be handled within one unified sub-module. It efficiently
creates colored references by building the correlations between pixels of the
same semantic regions. Then with a blending UNet, seamless and realistic head
swapping results can be rendered.

We summarize our main contributions as follows: 1) We introduce a Head2Head
Aligner that holistically migrates position and expression information from
the target to the source head by examining multi-scale information. 2) We
design a Head2Scene Blender to simultaneously handle facial skin color and
background texture mismatches. 3) Our proposed Head Swapper (HeSer) produces
photo-realistic head swapping results on different scenes. To the best of our
knowledge, this is one of the earliest methods to achieve few-shot head
swapping in the wild.

##  2 Related Work

The proposed two-stage Head Swapper is closely related to the topics of face
swapping, head reenactment and image blending, which will be discussed below.

Face and Head Swapping. Many methods have been proposed for face swapping [26,
25, 2, 22, 4, 52, 40, 39]. While few studies tackle the task in a source-
oriented manner [26, 25] that aims at reenacting and blending the source into
the target, most methods are performed in a target-oriented manner [2, 22, 4,
40, 52]. They extract identity representation from the source image and inject
it into the target image in a generative model. Face swapping approaches
produce immutable hairstyles and head shapes, which limits the overall
similarity between the generated results and the source.

The task of head swapping has been rarely studied. Deepfacelab [27] is the
first work to tackle head swapping. However, it suffers from the following
issues: 1) huge amounts of source data are required; 2) color transfer can
only be poorly performed; and 3) the regions that require inpainting the
fusion look unnatural. Later StylePoseGAN [29], which is specially designed
for re-rendering using pose-/appearance-conditioned StyleGAN [34], potentially
has the capability of head swapping. However, it tends to perform poorly for
in-the-wild scenarios. Our proposed head swapping pipeline is the first work
committed to achieving few-shot head swapping in the wild.

Few-Shot Head Reenactment. Head swapping requires placing the source head at
the target head’s position with the same pose and expression, which is similar
to the task of talking head generation and head reenactment [19, 35, 8, 31,
32, 7, 5, 44, 47, 49, 33, 16, 24, 23]. Here we will briefly discuss the
visual-driven few-shot reenactment methods.

Recent few-shot methods can be roughly divided into warping-based and
reconstruction-based methods. Warping-based approaches[31, 32, 33] deform the
source images to imitate the motion of driving ones, they tend to work poorly
for the case with a big range of motion. Reconstruction-based approaches
mostly leverage generative adversarial networks (GANs) [9]. A great number of
studies use intermediate representations such as landmarks [42, 5] and 3D
models [47]. However, the inaccuracy of the structural information might lead
to error accumulation. Certain recent studies focuses on the extraction and
disentanglement of pose and identity information in the latent space [48, 7,
49, 23]. Particularly, LPD [7] verifies that a style-based generator can
explicitly handles such information in latent vectors. Inspired by this type
of work, we introduce the Head2Head Aligner to find the latent representation
of the whole head as well as the facial details in a reconstruction-based
learning pipeline.

Figure 1: The overall architecture of our proposed pipeline for testing. First
stage: the Head2Head Aligner $G_{A}$ (detailed in Fig. 2) generates the
animated portrait ${I_{A}}$, depicts the ${K}$-shot source image
${I_{{ID}_{k}}}(k=1,...,K)$ at the same pose and expression of the target
image ${I_{T}}$. Second stage: we concatenate grayscale head $I_{GH}$, head
mask ${M_{H}}$, inpainting mask $M_{I}$, reference source $I_{T}$ and
background $I_{BG}$ to go through the Head2Scene blender $G_{B}$ (detailed in
Fig. 4) to generate the blending output $I_{B}$ (i.e., the head swapping
result) in photo-realistic quality.

Color Transfer. As we need to transfer the skin color of the source face to
the target one, it is similar to the task of makeup transfer. Early makeup
transfer methods utilize facial landmarks or parsing results as prior
information [3, 14, 15, 10, 36], however, they cannot keep the facial
expressions unchanged. The subsequently PSGAN [17, 30] and SCGAN [11] solve
the pose misalignment problem above. However, they tend to change the
background color after the color transfer, making them unsuitable for head
swapping where the background of the target must not be changed.

##  3 Method

In this work, we introduce the Head Swapper (HeSer), which is illustrated in
Fig. 1. Let $I_{S}=\\{I_{S_{k}}|k\in[1,K]\\}$ denote the set of source images,
and let ${I_{T}}$ be the image of the target person. We aim to create a new
blending output (i.e., head swapped result) ${I_{B}}$, with the head of $S$ on
the body and background of ${I_{T}}$. Notably, the pose, expression, and skin
color of ${I_{B}}$ should remain consistent with ${I_{T}}$, but the identity,
head structure and hairstyle should be the same as $S$.

In this section, we first design a Head2Head Aligner (Sec. 3.1) to produce an
_animated portrait_ image $I_{A}$ as an intermediate representation. Then
$I_{B}$ is derived by blending $I_{A}$ with $I_{T}$ in the Head2Scene Blender
(Sec. 3.2).

Figure 2:  Overview of the proposed Head2Head Aligner for training. The
average portrait vector ${{\it{\bar{f}_{por}}}}$, average identity vector
${{\it{\bar{f}}_{id}}}$, pose vector ${{\it{{f}}_{pose}}}$ and expression
vector ${{\it{{f}}_{exp}}}$ are obtained from the portrait encoder, identity
encoder, pose encoder and expression encoder respectively. Then the vectors
above are sent into the animated portrait generation.

###  3.1 Head2Head Aligner

Instead of directly building $I_{B}$, our first step is to produce an image
which is positionally and emotionally aligned with $I_{T}$, but with the same
identity as $S$. We name this intermediate _animated portrait_ image $I_{A}$.
We identify that all desired information required for $I_{A}$ can be extracted
from multi-scale perspectives on both the source and target, including: 1) the
global head and hair structure information of $I_{S}$; 2) the identity details
on the faces of $I_{S}$; 3) the global pose of $I_{T}$; and 4) the detailed
expressions of $I_{T}$.

To this end, we design the Head2Head Aligner as illustrated in Fig. 2. The
Head2Head Aligner consists of a portrait encoder ${E_{por}}$, an identity
encoder ${E_{id}}$, a pose encoder ${E_{pose}}$, an expression encoder
${\it{E}_{exp}}$, and a generator $G$.

It is worth noting that during training, the source image and target image are
obtained from the same video of a particular person (illustrated in Fig. 2),
and they are derived from videos of different people during testing. Thus the
information disentanglement should be taken into consideration. In the
training stage, we randomly select $K+1$ frames from the same video sequence
of an individual person, where $K$ frames are set as the $K$-shot source
images $I_{S}$, while the remaining frame is set as the target image $I_{T}$.

Source Encoding. Both the coarse and fine scales are leveraged for information
encoding from the source and target. As for the source image, global portrait
information including the head and hair, is directly extracted by an encoder
${E_{por}}$ to a ${d_{por}}$-dimensional vector
${f_{por}}_{k}=E_{por}(I_{{S}_{\scriptsize k}})$.

On the other hand, the identity encoder is adopted from a pretrained state-of-
the-art face recognition model [6], which can provide more representative
identity embedding [21, 4, 49]. More specifically, for each source image
$I_{{S}_{\scriptsize k}}$, it first undergoes a central cropping
transformation $C$, after which $C(I_{{S}_{\scriptsize k}})$ is passed through
the identity encoder $E_{id}$ to generate a $d_{id}$-dimensional identity
embedding ${f_{id}}_{k}={E_{id}(C(I_{{S}_{\scriptsize k}}))}$.

For randomly fetched $K$ frames of source images ${{I_{{S}_{\scriptsize
1}}},{I_{{S}_{\scriptsize 2}}},...,{I_{{S}_{\scriptsize K}}}}$, a total of $K$
portrait vectors ${f_{por}}_{k}$ and $K$ identity embeddings ${{f_{id}}_{k}}$
are produced; we acquire ${{\it{\bar{f}_{por}}}}$ and ${{\it{\bar{f}}_{id}}}$
by taking the average of $\\{{{f_{por}}_{1}},...,{{f_{por}}_{K}}\\}$ and
$\\{{{f_{id}}_{1}},...,{{f_{id}}_{K}}\\}$ respectively.

Target Encoding. To obtain both facial expression details and the global head
pose, as shown at the bottom-left of Fig. 2, we apply the pose encoder
${\it{E}_{pose}}$ and the expression encoder ${\it{E}_{exp}}$ to extract the
pose and detailed facial expression information respectively from the target
image ${I}_{T}$. It first undergoes a series of augmentations [7] $\it{A}$
before being sent into the pose encoder ${\it{E}_{pose}}$. An identity-
agnostic $\it{d}_{pose}$-dimensional pose descriptor
${f_{pose}}={{\it{E}_{pose}}(A({I}_{T}))}$ is thus encoded. Then the augmented
image is also processed with central cropping $\it{C}$ mentioned above. The
result ${C(A({I}_{T}))}$ is fed into the expression encoder $E_{exp}$ to
produce a ${d_{exp}}$-dimensional expression vector $f_{exp}$.

Animated Portrait Generation. The feature descriptors containing information
regarding the body, identity, pose and expression are then composed together
to produce the _animated portrait_. Specifically, the features
${{\it{\bar{f}_{por}}}}$, ${{\it{\bar{f}}_{id}}}$, ${{\it{{f}}_{pose}}}$ and
${{\it{{f}}_{exp}}}$ are concatenated and processed by a generator via
AdaIN[13] to produce the desired output $I_{A}$:

$\displaystyle
I_{A}=G({{\it{\bar{f}_{por}}}},{{\it{\bar{f}_{id}}}},f_{pose},f_{exp})$ (1)

The loss functions used are basically the reconstruction loss, perceptual
loss, adversarial loss and identity loss, which are the same as [7]. Please
refer to supplementary material for more details.

Figure 3: Illustration of data preprocessing for head2scene blender. Figure 4:
Illustration of our proposed Head2Scene Blender in the testing setting. The
head-color reference $I_{T\rightarrow A}^{HR}$ and inpainting reference
$I_{T\rightarrow A}^{IR}$ are first obtained using the Reference Creation sub-
module. Then $I_{T\rightarrow A}^{HR},I_{T\rightarrow
A}^{IR},M_{A}^{H},I_{T}^{BG},M_{A}^{I}$, and $I_{A}^{GH}$ are fed into the
blending UNet to produce the final result ${I_{B}}$. In the training phase,
the same image is used for $I_{A}$, $I_{T}$ and $I_{B}$.

###  3.2 Head2Scene Blender

With the _animated portrait_ image $I_{A}$ above, our second step is to
produce the blending output $I_{B}$. It should be a seamless combination of
the head from $I_{A}$ and the background and body in $I_{T}$. The challenges
are 1) the head shape on $I_{A}$ could be significantly different from that of
$I_{T}$, thus the blending procedure requires amending the “outliners”, i.e.,
masking out the areas of the original head and fill in the gaps led by head
shape mismatches; and 2) the color of the $I_{B}$ should be consistent with
$I_{T}$.

To this end, we design the Head2Scene Blender, which simultaneously solves
both challenges as illustrated in Fig. 4. The key is to build “ _color
references_ ” which serve as guidance for both skin color transfer and
background gap inpainting with a _Semantic-Guided Color Reference Creation_
sub-module. Then the final results are generated through a _Blending UNet_.
Below we will first introduce the necessary _Data preprocessing_ steps for
producing the _color references_ and then introduces the sub-modules
respectively.

Data preprocessing. The data preprocessing procedure prepares the materials
for performing the color transfer and inpainting, which are binary masks that
identifies the areas in need and the cropped images. Particularly, instead of
directly modifying the animated head in $I_{A}$, we argue that it would be
easier to reconfigure the color transfer problem into the problem of re-
coloring its grey-scale version of the head $I_{A}^{GH}$ only.

Detailedly, given the _animated portrait_ $I_{A}$, we employ a parsing tool to
obtain the segmentation masks $M_{A}$, which indicates the semantic areas of
the portrait image. Then the head mask $M_{A}^{H}$ of the target image is
obtained by combining the masks belonging to the head area. Similarly, the
segmentation masks $M_{T}$ and head mask $M_{T}^{H}$ of the target image
$I_{T}$ are obtained in the same way. The greyscale head can be computed as
$I_{A}^{GH}=Gray(I_{A}*M_{A}^{H})$.

Furthermore, the head mask $M_{T}^{H}$ is dilated to an enlarged version
$\hat{M}_{T}^{H}$, then the _target inpainting mask_
$M_{T}^{I}=\hat{M}_{T}^{H}-M_{T}^{H}$ which will be further used for creating
the _color reference_ for inpainting. Different from the target inpainting
mask above, the _animated inpainting mask_ $M_{A}^{I}$ is obtained via both
$M_{A}^{H}$ and $M_{T}^{H}$. More especially, the union set of ${M_{A}^{H}}$
and ${M_{T}^{H}}$ is dilated to an enlarged mask ${\it{\hat{M}}_{A}^{H}}$. Let
the _animated inpainting mask_ $M_{A}^{I}={\it{\hat{M}}_{A}^{H}}-M_{A}^{H}$
denotes the area that needs inpainting when blending $I_{A}$ with $I_{T}$.
$I^{BG}_{T}=I_{T}*(1.0-\it{\hat{M}}_{A}^{H})$ is the background image without
the target head.

Semantic-Guided Color Reference Creation. After the data processing steps, we
need to re-color the greyscale head $I_{A}^{GH}$ and fill the _animated
inpainting mask_ $M_{A}^{I}$. As stated before, our intuition is to provide a
color reference for both of them, namely the _head-color reference_
$I_{T\rightarrow A}^{HR}$ and the _inpainting reference_ $I_{T\rightarrow
A}^{IR}$, respectively.

Inspired by previous work [45, 43, 17], we identify that each _color
reference_ should spatially match the target region and provide pixel-wise
color guidance. As the desired color should be consistent with the target
image $I_{T}$, the color of the _references_ could be derived by querying the
coherent color from $I_{T}$ through correlation learning. This is achieved in
the Reference Creation sub-module $RC$.

Due to the lack of paired data, the Reference Creation sub-module is still
trained in a self-driven manner with $I_{A}$ and $I_{T}$ sourcing from the
same image. As shown in the left-hand diagram of Fig. 4, in the training
phase, the _animated portrait_ $I_{A}$ and target image $I_{T}$ first undergo
a random color augmentation $C^{\prime}$ and a random horizontal flip
augmentation $F$ respectively, which prevents the networks from directly
leveraging the pixels from the same position. The augmented _animated
portrait_ $C^{\prime}(I_{A})$ and the random flipped _target image_
${F(I_{T})}$ are then both sent into a feature pyramid network ${FPN}$ [45] to
produce the following semantic representation:

$\displaystyle{f}_{A}$ $\displaystyle=FPN(C^{\prime}(I_{A})).$ (2)
$\displaystyle{f}_{T}$ $\displaystyle=FPN(F(I_{T})).$ (3)

The next step to colorization is to compute the correlations between extracted
features on each spatial location, and resample pixel colors from the target
image to the _color references_. Previous approaches [45, 43, 17] produce a
prohibitively large memory footprint when estimating the correspondence due to
the fact that the pairwise similarities are computed among all locations of
the feature maps, despite their semantic independence. As the pixels in the
image pairs with different semantic labels make little contribution to the
correspondence matrix, we only compute the correlations among the same
semantic regions individually. Our practice not only alleviates memory
consumption but also avoids mismatched correlations.

Figure 5: The illustration of our Semantic-Guided Color Reference Creation
module, best viewed in color.

Concretely, as shown in Fig. 5, for each semantic region $r$ ${\in}$
$\\{face,hair,eye,nose,lip,tooth,inpainting\\}$, we compute a correlation
matrix $\it{\Gamma}^{r}\in\rm{R}^{\it{N}_{A}^{r}\times\it{N}_{T}^{r}}$, of
which each element $\it{\Gamma^{r}}(u,v)$ is a pairwise feature correlation
calculated as:

$\displaystyle\it{\Gamma^{r}}(u,v)=\frac{\bm{\bar{f}}^{r}_{A}(u)^{T}\bar{\bm{f}}^{r}_{T}(v)}{\|{\bar{\bm{f}}^{r}_{A}(u)}\|\
\|{\bar{\bm{f}}^{r}_{T}(v)}\|},u\in M_{A}^{r},v\in M_{T}^{r},$ (4)

where $\bar{{f}}_{A}^{r}(u)$ and $\bar{{f}}_{T}^{r}(v)\in R^{C}$ represent the
channel-wise centralized feature of $\it{f}_{A}^{r}$ and $\it{f}_{T}^{r}$ in
positions $u$ and $v$ respectively, _i.e_.,
$\bar{f}_{A}^{r}(u)=\it{f}_{A}^{r}(u)-\text{mean}(f_{A}^{r}(u))$ and
$\bar{f}_{T}^{r}(v)=f_{T}^{r}(v)-\text{mean}(f_{T}^{r}(v))$, and the
dimensions $\it{N}_{A}^{r}$ and $\it{N}_{T}^{r}$ are the pixel number of the
semantic region $\it{r}$ in the animated portrait image and target image
respectively. Compared to the original correlation maps in prior works [45,
17], our approach reduces the computational complexity from $\it{O}(wh*wh)$ to
$\it{O}(\it{N}_{S}^{r}*\it{N}_{R}^{r})$, where $w$ and $h$ denote the spatial
size of the original feature map.

As $\it{\Gamma^{r}}(u,v)$ indicates the similarity between $M_{A}(u)$ and
$M_{T}(v)$, we normalize the $\it{\Gamma^{r}}(u,v)$ via softmax and compute
their weighted contribution of these variables from the target image to the
head and inpainting regions of the animated portrait image, as follows:

$I_{T\rightarrow A}^{r}(u)=\sum_{v\in
M_{T}^{r}}{softmax}_{v}(\Gamma^{r}(u,v)/{\tau})\cdot I_{T}(v),u\in M_{A}^{r}.$
(5)

where $\tau$ is the temperature coefficient. Finally, as shown in Fig. 5, the
created inpainting reference $I_{T\rightarrow A}^{IR}$ is derived from $I_{T}$
and $M_{T}^{I}$, while the warped head-color reference $I_{T\rightarrow
A}^{HR}$ is obtained from $I_{T}$ and the combination of $M_{T}^{r}$, where
$r$ ${\in}$ $\\{face,hair,eye,nose,lip,tooth\\}$. We denote the overall
estimation procedure as:

$\displaystyle I_{T\rightarrow A}^{HR}$
$\displaystyle=RC(f_{A},M_{A},f_{T},M_{T},I_{T}*M_{T}^{H}).$ (6)
$\displaystyle I_{T\rightarrow A}^{IR}$
$\displaystyle=RC(f_{A},M_{A}^{I},f_{T},M_{T}^{I},I_{T}*M_{T}^{I}).$ (7)

Blending UNet. After acquiring the created head-color/inpainting reference
$I_{T\rightarrow A}^{HR}$/$I_{T\rightarrow A}^{IR}$, the head mask
${M_{A}^{H}}$, background ${I_{T}^{BG}}$, inpainting mask ${M_{A}^{I}}$ and
gray-scale head ${I_{A}^{GH}}$ produced by the data processing, we concatenate
them all channel-wisely, and pass them to a Blending UNet. Our goal is that:
$1)$ the head-color distilled from the target image is transferred to the
gray-scale head ${I_{A}^{GH}}$, where the identity is retained while the color
is kept consistent with the remaining skin (such as neck) exposed in the rest
of the reference body; $2)$ the missing region masked by the inpainting mask
can be estimated. Consequently, the overall process of the Blending UNet $B$
can be formulated as follows:

$I_{B}=B(I_{T\rightarrow A}^{HR},I_{T\rightarrow
A}^{IR},M_{A}^{H},I_{T}^{BG},M_{A}^{I},I_{A}^{GH}).$ (8)

Cycle Loss: The training relies on the reconstruction loss, perceptual loss,
adversarial loss. Additionally, in order to guarantee that the warped
headcolor/inpainting exemplars could learn a meaningful correspondence matrix,
we introduce the cycle consistent loss.

$L_{c}=\lambda_{c}\left\|{I_{T\rightarrow A\rightarrow T}-I_{T}}\right\|_{1},$
(9)

where $I_{T\rightarrow A\rightarrow T}$ is the color reference after cycled,
and $I_{T\rightarrow A\rightarrow T}^{k}(u)=\sum_{v\in
M_{A}^{k}}{softmax}_{v}(\Gamma^{k}(u,v)/{\tau})\cdot I_{T\rightarrow
A}(v),u\in M_{T}^{k}$. Besides, the additional target image $I_{T}^{\prime}$
coming from different image compared to $I_{A}$ is also utilized to ensure the
meaningful of warped exemplar:

$L_{c^{\prime}}=\lambda_{c}\left\|{I_{T^{\prime}\rightarrow A\rightarrow
T^{\prime}}-I_{T}}\right\|_{1}.$ (10)

Please refer to the supplementary materials for more details.

##  4 Experiments

Detailed descriptions of the data collection, along with more experimental
details, are elaborated in the supplementary material.

###  4.1 Headswap Results

We first evaluate our head swapping result. We compare our proposed HeSer with
the state-of-the-art face swapping model [4] and Deepfacelab [27] as follows.

Qualitative comparison. The results in Fig.
LABEL:fig:compare_of_headswap_and_faceswap_and_deepfacelab demonstrate that
our method can outperform other methods in multiple aspects, e.g., identity
preservation, pose and expression consistency, skin color alignment, head-
background coherence, and fidelity. The source image for face swapping is one
of the 32-shot images, while the source images for Deepfacelab and our head
swapping model are the same 32 frames. The face swapping method can swap the
facial elements from source image to the target image, and the mustache can be
slightly transferred; however, the immutable face shape and hairstyle restrict
the identity similarity in human cognition. Deepfacelab tends to animate
poorly when there is no source image with a similar pose to target image;
furthermore, it is incapable of inpainting the missing pixel and the
performance of head-color alignment is significantly inferior to ours. For
more vividly head swapping results generated by our method please refer to the
supplementary video.

Quantitative Comparison. We conduct a user study for quantitative comparison
of our HeSer with the face swapping method [4] and Deepfacelab [27]. We ask
the users to rank 1) how well the ID information is preserved in each method.
2) The emotion and pose similarity with the target image, we denote the
results as Exp. 3) The consistency of the Skin Color between generated
examples and the target torso. 4) The Inpainting smoothness between generated
head and the background. 5) The Holistic quality of the generated frames.

We randomly select 25 groups of source images and 10 target images from the
VoxCeleb2 test set, and totally 250 results are obtained for each method. Then
20 different workers, including 15 normal users and 5 experts, are asked to
evaluate randomly selected 25 results for every dimension. The workers rank
the methods according to the question, and the rankings are scaled into scores
of zero to five, where worst is zero and best is five. Table 1 shows the
average score result from the user study. Our HeSer outperforms other methods
by a large margin, except for the expression consistency and skin color
alignment. Face swapping [4] directly injects the identity information into
the target image thus naturally keeping better facial structures and skin
colors.

Table 1: Average score from the user study, rating from 0 to 5.

Method |  ID $\uparrow$ |  Exp $\uparrow$ |  Skin Color $\uparrow$ |  Inpainting $\uparrow$ |  Holistic $\uparrow$
---|---|---|---|---|---
FaceSwap [4] | 1.22 | 3.02 | 3.36 | – | 1.42
Deepfacelab [27] | 2.16 | 1.47 | 1.53 | 0.05 | 2.26
Ours | 4.12 | 3.01 | 2.61 | 4.95 | 3.82

###  4.2 Superiority of the Head2Head Aligner

In this subsection, we evaluate Head2Head Aligner independently. As the
Head2Head Aligner is devised to animate the source image with the pose and
expression of the target image, which is similar to the task of reenactment,
different reenactment strategies are compared. The performance is discussed
from two perspectives: K-shot inputs, and fine-tuning or not. As some of our
2D-based animated competitors (FOMM [32] and Siarohin et al. [33]) do not
support few-shot inputs and fine-tuning, we split our evaluations into 1) a
one-shot setting, where all competitors are compared; 2) otherwise, only LPD
is compared.

Table 2: Quantitative comparison on face reenactment.

Method |  $E_{ID}$ $\downarrow$ |  $E_{P}$ $\downarrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$
---|---|---|---|---|---
FOMM [32] | 0.49 | 0.275 | 0.76 | 0.18 | 30.92
LPD [7] | 0.71 | 0.063 | 0.52 | 0.50 | 28.84
Siarohin et al [33] | 0.71 | 0.137 | 0.73 | 0.20 | 30.01
Ours | 0.53 | 0.026 | 0.77 | 0.19 | 31.33
LPD (32-shot + ft) | 0.23 | 0.024 | 0.62 | 0.36 | 29.46
Ours (32-shot + ft) | 0.22 | 0.024 | 0.89 | 0.12 | 33.26

\begin{overpic}[width=390.25534pt]{fig/reenact-
compare-3rd/4_plot_fig_32_shot_iterations-cross-id-00600step-updated.jpg}
\put(36.0,81.0){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize 32-shot, ft600,
cross-id}}} \put(-3.5,65.0){\rotatebox{90.0}{\color[rgb]{0,0,0}{\scriptsize
Source}}} \put(-3.5,47.0){\rotatebox{90.0}{\color[rgb]{0,0,0}{\scriptsize
Target}}} \put(-3.5,25.0){\rotatebox{90.0}{\color[rgb]{0,0,0}{\scriptsize LPD
\cite[cite]{[\@@bibref{}{Burkov2020lpd}{}{}]}}}}
\put(-3.5,7.0){\rotatebox{90.0}{\color[rgb]{0,0,0}{\scriptsize Our}}}
\end{overpic} Figure 7: Qualitative cross-id animated results of 32-shot fine-
tuned model with 600 iterations. Source, target, and our animated result are
re-aligned following the instruction of LPD.
\begin{overpic}[width=447.16759pt]{fig/blending_examplar_3.jpg}
\put(5.0,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Target}}}
\put(15.0,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Animated
portrait}}} \put(30.0,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize
Inpainting mask}}}
\put(46.5,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Reference}}}
\put(56.0,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Inpainting
reference}}} \put(71.0,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize
Head-color reference}}}
\put(87.2,43.5){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Blending
result}}} \end{overpic} Figure 8: Illustration of head-color/inpainting
reference in our Head2Scene Blender. Best viewed in color.

Evaluation Metrics. Five metrics are included in terms of head reenactment
evaluation: (1) Identity error ($E_{ID}$) using the cosine similarity between
face embeddings. (2) Pose reconstruction error ($E_{P}$), measuring the pose
error between the synthesized and ground truth images as performed in [7, 41].
(3) Pixel-wise Reconstruction fidelity using PSNR. (4) Semantic perceptual
similarity via the AlexNet-based LPIPS metric [46]. (5) The perceived quality
via the structural similarity index measure (SSIM) [37].

One-shot setting Evaluation. We first compare the performance of our animated
method with four competitors via five evaluation metrics under the one-shot
face reenactment setting. The quantitative results are shown in Table. 2. Our
method substantially outperforms all other animated models on every metric,
except for the identity error and LPIPS, which is slightly lower than FOMM.
For more qualitative comparison results please refer to supplementary
material.

Impact of K-shot and fine-tuning. We further analyze the effect of increasing
the K-shot number with subject-specific fine-tuning or a meta-learned model,
which is the setting illustrated in Fig. LABEL:fig:fig1. Specifically, we set
the model that takes 32 frames as input and leverage 600 iterations of
finetuning as the standard-setting. The quantitative comparisons are
illustrated in Table 2. More quantitative evaluations on the influences of
shot number and finetuning iterations are plotted in the supplementary
material. It shows that our method almost outperforms other methods on all
metrics with various settings. The performances of all metrics tend to improve
as the k-shot number increases and fine-tuning is added respectively, while
the identity error and pose reconstruction error of our animated method
exhibit prominent improvement relative to LPD, demonstrating that our proposed
scheme is capable of generating animated portraits with smaller identity gap
and higher pose consistency.

It is further noteworthy that our animated scheme outperforms LPD with only
about one-fifth amounts of the training data in LPD. Though the identity error
of our method and LPD is almost equal when the number of tuning steps is
sufficiently large (such as 600), the facial expressions and emotions of our
animated portraits are significantly better than those of LPD, as illustrated
in Fig. 7. Please refer to supplementary material for more detailed
qualitative results.

###  4.3 Effectiveness of Head2Scene Blender

The representative blending results are shown in the last column of Fig. 8.
Our Head2Scene Blender can produce head swapping portraits of photo-realistic
quality. The head-color reference (sixth column in Fig. 8) is created from the
target image, while the head structure is consistent with the animated head.
Similar behavior is observed in the inpainting reference (fifth column in Fig.
8), where the estimated region style is faithful to the reference background.
It is noteworthy that the hat region of the inpainting reference in the second
row is constructed from the source background, which further demonstrates the
inpainting effectiveness of our blending module.

##  5 Conclusion and Discussion

Conclusion. In this paper, we propose the Head Swapper (Heser), which achieves
few-shot head swapping for in-the-wild scenarios. Specifically, our Head2Head
Aligner generates high-fidelity reenact results with high pose and expression
consistency, and the Head2Scene Blender seamlessly blends the aligned source
head to the target image while maintaining the color of the target person.
Extensive experiments demonstrate that the HeSer can achieve superior head
swapping results on a variety of scenes.

Broader Impact. Vivid video synthesis technologies create possibilities for
immoral behaviors. Recent generative models have greatly influenced identity
safe, image authenticity, etc. We will share the results of HeSer to the
face/head forgery detection community for the healthy development of the AI
technology.

Acknowledgement. This work is supported by the CCF-Baidu Open Fund.

## References

  * [1] Brock Andrew, Donahue Jeff, and Simonyan Karen.  Large scale GAN training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [2] Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, and Gang Hua.  Towards open-set identity preserving face synthesis.  In IEEE Conf. Comput. Vis. Pattern Recog., 2018.
  * [3] Huiwen Chang, Jingwan Lu, Fisher Yu, and Adam Finkelstein.  Pairedcyclegan: Asymmetric style transfer for applying and removing makeup.  cvpr, 2018.
  * [4] Renwang Chen, Xuanhong Chen, and Yanhao. Ge.  Simswap: An efficient framework for high fidelity face swapping.  In ACM Int. Conf. Multimedia, 2020.
  * [5] Zhuo Chen, Chaoyue Wang, Bo Yuan, and Dacheng Tao.  Puppeteergan: Arbitrary portrait animation with semantic-aware appearance transformation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [6] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos. Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [7] Burkov Egor, Pasechnik Igor, Grigorev Artur, and Lempitsky Victor.  Neural head reenactment with latent pose descriptors.  In IEEE Conf. Comput. Vis. Pattern Recog., 2020.
  * [8] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nießner.  Dynamic neural radiance fields for monocular 4d facial avatar reconstruction.  arXiv preprint arXiv:2012.03065, 2020.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [10] Qiao Gu, Guanzhi Wang, Tik Chiu Mang, YuWing Tai, and ChiKeung Tang.  Ladn: Local adversarial disentangling network for facial makeup and de-makeup.  iccv, 2019.
  * [11] Han, Chu Deng, Hongmin Han, Guoqiang cai, Shengfeng Han, and He.  Spatially-invariant style-codes controlled makeup transfer.  In CVPR, 2021.
  * [12] Zhang Han, Goodfellow Ian, Metaxas Dimitris, and Odena Augustus.  Self-attention generative adversarial networks.  arXiv preprint arXiv:1805.08318, 2018.
  * [13] Xun Huang and Serge. Belongie.  Arbitrary style transfer in real-time with adaptive instance normalization.  In IEEE Conf. Conference of the International Speech Communication Association., 2017.
  * [14] Huiwen, Jingwan Chang, Fisher Lu, Adam Yu, and Finkelstein.  Pairedcyclegan: Asymmetric style transfer for applying and removing makeup.  In CVPR, 2018.
  * [15] Hungjen, Kaming Chen, Szuyu Hui, Liwu Wang, Honghan Tsao, Wenhuang Shuai, and Cheng.  Beautyglow: On-demand makeup transfer framework with reversible generative network.  In CVPR, 2019.
  * [16] Xinya Ji, Hang Zhou, Kaisiyuan Wang, Wayne Wu, Chan Change Loy, Xun Cao, and Feng Xu.  Audio-driven emotional video portraits.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [17] Wentao Jiang, Si Liu, Chen Gao, Jie Cao, Ran He, Jiashi Feng, and Shuicheng Yan.  X2face: A network for controlling face generation by using images, audio, and pose codes.  cvpr, 2020.
  * [18] Son Chung Joon, Nagrani Arsha, and Zisserman. Andrew.  Voxceleb2: Deepspeaker recognition.  In IEEE Conf. Conference of the International Speech Communication Association., 2018.
  * [19] Hyeongwoo Kim, Pablo Garrido, Ayush Tewari, Weipeng Xu, Justus Thies, Matthias Niessner, Patrick Pérez, Christian Richardt, Michael Zollhöfer, and Christian Theobalt.  Deep video portraits.  SIGGRAPH, 2018.
  * [20] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [21] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang. Wen.  Faceshifter: Towards high fidelity and occlusion aware face swapping.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [22] Lingzhi Li, Jianmin Bao, Hao Yang, Dong Chen, and Fang Wen.  Faceshifter: Towards high fidelity and occlusion aware face swapping.  In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
  * [23] Borong Liang, Yan Pan, Zhizhi Guo, Hang Zhou, Zhibin Hong, Xiaoguang Han, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang.  Expressive talking head generation with granular audio-visual control.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [24] Yuanxun Lu, Jinxiang Chai, and Xun Cao.  Live speech portraits: real-time photorealistic talking-head animation.  ACM Transactions on Graphics (TOG), 40(6):1–17, 2021.
  * [25] Yuval Nirkin, Yosi Keller, and Tal Hassner.  Fsgan: Subject agnostic face swapping and reenactment.  iccv, 2019.
  * [26] Yuval Nirkin, Iacopo Masi, Anh Tran Tuan, Tal Hassner, and Gerard Medioni.  On face segmentation, face swapping, and face perception.  In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 98–105. IEEE, 2018.
  * [27] Ivan Perov, Daiheng Gao, Nikolay Chervoniy, Kunlin Liu, Sugasa Marangonda, Chris Um, Mr Dpfks, Carl Shift Facenheim, Luis RP, Jian Jiang, et al.  Deepfacelab: Integrated, flexible and extensible face-swapping framework.  arXiv preprint arXiv:2005.05535, 2020.
  * [28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In IEEE Conf. Comput. Vis. Pattern Recog., June 2018.
  * [29] Kripasindhu Sarkar, Vladislav Golyanik, Lingjie Liu, and Christian Theobalt.  Style and pose control for image synthesis of humans from a single monocular view.  arXiv preprint arXiv:2102.11263, 2021.
  * [30] Si, Wentao Liu, Chen Jiang, Ran Gao, Jiashi He, Bo Feng, Shuicheng Li, and Yan.  Psgan++: Robust detail-preserving makeup transfer and removal.  In PAMI, 2021.
  * [31] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  Animating arbitrary objects via deep motion transfer.  CVPR, 2019.
  * [32] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe.  First order motion model for image animation.  neurips, 2019.
  * [33] Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Motion representations for articulated animation.  In CVPR, 2021.
  * [34] Tero, Samuli Karras, Miika Laine, Janne Aittala, Jaakko Hellsten, Timo Lehtinen, and Aila.  Analyzing and improving the image quality of stylegan.  In CVPR, 2020.
  * [35] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM Trans. Graph., 2019.
  * [36] Tingting, Ruihe Li, Chao Qian, Si Dong, Qiong Liu, Wenwu Yan, Liang Zhu, and Lin.  Beautygan: Instance-level facial makeup transfer with deep generative adversarial network.  In ACMMM, 2018.
  * [37] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli.  Image quality assessment: from error visibility to structural similarity.  In IEEE transactions on image processing, 2004.
  * [38] Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  In IEEE Conf. Comput. Vis. Pattern Recog., July 2017.
  * [39] Zhiliang Xu, Zhibin Hong, Changxing Ding, Zhen Zhu, Junyu Han, Jingtuo Liu, and Errui Ding.  Mobilefaceswap: A lightweight framework for video face swapping.  AAAI, 2022.
  * [40] Zhiliang Xu, Xiyu Yu, Zhibin Hong, Zhen Zhu, Junyu Han, Jingtuo Liu, Errui Ding, and Xiang Bai.  Facecontroller: Controllable attribute editing for face in the wild.  AAAI, 2021.
  * [41] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky.  Fast bi-layer neural synthesis of one-shot realistic head avatars.  In Eur. Conf. Comput. Vis., August 2020.
  * [42] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky.  Few-shot adversarial learning of realistic neural talking head models.  In Int. Conf. Comput. Vis., October 2019.
  * [43] Bo Zhang, Mingming He, Jing Liao, V. Sander Pedro, Lu Yuan, Amine Bermak, and Dong Chen.  Deep exemplar-based video colorization.  In CVPR, pages 8052–8061, 2019.
  * [44] Jiangning Zhang, Xianfang Zeng, Mengmeng Wang, Yusu Pan, Liang Liu, Yong Liu, Yu Ding, and Changjie Fan.  Freenet: Multi-identity face reenactment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [45] Pan Zhang, Bo Zhang, Dong Chen, Lu Yuan, and Fang Wen.  Cross-domain correspondence learning for exemplar-based image translation.  In CVPR, pages 5143–5153, 2020.
  * [46] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In IEEE Conf. Comput. Vis. Pattern Recog., pages 586–595, 2018\.
  * [47] Zhimeng Zhang, Lincheng Li, Yu Ding, and Changjie Fan.  Flow-guided one-shot talking face generation with a high-resolution audio-visual dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [48] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang.  Talking face generation by adversarially disentangled audio-visual representation.  In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2019.
  * [49] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei Liu.  Pose-controllable talking face generation by implicitly modularized audio-visual representation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [50] Tong Zhou, Changxing Ding, Shaowen Lin, Xinchao Wang, and Dacheng Tao.  Learning oracle attention for high-fidelity face completion.  CVPR, 2020.
  * [51] Yuqian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi.  Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations.  CVPR, pages 2266–2276, 2021.
  * [52] Yuhao Zhu, Qi Li, Jian Wang, Cheng-Zhong Xu, and Zhenan Sun.  One shot face swapping on megapixels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4834–4844, 2021.

## Appendices

##  Appendix A Head2Head Aligner Training Details

In the training stage of the Head2Head Aligner, the target image $I_{T}$ and
the source image set $I_{S}$ are all sampled from the same identity. We would
expect the output animated portrait $I_{A}$ of our model to be the same as
$I_{T}$, thus the whole training procedure is based on frame reconstruction.
The loss functions are as follows:

Pixel-wise Reconstruction Loss. The reconstruction loss in the first stage
$L^{1}_{L1}$ encourages the pixel-wise similarity between the reenactment
output $I_{R}$ and the target image $I_{T}$ via L1 loss:

$\displaystyle L^{1}_{L1}=\lambda_{L1}\left\|{I_{T}-I_{A}}\right\|_{1}.$ (11)

where $\lambda_{rec}$ is the loss weight.

Perceptual Loss. We utilize the perceptual loss to minimize the semantic
discrepancy between the animated portrait $I_{A}$ and the target image
$I_{T}$. The feature matching loss is used on a pre-trained VGG-19 network, a
pre-trained ResNet with ArcFace [6] $E_{id}$, and the discriminator $D_{A}$:

$L^{1}_{per}=\sum^{K}_{k=1}\sum^{L_{k}}_{l=1}\lambda^{k}_{l}\left\|{\phi^{k}_{l}(I_{T})-\phi^{k}_{l}(I_{A})}\right\|_{1},$
(12)

where $\lambda^{k}_{l}$ balances the terms, $\phi^{k}_{l}$ represents the
activation of layer $l$. $K=3$ represents the three networks repectively.

Identity Loss. Moreover, to further enhance the ability of identity
preservation, we leverage an additional identity loss to minimize the identity
gap between the generated output and the target:

$\displaystyle L_{id}=\lambda_{id}(1-\cos{(E_{id}(I_{T}),E_{id}(I_{A}))}).$
(13)

where $\lambda_{id}$ is the loss weight, $\cos$ denotes the cosine distance of
identity embeddings.

Adversarial Loss. The discriminator $D_{A}$ is imposed to make the animated
portrait $I_{A}$ look indistinguishable from the target image $I_{T}$. The
training losses are defined as follows:

$\displaystyle L^{D_{A}}_{adv}$
$\displaystyle=-E[h(D_{A}(I_{T}))]-E[h(-D_{A}(I_{A}))].$ (14) $\displaystyle
L^{G_{A}}_{adv}$ $\displaystyle=-E[D_{A}(I_{A})],$ (15)

where $h(x)=\min(0,-1+x)$ is a hinge function used to regularize the
discriminator [12, 1].

##  Appendix B Head2Scene Blender Training Details

We jointly train the Semantic-Guided Color Reference Creation module and the
Blending UNet via the loss functions below, expecting the sequential two jobs
to facilitate each other.

Perceptual Loss. Reconstruct training is also leveraged to minimize the
difference between the ultimate blending output ${I_{B}}$ and the target image
${I_{T}}$ by minimizing the perceptual loss.

$L^{2}_{per}=\sum^{L_{1}}_{l=1}\lambda^{1}_{l}\left\|{\phi^{1}_{l}(I_{B})-\phi^{1}_{l}(I_{T})}\right\|_{1}.$
(16)

here $\lambda^{1}_{l}$ balance the terms layer-wise, $\phi^{1}_{l}$ represents
the activation of layer $l$ in the pre-trained VGG-19 as illustrated in Eq.
12.

Reconstruction Loss. Note that the above feature matching loss excels in
capturing fine details while missing the low-frequency image content. This
could result in inaccurate colors. Consequently, the L1 loss is also applied
for color consistency:

$L^{2}_{L1}=\lambda_{1}\left\|{I_{B}-I_{T}}\right\|_{1}.$ (17)

Cycle Loss: In order to guarantee that the warped headcolor/inpainting
references could learn a meaningful correspondence matrix, we introduce the
cycle consistent loss.

$L_{c}=\lambda_{c}\left\|{I_{T\rightarrow A\rightarrow T}-I_{T}}\right\|_{1},$
(18)

where $I_{T\rightarrow A\rightarrow T}$ is the reference after cycled
warpping, and $I_{T\rightarrow A\rightarrow T}^{k}(u)=\sum_{v\in
M_{A}^{k}}{softmax}_{v}(\Gamma^{k}(u,v)/{\tau})\cdot I_{T\rightarrow
A}(v),u\in M_{T}^{k}$. Besides, the additional target image $I_{T}^{\prime}$
coming from different image compared to $I_{A}$ is also utilized to ensure the
meaningful of created reference:

$L_{c^{\prime}}=\lambda_{c}\left\|{I_{T^{\prime}\rightarrow A\rightarrow
T^{\prime}}-I_{T}}\right\|_{1}.$ (19)

Adversarial Loss. We utilize another discriminator $D_{B}$ to distinguish the
blending outputs and the real samples from ground truth, with head mask
$M_{A}^{H}$ and inpainting mask $M_{A}^{I}$ concatenated as conditions for
further improving the fidelity of our blending outputs. The adversarial
objectives are optimized by hinge loss:

$\displaystyle\begin{split}L^{D_{B}}_{adv}&=-E[h(D_{B}(I_{T}\odot
M_{A}^{H}\odot M_{A}^{I}))]\\\ &-E[h(-D_{B}(I_{B}\odot M_{A}^{H}\odot
M_{A}^{I}))].\end{split}$ (20) $\displaystyle L^{B}_{adv}$
$\displaystyle=-E[D_{B}(I_{B}\odot M_{A}^{H}\odot M_{A}^{I})].$ (21)

where $\odot$ denotes the concatenation along the dimensionality of channel.

Figure 10:  Quantitative result of increasing the K-shot number with subject-
specific fine-tuning or a meta-learned model. Meta denotes the meta-learned
model, ft600 indicates the fine-tuned model under 600 iterations. The
transparent area around the dotted and solid lines represents the variance.

##  Appendix C Experimental Details

In this section, we describe the data collection and the experimental details,
$i.e.$, evaluation metrics, competitors and implementation details.

###  C.1 Data collection

In terms of Head2Head Aligner, We re-download the 1080P videos with the urls
provided by the VoxCeleb2 dataset [18] from YouTube. The frames are aligned by
detected landmarks and processed to ${512\times 512}$. Compared with previous
studies [7, 42], a larger cropping window size is used. Under such a setting,
we manage to collect overall 28,367 videos with 5,478 different identities,
which is much less than the original training set of 145,569 videos with 5,994
different identities in [7, 42]. Besides, totally 805 videos with 86 different
identities are gathered as the test set.

###  C.2 Implementation Details.

The portrait encoder $E_{por}$ is ResNeXt-50 [38], the pose and expression
encoder are both constructed by the MobileNetV2 [28]. The size of the latent
embedding, $i.e.$ $d_{1}$, $d_{2}$, $d_{3}$ and $d_{4}$, are 512, 512, 256 and
256 respectively. The MLP module that transforms them into AdaIN parameters is
a 2-layer ReLU perceptron with spectral normalization, where the output of the
intermediate layer is 768. The generator is borrowed from [7]. We add
upsampling residual blocks after the last layer to generate the reenacted
image with $512\times 512$ resolution. The first module is trained for roughly
three weeks with batch size to be 6 on six 32G Tesla V100 GPUs.

As for the Head2Scene Blender, we use a VGG-19 for feature extraction. The
Blending UNet is a seven-layer deep residual U-type network. The training
image size is 512 $\times$ 512 and two 48G RTX8000 GPUs is used to train the
model for 5 days.

For both models, Adam [20] optimizer is used with $\beta_{1}=0.9$ and
$\beta_{2}=0.999$, the imbalanced learning rate for generator and
discriminator is set to be 1e-4 and 4e-4 respectively. Spectral normalization
is applied to all operators in the system for stabilizing the adversarial
training.

##  Appendix D More Studies on Head2Head Aligner

This section is the supplementary materials for more quantitative and
qualitative reenactment comparison under different settings:

One-shot meta-learning evaluation. Fig. LABEL:fig:1shot_meta_crossid_sup is
the supplementary qualitative results for cross-id animated portraits in one-
shot meta-learning setting. Significant artifacts exhibited in FOMM and
Siarohin et al [33]; our method can generate the animated portraits with
higher identity similarity and pose consistency, besides, the elaborated
emotions (such as happiness in 1st, 4th, and 8th row of Fig.
LABEL:fig:1shot_meta_crossid_sup) are also well animated.

Impact of K-shot and fine-tuning. Quantitative results of increasing the
K-shot number with subject-specific fine-tuning or a meta-learned model are
shown in Fig. 10. The qualitative impact of fine-tuning is illustrated in Fig.
LABEL:fig:1shot_ft600step_crossid_sup, where better identity preservation is
obtained compared to the animated portrait from the meta-learned model, while
the pose error is increased for the fine-tuned model overfits the one-shot
source. Then we increase the K-shot number, as depicted in Fig.
LABEL:fig:32shot_ft600step_crossid_sup, and the identity similarity and pose
consistency are further improved.

##  Appendix E More Studies on Head2Scene Blender

This section is the supplementary materials for the discussion of 1) the
memory-saving of our Semantic-Guided Color Reference Creation module, 2) skin
color alignment and 3) inpainting performance in the blending network.

Semantic-Guided Color Reference Creation. To verify the mechanism of our
proposed calculation-reducing method, we present the semantic-specific
contribution that the target image made in the correlation matrix. Specially,
the accumulated attention distribution, denoting for pixels in source image
with semantic label $k$, is computed via $\sum_{v\in
M_{T}^{k}}{softmax}_{v}(\Gamma^{k}(u,v)/\tau)(u\in M_{A}^{k})$, and the
qualitative results are exhibited in Fig. 13. Obviously, it demonstrates that
the pixels in the target image with different semantic label almost has no
contribution in the correspondence matrix to that in the source image. Thus
the correspondence between pixels from source and target image with different
semantic labels is redundant. Following our method, roughly averaged 9G GPU
memory usage is saved in 512$\times$512 training setting.

\begin{overpic}[width=433.62pt]{fig/fig_contribut_nolabel_reange.jpg}
\par\put(6.0,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize source}}}
\put(24.0,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize skin}}}
\put(40.0,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize nose}}}
\put(56.0,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize right eye}}}
\put(73.5,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize up lip}}}
\put(89.0,37.1){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize low lip}}}
\par\put(5.0,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize reference}}}
\put(23.0,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize left eye}}}
\put(39.0,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize left eye}}}
\put(56.0,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize left brow}}}
\put(73.5,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize hair}}}
\put(90.0,18.6){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize output}}}
\par\end{overpic} Figure 13:  The accumulated attention distribution for
diverse semantic label, it shows an intuitive phenomenon: the highlighted
regions correspond to semantic label blow the sub figure, and the remain
regions are pitch-dark.

Skin Color Alignment. For effective comparison, we overlay the animated
portrait on our head swapping result and using the target head as a reference.
The qualitative results of skin color alignment with different method on
VoxCeleb2 test set is illustrated in Fig. 15, where the remarkable performance
advantage is depicted. Our method outperforms deepfacelab, SCGAN and PSGAN
with more similar skin color to target image.

Performance of Inpainting. As the missing pixels are affected by the
surroundings, for a fair comparison, we excavate neck and background around
the head from our head swapping results and take them as input for inpainting,
making facial skin color consistent with the skin color. Since the source code
of reference-guided competitors ( LOA[50] and TransFill[51]) are not released
publicly, we send the testing data to authors for the compared result
respectively. Qualitative comparison results are shown in
Fig.LABEL:fig:paperfig_plot_for_inpaintingCompare.

##  Appendix F Limitations

We briefly discuss the limitations of our two modules. 1) Although the
Head2Head Aligner in HeSer can achieve better identity preservation and pose
consistency than the previous 2D-based reenactment methods, it still needs a
subject-specific fine-tuning procedure to generate an emotionally and
positionally aligned head. 2) Segmentation masks are required for every frame
of the target video, thus might lead to temporal inconsistency and increasing
computational complexity. Moreover, HeSer performs poorly on long hair cases,
which might be caused by the inaccuracy of segmentation results on the end of
the hair.

Though there are certain limitations of our work, it is still one of the
earliest attempts at few-shot head swapping, we hope it can contribute to the
community and boost future head swapping techniques for solving the
limitations above.

\begin{overpic}[width=433.62pt]{fig/makeuptransfer_voxceleb_half.jpg}
\put(3.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Source}}}
\put(12.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Reference}}}
\par\put(22.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize IDT
\cite[cite]{[\@@bibref{}{Perov2020deepfacelab}{}{}]}}}}
\put(33.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize LCT
\cite[cite]{[\@@bibref{}{Perov2020deepfacelab}{}{}]}}}}
\put(42.5,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize MKL
\cite[cite]{[\@@bibref{}{Perov2020deepfacelab}{}{}]}}}}
\put(52.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize RCT
\cite[cite]{[\@@bibref{}{Perov2020deepfacelab}{}{}]}}}}
\put(62.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize SOT
\cite[cite]{[\@@bibref{}{Perov2020deepfacelab}{}{}]}}}}
\par\put(71.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize
PSGAN\cite[cite]{[\@@bibref{}{jiang2020psgan}{}{}]}}}}
\put(82.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize
SCGAN\cite[cite]{[\@@bibref{}{deng2021scgan}{}{}]}}}}
\put(93.0,40.2){\rotatebox{0.0}{\color[rgb]{0,0,0}{\scriptsize Ours}}}
\end{overpic} Figure 15:  Qualitative comparsion of facial skin color
alignment with different method on voxceleb2 test dataset, best viewed in
color.