# FlowFace: Semantic Flow-guided Shape-aware Face Swapping

Hao Zeng1, Wei Zhang1, Changjie Fan1, Tangjie Lv1, Suzhen Wang1,
Zhimeng Zhang1, Bowen Ma1, Lincheng Li1, Yu Ding1,3,, Xin Yu2 Yu Ding is the
corresponding author.

###### Abstract

In this work, we propose a semantic flow-guided two-stage framework for shape-
aware face swapping, namely FlowFace. Unlike most previous methods that focus
on transferring the source inner facial features but neglect facial contours,
our FlowFace can transfer both of them to a target face, thus leading to more
realistic face swapping. Concretely, our FlowFace consists of a face reshaping
network and a face swapping network. The face reshaping network addresses the
shape outline differences between the source and target faces. It first
estimates a semantic flow (_i.e._ , face shape differences) between the source
and the target face, and then explicitly warps the target face shape with the
estimated semantic flow. After reshaping, the face swapping network generates
inner facial features that exhibit the identity of the source face. We employ
a pre-trained face masked autoencoder (MAE) to extract facial features from
both the source face and the target face. In contrast to previous methods that
use identity embedding to preserve identity information, the features
extracted by our encoder can better capture facial appearances and identity
information. Then, we develop a cross-attention fusion module to adaptively
fuse inner facial features from the source face with the target facial
attributes, thus leading to better identity preservation. Extensive
quantitative and qualitative experiments on in-the-wild faces demonstrate that
our FlowFace outperforms the state-of-the-art significantly.

##  1 Introduction

Face swapping refers to transferring the identity information of a source face
to a target face while maintaining the attributes (e.g., expression, pose,
hair, lighting, and background) of the target. It has attracted many interests
due to its wide applications, such as portrait reenactment, film production,
and virtual reality.

Recent works (Li et al. 2019; Chen et al. 2020; Xu et al. 2021; Li et al.
2021) have made great efforts to achieve promising face swapping results.
However, these methods often focus on inner facial feature transferring but
neglect facial contour reshaping. We observe that facial contours also carry
the identity information of a person, but few efforts (Wang et al. 2021) have
been made on facial contours transferring. Facial shape transferring is still
a challenge for authentic face swapping.

To solve the shape transferring problem, we propose a semantic flow-guided
two-stage framework, dubbed FlowFace. Unlike existing methods, FlowFace is a
shape-aware face swapping network. In a nutshell, we first present a face
reshaping network to warp the target face referring to the source face shape
at the first stage. Then, based on the reshaped facial contour, we further
employ a face swapping network to transfer the inner facial features to the
reshaped target face.

Our face reshaping network addresses the shape outline discrepancy between the
source face and the target face. Specifically, we use a 3D face reconstruction
model (_i.e._ , 3DMM (Blanz and Vetter 1999)) to obtain shape coefficients of
the source and target faces and then project the obtained 3D shapes to 2D
facial landmarks. To accurately warp the target face, we need to obtain dense
motion between the source and the target faces. Subsequently, we design a
semantic guided generator to transform the sparse 2D facial landmarks into the
dense flow. The estimated flow, called semantic flow, will be exploited to
warp the target face shape explicitly in a pixel-wise manner. In addition, we
propose a semantic-guided discriminator to enforce our face reshaping network
to produce accurate semantic flow.

After reshaping the target face, we introduce a face swapping network for
transferring the inner facial features of the source face to the target ones.
Prior works usually use a face recognition model to extract the identity
embedding of the source face and then transfer it to the target face. We argue
this would lose some personalized appearances during transferring because the
identity embedding is often trained under discriminative tasks and thus may
ignore intra-class variations (Kim, Lee, and Zhang 2022). Thus, we opt to
employ a pre-trained masked autoencoder (MAE) (He et al. 2022) to extract
facial features that better capture facial appearances and identity
information. Moreover, unlike prior arts that widely employ AdaIN (Liu et al.
2017b) to infuse the source identity embedding to the target face, we develop
a cross-attention fusion module to adaptively fuse the source and target
features. In doing so, we achieve better face swapping performance.

Extensive quantitative and qualitative experiments validate the effectiveness
of our FlowFace on in-the-wild faces and our FlowFace outperforms the state-
of-the-art. Overall, our contributions are summerized as follows:

  * •

We propose a two-stage framework for shape-aware face swapping, namely
FlowFace. It can effectively transfer both the inner facial features and the
facial outline to a target face, thus achieving authentic face swapping
results.

  * •

We design a semantic flow-guided face reshaping network and validate its
effectiveness in transferring the source face shapes to the target ones. The
reshaped target faces are more similar to the source faces in terms of face
contours.

  * •

We design a pre-trained face masked autoencoder based face swapping network.
The encoder captures not only identity information but also facial appearance,
thus allowing us to transfer richer information from the source face to the
target and achieve identity similarity.

  * •

We design a cross-attention fusion module to adaptively fuse the source and
target features. To the best of our knowledge, we are the first to perform
face swapping in the latent space of the pre-trained masked autoencoder.

##  2 Related Work

The previous face swapping methods can be classified as the target attribute-
guided and source identity-guided methods.

Target attribute-guided methods edit the source face first and then blend it
to the target background. Early methods (Bitouk et al. 2008; Chen et al. 2019;
Lin et al. 2012) directly warp the source face according to the target facial
landmarks, thus failing to address large posture differences and expression
differences. 3DMM-based methods (Blanz et al. 2004; Thies et al. 2016;
MarekKowalski 2021; Nirkin et al. 2018) swap faces by 3D-fitting and re-
rendering. However, these methods often cannot handle skin color or lighting
differences and suffer from poor fidelity. Later, GAN-based methods improve
the fidelity of the generated faces. Deepfakes (DeepFakes 2019) transfers the
target attributes to the source face by an encoder-decoder structure while
being constrained by two specific identities. FSGAN(Nirkin, Keller, and
Hassner 2019) employs the target facial landmarks to animate the source face
and proposes a blending network to fuse the generated source face to the
target background. However, it fails to tackle drastic skin color differences.
Later, AOT (Zhu et al. 2020) focuses on swapping faces with large differences
in skin color and lighting by formulating appearance mapping as an optimal
transport problem. These methods always need a facial mask to blend the
generated face with the target background. However, the mask-guided blending
restricts the face shape change.

Source identity-guided methods usually adopt the identity embedding or the
latent representation of StyleGAN2 (Karras et al. 2020) to represent the
source identity and inject in into the target face. FaceShifter (Li et al.
2019) designs an adaptive attentional denormalization generator to integrate
the source identity embedding and the target features. SimSwap (Chen et al.
2020) introduces a weak feature matching loss to help preserve the target
attributes. MegaFS (Zhu et al. 2021), RAFSwap (Xu et al. 2022a) and HighRes
(Xu et al. 2022b) utilize the pre-trained StyleGAN2 to swap faces and can
achieve high-resolution face swapping. FaceController (Xu et al. 2021)
exploits the identity embedding with 3D priors to represent the source
identity and design a unified framework for identity swapping and attribute
editing. InfoSwap (Gao et al. 2021) leverages the information bottleneck
principle to disentangle the identity and identity-irrelevant information.
FaceInpainter (Li et al. 2021) also utilizes the identity embedding with 3D
priors to implement controllable face in-painting under heterogeneous domains.
Smooth-Swap (Kim, Lee, and Zhang 2022) builds smooth identity embedding that
makes the training of face swapping fast and stable.

However, most of these methods neglect the facial outlines during face
swapping. Recently, HifiFace (Wang et al. 2021) can control the face shape
using a 3D shape-aware identity. However, it injects the shape representation
into the latent feature space, making it hard for the model to correctly
decode the face shape. Moreover, these methods always need a pre-trained face
recognition model during the inference time, which is not friendly to
deployment.

##  3 Proposed Method

Figure 1: Overview of our two-stage FlowFace. In the first stage, the face
reshaping network ($F^{res}$) transfers the shape of the source face $I_{s}$
to the target face $I_{t}$ by warping $I_{t}$ explicitly with an estimated
semantic flow $V_{t}$. In the second stage, the face swapping network
($F^{swa}$) generates the inner facial details by manipulating the latent face
representation $e_{s}$ and $e_{t}$ using our designed cross-attention fusion
module. It should be noted that ⓒ in the figure represents the concatenation
operation.

The face swapping task aims to generate a face with the identity of the source
face and the attributes of the target face. This paper proposes a semantic
flow-guided two-stage framework for shape-aware face swapping, namely
FlowFace. As shown in Figure 1, FlowFace consists of a face reshaping network
$F^{res}$ and a face swapping network $F^{swa}$. Let $I_{s}$ be the source
face and $I_{t}$ be the target face. $F^{res}$ first transfers the shape of
$I_{s}$ to the target face $I_{t}$. The reshaped image is denoted as
$I^{res}_{t}$. Then $F^{swa}$ generates the inner face of $I^{res}_{t}$ and
outputs the result image $I_{o}$.

### Face Reshaping Network

We design the face reshaping network, $F^{res}$, to address the shape
discrepancy between the source and target faces. It warps the target face
shape explicitly pixel-wise with an estimated semantic flow. To achieve this
goal, $F^{res}$ requires a face shape representation that models the shape
differences between the source and target faces. Then it estimates a semantic
flow according to the above shape differences. Finally, the semantic flow is
used to warp the target face shape.

#### Face Shape Representation.

Since our face reshaping network needs to warp the face shape pixel-wisely, we
choose the explicit facial landmarks as the shape representation. We use a 3D
face reconstruction model to obtain facial landmarks. As shown in Figure 1,
the 3D face reconstruction model $E_{3D}$ extracts 3D coefficients of the
source and target:

$\begin{split}(\beta_{*},\theta_{*},\psi_{*},c_{*})=E_{3D}(I_{*}),\end{split}$
(1)

where $\beta_{*},\theta_{*},\psi_{*},c_{*}$ are the FLAME coefficients (Li et
al. 2017) representing the face shape, pose, expression, and camera,
respectively. $*$ is $s$ or $t$, representing the source or the target,
respectively. With these coefficients, the target face can be modeled as:

$M_{t}(\beta_{t},\theta_{t},\psi_{t})=W\left(T_{P}(\beta_{t},\theta_{t},\psi_{t}),\mathbf{J}(\beta_{t}),\theta_{t},\mathcal{W}\right),$
(2)

where $M_{t}$ represents the 3D face mesh of the target face. $W$ is a linear
blend skinning (LBS) function that is applied to rotate the vertices of
$T_{P}$ around joint $J$. $\mathcal{W}$ is the blend weights. $T_{P}$ denotes
the template mesh $\overline{T}$ with shape, pose, and expression offsets (Li
et al. 2017).

Then, we reconstruct the source face similarly, except that the source pose
and expression coefficients are replaced with the target ones. The obtained 3D
face mesh is denoted as $M_{s2t}$. Finally, we sample 3D facial landmarks from
$M_{t}$ and $M_{s2t}$ and project these 3D points to 2D facial landmarks with
the target camera parameter $c_{t}$:

$\begin{split}P_{t}=s\Pi\left(M_{t}^{i}\right)+t,\\\
P_{s2t}=s\Pi\left(M_{s2t}^{i}\right)+t,\end{split}$ (3)

where $M_{*}^{i}$ is a vertex in $M_{*}$, $\Pi$ is an orthographic 3D-2D
projection matrix, and $s$ and $t$ are parameters in $c_{t}$, indicating
isotropic scale and 2D translation. $P_{*}$ denotes the 2D facial landmarks.
It should be noted that we only use the landmarks at the facial contours as
the shape representation since inner facial landmarks contain identity
information that may influence the reshaping result.

#### Semantic Flow Estimation.

The relative displacement between $P_{t}$ and $P_{s2t}$ only describes sparse
movement. To accurately warp the target face, we need to obtain dense motion
between the source and the target faces. Therefore, we propose the semantic
flow, which models the semantic correspondences between two faces, to achieve
pixel-wised movement. We design a semantic guided generator $G^{res}$ to
estimate the semantic flow. Specifically, $G^{res}$ requires three inputs:
$P_{s2t}$, $P_{t}$ and $S_{t}$, where $P_{s2t}$ and $P_{t}$ are the 2D facial
landmarks obtained above. $S_{t}$ is the target face segmentation map that
complements the semantic information lost in facial landmarks. The output of
$G^{res}$ is the estimated semantic flow $V_{t}$, the formulation is:

$V_{t}=G^{res}(P_{s2t},P_{t},S_{t}).$ (4)

Then, a warping module is introduced to generate the warped faces using
$V_{t}$. We find that an inaccurate flow is likely to produce unnatural
images, and therefore, we design a semantic guided discriminator $D^{res}$
that ensures $G^{res}$ to produce a more accurate flow. Specifically, the
warping operation is conducted on both $I_{t}$ and $S_{t}$:

$(I_{t}^{res},S_{t}^{res})=F(V_{t},I_{t},S_{t}),$ (5)

where $F$ is the warping function in the warping module. We feed the
concatenation of the warped face $I_{t}^{res}$ and the warped segmentation map
$S_{t}^{res}$ to $D^{res}$. Thus, $D^{res}$ is able to discriminate whether
the input is real or fake from the semantic level and the image level. It
should be noted that $S_{t}^{res}$ and $D^{res}$ are only used during
training.

#### Training Loss.

We employ three loss functions for $F^{res}$:

$\mathcal{L}^{res}=\mathcal{L}_{adv}+\lambda_{rec}\mathcal{L}_{rec}+\lambda_{ldmk}\mathcal{L}_{ldmk},$
(6)

where $\lambda_{ldmk}$ and $\lambda_{rec}$ are hyperparameters for each term.
In our experiments, we set $\lambda_{ldmk}$=800 and $\lambda_{rec}$=10.

Adversarial Loss. To make the resultant images more realistic, we adopt the
hinge version adversarial loss(Lim and Ye 2017) for training, denoted by
$L_{adv}$:

$\mathcal{L}_{adv}=-\mathbb{E}[D^{res}([I_{t}^{res},S_{t}^{res}])],$ (7)

where $D^{res}$ is the discriminator which is trained with:

$\begin{split}\mathcal{L}_{D}=\mathbb{E}[\max(0,1-D([I_{t},S_{t}]))]\\\
+\mathbb{E}[\max(0,1+D([I_{t}^{res},S_{t}^{res}])].\end{split}$ (8)

Reconstruction Loss. Since there is no ground-truth for face reshaping
results, we enforce $I_{s}=I_{t}$ with a certain probability when training
$G^{res}$. Then the face reshaping task becomes a reconstruction task, and we
introduce a pixel-wise reconstruction loss:

$\mathcal{L}_{rec}=\left\|{I_{t}^{res}-I_{t}}\right\|_{2},$ (9)

where $\left\|*\right\|_{2}$ denotes the euclidean distance.

Landmark Loss. Since there is not pixel-wised ground truth for $I_{t}^{res}$,
we exploit the 2D facial landmarks $P_{s2t}$ to constrain the shape of
$I_{t}^{res}$. Specifically, we first use a pre-trained facial landmark
detector (Sun et al. 2019) to predict the the facial landmarks of
$I_{t}^{res}$, denoted as $P_{t}^{res}$. Then the loss is computed as:

$\mathcal{L}_{ldmk}=\left\|P_{t}^{res}-P_{s2t}\right\|_{2}.\ $ (10)

At this point, our designed face reshaping network is able to transfer the
face shape of the source to the target face. However, the inner facial
features are still unchanged.

### Face Swapping Network

The face swapping network $F^{swa}$ is used to generate the inner face of
$I_{t}^{res}$ ($I_{t}$). As shown in Figure 1, we first utilize a shared face
encoder $E_{f}$ to map both $I_{s}$ and $I_{t}^{res}$ into patch embeddings
$e_{s}$ nad $e_{t}$. Then a cross-attention fusion module is designed to
adaptively fuse the identity information of the source face and the attribute
information of the target. Finally, the facial decoder, fed with the
manipulated embeddings $e_{o}$, outputs the final face swapping result
$I_{o}$.

#### Shared Face Encoder.

Most previous face swapping methods map the source face into an ID embedding
with a pre-trained face recognition model and extract the target face
attributes with another face encoder. However, we argue that using two
different encoders is unnecessary and even makes deploying more complex.
Moreover, the ID embedding is trained on purely discriminative tasks and may
lose some personalized appearances during transferring.

Therefore, we employ a shared encoder to project both the source face and the
target face into a common latent representation. The encoder is designed
following MAE (He et al. 2022) and pre-trained on a large-scale face dataset
using the masked training strategy. Compared to the compact latent code of
StyleGAN2 (Karras et al. 2020) and the identity embedding, the latent space of
MAE can better capture facial appearances and identity information, because
masked training requires reconstructing masked image patches from visible
neighboring patches, thus ensuring each patch embedding contains rich topology
and semantic information.

Based on the pre-trained encoder $E_{f}$, we can project a facial image
$I_{*}$ into a latent representation, also known as patch embeddings:

$e_{*}=E_{f}(I_{*}),$ (11)

where $e_{*}\in R^{N*L}$. $N$ and $L$ denote the number of patches and the
dimension of each embedding, respectively.

#### Cross-Attention Fusion Module.

The shared face encoder projects the source face and the target face into a
representative latent space. The subsequent operation is to fuse the source
identity information with the target attribute in this latent space.
Intuitively, identity information should be transferred between related
patches (_e.g._ , nose to nose, etc.). Therefore, we design a cross-attention
fusion module (CAFM) to adaptively aggregate identity information from the
source and fuse it into the target.

As shown in Figure 1, our CAFM consists of a cross-attention block and two
standard transformer blocks (Dosovitskiy et al. 2020). Given the source patch
embeddings $e_{s}$ and the target patch embeddings $e_{t}$, we first compute
$Q,K,V$ for each patch embedding in $e_{s}$ and $e_{t}$. Then the cross
attention is computed by:

$\operatorname{CA}(Q_{t},K_{s})=\operatorname{softmax}\left(\frac{Q_{t}K_{s}^{T}}{\sqrt{d_{k}}}\right),$
(12)

where $\operatorname{CA}$ represents Cross Attention, $Q_{*},K_{*},V_{*}$ are
predicted by attention heads, and $d_{k}$ is the dimension of $K_{*}$. The
cross attention describes the relation between each target patch and the
source patches. Next, the source identity information is aggregated based on
the computed $\operatorname{CA}$ and fused to the target values via addition:

$V_{fu}=\operatorname{CA}*V_{s}+V_{t}.$ (13)

Then, $V_{fu}$ are normalized by a layer normalization (LN) and processed by
multi-layer perceptrons (MLP). The Cross Attention and MLP are along with skip
connections. The fused embeddings $e_{fu}$ are further fed into two
transformer blocks to obtain the final output $e_{o}$.

Finally, we utilize a a convolutional decoder to generate the final swapped
face image $I_{o}$ from $e_{o}$. In contrast to the ViT decoder in MAE, we
find the convolutional decoder achieves more realistic results.

#### Training Loss.

We employ six loss functions to train our face swapping network $F^{swa}$:

$\begin{split}\mathcal{L}^{swa}=\mathcal{L}_{adv}+\lambda_{rec}\mathcal{L}_{rec}+\lambda_{id}\mathcal{L}_{id}+\lambda_{exp}\mathcal{L}_{exp}\\\
+\lambda_{ldmk}\mathcal{L}_{ldmk}+\lambda_{perc}\mathcal{L}_{perc}\end{split},$
(14)

where $\lambda_{rec}$, $\lambda_{id}$, $\lambda_{exp}$, $\lambda_{ldmk}$,
$\lambda_{attr}$ are hyperparameters for each term. In our experiment, we set
$\lambda_{rec}$=10, $\lambda_{id}$=5, $\lambda_{exp}$=10,
$\lambda_{ldmk}$=5000 and $\lambda_{attr}$=2.

As in the face reshaping stage, the adversarial loss is used to make the
resultant images more realistic, and the reconstruction loss between $I_{o}$
and $I_{t}^{res}$ is used for self-supervision since there is also no ground-
truth for face swapping results.

Identity Loss. The identity loss is used to improve the identity similarity
between $I_{s}$ and $I_{o}$:

$\mathcal{L}_{id}=1-cos(E_{id}(I_{o}),E_{id}(I_{s})),$ (15)

where $E_{id}$ denotes a face recognition model (Deng et al. 2019) and $cos$
denotes the cosine similarity.

Posture Loss. We adopt the landmark loss to constrain the face posture during
face swapping:

$\mathcal{L}_{ldmk}=\left\|P_{t}^{res}-P_{o}\right\|_{2},\ $ (16)

where $P_{o}$ represents the landmarks of $I_{o}$.

Perceptual Loss. Since high-level feature maps contain semantic information,
we employ the feature maps from the last two convolutional layers of pre-
trained VGG as the facial attribute representation. The loss is formulated as:

$\mathcal{L}_{perc}=\left\|VGG(I_{t}^{res})-VGG(I_{o})\right\|_{2}.\ $ (17)

Expression Loss. We utilize a novel fine-grained expression loss (Zhang et al.
2021) that penalizes the $\mathcal{L}_{2}$ distance of two expression
embeddings:

$\mathcal{L}_{exp}=\left\|{E_{exp}(I_{o})-E_{exp}(I_{t})}\right\|_{2}.$ (18)

##  4 Experiments

Table 1: Quantitative comparisons with state-of-the-art methods on FF++. ”†”
means the results are cited from their papers.

Methods | ID Acc($\%$) $\uparrow$ | Shape$\downarrow$ | Expr.$\downarrow$ | Pose.$\downarrow$
---|---|---|---|---
CosFace | SphereFace | Avg
Deepfakes | 83.55 | 86.60 | 85.08 | 1.78 | 0.54 | 4.05
FaceSwap | 70.95 | 76.77 | 73.86 | 1.85 | 0.40 | 2.21
FSGAN | 48.86 | 53.85 | 51.36 | 2.18 | 0.27 | 2.20
FaceShifter | 97.38† | 80.64 | 89.01 | 1.68 | 0.33 | 2.28
SimSwap | 93.63 | 96.22 | 94.43 | 1.74 | 0.26 | 1.40
$F^{res}$+SimSwap | 94.31 | 96.82 | 95.56 | 1.67 | 0.27 | 2.27
HifiFace | 98.48† | 90.76 | 94.62 | 1.62 | 0.30 | 2.29
$F^{swa}$ | 99.18 | 98.23 | 98.70 | 1.43 | 0.21 | 1.99
Ours | 99.26 | 98.40 | 98.83 | 1.17 | 0.22 | 2.66

Figure 2: Qualitative comparisons with Deepfakes, FaceSwap, FSGAN,
FaceShifter, SimSwap and HifiFace on FF++. Our FlowFace outperforms the other
methods significantly, especially in preserving face shapes, identities, and
expressions.

Our method is validated through qualitative and quantitative comparisons with
state-of-the-art ones and a user study. Moreover, several ablation experiments
are also reported to validate our design of FlowFace.

### Implementation Details

Dataset. The training dataset is collected from three commonly-used face
datasets: CelebA-HQ (Karras et al. 2017), FFHQ (Karras, Laine, and Aila 2019),
and VGGFace2 (Cao et al. 2018). Faces are aligned and cropped to $256\times
256$. Particularly, low-quality faces are removed to ensure high-quality
training. The final dataset contains 350K face images, and 10K images are
randomly sampled as the validation dataset. For the comparison experiments, we
construct the test set by sampling FaceForensics++(FF++) (Rössler et al.
2019), following (Li et al. 2019). Specifically, FF++ consists of 1000 video
clips, and the test set is collected by sampling ten frames from each clip of
FF++, in a total of 10000 images.

Training. Our FlowFace is trained in a two-stage manner. Specifically,
$F^{res}$ is first trained for 32K steps with a batch size of eight. As for
$F^{swa}$, we first pre-trained the face encoder following the training
strategy of MAE on our face dataset. Then we fix the encoder and train other
components of $F^{swa}$ for 640K steps with a batch size of eight. We adopt
Adam (Kingma and Ba 2014) optimizer with $\beta_{1}$=0 and $\beta_{2}$=0.99
and the learning rate is set to 0.0001. More details are in the supplementary
materials and our codes will be made publicly available upon publication of
the paper.

Metrics. The quantitative evaluations are performed in terms of four metrics:
identity retrieval accuracy (ID Acc), shape error, expression error (Expr
Error), and pose error. We follow the same test protocol in (Li et al. 2019;
Wang et al. 2021). However, since some pre-trained models used in their
testing are not available, we leverage different ones. For ID Acc, we employ
two face recognition models, including CosFace (Wang et al. 2018) and
SphereFace (Liu et al. 2017a), to perform identity retrieval for a more
comprehensive comparison. For expression error, we adopt a different
expression embedding model (Vemulapalli and Agarwala 2019) to compute the
euclidean distance of expression embeddings between the target and swapped
faces.

Figure 3: Qualitative comparisons with more methods including MegaFS (Zhu et
al. 2021), FaceInpainter (Li et al. 2021), HighRes (Xu et al. 2022b) and
SmoothSwap (Kim, Lee, and Zhang 2022). The shown images of the compared
methods are cropped from their original papers or their released results.

### Comparisons with the State-of-the-art

#### Quantitative Comparisons.

Our method is compared with six methods including Deepfakes (DeepFakes 2019),
FaceSwap (MarekKowalski 2021), FSGAN (Nirkin, Keller, and Hassner 2019),
FaceShifter (Li et al. 2019), SimSwap (Chen et al. 2020), and HifiFace (Wang
et al. 2021). For Deepfakes, FaceSwap, FaceShifter, and HifiFace, we use their
released face swapping results of the sampled 10,000 images. For FSGAN and
SimSwap, the face swapping results are generated with their released codes.

Table 1 shows that our method achieves the best scores under most evaluation
metrics, including ID Acc, shape error, and Expr Error. These results validate
the superiority of our FlowFace. We obtain a slightly worse result than other
methods for the pose error, which can be attributed to our FlowFace changing
the face shape while the employed head pose estimator is sensitive to face
shapes.

Figure 4:  Qualitative ablation results of FlowFace.

#### Qualitative Comparisons.

The qualitative comparisons are conducted on the same FF++ test set collected
in the quantitative comparisons. As shown in Figure 2, our FlowFace maintains
the best face shape consistency. Note that most methods do nothing to transfer
the face shape, so their resulting face shapes are similar to the target ones.

Although HifiFace is specifically designed to change the face shape, our
method still obtains better results. As observed in Figure 2, our generated
face shapes are more similar to the source ones than HifiFace. Since HifiFace
injects the shape representation into the latent feature space, it is harder
to accurately decode the face shape from the latent feature than our explicit
semantic flow. Meanwhile, our method can better preserve the fine-grained
target expressions (marked with red boxes in rows 1,3).

Table 2: Subjective comparisons with SimSwap and HifiFace on FF++.

Method | Shape. ($\%$)$\uparrow$ | ID. ($\%$)$\uparrow$ | Exp. ($\%$)$\uparrow$ | Realism ($\%$)$\uparrow$
---|---|---|---|---
SimSwap | 20.67 | 27.78 | 34.44 | 14.67
HifiFace | 35.11 | 34.67 | 30.45 | 41.78
Ours | 44.22 | 37.55 | 35.11 | 43.55

Figure 5: Qualitative ablation results of each component in $F^{res}$. Figure
6: Qualitative ablation study of $F^{swa}$. Figure 7: Visualize the cross-
attention of different facial parts. For each part in the target, our CAFM can
accurately focus on the corresponding parts in the source.

We further compare our methods with four more SOTA face swapping methods: (Zhu
et al. 2021), FaceInpainter (Li et al. 2021), HighRes (Xu et al. 2022b) and
SmoothSwap (Kim, Lee, and Zhang 2022). Among them, MegaFS and HighRes are
based-on the latent space of StyleGAN2. As shown in Figure 3, our method can
better transfer the shape of the source to the target than all other methods.
Although SmoothSwap can change the face shape, it destroys the target
attributes (_e.g._ , hairstyle and hair color). Besides, our results are also
more similar to the source face in terms of inner facial features (_e.g._ ,
beard), validating that our face encoder can better capture facial appearances
than the identity embedding or the latent code of StyleGAN2. Moreover, our
method also preserves the target attributes (_e.g._ , skin color, lighting,
and expression) better than other methods.

#### User Study.

To further validate our FlowFace, we conduct a subjective comparison with
SimSwap and HifiFace, two SOTA methods that release their codes or results.
Fifteen participants are instructed to choose the best result in terms of
shape consistency, identity consistency, expression consistency, or image
realism, involving comparisons of 30 swapped faces by three methods. Table 2
shows that our method outperforms the two baselines in terms of all four
metrics, validating the superiority of our method.

### Analysis of FlowFace

Three ablation studies are conducted to validate our two-stage FlowFace
framework and several components used in $F^{res}$ and $F^{swa}$,
respectively.

#### Ablation Study on FlowFace.

We conduct ablation experiments to validate the design of our two-stage
framework. Figure 4 shows the swapped images by only $F^{res}$, only $F^{swa}$
and the full model (FlowFace). It can be seen that $F^{res}$ transforms the
face shape naturally according to the source, while $F^{swa}$ is good at
capturing the identity of the source inner face and other facial attributes of
the target. Benefiting from the strengths of both $F^{res}$ and $F^{swa}$, our
FlowFace is able to create results with accurate identity and consistent
facial attributes. Table 1 records the quantitative results, which further
illustrates the effectiveness of our two-stage framework. The above
observation validates the effectiveness of $F^{res}$ and confirms that face
shapes are essential for identifying a person.

To further validate our $F^{res}$, we plug it into the open-sourced SimSwap.
As shown in Figure 2 and Table 1, after reshaping by $F^{res}$, the face
swapping result of SimSwap is more similar to the source face in terms of face
contours. The ID Acc. also rises from 93.63% to 94.31%. The results
demonstrate the effectiveness of our $F^{res}$ and also reveal that the face
shape carries the identity information, thus improving identity similarity.

#### Ablation study on $F^{res}$.

We first conduct an ablation experiment to validate our proposed semantic
guided generator $G^{res}$. Specifically, we remove the semantic input $S_{t}$
of $G^{res}$ ($G^{res}$ w/o Seg). It can be seen from Figure 5 that some
inaccurate flow occurs in the generated face, which implies that only facial
landmarks cannot guide $G^{res}$ to produce accurate dense flow due to the
lack of semantic information. The results also demonstrate that the semantic
information is beneficial for accurate flow estimation and validates the
effectiveness of $G^{res}$.

Then, we conduct two ablation experiments to validate $D^{res}$: (1) removing
the semantic inputs ($S_{t}$ and $S_{t}^{res}$) of $D^{res}$ ($D^{res}$ w/o
Seg). Compared with $F^{res}$, the generated faces suffer from unnaturalness,
like the eyes are stretched, as observed in Figure 5. It implies that
structured information in the semantic inputs can provide more fine-grained
discriminative signals, thus enforcing $G^{res}$ to produce a more accurate
flow. (2) removing $D^{res}$ (w/o $D^{res}$). As observed in Figure 5,
compared with $F^{res}$, there are many artifacts in the generated images, and
the estimated flow also contains many noises. The above observation validates
the effectiveness of our proposed $D^{res}$.

Table 3: Quantitative ablation study of $F^{swa}$ on FF++.

Methods | ID Acc | Expr | Pose
---|---|---|---
CosFace | SphereFace | Avg
Addition | 99.38 | 99.44 | 99.41 | 0.43 | 4.90
AdaIN | 97.31 | 97.15 | 97.23 | 0.33 | 3.27
Id Embed. | 97.10 | 96.90 | 97.00 | 0.22 | 2.10
Vit Decoder | 98.44 | 97.73 | 98.09 | 0.23 | 2.80
$F^{swa}$ | 99.18 | 98.23 | 98.71 | 0.21 | 1.99

#### Ablation study on $F^{swa}$.

Three ablation experiments are conducted to evaluate the design of $F^{swa}$:

(1) Choices on CAFM, Addition and AdaIN. To verify the effectiveness of CAFM,
we compare with two other methods: Addition that directly adds the source
values to the target values; AdaIN that first averages source patch embeddings
and then injects it into the target feature map using AdaIN residual blocks.
As shown in Figure 6 and Table 3, Addition simply brings all information of
the source face to the target face, thus leading to severe pose and expression
mismatch. AdaIN impacts the non-face parts (_e.g._ , hair) due to its global
modulation. In contrast, $F^{res}$ with CAFM obtains a high ID Acc and
preserves the target attribute well, which proves that CAFM can accurately
extract identity information from the source face and adaptively infuse it
into the target counterpart.

To further validate the effectiveness of our CAFM, we visualize the cross
attention computed by CAFM. As shown in Figure 7, given a specific part
(marked by red boxes) of the target face, CAFM accurately focuses on the
corresponding parts of the source face, validating our CAFM can adaptively
transfer the identity information from the source patches to corresponding
target patches.

(2) Latent Representation vs. ID Embedding (ID Embed.). To verify the
superiority of using the latent representation of MAE, we train a new model
which adopts the identity embedding as the identity representation and employs
AdaIN as the injection method. As can be seen from Figure 6, ID Embed. misses
some fine-grained face appearances, such as eyes color, beard. In contrast,
$F^{swa}$ contains richer identity information and achieves higher ID Acc, as
shown in Tab 3.

(3) Convolutional Decoder vs. ViT Decoder (ViT Decoder). We try two different
decoders to find out the better one. As shown in Figure 6, the results of ViT
Decoder contains a lot of artifacts. In contrast, Convolutional Decoder
achieves realistic results with high fidelity.

##  5 Conclusion

This work proposes a semantic flow-guided two-stage framework, FlowFace, for
shape-aware face swapping. In the first stage, the face reshaping network
transfers the shape of the source face to the target face by warping the face
pixel-wisely using semantic flow. In the second stage, we employ a pre-trained
masked autoencoder to extract facial features that better capture facial
appearances and identity information. Then, we design a cross-attention fusion
module to better fuse the source and the target features, thus leading to
better identity preservation. Extensive quantitative and qualitative
experiments are conducted on in-the-wild faces, demonstrating that our
FlowFace outperforms the state-of-the-art significantly.

##  6 Acknowledgments

This work is supported by the 2022 Hangzhou Key Science and Technology
Innovation Program (No. 2022AIZD0054), and the Key Research and Development
Program of Zhejiang Province (No. 2022C01011), the ARC-Discovery grants
(DP220100800) and ARC-DECRA (DE230100477).

## References

  * Bitouk et al. (2008) Bitouk, D.; Kumar, N.; Dhillon, S.; Belhumeur, P.; and Nayar, S. K. 2008.  Face swapping: automatically replacing faces in photographs.  In _ACM SIGGRAPH 2008 papers_ , 1–8.
  * Blanz et al. (2004) Blanz, V.; Scherbaum, K.; Vetter, T.; and Seidel, H.-P. 2004.  Exchanging faces in images.  In _Computer Graphics Forum_ , volume 23, 669–676. Wiley Online Library.
  * Blanz and Vetter (1999) Blanz, V.; and Vetter, T. 1999.  A morphable model for the synthesis of 3D faces.  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , 187–194.
  * Cao et al. (2018) Cao, Q.; Shen, L.; Xie, W.; Parkhi, O. M.; and Zisserman, A. 2018.  Vggface2: A dataset for recognising faces across pose and age.  In _2018 13th international conference on automatic face & gesture recognition (FG 2018)_, 67–74. IEEE.
  * Chen et al. (2019) Chen, D.; Chen, Q.; Wu, J.; Yu, X.; and Jia, T. 2019.  Face swapping: realistic image synthesis based on facial landmarks alignment.  _Mathematical Problems in Engineering_ , 2019.
  * Chen et al. (2020) Chen, R.; Chen, X.; Ni, B.; and Ge, Y. 2020.  SimSwap: An Efficient Framework For High Fidelity Face Swapping.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , 2003–2011.
  * Choi et al. (2020) Choi, Y.; Uh, Y.; Yoo, J.; and Ha, J.-W. 2020.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 8188–8197.
  * DeepFakes (2019) DeepFakes. 2019.  DeepFakes.  https://github.com/deepfakes/faceswap.  Online; Accessed March 1, 2021.
  * Deng et al. (2019) Deng, J.; Guo, J.; Xue, N.; and Zafeiriou, S. 2019.  Arcface: Additive angular margin loss for deep face recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4690–4699.
  * Dosovitskiy et al. (2020) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Feng et al. (2020) Feng, Y.; Feng, H.; Black, M. J.; and Bolkart, T. 2020.  Learning an Animatable Detailed 3D Face Model from In-The-Wild Images.  _arXiv preprint arXiv:2012.04012_.
  * Gao et al. (2021) Gao, G.; Huang, H.; Fu, C.; Li, Z.; and He, R. 2021.  Information bottleneck disentanglement for identity swapping.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 3404–3413.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16000–16009.
  * Isola et al. (2017) Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A. 2017.  Image-to-image translation with conditional adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1125–1134.
  * Karras et al. (2017) Karras, T.; Aila, T.; Laine, S.; and Lehtinen, J. 2017.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_.
  * Karras, Laine, and Aila (2019) Karras, T.; Laine, S.; and Aila, T. 2019.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4401–4410.
  * Karras et al. (2020) Karras, T.; Laine, S.; Aittala, M.; Hellsten, J.; Lehtinen, J.; and Aila, T. 2020\.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8110–8119.
  * Kim, Lee, and Zhang (2022) Kim, J.; Lee, J.; and Zhang, B.-T. 2022.  Smooth-Swap: A Simple Enhancement for Face-Swapping with Smoothness.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10779–10788.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Li et al. (2021) Li, J.; Li, Z.; Cao, J.; Song, X.; and He, R. 2021.  FaceInpainter: High Fidelity Face Adaptation to Heterogeneous Domains.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5089–5098.
  * Li et al. (2019) Li, L.; Bao, J.; Yang, H.; Chen, D.; and Wen, F. 2019.  Faceshifter: Towards high fidelity and occlusion aware face swapping.  _arXiv preprint arXiv:1912.13457_.
  * Li et al. (2017) Li, T.; Bolkart, T.; Black, M. J.; Li, H.; and Romero, J. 2017.  Learning a model of facial shape and expression from 4D scans.  _ACM Transactions on Graphics, (Proc. SIGGRAPH Asia)_ , 36(6): 194:1–194:17.
  * Lim and Ye (2017) Lim, J. H.; and Ye, J. C. 2017.  Geometric gan.  _arXiv preprint arXiv:1705.02894_.
  * Lin et al. (2012) Lin, Y.; Lin, Q.; Tang, F.; and Wang, S. 2012.  Face replacement with large-pose differences.  In _Proceedings of the 20th ACM international conference on Multimedia_ , 1249–1250.
  * Liu et al. (2017a) Liu, W.; Wen, Y.; Yu, Z.; Li, M.; Raj, B.; and Song, L. 2017a.  Sphereface: Deep hypersphere embedding for face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 212–220.
  * Liu et al. (2017b) Liu, X.; Vijaya Kumar, B.; You, J.; and Jia, P. 2017b.  Adaptive deep metric learning for identity-aware facial expression recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 20–29.
  * MarekKowalski (2021) MarekKowalski, M. 2021.  FaceSwap.  [EB/OL].  https://github.com/MarekKowalski/FaceSwap Accessed March 1, 2021\.
  * Miyato et al. (2018) Miyato, T.; Kataoka, T.; Koyama, M.; and Yoshida, Y. 2018.  Spectral normalization for generative adversarial networks.  _arXiv preprint arXiv:1802.05957_.
  * Nirkin, Keller, and Hassner (2019) Nirkin, Y.; Keller, Y.; and Hassner, T. 2019.  Fsgan: Subject agnostic face swapping and reenactment.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 7184–7193.
  * Nirkin et al. (2018) Nirkin, Y.; Masi, I.; Tuan, A. T.; Hassner, T.; and Medioni, G. 2018.  On face segmentation, face swapping, and face perception.  In _IEEE International Conference on Automatic Face & Gesture Recognition_, 98–105. IEEE.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Kopf, A.; Yang, E.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , 8024–8035. Curran Associates, Inc.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _International Conference on Medical image computing and computer-assisted intervention_ , 234–241. Springer.
  * Rössler et al. (2019) Rössler, A.; Cozzolino, D.; Verdoliva, L.; Riess, C.; Thies, J.; and Nießner, M. 2019.  FaceForensics++: Learning to Detect Manipulated Facial Images.  In _International Conference on Computer Vision (ICCV)_.
  * Sun et al. (2019) Sun, K.; Zhao, Y.; Jiang, B.; Cheng, T.; Xiao, B.; Liu, D.; Mu, Y.; Wang, X.; Liu, W.; and Wang, J. 2019.  High-resolution representations for labeling pixels and regions.  _arXiv preprint arXiv:1904.04514_.
  * Thies et al. (2016) Thies, J.; Zollhofer, M.; Stamminger, M.; Theobalt, C.; and Nießner, M. 2016\.  Face2face: Real-time face capture and reenactment of rgb videos.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2387–2395.
  * Vemulapalli and Agarwala (2019) Vemulapalli, R.; and Agarwala, A. 2019.  A compact embedding for facial expression similarity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 5683–5692.
  * Wang et al. (2018) Wang, H.; Wang, Y.; Zhou, Z.; Ji, X.; Gong, D.; Zhou, J.; Li, Z.; and Liu, W. 2018\.  Cosface: Large margin cosine loss for deep face recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5265–5274.
  * Wang et al. (2021) Wang, Y.; Chen, X.; Zhu, J.; Chu, W.; Tai, Y.; Wang, C.; Li, J.; Wu, Y.; Huang, F.; and Ji, R. 2021.  HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping.  _arXiv preprint arXiv:2106.09965_.
  * Xu et al. (2022a) Xu, C.; Zhang, J.; Hua, M.; He, Q.; Yi, Z.; and Liu, Y. 2022a.  Region-Aware Face Swapping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7632–7641.
  * Xu et al. (2022b) Xu, Y.; Deng, B.; Wang, J.; Jing, Y.; Pan, J.; and He, S. 2022b.  High-resolution face swapping via latent semantics disentanglement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 7642–7651.
  * Xu et al. (2021) Xu, Z.; Yu, X.; Hong, Z.; Zhu, Z.; Han, J.; Liu, J.; Ding, E.; and Bai, X. 2021\.  FaceController: Controllable Attribute Editing for Face in the Wild.  _arXiv preprint arXiv:2102.11464_.
  * Zhang et al. (2021) Zhang, W.; Ji, X.; Chen, K.; Ding, Y.; and Fan, C. 2021.  Learning a Facial Expression Embedding Disentangled From Identity.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6759–6768.
  * Zhu et al. (2020) Zhu, H.; Fu, C.; Wu, Q.; Wu, W.; Qian, C.; and He, R. 2020.  AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection.  In _Neural Information Processing Systems (NeurIPS)_.
  * Zhu et al. (2021) Zhu, Y.; Li, Q.; Wang, J.; Xu, C.-Z.; and Sun, Z. 2021.  One Shot Face Swapping on Megapixels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 4834–4844.

## Supplementary Material

##  Appendix A Implementation Details

All models are implemented with PyTorch (Paszke et al. 2019). We train these
models in a two-stage manner. We first train the face reshaping network and
then the face swapping network. For both stages, we adopt Adam (Kingma and Ba
2014) optimizer with $\beta_{1}$=0 and $\beta_{2}$=0.99 and the learning rate
is set to 0.0001. For stable training, we apply spectral normalization (Miyato
et al. 2018) to discriminators in both stages. All experiments are conducted
on NVIDIA RTX A30 GPUs.

#### Face Reshaping Network

Our face reshaping network consists of a semantic guided generator and a
semantic guided discriminator. As shown in Figure 8, we only show the detailed
structure of the semantic guided generator, which is based on a U-net
(Ronneberger, Fischer, and Brox 2015). We design the semantic guided
discriminator by following PatchGAN (Isola et al. 2017). Specifically, we
change the input channel from 3 to 22, since the input of our semantic guided
discriminator contains an additional face segmentation map, which has 19
channels.

#### Face Swapping Network

As shown in Figure 9, we draw the details of each component in the face
swapping network, including the shared facial encoder $E_{f}$, the cross
attention fusion module $\operatorname{CAFM}$ and the facial decoder $D_{f}$.
The discriminator and the residual blocks are the same as StarGANv2 (Choi et
al. 2020). The transformer blocks are standard as in ViT (Dosovitskiy et al.
2020).

Figure 8:  The architecture of our semantic guided generator in the face
swapping network. We adopt 17 landmarks to represent the face shape and these
landmarks are transformed into heatmaps. The segmentation map is a tensor with
19 channels, representing 19 components in a face image. Therefore, the input
tensor of the generator has 53 channels.  Figure 9:  The architecture of our
face swapping network. We draw the details of each component in the face
swapping network, including the facial encoder $E_{f}$, the mask decoder
$D_{m}$, the facial decoder $D_{f}$ and the identity compound embedding
encoder $E_{c}$.

#### Face Reshaping Network Training.

We only sample 10000 face images from our constructed dataset for training,
and the 3DMM coefficients of these faces are extracted by the pre-trained DECA
model111https://github.com/YadiraF/DECA (Feng et al. 2020) offline. During
training, we first construct a batch size by randomly sampling eight source-
target image pairs and their corresponding coefficients. The probability for
$I_{s}=I_{t}$ is set to 10%. Then, the facial landmarks $P_{t}$ and $P_{s2t}$
are computed online. The facial segmentation maps are also obtained online
using the pre-trained face parsing model222https://github.com/zllrunning/face-
parsing.PyTorch. For the landmark loss, we adopt
HRNet333https://github.com/HRNet/HRNet-Facial-Landmark-Detection (Sun et al.
2019). The face reshaping network is trained for 25 epochs.

Figure 10:  More face swapping results of female faces. The images contain
various postures, lighting conditions, expressions, skin colors, and makeup.
Moreover, our FlowFace can also tackle artistic paintings. Please zoom in for
more details.  Figure 11:  More face swapping results of male faces.

#### Face Swapping Network Training.

We first train our face masked autoencoder, following the same settings in
MAE444https://github.com/facebookresearch/mae (He et al. 2022). Then, we
preserve the encoder part of MAE as our shared face encoder and fix it all the
time during training. We train our face swapping network for 30 epochs on our
constructed training set. The batch size is set to eight, and the probability
for $I_{s}=I_{t}$ is set to 25%.

##  Appendix B Additional Results

Figure 10 and Figure 11 illustrate more face swapping results generated by our
FlowFace. The images are captured under various conditions (e.g., postures,
expressions, skin colors, lighting conditions, makeup, etc.), and the faces
also come from various domains (e.g., human faces, cartoon faces, and art
painting faces).

Figure 12:  Some failure cases of FlowFace. Occlusions may lead to inaccurate
semantic flow in the face reshaping network. However, our face swapping
network preserves the occlusion well.

##  Appendix C Limitations and Future Work

Although our method achieves superior visual results to prior arts, it also
shares some limitations. As shown in Figure 12, our face reshaping network may
be influenced by some occlusions, even though the face swapping network can
well preserve the occlusions. However, this limitation can be addressed by the
refinement network as in (Li et al. 2019) or an occlusion-specific attribute
loss. Besides, the resultant video may suffer from flicker when our FlowFace
is applied to video sequences frame by frame. This is because the reshaping
network relies on 3D face reconstruction and facial landmarks, and neither of
them can preserve the temporal consistency. However, we will address the
temporal consistency problem in our future work by either a post-processing
operation or a new temporal consistent framework.


aainstitutetext: Department of Physics, Liaoning University, Shenyang 110036 ,
Chinabbinstitutetext: School of Physics, Beihang University, Beijing 100191,
Chinaccinstitutetext: CAS Center for Excellence in Particle Physics, Beijing
100049, China

#  $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ and its impact on $B_{s,d}\to\mu^{+}\mu^{-}$

Guang-Zhi Xu<EMAIL_ADDRESS>b , Yue Qiu<EMAIL_ADDRESS>b, c , Cheng-
Ping Shen<EMAIL_ADDRESS>b, c , Yu-Jie Zhang<EMAIL_ADDRESS>
###### Abstract

The decay of $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ and its impact on
$B_{s,d}\to\mu^{+}\mu^{-}$ is studied here. The $\mu^{+}\mu^{-}$ decay widths
of vector mesons $B^{*}_{s,d}$ are about a factor of 700 larger than the
corresponding scalar mesons $B_{s,d}$. The obtained ratio of the branching
fractions
$Br({B_{s,d}^{*}\to\mu^{+}\mu^{-}})/{Br({B_{s,d}\to\mu^{+}\mu^{-}})}$ is about
$\frac{0.3\times{\rm eV}}{{\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)}}$. At the
same time, the hadronic contribution $B_{s,d}\to
B^{*}_{s,d}\gamma\to\mu^{+}\mu^{-}$ is estimated too. The relative increase of
the amplitude of $B_{s,d}\to\mu^{+}\mu^{-}$ is about $(0.01\pm
0.006)\sqrt{\frac{{\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)}}{{100~{}{\rm eV}}}}$.
If we choose $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)=2~{}$eV, the branching
fractions of the vector mesons to lepton pair are $(6.2\pm 0.6)\times
10^{-10}$ and $(1.7\pm 0.2)\times 10^{-11}$ for $B^{*}_{s}$ and $B^{*}_{d}$
respectively. If we choose $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)=200~{}$eV,
the updated branching fractions of the scalar mesons to muon pair are
$(3.78\pm 0.25)\times 10^{-9}$ and $(1.09\pm 0.09)\times 10^{-10}$ for $B_{s}$
and $B_{d}$ respectively. Further studies on $B^{*}_{s,d}$ are usefully here,
including dielectron decay, two-body decay with $J/\psi$, and so on.

##  1 Introduction

The leptonic decays of the $B_{s,d}$ mesons play an important role in the
standard model (SM) and the new physics (NP) Bobeth:2013uxa ; CMS:2014xfa .
They are highly suppressed in the SM since the flavor changing neutral current
decays are generated through W-box and Z-penguin diagrams. Furthermore, the
branching fractions of leptonic decays of scalar meson undergo an additional
helicity suppression factor by $m_{\mu}^{2}/M_{S}^{2}$, where $m_{\mu}$ and
$M_{S}$ denote masses of the muon lepton and the scalar meson, respectively.
The suppression factor can be removed in some NP models, such as the two-Higgs
doublet models Cheng:2015yfu , the minimal supersymmetric standard model
(MSSM) Babu:1999hn , the next minimal supersymmetric standard model (NMSSM)
Li:2015dil , the dark matter Belanger:2015nma , the universal extra
dimensional model Datta:2015aka , lepton universality violation model
Glashow:2014iga , the four generations fermion Hou:2014nna , and so on
Arbey:2012ax . The branching fractions of $B_{s,d}\to\mu^{+}\mu^{-}$ measured
by the CMS and LHCb Collaborations CMS:2014xfa  and predicted within the SM
Bobeth:2013uxa  with NNLO QCD Hermann:2013kca  and NLO EW Bobeth:2013tba
corrections included are collected in Table 1.

Since the experimental branching fractions of $B_{s,d}\to\mu^{+}\mu^{-}$ are
measured from the dimuon distributions by the CMS and LHCb Collaborations
CMS:2014xfa , the process $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ will enhance the
dimuon distributions for the mass splitting are about 45 MeV between $B_{s,d}$
and $B^{*}_{s,d}$. In another hand, the hadronic contribution $B_{s,d}\to
B^{*}_{s,d}\gamma\to\mu^{+}\mu^{-}$ are missed in the theoretical prediction
Bobeth:2013uxa . So we study $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ and its impact on
$B_{s,d}\to\mu^{+}\mu^{-}$ within SM here. The process $B_{s}\to
B_{s}^{*}\gamma\to\mu^{+}\mu^{-}\gamma$ was considered in Ref. Aditya:2012im .
Recently, $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ are considered in Ref.
Grinstein:2015aua ; Khodjamirian:2015dda . And hadronic contribution from
charmonium in $B\to K^{(*)}\ell^{+}\ell^{-}$ and $B\to X_{s}\gamma$ had been
studied in Refs. Falk:1993dh ; Khodjamirian:2010vf .

Table 1: The branching fractions of $B_{s,d}\to\mu^{+}\mu^{-}$ measured by the CMS and LHCb Collaborations CMS:2014xfa  and predicted within the SM Bobeth:2013uxa  with NNLO QCD Hermann:2013kca  and NLO EW Bobeth:2013tba  corrections included.  | EX CMS:2014xfa  | SM Bobeth:2013uxa  | Deviations
---|---|---|---
$Br(B_{d}\to\mu^{+}\mu^{-})$ | $(3.9^{+1.6}_{-1.4})\times 10^{-10}$ | $(1.06\pm 0.09)\times 10^{-10}$ | $2.2\sigma$
$Br(B_{s}\to\mu^{+}\mu^{-})$ | $(2.8^{+0.7}_{-0.6})\times 10^{-9}$ | $(3.66\pm 0.23)\times 10^{-9}$ | $1.2\sigma$
$\frac{Br(B_{d}\to\mu^{+}\mu^{-})}{Br(B_{s}\to\mu^{+}\mu^{-})}$ | $0.14^{+0.08}_{-0.06}$ | $0.0295^{+0.0028}_{-0.0025}$ | $2.3\sigma$

##  2 The Decay of $B_{s}^{*}(B_{d}^{*})\to\mu^{+}\mu^{-}$

Within the SM, effective Lagrangian related with $b\bar{s}\to\mu^{+}\mu^{-}$
is given in Ref. Grinstein:2004vb ; Descotes-Genon:2013wba

$\displaystyle\mathcal{L}=\mathcal{N}\biggl{[}C_{7}^{eff}(\mu_{f})\mathcal{O}^{\gamma}_{7}+C_{9}(\mu_{f})\mathcal{O}^{V}_{9}+C_{10}(\mu_{f})\mathcal{O}^{A}_{10}\biggl{]},$
(1)

where
$\mathcal{N}=\frac{G_{F}}{\sqrt{2}}V_{tb}V^{*}_{ts}\frac{e^{2}}{4\pi^{2}}$,
and the operators $\mathcal{O}_{7,9,10}$ read as

$\displaystyle\mathcal{O}^{\gamma}_{7}$ $\displaystyle=$
$\displaystyle-\frac{2im_{b}(p_{\mu}^{\nu}+p_{\bar{\mu}}^{\nu})}{(p_{\mu}+p_{\bar{\mu}})^{2}}(\bar{s}\sigma_{\rho\nu}P_{R}b)(\bar{\mu}\gamma^{\rho}\mu),$
(2) $\displaystyle\mathcal{O}^{V}_{9}$ $\displaystyle=$
$\displaystyle(\bar{s}\gamma_{\rho}P_{L}b)(\bar{\mu}\gamma^{\rho}\mu),$ (3)
$\displaystyle\mathcal{O}^{A}_{10}$ $\displaystyle=$
$\displaystyle(\bar{s}\gamma_{\rho}P_{L}b)(\bar{\mu}\gamma^{\rho}\gamma_{5}\mu),$
(4)

where $P_{L}=(1-\gamma_{5})/2$ , $P_{R}=(1+\gamma_{5})/2$. And the Wilson
coefficients are $C_{7,9,10}(\mu_{f})=(-0.316,4.403-0.47i,-4.493)$ at
$\mu_{f}=m_{b}=4.5$ GeV Khodjamirian:2015dda . The superscript $\gamma$, $V$,
and $A$ denote the contributions from photon, vector current, and axial vector
current, respectively.

The relations between the quark level operators and the meson are described as

$\displaystyle\langle 0|\bar{s}\gamma^{\mu}b|B_{s}^{*}(q,\varepsilon)\rangle$
$\displaystyle=$ $\displaystyle
m_{B_{s}^{*}}\,f_{B_{s}^{*}}\,\varepsilon^{\mu},$ (5) $\displaystyle\langle
0|\bar{s}\sigma^{\mu\nu}b|B_{s}^{*}(q,\varepsilon)\rangle$ $\displaystyle=$
$\displaystyle-i\,f_{B_{s}^{*}}^{T}(q^{\mu}\varepsilon^{\nu}-\varepsilon^{\mu}q^{\nu}),$
(6) $\displaystyle\langle 0|\bar{s}\gamma^{\mu}\gamma_{5}b|B_{s}(q)\rangle$
$\displaystyle=$ $\displaystyle-if_{B_{s}}\,q^{\mu},$ (7)

where the three decay constants $f_{B_{s}},~{}f_{B_{s}^{*}}$ and
$f_{B_{s}^{*}}^{T}$ depend on the renormalization scale. Their relations have
been studied in Ref. Manohar:2000dt  in the heavy-quark limit. Ignoring the
mass difference between $B_{s}$ and $B_{s}^{*}$ and the high order QCD
corrections, we have

$\displaystyle f_{B_{s}^{*}}=f_{B_{s}^{*}}^{T}=\,f_{B_{s}}.$ (8)

Then the amplitudes of $B_{s}^{*}(B_{s})\to\mu^{+}\mu^{-}$ are Aditya:2012im

$\displaystyle\mathcal{M}({B_{s}^{*}\to\mu^{+}\mu^{-}})$ $\displaystyle=$
$\displaystyle
f_{B_{s}^{*}}\frac{\mathcal{N}}{2}m_{B_{s}^{*}}\bar{\mu}/{\varepsilon}\Big{[}C_{V}^{eff}+C_{10}\gamma_{5}\Big{]}\mu,$
$\displaystyle\mathcal{M}({B_{s}\to\mu^{+}\mu^{-}})$ $\displaystyle=$
$\displaystyle-if_{B_{s}}\mathcal{N}C_{10}m_{\mu}\bar{\mu}\gamma_{5}\mu,$ (9)

where

$\displaystyle C_{V}^{eff}=C_{9}+2\frac{m_{b}}{m_{B_{s}^{*}}}C_{7}^{eff}.$
(10)

The helicity suppression factor $m_{\mu}^{2}/m_{M}^{2}$ in the decay width is
removed in the vector meson decay. Then we can get the decay widths of
$B_{s}^{*}(B_{s})\to\mu^{+}\mu^{-}$

$\displaystyle\Gamma({B_{s}^{*}\to\mu^{+}\mu^{-}})$ $\displaystyle=$
$\displaystyle\frac{G_{f}^{2}\alpha_{em}^{2}}{96\pi^{3}}\left|V_{tb}V_{ts}^{*}\right|^{2}\Big{(}\left|C_{10}\right|^{2}+\left|C_{V}^{eff}\right|^{2})m_{B_{s}^{*}}^{3}f_{B_{s}^{*}}^{2}$
$\displaystyle\Big{(}1+{\cal{O}}(m_{\mu}^{2}/m_{B_{s}}^{2})\Big{)}$
$\displaystyle\Gamma({B_{s}\to\mu^{+}\mu^{-}})$ $\displaystyle=$
$\displaystyle\frac{G_{f}^{2}\alpha_{em}^{2}}{16\pi^{3}}\left|V_{tb}V_{ts}^{*}\right|^{2}\left|C_{10}\right|^{2}m_{\mu}^{2}m_{B_{s}}f_{B_{s}}^{2}$
(11) $\displaystyle\Big{(}1+{\cal{O}}(m_{\mu}^{2}/m_{B_{s}}^{2})\Big{)}$
$\displaystyle\frac{\Gamma({B_{s}^{*}\to\mu^{+}\mu^{-}})}{\Gamma({B_{s}\to\mu^{+}\mu^{-}})}$
$\displaystyle=$
$\displaystyle\frac{\left(\left|C_{10}\right|^{2}+\left|C_{V}^{eff}\right|^{2}\right)m_{B_{s}^{*}}^{3}f_{B_{s}^{*}}^{2}}{6\left|C_{10}\right|^{2}m_{\mu}^{2}m_{B_{s}}f_{B_{s}}^{2}}\Big{(}1+{\cal{O}}(m_{\mu}^{2}/m_{B_{s}}^{2})\Big{)}$

The ratio of decay width is about 700 for $B_{s}^{(*)}$ and $B_{d}^{(*)}$
both.

##  3 The impact of $B_{s}^{*}(B_{d}^{*})\to\mu^{+}\mu^{-}$ on
$B_{s}(B_{d})\to B_{s}^{*}(B_{d}^{*})\gamma\to\mu^{+}\mu^{-}$

Figure 1:  Feynman diagrams of $B_{s,d}\to
B^{*}_{s,d}\gamma^{*}\to\mu^{+}\mu^{-}$.

Meanwhile, $B^{*}_{s,d}$ will impact on the leptonic decay of $B_{s,d}$
through the loop contribution $B_{s,d}\to
B^{*}_{s,d}\gamma^{*}\to\mu^{+}\mu^{-}$. The Feynman diagrams are given in
Fig. 1. The vertex of $B_{s,d}\to B^{*}_{s,d}\gamma$ is given as the operator
Peskin:1995ev ; Ebert:2002xz

$\displaystyle\mathcal{M}_{B_{s}B_{s}^{*}\gamma}$ $\displaystyle=$
$\displaystyle\sum_{q=s,b}<B_{s}^{*}\gamma|iee_{q}\bar{q}(p_{\bar{q}})\gamma_{\mu}q(p_{q})|B_{s}>$
$\displaystyle=$
$\displaystyle\sum_{q=s,b}\varepsilon^{\mu}_{\gamma}p_{\gamma}^{\nu}<B_{s}^{*}|iee_{q}\bar{q}(p_{\bar{q}})\frac{i\sigma_{\mu\nu}}{2m_{q}}q(p_{q})|B_{s}>.$

We can simplify the matrix element
$<B_{s}^{*}|\bar{q}(p)\sigma_{\mu\nu}q(p)|B_{s}>$ with the procedure in Refs.
Beneke:2000wa ; Cheung:2014cka 111The decay constants defined of vector meson
in Eq. (5) is different with Eq.(44) in Ref. Beneke:2000wa  with an additional
$i$. ,

$\displaystyle\mathcal{M}_{B_{s}B_{s}^{*}\gamma}$ $\displaystyle=$
$\displaystyle\sum_{q=s,b}\frac{-ee_{q}}{2m_{q}}\varepsilon^{\mu}_{\gamma}p_{\gamma}^{\nu}<B_{s}^{*}|\bar{q}(p)\sigma_{\mu\nu}q(p)|B_{s}>$
(12) $\displaystyle=$ $\displaystyle
i\epsilon_{\mu\nu\alpha\beta}\varepsilon^{\mu}_{\gamma}p^{\nu}_{\gamma}\varepsilon^{\alpha}_{B^{*}_{s}}p^{\beta}_{B^{*}_{s}}\sum_{q=s,b}\left(\frac{ee_{q}}{m_{q}}\right){\cal{I}}.$

$\cal{I}$ is related with the wave functions of $B^{*}_{s}$ and $B_{s}$
Beneke:2000wa , and it is ${\cal{I}}=<B_{s}|j_{0}(p_{\gamma}r)|B_{s}^{*}>\sim
1$ in the non-relativistic limit Lahde:1999ih . We can rewrite Eq. (12) with

$\displaystyle\mathcal{M}_{B_{s}B_{s}^{*}\gamma}$ $\displaystyle=$
$\displaystyle
i\frac{g_{B_{s}B^{*}_{s}\gamma}}{m_{B_{s}^{*}}}\epsilon_{\mu\nu\alpha\beta}\varepsilon^{\mu}_{\gamma}p^{\nu}_{\gamma}\varepsilon^{\alpha}_{B^{*}_{s}}p^{\beta}_{B^{*}_{s}}$
(13)

Where the dimensionless vector-scalar-photon coupling constant
$g_{B_{s}B^{*}_{s}\gamma}$ is related with the magnetic moments of $b$ and $s$
quarks. And the phase factor $i$ is consistent with the amplitude of
$\gamma^{*}\to VP$ in Ref. Ma:2004qf .

There are ultraviolet (UV) logarithmical divergences in the evaluation of loop
integrals. Then we introduce a cut off regularization scheme for the UV
divergence integral

$\displaystyle\int\frac{d^{4}q}{(2\pi)^{4}}\frac{1}{\left(q_{i}^{2}-m_{i}^{2}\right)\left(q_{j}^{2}-m_{j}^{2}\right)}$
(14) $\displaystyle\to$
$\displaystyle\int\frac{d^{4}q}{(2\pi)^{4}}\left[\frac{1}{\left(q_{i}^{2}-m_{i}^{2}\right)\left(q_{j}^{2}-m_{j}^{2}\right)}\right.$
$\displaystyle\left.-\frac{1}{\left(q_{i}^{2}-(m_{i}+\Lambda)^{2}\right)\left(q_{j}^{2}-(m_{j}+\Lambda)^{2}\right)}\right],$

where $i,j=B_{s}^{*},~{}\gamma$ or $\mu$, and $q_{i}$ corresponds the momentum
of $i$ in loop. $\Lambda\ll M_{W}$ for the amplitude is UV finite when W boson
are involved. Since the hadronic contribution will be suppressed when
$\sqrt{q_{j}^{2}}-m_{j}\gg\Lambda_{QCD}$, $\Lambda$ is about several
$\Lambda_{QCD}$. The cut off regularization scheme is similar Pauli-Villars
regularization scheme but acts on two propagators. The Pauli-Villars
regularization scheme of the UV divergence integral is the same with the form
factor ${\cal F}$ which is introduced in the $B_{s}B^{*}_{s}\gamma$ vertex in
Ref. Li:2007au

$\displaystyle{\cal
F}=\left(\frac{\Lambda^{2}-m_{B^{*}_{s}}^{2}}{\Lambda^{2}-q_{B^{*}_{s}}^{2}}\right),$
(15)

for

$\displaystyle\frac{1}{q_{B^{*}_{s}}^{2}-m_{B^{*}_{s}}^{2}}{\cal
F}=\frac{1}{q_{B^{*}_{s}}^{2}-m_{B^{*}_{s}}^{2}}-\frac{1}{q_{B^{*}_{s}}^{2}-\Lambda^{2}}.$
(16)

But here it acts on the UV divergence term only and the two propagators. Then
the soft contribution will be maintained in our calculation.

The amplitude from $B_{s}\to B^{*}_{s}\gamma^{*}\to\mu^{+}\mu^{-}$ can be
written as

$\displaystyle\mathcal{M}({B_{s}\to B^{*}_{s}\gamma^{*}\to\mu^{+}\mu^{-}})$
$\displaystyle=$ $\displaystyle
ie\mathcal{N}g_{B_{s}B^{*}_{s}\gamma}R(\Lambda)C_{V}^{eff}f_{B_{s}^{*}}m_{\mu}\bar{\mu}\gamma_{5}\mu.$
(17)

$m_{\mu}$ reappears in the amplitude of leptonic decay of scalar meson. The
factor $R(\Lambda)$ serves as a function of the high energy cut is shown in
Fig.2. More information about the factor $R(\Lambda)$ are given in the
Appendix.

Figure 2:  $R(\Lambda)$ of $B_{s}\to\mu^{+}\mu^{-}$ defined in Eq.(17) as a
function of the cut off energy.

Only the $C_{10}$ term is taken into account in the calculation of the NNLO
QCD Hermann:2013kca  and NLO EW Bobeth:2013tba  corrections of
$B_{s,d}\to\mu^{+}\mu^{-}$ within the SM Bobeth:2013uxa . But the contribution
from $B_{s}^{*}$ is missed in the previous calculation, which is also related
with $C_{7}$ and $C_{9}$ terms.

Compared with Eq.(2), the previous amplitude is added a factor $F$,

$\displaystyle F({B_{s}^{*}})$ $\displaystyle=$
$\displaystyle\frac{\mathcal{M}({B_{s}\to
B^{*}_{s}\gamma^{*}\to\mu^{+}\mu^{-}})}{\mathcal{M}({B_{s}\to\mu^{+}\mu^{-}})}$
(18) $\displaystyle=$
$\displaystyle-\frac{C_{V}^{eff}f_{B_{s}^{*}}}{C_{10}f_{B_{s}}}{eg_{B_{s}B^{*}_{s}\gamma}}R(\Lambda)$

We can estimate $g_{B_{s}B^{*}_{s}\gamma}$ in several ways, including the
heavy-quark and chiral effective theories Cho:1992nt ; Amundson:1992yp  with
the radiative and pion transition widthes of $D^{*+}$, light cone QCD sum
rules Zhu:1998ih ; Zhu:1996qy , and the radiative M1 decay widths of
$B^{*}_{s}\to B_{s}\gamma$ from potential model Lahde:1999ih ; Goity:2000dk .
The radiative M1 decay width of $B^{*}_{s}\to B_{s}\gamma$ is

$\displaystyle
g_{B_{s}B^{*}_{s}\gamma}=-m_{B^{*}_{s}}\left(\frac{12\pi}{E_{\gamma}^{3}}\Gamma(B^{*}_{s}\to
B_{s}\gamma)\right)^{1/2}.$ (19)

The predicted M1 widths are $0.15-400~{}$eV and $10-300~{}$eV for
$B^{*}_{s}\to B_{s}\gamma$ and $B^{*}_{d}\to B_{d}\gamma$, respectively
Lahde:1999ih ; Goity:2000dk ; Ebert:2002xz ; Cheung:2014cka ; Cheung:2015rya .

##  4 Numerical Result

The parameters in the numerical calculation are chosen as Agashe:2014kda

$\displaystyle\Lambda$ $\displaystyle=$ $\displaystyle 1.2~{}{\rm GeV},$
$\displaystyle m_{b}$ $\displaystyle=$ $\displaystyle 4.2~{}{\rm GeV},$
$\displaystyle\alpha_{em}$ $\displaystyle=$ $\displaystyle 1/137.$ (20)

The branch fraction of $B^{*}_{s,d}$ weak decay is much less than the M1
decay, $\Gamma_{tot}(B^{*}_{s,d})\approx\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)$.
We can get the ratio

$\displaystyle\frac{Br({B_{s}^{*}\to\mu^{+}\mu^{-}})}{Br({B_{s}\to\mu^{+}\mu^{-}})}$
$\displaystyle=$ $\displaystyle(0.34\pm 0.03)\times\frac{\rm
eV}{\Gamma(B^{*}_{s}\to B_{s}\gamma)},$
$\displaystyle\frac{Br({B_{d}^{*}\to\mu^{+}\mu^{-}})}{Br({B_{d}\to\mu^{+}\mu^{-}})}$
$\displaystyle=$ $\displaystyle(0.33\pm 0.03)\times\frac{\rm
eV}{\Gamma(B^{*}_{d}\to B_{d}\gamma)}.$ (21)

The main uncertainty resulting from the value of $f_{B_{s,d}}^{*}$. The
distribution of the dimuon invariant mass measured by CMS and LHCb
Collaborations in Ref. CMS:2014xfa  should include the contributions of
$B^{*}_{s,d}\to\mu^{+}\mu^{-}$. If $\Gamma(B^{*}_{s}\to B_{s}\gamma)=1~{}$eV
Lahde:1999ih , we can get
$\frac{Br({B_{s}^{*}\to\ell^{+}\ell^{-}})}{Br({B_{s}\to\mu^{+}\mu^{-}})}=0.34$
for $\ell=e,~{}\mu$. Then $B_{s}^{*}\to e^{+}e^{-}$ may be searched by CMS and
LHCb experiments with larger data samples.

In another hand, if $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)\sim 100$ eV, we can
find that the amplitude of $B_{s,d}\to\mu^{+}\mu^{-}$ will be modified by the
contributions of $B_{s,d}^{*}$ with a factor

$\displaystyle F({B_{s,d}^{*}})$ $\displaystyle=$ $\displaystyle(0.011\pm
0.006)\sqrt{\frac{\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)}{100{\rm eV}}},$ (22)

The main uncertainty resulting from the value of $\Lambda$. Then the new
predictions of $\Gamma(B_{s,d}\to\mu^{+}\mu^{-})$ are

$\displaystyle Br(B_{s}\to\mu^{+}\mu^{-})$ $\displaystyle=$
$\displaystyle(36.6\pm 2.3)\times\left(1+(0.023\pm
0.012)\sqrt{\frac{\Gamma_{tot}(B^{*}_{s})}{100{\rm eV}}}\right)\times
10^{-10},$ $\displaystyle Br(B_{d}\to\mu^{+}\mu^{-})$ $\displaystyle=$
$\displaystyle(10.6\pm 0.9)\times\left(1+(0.023\pm
0.012)\sqrt{\frac{\Gamma_{tot}(B^{*}_{d})}{100{\rm eV}}}\right)\times
10^{-11}.$

If $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)=200$ eV, this factor will increase
the decay width $\Gamma(B_{s,d}\to\mu^{+}\mu^{-})$ by a factor $(3.3\pm
1.7)\%$, which is about a factor of 10 larger than the neglect NLO EW
correction factor $0.3\%$ at the decay width in Ref. Bobeth:2013uxa . And the
corresponded $g_{B_{s,d}B^{*}_{s,d}\gamma}=-1.5$, about a factor of 15 larger
than the $e_{q}e=-1/3\sqrt{4\pi\alpha_{em}}=-0.10$. The $\Gamma(B^{*}_{s,d}\to
B_{s,d}\gamma)$ may be measured through two-body decay $B^{*}_{s,d}\to
J/\psi+M$ by CMS and LHCb.

##  5 Summary

In summary, $B^{*}_{s,d}\to\mu^{+}\mu^{-}$ in the dimuon distributions and the
hadronic contribution $B_{s,d}\to B^{*}_{s,d}\gamma\to\mu^{+}\mu^{-}$ are
studied here. The $\mu^{+}\mu^{-}$ decay widths of vector mesons $B^{*}_{s,d}$
are about a factor of 700 larger than the corresponding scalar mesons
$B_{s,d}$. The obtained ratio of the branching fractions
$Br({B_{s,d}^{*}\to\mu^{+}\mu^{-}})/{Br({B_{s,d}\to\mu^{+}\mu^{-}})}$ is about
$0.3\times{\rm eV}/{\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)}$. At the same time,
the hadronic contribution $B_{s,d}\to B^{*}_{s,d}\gamma\to\mu^{+}\mu^{-}$ is
calculated too. The amplitude of $B_{s,d}\to\mu^{+}\mu^{-}$ is enhanced by a
factor of $0.01\sqrt{{\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)}/{100~{}{\rm
eV}}}$. If we choose $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)=2~{}$eV, the
branching fractions of the vector mesons to lepton pair are $5.3\times
10^{-10}$ and $1.6\times 10^{-11}$ for $B^{*}_{s}$ and $B^{*}_{d}$
respectively. If we choose $\Gamma(B^{*}_{s,d}\to B_{s,d}\gamma)=200~{}$eV,
the updated branching fractions of the scalar mesons to muon pair are
$(3.78\pm 0.25)\times 10^{-9}$ and $(1.09\pm 0.09)\times 10^{-10}$ for $B_{s}$
and $B_{d}$ respectively. Further studies on $B^{*}_{s,d}$ are usefully here,
including dielectron decay, two-body decay with $J/\psi$, and so on.

###### Acknowledgements.

We would like to thank K.T. Chao, Y.Q. Ma, C. Meng, Q. Zhao, and S.L. Zhu for
useful discussions. This work is supported by the National Natural Science
Foundation of China (Grants No. 11375021, 11575017), the New Century Excellent
Talents in University (NCET) under grant NCET-13-0030, the Major State Basic
Research Development Program of China (No. 2015CB856701), and the Fundamental
Research Funds for the Central Universities.

##  6 Appendix: $R(\Lambda)$

$R(\Lambda)$ of $B_{s}\to B^{*}_{s}\gamma^{*}\to\mu^{+}\mu^{-}$ defined in
Eq.(17) is given as:

$\displaystyle R(\Lambda)$ $\displaystyle=$
$\displaystyle\frac{1}{32\pi^{2}{m_{B_{s}}}^{2}m_{\mu}^{2}}\left\\{3{m_{B_{s}}}^{2}m_{\mu}^{2}-2m_{\mu}^{2}\left({m_{B_{s}}}^{2}-{m_{B^{*}_{s}}}^{2}\right)^{2}{C_{0}}\left({m_{B_{s}}}^{2},m_{\mu}^{2},m_{\mu}^{2},{m_{B^{*}_{s}}}^{2},0,m_{\mu}^{2}\right)\right.$
(23)
$\displaystyle\left.+{m_{B_{s}}}^{2}(m_{\mu}^{2}-{m_{B^{*}_{s}}}^{2})\left({B_{0}}\left(0,m_{\mu}^{2},{m_{B^{*}_{s}}}^{2}\right)-{B_{0}}\left(0,({\Lambda}+m_{\mu})^{2},({\Lambda}+{m_{B^{*}_{s}}})^{2}\right)\right)\right.$
$\displaystyle\left.+\left(m_{B_{s}}^{2}\left(2m_{\mu}^{2}+{m_{B^{*}_{s}}}^{2}\right)+2m_{\mu}^{2}{m_{B^{*}_{s}}}^{2}\right)\left({B_{0}}\left(m_{\mu}^{2},m_{\mu}^{2},{m_{B^{*}_{s}}}^{2}\right)-{B_{0}}\left(m_{\mu}^{2},(\Lambda+m_{\mu})^{2},({\Lambda}+{m_{B^{*}_{s}}})^{2}\right)\right)\right.$
$\displaystyle\left.+m_{\mu}^{2}\left(3{m_{B_{s}}}^{2}-2{m_{B^{*}_{s}}}^{2}\right)\left({B_{0}}\left(m_{\mu}^{2},0,m_{\mu}^{2}\right)-{B_{0}}\left(m_{\mu}^{2},{\Lambda}^{2},(\Lambda+m_{\mu})^{2}\right)\right)\right\\}$

The scalar functions ${B_{0}}$ and ${C_{0}}$ are given in Ref. 'tHooft:1978xw
; vanOldenborgh:1989wn ; vanOldenborgh:1990yc . As a numerical fit between
$0.5-2$ GeV, we can get

$\displaystyle R(\Lambda)=0.022+0.062\times
ln(\frac{\Lambda+m_{B_{s}}}{m_{B_{s}}}).$ (24)

## References

  * (1) C. Bobeth, M. Gorbahn, T. Hermann, M. Misiak, E. Stamou, and M. Steinhauser, $B_{s,d}\to l^{+}l^{-}$ in the Standard Model with Reduced Theoretical Uncertainty, Phys. Rev. Lett. 112 (2014) 101801, [ arXiv:1311.0903].
  * (2) LHCb, CMS Collaboration, V. Khachatryan et al., Observation of the rare $B^{0}_{s}\to\mu^{+}\mu^{-}$ decay from the combined analysis of CMS and LHCb data, Nature 522 (2015) 68–72, [ arXiv:1411.4413].
  * (3) X.-D. Cheng, Y.-D. Yang, and X.-B. Yuan, Revisiting $B_{s}\to\mu^{+}\mu^{-}$ in the two-Higgs doublet models with $Z_{2}$ symmetry,  arXiv:1511.01829.
  * (4) K. S. Babu and C. F. Kolda, Higgs mediated $B^{0}\to\mu^{+}\mu^{-}$ in minimal supersymmetry, Phys. Rev. Lett. 84 (2000) 228, [ hep-ph/9909476].
  * (5) T. Li, S. Raza, and X.-C. Wang, The Super-Natural Supersymmetry and Its Classic Example: M-Theory Inspired NMSSM,  arXiv:1510.06851.
  * (6) G. B langer, C. Delaunay, and S. Westhoff, A Dark Matter Relic From Muon Anomalies, Phys. Rev. D92 (2015) 055021, [ arXiv:1507.06660].
  * (7) A. Datta and A. Shaw, Non-minimal UED confronts $B_{s}\rightarrow\mu^{+}\mu^{-}$,  arXiv:1506.08024.
  * (8) S. L. Glashow, D. Guadagnoli, and K. Lane, Lepton Flavor Violation in $B$ Decays?, Phys. Rev. Lett. 114 (2015) 091801, [ arXiv:1411.0565].
  * (9) W.-S. Hou, M. Kohda, and F. Xu, Correlating $B_{q}^{0}\to\mu^{+}\mu^{-}$ and $K_{L}\to\pi^{0}\nu\bar{\nu}$ Decays with Four Generations, Phys. Lett. B751 (2015) 458, [ arXiv:1411.1988].
  * (10) A. Arbey, M. Battaglia, F. Mahmoudi, and D. Mart nez Santos, Supersymmetry confronts $B_{s}\to\mu^{+}\mu^{-}$ : Present and future status, Phys. Rev. D87 (2013) 035026, [ arXiv:1212.4887].
  * (11) T. Hermann, M. Misiak, and M. Steinhauser, Three-loop QCD corrections to $B_{s}\to\mu^{+}\mu^{-}$, JHEP 12 (2013) 097, [ arXiv:1311.1347].
  * (12) C. Bobeth, M. Gorbahn, and E. Stamou, Electroweak Corrections to $B_{s,d}\to\ell^{+}\ell^{-}$, Phys. Rev. D89 (2014) 034023, [ arXiv:1311.1348].
  * (13) Y. G. Aditya, K. J. Healey, and A. A. Petrov, Faking $B_{s}\to\mu^{+}\mu^{-}$, Phys. Rev. D87 (2013) 074028, [ arXiv:1212.4166].
  * (14) B. Grinstein and J. M. Camalich, Weak decays of unstable $b$-mesons,  arXiv:1509.05049.
  * (15) A. Khodjamirian, T. Mannel, and A. A. Petrov, Direct probes of flavor-changing neutral currents in $e^{+}e^{-}$ collisions,  arXiv:1509.07123.
  * (16) A. F. Falk, M. E. Luke, and M. J. Savage, Nonperturbative contributions to the inclusive rare decays $B\to X_{s}\gamma$ and $B\to X_{s}\ell^{+}\ell^{-}$, Phys. Rev. D49 (1994) 3367, [ hep-ph/9308288].
  * (17) A. Khodjamirian, T. Mannel, A. A. Pivovarov, and Y. M. Wang, Charm-loop effect in $B\to K^{(*)}\ell^{+}\ell^{-}$ and $B\to K^{*}\gamma$, JHEP 09 (2010) 089, [ arXiv:1006.4945].
  * (18) B. Grinstein and D. Pirjol, Exclusive rare $B\to K^{*}e^{+}e^{-}$ decays at low recoil: Controlling the long-distance effects, Phys. Rev. D70 (2004) 114005, [ hep-ph/0404250].
  * (19) S. Descotes-Genon, J. Matias, and J. Virto, Understanding the $B\to K^{*}\mu^{+}\mu^{-}$ Anomaly, Phys. Rev. D88 (2013) 074002, [ arXiv:1307.5683].
  * (20) A. V. Manohar and M. B. Wise, Heavy quark physics, Camb. Monogr. Part. Phys. Nucl. Phys. Cosmol. 10 (2000) 1.
  * (21) M. E. Peskin and D. V. Schroeder, An Introduction to quantum field theory.  1995\.
  * (22) D. Ebert, R. N. Faustov, and V. O. Galkin, Radiative M1 decays of heavy light mesons in the relativistic quark model, Phys. Lett. B537 (2002) 241, [ hep-ph/0204089].
  * (23) M. Beneke and T. Feldmann, Symmetry breaking corrections to heavy to light B meson form-factors at large recoil, Nucl. Phys. B592 (2001) 3, [ hep-ph/0008255].
  * (24) C.-Y. Cheung and C.-W. Hwang, Strong and radiative decays of heavy mesons in a covariant model, JHEP 04 (2014) 177, [ arXiv:1401.3917].
  * (25) T. A. Lahde, C. J. Nyfalt, and D. O. Riska, Spectra and M1 decay widths of heavy light mesons, Nucl. Phys. A674 (2000) 141, [ hep-ph/9908485].
  * (26) J. P. Ma and Z. G. Si, Predictions for $e^{+}e^{-}\to J/\psi\eta_{c}$ with light-cone wave- functions, Phys. Rev. D70 (2004) 074007, [ hep-ph/0405111].
  * (27) G. Li, Q. Zhao, and B.-S. Zou, Isospin violation in phi, $J/\psi$, $\psi^{\prime}\to\omega\pi^{0}$ via hadronic loops, Phys. Rev. D77 (2008) 014010, [ arXiv:0706.0384].
  * (28) P. L. Cho and H. Georgi, Electromagnetic interactions in heavy hadron chiral theory, Phys. Lett. B296 (1992) 408, [ hep-ph/9209239]. [Erratum: Phys. Lett. B300 (1993) ,410].
  * (29) J. F. Amundson, C. G. Boyd, E. E. Jenkins, M. E. Luke, A. V. Manohar, J. L. Rosner, M. J. Savage, and M. B. Wise, Radiative D* decay using heavy quark and chiral symmetry, Phys. Lett. B296 (1992) 415, [ hep-ph/9209241].
  * (30) S.-L. Zhu and Y.-B. Dai, Radiative decays of heavy hadrons from light cone QCD sum rules in the leading order of HQET, Phys. Rev. D59 (1999) 114015, [ hep-ph/9810243].
  * (31) S.-L. Zhu, W.-Y. P. Hwang, and Z.-s. Yang, $D^{*}\to D\gamma$ and $B^{*}\to B\gamma$ as derived from QCD sum rules, Mod. Phys. Lett. A12 (1997) 3027, [ hep-ph/9610412].
  * (32) J. L. Goity and W. Roberts, Radiative transitions in heavy mesons in a relativistic quark model, Phys. Rev. D64 (2001) 094007, [ hep-ph/0012314].
  * (33) C.-Y. Cheung and C.-W. Hwang, Three symmetry breakings in strong and radiative decays of strange heavy mesons,  arXiv:1508.07686.
  * (34) Particle Data Group, K. A. Olive et al., Review of Particle Physics, Chin. Phys. C38 (2014) 090001.
  * (35) G. ’t Hooft and M. J. G. Veltman, Scalar One Loop Integrals, Nucl. Phys. B 153(1979) 365 .
  * (36) G. J. van Oldenborgh and J. A. M. Vermaseren, New Algorithms for One Loop Integrals, Z. Phys. C 46 (1990) 425.
  * (37) G. J. van Oldenborgh, FF: A Package to evaluate one loop Feynman diagrams, Comput. Phys. Commun.  66 (1991) 1.