# SARA: Controllable Makeup Transfer with Spatial Alignment and Region-
Adaptive Normalization

Xiaojing Zhong†, Xinyi Huang†, Zhonghua Wu, Guosheng Lin∗, Qingyao Wu∗ †Co-
first author.∗Corresponding author.Xiaojing Zhong, XinyiHuang and Qingyao Wu
are with School of Software Engineering, South China University of Technology,
Guangzhou, China. (email<EMAIL_ADDRESS><EMAIL_ADDRESS>qyw@scut.edu.cn)Zhonghua Wu and Guosheng Lin are with School of Computer
Science and Engineering, Nanyang Technological University, Singapore. (email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Makeup transfer is a process of transferring the makeup style from a reference
image to the source images, while preserving the source images’ identities.
This technique is highly desirable and finds many applications. However,
existing methods lack fine-level control of the makeup style, making it
challenging to achieve high-quality results when dealing with large spatial
misalignments. To address this problem, we propose a novel Spatial Alignment
and Region-Adaptive normalization method (SARA) in this paper. Our method
generates detailed makeup transfer results that can handle large spatial
misalignments and achieve part-specific and shade-controllable makeup
transfer. Specifically, SARA comprises three modules: Firstly, a spatial
alignment module that preserves the spatial context of makeup and provides a
target semantic map for guiding the shape-independent style codes. Secondly, a
region-adaptive normalization module that decouples shape and makeup style
using per-region encoding and normalization, which facilitates the elimination
of spatial misalignments. Lastly, a makeup fusion module blends identity
features and makeup style by injecting learned scale and bias parameters.
Experimental results show that our SARA method outperforms existing methods
and achieves state-of-the-art performance on two public datasets.

###### Index Terms:

Makeup transfer, Style transfer, Generative models, Correspondence matching,
Normalization.

††publicationid: pubid: 0000–0000/00$00.00 © 2021 IEEE

##  I Introduction

Given a facial image with a certain makeup style, the makeup transfer task
aims to transfer the style to a target face. This is widely demanded in many
real-world scenarios where users would like to beautify themselves. Although
makeup and non-makeup images can be easily obtained from the internet, it is
not easy to obtain paired images because an absolutely identical face equipped
with different makeup styles is unavailable. To address this issue, one
possible way [1, 2, 3, 4] is to capitalize on the idea of CycleGAN [5] to
construct the training in a cyclic manner by bridging two networks, one that
transfers makeup styles and another that removes makeup.

Figure 1: SARA supports flexible operations. (a) Compared to SCGAN [6], SARA
better deals with large spatial misalignment by incorporating spatial
information into the transfer framework. The highlights are the unnatural
results generated by SCGAN. (b) Users are allowed to select partial makeup
styles from various reference images. (c) SARA can adjust the degree of the
transferred makeup styles.

However, existing makeup transfer approaches often treat different semantic
regions equally, which can limit fine-grained control over makeup styles - an
essential factor in achieving realistic makeup transfer results. Recent
advancements [7, 8, 9, 10] in style transfer, such as the control of affine
transformation parameters in AdaIN layers [11], have shown great potential in
virtual try-on applications [12, 13, 14], and we believe that these techniques
can be extended to virtual makeup transfer as well. For example, Deng et al[6]
propose a method for separating semantic regions and arranging corresponding
style codes to empower AdaIN layers with spatial control. However, a
decoupling of shape and style without proper preservation of spatial context
can result in unnatural-looking makeup transfer (as seen in Fig. 1). [15, 16]
compute the local correlation of key facial landmarks, but the resulting
attention map is often too sparse to handle complex makeup styles.

To overcome the mentioned limitations, we propose a novel makeup transfer
framework called Controllable Makeup Transfer with Spatial Alignment and
Region-Adaptive normalization (SARA), which comprises three modules.

Method | Property
---|---
Shade | Part | Misalign | Landmarks free
BGAN[1] | ✗ | ✗ | ✗ | ✔
BGlow[17] | ✔ | ✗ | ✗ | ✔
LADN[18] | ✔ | ✗ | ✗ | ✗
PSGAN [15] | ✔ | ✔ | ✔ | ✗
PSGAN++[16] | ✔ | ✔ | ✔ | ✗
SSAT[19] | ✔ | ✔ | ✗ | ✔
Ours | ✔ | ✔ | ✔ | ✔
TABLE I: ”Shade” refers to methods that allow for the control of makeup shade
during transfer. ”Part” indicates the ability to perform partial transfer.
”Misalign” refers to methods that are robust even when there is a significant
spatial misalignment between the source and reference images. ”Landmarks free”
describes methods that can perform makeup transfer without relying on facial
landmarks.

First, a Spatial Alignment Module (SAM) is introduced to warp the reference
image (with makeup) to be aligned with the source image (with non-makeup) to
preserve the spatial context relationship. The matching correspondence is
densely calculated in a pixel-to-pixel manner, beneficial for dealing with the
makeup over a large area. Besides, the computed correspondence matrix is also
used to generate the target semantic map, which serves as the broadcast mask
in the following normalization. Due to the fact that the coupling of the shape
and style of makeup is not conducive to accurate transfer, a Region-Adaptive
normalization Module (RAM) is proposed to extract the per-region styles from
the reference image and broadcast the style codes to the target semantic map.
By incorporating the aligned features generated from SAM, RAM dynamically
combines the shape-independent styles with the spatial context. Final, a
Makeup Fusion Module (MFM) is responsible for allocating the learned scale and
bias parameters to the fusion blocks, progressively fusing the identity
features with the makeup styles to render the fine-grained results.

Overall, SARA is capable of achieving flexible makeup transfer controls within
one unified framework. (i) SARA can deal with different poses and expressions
between reference and source images, while many existing methods merely work
well on frontal facial images and the same expressions. (ii) SARA can perform
high-quality makeup removal when the reference and source images are inverse.
(iii) SARA can realize partial makeup transfer in the scenario where users
would like to transfer one part, such as the lip or eyebrows. (iv) SARA can
adjust the degree of makeup styles from light to heavy. Tab I demonstrates the
properties of various makeup transfer methods in summary.

Our contributions can be summarized as follows:

  * •

We design a unified framework SARA for the controllable makeup transfer tasks,
which supports partial, shade-control makeup transfer and removal at the same
time. With our carefully designed three main modules, our proposed SARA
achieves state-of-the-art performance on the MT and M-wild datasets.

  * •

We propose to incorporate spatial alignment into region-adaptive normalization
to simultaneously eliminate the spatial gap between the reference style and
the source identity and preserve the spatial context relationship.

  * •

We estimate a correspondence matrix in a pixel-to-pixel manner to realize the
spatial alignment, which is dense enough to represent detailed makeup styles.
The matrix is also used to guide the style codes broadcasted to the target
semantic region.

##  II Related Work

Figure 2: Overview of our proposed method. It mainly has three modules: (i)
Spatial Alignment Module (SAM) warps the reference image $y_{r}$ and the
partial reference semantic map $y_{ri}^{l}$ to be aligned with $x_{s}^{l}$
through estimating the dense correspondence between $y_{r}$ and the source
semantic map $x_{s}^{l}$, where $i=\\{lip,skin,eyes\\}$. (ii) Region-Adaptive
normalization Module (RAM) decouples the shape and style of makeup via a
region-wise average pooling layer, broadcasting the shape-independent style
codes to the target semantic map $W_{y_{ri}^{l}}$ to generate the style matrix
$ST$. Besides, it incorporates $W_{y_{r}}^{out}$ into the normalization, which
provides spatial context relationship during the transfer. (iii) Makeup Fusion
Module (MFM) modulates scale and bias parameters of the fusion blocks in
multi-level, progressively fusing makeup styles with the identity features, to
generate the fine-grained result $\hat{y}_{s}$.

###  II-A Makeup Transfer

Makeup transfer has gained significant interest in the last decade [20, 21,
22, 23], with several methods proposed for transferring makeup styles from
reference images to source images. Some of the popular methods include
CycleGAN [5], which is an influential method for learning domain-to-domain
translation, but it lacks direct constraints between input and output images,
and PairedCycleGAN [2], which extends CycleGAN by introducing a paired cycle
GAN for makeup transfer and removal. Li et al. tackle the problem by designing
a dual input/output GAN to train makeup transfer and removal jointly [1]. They
also propose to measure similarity by matching the color histograms of makeup
regions between images.

Other studies, such as [24, 18], focus on dramatic makeup styles, while lyu et
al. introduce a 3D-aware GAN to unwarp the texture of the face and refine it
based on the symmetry of the face to solve the shadow and occlusion problem
[25]. Wan et al. improve the fidelity of local regions sensitive to color by
consolidating the transformer architecture into makeup transfer [26], and [19]
facilitates the render quality in the correct position. However, these methods
lack user control over the transfer process. To address this, Chen et al.
propose BeautyGlow [17] to disentangle makeup style and facial identity of
latent vectors, enabling flexible control of style degree. PSGAN [15] and
improved PSGAN++ [16] propose to handle the spatial misalignment problem
facing non-frontal images by incorporating visual appearances and locations
into the attention mechanism. Our proposed method, SARA, performs better in
several aspects. First, we depart from the need for facial landmarks that are
easily inaccurate and computational-expensive. Second, local correspondence
cannot model the global context relationship. EleGant [27] uses a pyramid
structure with a high-resolution feature map to preserve high-frequency makeup
features beyond color distributions, but it lacks careful consideration of
large misalignment. SCGAN [6] separates facial component attributes via
semantic layouts, which are mapped into latent space and recombined in a
specific order. However, decoupling the shape and style of makeup without any
consideration of space may lead to unnatural artifacts [28, 29]. Therefore, we
propose incorporating spatial alignment to retain the context that provides
details.

###  II-B Style Transfer

Conventional style transfer methods rely on hand-crafted algorithms to render
an image in fixed styles [30, 31], or match two images with hand-crafting
features [32, 33]. More recent advances in style transfer have shifted towards
the use of deep learning techniques, specifically the increasingly effective
generative models [34, 35, 36]. AdaIN [11] and DIN [37] use conditional
instance normalization to align content and style feature statistics, but
dynamic generation of affine parameters may cause distortion artifacts. Other
methods use an autoencoder-based framework with feature transformation and/or
fusion [8, 38, 39, 40]. These models have shown impressive results in
generating realistic images in a given style or transferring the style of one
image onto another [41]. Importantly, controlling the output of style transfer
is vital for downstream applications. One approach is to decompose the global
attributes of images, such as hue, saturation, and luminance [42]. In contrast
to global style control, another sort of style control is spatial [43, 44],
which lets the user specify which areas of the output should be rendered
exclusively with features from a hand-picked region of the style image.
Despite that, there exists a gap between the generated and ideal results when
applying existing style transfer methods for makeup transfer, as the latter
are sensitive to semantic details and require more exquisite control.

##  III Methodology

###  III-A Problem Formulation and Notations

Let X and Y denote the domain of the images with non-makeup and the images
with makeup, respectively. Given a source image $x_{s}\in X$ and a reference
image $y_{r}\in Y$, our goal is to learn a mapping function:
$f(x_{s},y_{r}\rightarrow\hat{y}_{s})$, where $\hat{y}_{s}$ has the same
makeup style as $y_{r}$ while preserving the facial identity of $x_{s}$.
Furthermore, as makeup removal is a special case of makeup transfer, we also
learn a mapping function: $\hat{f}(y_{r},x_{s}\rightarrow\hat{x}_{r})$, where
$\hat{x}_{r}$ has the same makeup style as $x_{s}$ while preserving the facial
identity of $y_{r}$.

Figure 3: Illustration of the correspondence matching process.
$\mathcal{F}_{x}$ and $\mathcal{F}_{y}$ refer to the feature extraction
operators in $x_{s}^{l}$ and $y_{r}$, respectively. Then we flatten the
generated high-level features $f_{x}$ and $f_{y}$ and further normalize them
along the channel dimension. We estimate the dense correspondence matrix
$C\in\mathbb{R}^{HW\times HW}$ through the multiplication of matrices, also
regarded as the attention map.

###  III-B Network Structure

The overall framework of SARA is shown in Fig. 2, which consists of three
modules: a Spatial Alignment Module (SAM), a Region-Adaptive normalization
Module (RAM) and a Makeup Fusion Module (MFM). The content of each module will
be introduced in detail (Sec III-B1 $\sim$ Sec III-B3). The objective
functions of the whole model will be described in Sec III-C.

####  III-B1 Spatial Alignment Module

The spatial context relationship retains fine-level details of the makeup
style and should not be disregarded during the transfer. Warping the reference
style to be aligned with the source identity preserves the spatial context
while mitigating the spatial misalignment. SAM starts with a reference image
$y_{r}$ and a source semantic map $x_{s}^{l}$, which is obtained by running an
off-the-shelf face parsing network [45] on $x_{s}$. We estimate a dense
correspondence matrix between them. Fig. 3 shows the process of correspondence
matching.

To be specific, we adopt two independent feature extraction operators,
$\mathcal{F}_{x}(x_{s}^{l})$ and $\mathcal{F}_{y}(y_{r})$, to obtain high-
level features $f_{x}\in\mathbb{R}^{H\times W\times C}$ and
$f_{y}\in\mathbb{R}^{H\times W\times C}$, respectively. We leverage cosine
similarity to match the aggregated features, which has been proven useful in
calculating pixel-wise semantic relevance [46, 47]. Then we flatten $f_{x}$
and $f_{y}$ to be $f_{x}^{{}^{\prime}}\in\mathbb{R}^{C\times HW}$ and
$f_{y}^{{}^{\prime}}\in\mathbb{R}^{C\times HW}$, respectively. Following the
same setting as [48], the correspondence matrix $C\in\mathbb{R}^{HW\times HW}$
can be computed as:

$C(i,j)=\frac{(f_{x}^{{}^{\prime}}(i)-u_{x})^{T}(f_{y}^{{}^{\prime}}(j)-u_{y})}{\|(f_{x}^{{}^{\prime}}(i)-u_{x})\|\|(f_{y}^{{}^{\prime}}(j)-u_{y})\|},$
(1)

where $u_{x}$ and $u_{y}$ represent the corresponding mean vectors along the
channel dimension. i and j represent $f_{x}^{{}^{\prime}}(i)$ at position i
and $f_{y}^{{}^{\prime}}(j)$ at position j.

Accordingly, SAM is able to warp $y_{r}$ under the guidance of the dense
correspondence matrix $C$. The warped image $W_{y_{r}}(u)$ can be represented
as:

$W_{y_{r}}(u)=\sum\limits_{v}\mathop{softmax}\limits_{v}(\alpha C(u,v)\cdot
y_{r}(v)),$ (2)

$\alpha$ is a hyper-parameter that controls result sharpness and is set at
100.

Rather than using the global makeup style, we extract three specific
components of $W_{y_{r}}$, which are used for embodying makeup generally, to
realize partial makeup transfer. To extract the corresponding regions, we warp
the partial semantic mask $y_{ri}^{l}$ as below:

$W_{y_{ri}^{l}}(u)=\sum\limits_{v}\mathop{softmax}\limits_{v}(\alpha
C(u,v)\cdot y_{ri}^{l}(v)),$ (3)

where $i=\left\\{lip,skin,eyes\right\\}$. The filtered makeup regions can be
obtained as follows:

$W_{y_{r}}^{out}=W_{y_{ri}^{l}}\odot W_{y_{r}}$ (4)

where $\odot$ denotes element-wise multiplication for each pixel. Besides,
$W_{y_{ri}^{l}}$ is utilized to aid in the target semantic generation in the
per-region normalization. SAM is able to warp the makeup region that covers a
large area, like blush, since it models the global correspondence.

####  III-B2 Region-Adaptive normalization Module

Style can be regarded as the embedded codes to be filled into the affine
transformation parameters of the normalization [6, 49, 7, 12], independent of
shape. The decoupling of shape and style contributes to the high-quality color
transfer of the makeup. Inspired by [49], we takes the per-region styles
extracted from the source appearance and then broadcast them to the target
semantic regions on each facial part, which can be solved by per-region
encoding and normalization, respectively.

Figure 4: Details of region-adaptive normalization. The scale and bias
parameters $\alpha$ and $\beta$ are weighted from the combined warped out
$W^{out}_{y_{r}}$ and the style matrix $ST$.

Per-region encoding. We adopt the reference features $F_{r}$ obtained from the
spatial alignment module instead of the original reference image $y_{r}$ as
input. Intuitively, we contend that this could serve in maintaining the
consistency and continuity of the original features since using external data
may break it. To extract per-region style codes, we construct a style matrix
$ST\in\mathbb{R}^{256\times s}$ through conducting a region-wise average
pooling layer on the $F_{r}$, where s denotes the dimensional concatenation of
lip,skin and eyebrow. Each column of the matrix corresponds to the style code
of a semantic region. We further broadcast the style codes to their target
region to generate the target style map.

Per-region normalization. To be specific, we set $h^{i}\in\mathbb{R}^{B\times
C^{i}\times H^{i}\times W^{i}}$ as the activation value before the i-th
normalization layer, where $B$,$C^{i}$,$H^{i}$,$W^{i}$ represent batch size,
number of channels, height and width of the i-th feature map respectively. The
mean $\mu^{i}_{c}$ and standard deviation $\sigma^{i}_{c}$ are calculated in
the channel-wise manner. Then, the output of RAM in the i-th layer is given
by:

$h^{i+1}_{b,c,h,w}=\alpha^{i}_{c,h,w}\frac{h_{b,c,h,w}^{i}-\mu^{i}_{c}}{\sigma^{i}_{c}}+\beta^{i}_{c,h,w},$
(5)

where $\alpha^{i}_{c,h,w}$ and $\beta^{i}_{c,h,w}$ are the learnable
parameters. As shown in the Fig. 4, they are the weighted sum of these
features from the convolution output of the combined warped out
$W_{y_{r}}^{out}$ and the style matrix $ST$, which are defined as:

$\displaystyle\alpha^{i}_{c,h,w}=\theta_{\alpha}\cdot\alpha_{W}+(1-\theta_{\alpha})\cdot\alpha_{C}$
(6)
$\displaystyle\beta^{i}_{c,h,w}=\theta_{\beta}\cdot\beta_{W}+(1-\theta_{\beta})\cdot\beta_{C},$

where $\theta_{\alpha}$ and $\theta_{\beta}$ are the adaptation coefficients
that are dynamically weighted from two sources in the transfer.

Different from [49], which weights the $\theta_{\alpha}$ and $\theta_{\beta}$
from the style matrix and the semantic mask, we incorporate the spatial
alignment into the per-region normalization, using the warped image aligned
with the source identity to provide the spatial context of the style.

####  III-B3 Makeup Fusion Module

The MFM is composed of a face identity encoder, two upsample layers, five
fusion blocks, and a decoder. Compared to [49, 38], which begins with random
noises, the face identity encoder is expected to extract the identity feature
of $x_{s}$. Fusion blocks receive scale and bias parameters from RAM to
denormalize the feature maps, and the upsample layers are used to recover the
resolution of the original image. Finally, the decoder is responsible for
decoding the feature maps to generate the result $\hat{y}_{s}$.

###  III-C Loss Function

We train our model in a collaborative fashion to ensure that the various
components of the network work together to maximum effect. The total loss
$\mathcal{L}_{total}$ can be defined as below:

$\displaystyle\mathcal{L}_{total}=\lambda_{1}\mathcal{L}_{domain}+\lambda_{2}\mathcal{L}_{perc}+\lambda_{3}\mathcal{L}_{corr}$
(7)
$\displaystyle+\lambda_{4}\mathcal{L}_{makeup}+\lambda_{5}\mathcal{L}_{cycle}+\lambda_{6}\mathcal{L}_{adv}+\lambda_{7}\mathcal{L}_{id},$

where $\lambda_{1,2,4,5,6,7}$ are the hyper-parameters to control the weights
of each term.

Domain alignment loss. To support the correspondence learning across two
different domains, we need to ensure that the feature embeddings $f_{x}$ from
$F_{x}(x_{s}^{l})$ and $f_{y}$ from $F_{y}(y_{r})$ lie in the same domain.
$\mathcal{L}_{domain}$ can be defined as below:

$\mathcal{L}_{domain}=||F_{x}(x_{s}^{l})-F_{y}(y_{r})||_{1},$ (8)

where $||\cdot||_{1}$ is the L1-Norm. Please note that even if
$F_{x}(x_{s}^{l})=F_{y}(y_{r})$ in, we can still calculate the similarity
between them to get the correspondence matrix.

Perceptual loss. High-level features usually contain semantic information.
Thus, we utilize VGG-19 pretrained model[50] to extract features from the
generated output $\hat{y}_{s}$ and the input source image $x_{s}$,
respectively. We merely adopt the activation after $relu4\\_2$ layer which is
set as $\phi_{l}$ since $\hat{y}_{s}$ and $x_{s}$ are not aligned in pixel-
level, $\mathcal{L}_{perc}$ can be formulated as below:

$\mathcal{L}_{perc}=||\phi_{l}(\hat{y}_{s})-\phi_{l}(x_{s})||_{2},$ (9)

where $||\cdot||_{2}$ is the L2-Norm.

Correspondence regularization. Despite the domain alignment loss, there lacks
a powerful supervision to train correspondence learning indeed. To solve that,
we can construct a cyclic warping as [48] to ensure the learned correspondence
is cycle-consistent. We warp $W_{y_{r}}$ again with the correspondence matrix
to conduct L1 loss with the original image $y_{r}$. $\mathcal{L}_{corr}$ can
be computed as below:

$\mathcal{L}_{corr}=||\hat{W}_{y_{r}}-y_{r}||_{1},$ (10)

where $\hat{W}_{y_{r}}(u)=\sum\limits_{v}\mathop{softmax}\limits_{v}(\alpha
C(u,v)\cdot W_{y_{r}}(v))$.

Adversarial loss. We adopt a discriminator [51] to constrain the latent space
of output into the data distribution like $y_{r}$ in order to improve the
quality of the generated image $\hat{y}_{s}$. $\mathcal{L}_{GAN}^{G}$ and
$\mathcal{L}_{GAN}^{D}$ are respectively formulated as below:

$\displaystyle\mathcal{L}_{GAN}^{D}$
$\displaystyle=-\mathbb{E}[h(D(y))]-\mathbb{E}[h(-D(G(x,y)))]$ (11)
$\displaystyle\mathcal{L}_{GAN}^{G}$ $\displaystyle=-\mathbb{E}[D(G(x,y))],$

where the hinge function $h(t)=min(0,-1+t)$ is expected to regularize the
discriminator[52, 53].

Makeup loss. It is normally used to match the local histogram of the makeup
region (skin, lip and eyes) between the generated image and the pseudo ground
truth proposed by [1]. We denote the pseudo ground truth that has the identity
of x while equipping with the similar color distribution of y as $HM(x,y)$.
The opposite is $HM(y,x)$.

$\mathcal{L}_{makeup}$ can be computed as below:

$\displaystyle\mathcal{L}_{makeup}$ $\displaystyle=||G(x,y)-HM(x,y)||_{2}$
(12) $\displaystyle+||G(y,x)-HM(y,x)||_{2}$

Cycle consistency loss. We perform a global cycle loss[5] to maintain the
consistency of the mapping between two domains due to the lack of direct
supervision. $\mathcal{L}_{cycle}$ can be calculated as below:

$\displaystyle\mathcal{L}_{cycle}$ $\displaystyle=||G(G(y,x),y)-y||_{1}$ (13)
$\displaystyle+||G(G(x,y),x)-x||_{1}$

Identity loss. We construct the identity loss by referring to the reference,
source, and output images as the same images. This aims to maintain
consistency of the identity during the generation. $\mathcal{L}_{id}$ can be
formulated as below:

$\displaystyle\mathcal{L}_{id}$ $\displaystyle=||G(x,x)-x||_{1}$ (14)
$\displaystyle+||G(y,y)-y||_{1}$

| Adding Blocks | SARA
---|---|---
architecture | 1-3-2 | 1-2-3 | 2-2-2 | 0-3-3 | 1-2-2
step 1 | Face encoder | Face encoder | Face encoder | Face encoder | Face encoder
step 2 | FusionBlock(256,256) | FusionBlock(256,256) | FusionBlock(256,256) | – | FusionBlock(256,256)
step 3 | – | – | FusionBlock(256,256) | – | –
step 4 | Upsample | Upsample | Upsample | Upsample | Upsample
step 5 | FusionBlock(256,128) | FusionBlock(256,128) | FusionBlock(256,128) | FusionBlock(256,256) | FusionBlock(256,128)
step 6 | FusionBlock(128,128) | – | – | FusionBlock(256,128) | –
step 7 | FusionBlock(128,128) | FusionBlock(128,128) | FusionBlock(128,128) | FusionBlock(128,128) | FusionBlock(128,128)
step 8 | Upsample | Upsample | Upsample | Upsample | Upsample
step 9 | – | FusionBlock(128,128) | – | FusionBlock(128,128) | –
step 10 | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64)
step 11 | FusionBlock(64,64) | FusionBlock(64,64) | FusionBlock(64,64) | FusionBlock(64,64) | FusionBlock(64,64)
step 12 | Decoder | Decoder | Decoder | Decoder | Decoder

TABLE II: Adding one fusion block in 1-2-2 architecture (The number of fusion
blocks from bottom to top) with different orders.

| Reducing Blocks | SARA
---|---|---
architecture | 1-1-2 | 1-2-1 | 0-2-2 | 0-1-2 | 1-2-2
step 1 | Face encoder | Face encoder | Face encoder | Face encoder | Face encoder
step 2 | FusionBlock(256,256) | FusionBlock(256,256) | – | – | FusionBlock(256,256)
step 3 | Upsample | Upsample | Upsample | Upsample | Upsample
step 4 | FusionBlock(256,128) | FusionBlock(256,128) | FusionBlock(256,128) | FusionBlock(256,128) | FusionBlock(256,128)
step 5 | – | FusionBlock(128,128) | FusionBlock(128,128) | – | FusionBlock(128,128)
step 6 | Upsample | Upsample | Upsample | Upsample | Upsample
step 7 | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64) | FusionBlock(128,64)
step 8 | FusionBlock(64,64) | – | FusionBlock(64,64) | FusionBlock(64,64) | FusionBlock(64,64)
step 9 | Decoder | Decoder | Decoder | Decoder | Decoder

TABLE III: Reducing one fusion block in 1-2-2 architecture (The number of
fusion blocks from bottom to top) with different orders.

##  IV Experiments

###  IV-A Network architecture

We evaluated different architectures shown in Tab II and Tab III to choose the
best one. We represent the SARA architecture as 1-2-2 (the number of fusion
blocks from bottom to top) and demonstrate the architecture of adding blocks
and reducing blocks in two separate tables. The main differences between these
architectures are the number and the order of fusion blocks. All of the
architectures start with a face identity encoder and end with an image
decoder. Specifically, the fusion block consists of two components arranged in
sequence: a region-adaptive normalization layer, leaky ReLU activation, and
spectral normalization [54]. Its final output is obtained by connecting the
residual of the input and the result after normalization. The results
generated by the different architectures are shown in Fig. 5. We can see that
1-2-2 gets the best makeup transfer results in terms of the eye shadow and
shade. Besides, we adopt the discriminator from pix2pixeHD [55].

###  IV-B Implementation Setting and datasets

Datasets. Experiments are conducted on the Makeup Transfer (MT) dataset [1].
It contains 2719 makeup images, each of which has variations in poses,
expressions, and makeup styles. Besides, it contains 1115 non-makeup images
with different human identities. Following [1], we randomly selected 250
makeup images and 100 non-makeup images as the testing set, and the rest as
the training set. To further validate the effectiveness of our model, we also
tested it in the M-wild dataset [15], in which the facial images have complex
backgrounds.

Figure 5: Qualitative comparisons with different architectures. The above two
lines show a comparison of different orders for adding a fusion block, while
the below two lines show a comparison of different orders for reducing a
fusion block.

Implementation Details We train the model for 40 epochs with an Adam optimizer
($\beta_{1}=0,\beta_{2}=0.9$). We set imbalanced learning rates, 1e-4 and 4e-4
respectively, following the TTUR. The spectral norm is used in both the
generator and discriminator for stabilizing the training. All the experiments
were conducted on one NVIDIA GeForce RTX 2080 Ti GPU, and it takes roughly 2
days to train the whole model when batch size is set to 1. The reason we set
the batch size to 1 is because the computation of correspondences matching in
the spatial alignment module uses a lot of memory, and if we remove cycle loss
and identity loss from the loss function, we can adjust it to a larger batch
size, but it also has some impact on the results, which is a trade-off. We
empirically [15, 6] set the hyper-parameters of formula 7 as
($\lambda_{1}$=1.0,$\lambda_{2}$=0.001,$\lambda_{3}$=0.1,$\lambda_{4}$=50.0,$\lambda_{5}$=1.0,$\lambda_{6}$=10.0,$\lambda_{7}$=1.0).

Figure 6: Qualitative comparison in MT dataset. The compared methods include:
CycleGAN [ICCV’17], PairedCycleGAN [CVPR’18], BeautyGAN [ACMMM’18], BeautyGlow
[CVPR’19], LADN [ICCV’19], PSGAN [CVPR’20], SCGAN [CVPR’21], SSAT [AAAI’22].
SARA is able to generate more satisfactory results. Detailed anaysis are
elaborated in Sec IV-C. Figure 7: Qualitative comparisons with open source
models in M-Wild dataset. The compared methods include: BeautyGAN [ACMMM’18],
PSGAN [CVPR’20], SCGAN [CVPR’21], SSAT [AAAI’22], EleGANt [ECCV’22]. SARA is
good at handling large misalignment. Detailed anaysis are elaborated in Sec
IV-C.

###  IV-C Qualitative Results

In Fig. 6, we demonstrate the visual qualitative comparisons with other models
on the frontal facial images. For a fair comparison, we cropped the results of
their models directly from their papers since CycleGAN [2] and BeautyGlow [17]
have not released their code or pre-trained models. Both PairedCycleGAN [2]
and CycleGAN modify the image domain as a whole. This only matches the skin of
the face to the reference image, which can’t accurately transfer lipstick and
eye shadow. BeautyGAN [1] and BeautyGlow enforce the local transfer through
learning makeup loss. However, the skin color generated by BeautyGlow is not
close to the reference image, and the tooth texture of the reference image is
incorrectly transferred to the lips (the second row). LADN [18] is designed
for dramatic makeup transfer, which is not appropriate for the normal makeup
style. PSGAN [15] and SCGAN [6] show better transfer results than the above
methods. However, PSGAN generates an unnatural eye shadow since it mitigates
the misalignment caused by local correspondence. SCGAN transfers the makeup
style by decomposing the facial attributes, which ignores the spatial layout.
The background color of the image that is made is red, but the background
color of the image that is used as a reference is green. This is because it is
hard to tell where the background ends and the makeup begins during the
transfer. Eyeshadow and lipstick generated by SSAT [19] are not bright enough.
By contrast, SARA is capable of not only accurate spatial transfer but also of
preserving the details of makeup style. We further evaluated SARA with other
open source models on M-Wild [15] to validate its robustness under large
discrepancies in expressions and poses. The results generated by EleGANt [27],
shown in the first and third rows of Fig. 7, appear to have inaccurate eye
shadow shapes, while those shown in the second row have unnatural facial
shadows. Our method has superior spatial awareness to handle the large
misalignment of makeup transfer.

Model | Align Rank 1 | MisAlign Rank 1
---|---|---
BeautyGAN[1] | 10.6$\%$ | 3.1$\%$
PSGAN[15] | 15.2$\%$ | 19.2$\%$
SCGAN[6] | 30.6$\%$ | 28.4$\%$
SARA (Ours) | 43.6$\%$ | 49.3$\%$

TABLE IV: User study($\%$) on the MT dataset. Align Rank 1 and MisAlign Rank 1
mean that users select the best one when source images and reference images
are well aligned or not.

###  IV-D Quantitative Results

User study. In terms of quantitative comparison, we follow [27, 6, 19] to
evaluate the quality of the makeup transfer through a user study. Note that
most of the metrics used in the synthesis tasks are designed to evaluate the
global image quality rather than the quality of the makeup transfer itself. We
conducted a user study with 30 participants to compare our method with other
methods, using the pre-trained models provided by those other methods. They
are BeautyGAN, PSGAN, and SCGAN. PSGAN and SCGAN are both designed to deal
with spatial misalignment, so we conducted user studies on both aligned and
misaligned samples separately. A group of samples consists of four images
derived from four methods, respectively. We randomly selected 15 groups of
aligned samples and 15 groups of misaligned samples to ask users to rank the
methods, observing the realism of the generated images and the similarity of
makeup styles with the reference image. The comparison results of different
methods are shown in Table IV, which indicate that our method achieves the
best performance, especially with misaligned samples.

Model | BeautyGAN | PSGAN | SCGAN | Ours
---|---|---|---|---
Runtime | 0.1681 | 0.4218 | 0.2388 | 0.1523
TABLE V: Runtime(in second). Lower means better.

Runtime. As shown in Tab V, our model demonstrates the fastest inference speed
among all the compared methods.

Figure 8: Makeup transfer with four reference images by linear interpolation.

###  IV-E Controllable Makeup Transfer

####  IV-E1 Partial Makeup Transfer

Our makeup styles come from two sources: aligned warped out $W_{y}r^{out}$ and
non-aligned style matrix $ST$. Let $y_{ri}$ denote the reference images
providing makeup style for certain parts, where
$i=\left\\{lip,skin,eyes\right\\}$. In the inference stage, the combined
$W_{y_{ri}}^{out}$ can be replaced as below:

$\displaystyle W_{y_{r}i}^{out}=$ $\displaystyle(W_{y_{reyes}^{l}}\odot
W_{y_{r}eyes})+(W_{y_{rlip}^{l}}\odot W_{y_{r}lip})$ (15)
$\displaystyle+(W_{y_{rskin}^{l}}\odot W_{y_{r}skin}),$

where $\odot$ denotes element-wise multiplication for each pixel between the
partial warped semantic mask and corresponding warped images. The combined
style matrix $ST_{c}$ can be obtained by concatenating the style codes
distilled from corresponding reference features and further broadcasting the
combined style code according to the combined warped semantic map. Fig. 1(b)
demonstrates the partial makeup transfer when styles are obtained from the
lips in Ref 1, the skin in Ref 2, and the eyes in Ref 3.

####  IV-E2 Shade-controllable Makeup Transfer

To get shade-controllable makeup transfer, we apply linear interpolation on
two intermediately warped products (warped images $W_{y_{r}}$ and style code
$ST$) as below:

$\displaystyle W_{y}=\alpha W_{y_{r}}+(1-\alpha)W_{y_{s}}$ (16) $\displaystyle
ST=\alpha ST_{r}+(1-\alpha)ST_{s}$

The coefficient $\alpha\in[0,1]$ in the formula controls the degree of shade.
The example can be seen in the Fig. 1(c). SARA can fuse the makeup of multiple
images into one source image by replacing the source image in the formula with
another reference image.

$\displaystyle W_{y}=\alpha W_{y_{r1}}+(1-\alpha)W_{y_{r2}}$ (17)
$\displaystyle ST=\alpha ST_{r1}+(1-\alpha)ST_{r2}$

As Fig. 8 shows, when we change $\alpha$ from 0 to 1 with a gradient of 0.2,
we can get the results of which the styles are from four reference images in
varying degrees.

Figure 9: Makeup removal results. The inverse transfer works well with SARA.

####  IV-E3 Makeup Removal

The process of removing makeup is similar to transferring it, except that the
source and reference images are reversed. Specifically, by using a non-makeup
image as the reference ($y_{r}$) and a makeup image as the source ($x_{s}$),
we can achieve a makeup removal effect. Fig. 9 displays the results of our
method for makeup removal, demonstrating that SARA is effective at removing
makeup.

Figure 10: Effects of SAM. The third column is the intermediate warped output
generated from SAM. The fourth column is the generated results without using
SAM. The fifth column is the generated results using SNM with SPADE but
without SAM. Figure 11: Effects of RAM. In terms of the texture quality of
style, the full model outperforms w/o RAM.

###  IV-F Ablation Studies

Effectiveness of the Spatial-aware Alignment Module (SAM). SAM warp the
reference image to be aligned with the source image. The third column of Fig.
10 shows the immediate warped images generated by SAM, it can be seen that
makeup styles can be well preserved during the warping although the results
are not very sharp. w/o SAM denotes using the non-aligned semantic-adaptive
normalization without SAM. In particular, blusher cannot be transferred while
the color of lip is gloomy (first row) and artifacts of the eyes can be
observed (second row). To further explore the effectiveness of SAM, we also
conduct ablation experiments on combining the region-adaptive normalization
with SPADE [56] like [49]. The generated results are shown in the fifth
column. Although the semantic mask can mitigate the influence of unnatural
artifacts, it contributes little effect on the makeup transfer. Thus, the
results shown in the last column that have appropriate makeup styles are
benefited by SAM.

Effectiveness of the Region-Adaptive normalization Module (RAM). RAM performs
per-region normalization through adaptively fusing the makeup styles according
to the warped image providing spatial context and the style matrix equipped
with shape-independent style information. As shown in the Fig. 11, w/o RAM
indicates achieving per-region encoding and normalization merely on the warped
out generated from SAM. RAM steers the results to be more natural with the
better texture.

Effectiveness of the identity loss. In this paper, we introduce the identity
loss to control the continuity and consistency during generation, which
alleviates the production of unnatural results caused by occlusion or changes
in different lighting conditions. As shown in the Fig. 12, the makeup transfer
results generated from the model trained with identity loss will not be
affected by the occluded lips (the first and second rows). Furthermore, when
the image’s lighting with makeup is abnormal, identity loss can make the
transfer result free of mottled light and shadow, leading to the natural
results(the third and fourth rows). Overall, identity loss can make the model
competent for some unconventional transfers.

Figure 12: Effects of identity loss. The occlusion in the reference image may
cause artifacts in the corresponding regions, without identity loss. It can
also effectively eliminate any mottled light and shadow. Figure 13: Failure
cases in the dataset collected by [18] with complex and dramatic styles. The
transferred texture preservation is not ideal when the reference images have
large areas of camouflage or special symbols, then.

Limitation: In terms of dramatic styles, one potential limitation of our model
is that it may not be able to accurately transfer dramatic makeup looks filled
with blockbuster camouflage, as shown in Fig. 13. This could be because our
training data did not include enough examples of such complex makeup designs,
which may have made it difficult for the model to learn how to transfer them.
In order to improve the performance of our model in this area, we would need
to expand our training dataset to include a wider variety of makeup looks.

##  V Conclusion

To solve the problem of controllable makeup transfer, we introduced SARA, a
method that allows for freely transferring makeup styles. The proposed
architecture decouples the shape and color of the style while retaining the
spatial context relationship. We warp the partial semantic mask by
correspondence matching with a reference image to preserve the spatial context
rather than filling styles into non-makeup masks. Our proposed correspondence
matrix is calculated in a pixel-to-pixel manner, dense enough to represent
detailed and exquisite makeup styles. Eliminating spatial gaps will sacrifice
the precision and sharpness of styles, leading to a smooth result.

##  VI Acknowledgement

This work was supported by National Natural Science Foundation of China (NSFC)
62272172, Guangdong Basic and Applied Basic Research Foundation
2023A1515012920, Tip-top Scientific and Technical Innovative Youth Talents of
Guangdong Special Support Program 2019TQ05X200 and 2022 Tencent Wechat Rhino-
Bird Focused Research Program (Tencent WeChat RBFR2022008), and the Major Key
Project of PCL under Grant PCL2021A09.

## References

  * [1] T. Li, R. Qian, C. Dong, S. Liu, Q. Yan, W. Zhu, and L. Lin, “Beautygan: Instance-level facial makeup transfer with deep generative adversarial network,” in _Proceedings of the 26th ACM international conference on Multimedia_ , 2018, pp. 645–653.
  * [2] H. Chang, J. Lu, F. Yu, and A. Finkelstein, “Pairedcyclegan: Asymmetric style transfer for applying and removing makeup,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 40–48.
  * [3] Y. Choi, M. Choi, M. Kim, J.-W. Ha, S. Kim, and J. Choo, “Stargan: Unified generative adversarial networks for multi-domain image-to-image translation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8789–8797.
  * [4] X. Huang, M.-Y. Liu, S. Belongie, and J. Kautz, “Multimodal unsupervised image-to-image translation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 172–189.
  * [5] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [6] H. Deng, C. Han, H. Cai, G. Han, and S. He, “Spatially-invariant style-codes controlled makeup transfer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 6549–6557.
  * [7] J. Zhang, K. Li, Y.-K. Lai, and J. Yang, “Pise: Person image synthesis and editing with decoupled gan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7982–7990.
  * [8] V. Dumoulin, J. Shlens, and M. Kudlur, “A learned representation for artistic style,” _arXiv preprint arXiv:1610.07629_ , 2016.
  * [9] S. Choi, S. Park, M. Lee, and J. Choo, “Viton-hd: High-resolution virtual try-on via misalignment-aware normalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 14 131–14 140.
  * [10] Y. Lyu, Y. Jiang, B. Peng, and J. Dong, “Infostyler: Disentanglement information bottleneck for artistic style transfer,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [11] X. Huang and S. Belongie, “Arbitrary style transfer in real-time with adaptive instance normalization,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1501–1510.
  * [12] Z. Lv, X. Li, X. Li, F. Li, T. Lin, D. He, and W. Zuo, “Learning semantic person image generation by region-adaptive normalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 10 806–10 815.
  * [13] Y. Men, Y. Mao, Y. Jiang, W.-Y. Ma, and Z. Lian, “Controllable person image synthesis with attribute-decomposed gan,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5084–5093.
  * [14] X. Zhong, Z. Wu, T. Tan, G. Lin, and Q. Wu, “Mv-ton: Memory-based video virtual try-on network,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 908–916.
  * [15] W. Jiang, S. Liu, C. Gao, J. Cao, R. He, J. Feng, and S. Yan, “Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5194–5202.
  * [16] S. Liu, W. Jiang, C. Gao, R. He, J. Feng, B. Li, and S. Yan, “Psgan++: Robust detail-preserving makeup transfer and removal,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [17] H.-J. Chen, K.-M. Hui, S.-Y. Wang, L.-W. Tsao, H.-H. Shuai, and W.-H. Cheng, “Beautyglow: On-demand makeup transfer framework with reversible generative network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 042–10 050.
  * [18] Q. Gu, G. Wang, M. T. Chiu, Y.-W. Tai, and C.-K. Tang, “Ladn: Local adversarial disentangling network for facial makeup and de-makeup,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 10 481–10 490.
  * [19] Z. Sun, Y. Chen, and S. Xiong, “Ssat: a symmetric semantic-aware transformer network for makeup transfer and removal,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 36, no. 2, 2022, pp. 2325–2334.
  * [20] S. Wang and Y. Fu, “Face behind makeup,” in _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016.
  * [21] C. Li, K. Zhou, and S. Lin, “Simulating makeup through physics-based manipulation of intrinsic image layers,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 4621–4629.
  * [22] Y. Li, L. Song, X. Wu, R. He, and T. Tan, “Anti-makeup: Learning a bi-level adversarial network for makeup-invariant face verification,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, no. 1, 2018.
  * [23] D. Guo and T. Sim, “Digital face makeup by example,” in _2009 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2009, pp. 73–79.
  * [24] T. Nguyen, A. T. Tran, and M. Hoai, “Lipstick ain’t enough: beyond color matching for in-the-wild makeup transfer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 13 305–13 314.
  * [25] Y. Lyu, J. Dong, B. Peng, W. Wang, and T. Tan, “Sogan: 3d-aware shadow and occlusion robust gan for makeup transfer,” in _Proceedings of the 29th ACM International Conference on Multimedia_ , 2021, pp. 3601–3609.
  * [26] Z. Wan, H. Chen, J. An, W. Jiang, C. Yao, and J. Luo, “Facial attribute transformers for precise and robust makeup transfer,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2022, pp. 1717–1726.
  * [27] C. Yang, W. He, Y. Xu, and Y. Gao, “Elegant: Exquisite and locally editable gan for makeup transfer,” in _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI_. Springer, 2022, pp. 737–754.
  * [28] J. Liu, X. Fan, J. Jiang, R. Liu, and Z. Luo, “Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 1, pp. 105–119, 2021.
  * [29] Z. Zhao, B. Xiong, L. Wang, Q. Ou, L. Yu, and F. Kuang, “Retinexdip: A unified deep framework for low-light image enhancement,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1076–1088, 2021\.
  * [30] P. Haeberli, “Paint by numbers: Abstract image representations,” in _Proceedings of the 17th annual conference on Computer graphics and interactive techniques_ , 1990, pp. 207–214.
  * [31] A. Hertzmann, “Painterly rendering with curved brush strokes of multiple sizes,” in _Proceedings of the 25th annual conference on Computer graphics and interactive techniques_ , 1998, pp. 453–460.
  * [32] A. Hertzmann, C. E. Jacobs, N. Oliver, B. Curless, and D. H. Salesin, “Image analogies,” in _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , 2001, pp. 327–340.
  * [33] A. A. Efros and W. T. Freeman, “Image quilting for texture synthesis and transfer,” in _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_ , 2001, pp. 341–346.
  * [34] W. Tang, F. He, Y. Liu, Y. Duan, and T. Si, “Datfuse: Infrared and visible image fusion via dual attention transformer,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [35] W. Zhou, X. Lin, J. Lei, L. Yu, and J.-N. Hwang, “Mffenet: Multiscale feature fusion and enhancement network for rgb–thermal urban road scene parsing,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 2526–2538, 2021.
  * [36] P. Zhou, L. Xie, B. Ni, L. Liu, and Q. Tian, “Hrinversion: High-resolution gan inversion for cross-domain image synthesis,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2022.
  * [37] Y. Jing, X. Liu, Y. Ding, X. Wang, E. Ding, M. Song, and S. Wen, “Dynamic instance normalization for arbitrary style transfer,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 04, 2020, pp. 4369–4376.
  * [38] D. Y. Park and K. H. Lee, “Arbitrary style transfer with style-attentional networks,” in _proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 5880–5888.
  * [39] Y. Deng, F. Tang, W. Dong, H. Huang, C. Ma, and C. Xu, “Arbitrary video style transfer via multi-channel correlation,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 2, 2021, pp. 1210–1217.
  * [40] J. An, S. Huang, Y. Song, D. Dou, W. Liu, and J. Luo, “Artflow: Unbiased image style transfer via reversible neural flows,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 862–871.
  * [41] H. Chen, F. Shao, X. Chai, Y. Gu, Q. Jiang, X. Meng, and Y.-S. Ho, “Quality evaluation of arbitrary style transfer: subjective study and objective metric,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2022.
  * [42] L. A. Gatys, A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman, “Controlling perceptual factors in neural style transfer,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3985–3993.
  * [43] M. Lu, H. Zhao, A. Yao, F. Xu, Y. Chen, and L. Zhang, “Decoder network over lightweight reconstructed feature for fast semantic style transfer,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2469–2477.
  * [44] N. Kolkin, J. Salavon, and G. Shakhnarovich, “Style transfer by relaxed optimal transport and self-similarity,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 051–10 060.
  * [45] Z. Wei, Y. Sun, J. Wang, H. Lai, and S. Liu, “Learning adaptive receptive fields for deep image parsing network,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2434–2442.
  * [46] B. Zhang, M. He, J. Liao, P. V. Sander, L. Yuan, A. Bermak, and D. Chen, “Deep exemplar-based video colorization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 8052–8061.
  * [47] C. Luo, J. Zhan, X. Xue, L. Wang, R. Ren, and Q. Yang, “Cosine normalization: Using cosine similarity instead of dot product in neural networks,” in _International Conference on Artificial Neural Networks_. Springer, 2018, pp. 382–391.
  * [48] P. Zhang, B. Zhang, D. Chen, L. Yuan, and F. Wen, “Cross-domain correspondence learning for exemplar-based image translation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5143–5153.
  * [49] P. Zhu, R. Abdal, Y. Qin, and P. Wonka, “Sean: Image synthesis with semantic region-adaptive normalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 5104–5113.
  * [50] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [51] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [52] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena, “Self-attention generative adversarial networks,” in _International conference on machine learning_. PMLR, 2019, pp. 7354–7363.
  * [53] A. Brock, J. Donahue, and K. Simonyan, “Large scale gan training for high fidelity natural image synthesis,” _arXiv preprint arXiv:1809.11096_ , 2018\.
  * [54] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, “Spectral normalization for generative adversarial networks,” _arXiv preprint arXiv:1802.05957_ , 2018\.
  * [55] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-resolution image synthesis and semantic manipulation with conditional gans,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8798–8807.
  * [56] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 2337–2346.