# From hydro to jet quenching, coalescence and hadron cascade: a coupled
approach to solving the $R_{AA}\otimes v_{2}$ puzzle

Wenbin Zhao  Key Laboratory of Quark and Lepton Physics (MOE) & Institute of
Particle Physics,Central China Normal University, Wuhan 430079, China  Weiyao
Ke  Physics Department, University of California, Berkeley, California 94720,
USA  Nuclear Science Division MS 70R0319, Lawrence Berkeley National
Laboratory, Berkeley, California 94720, USA  Wei Chen  School of Nuclear
Science and Technology, University of Chinese Academy of Sciences, Beijing
100049, China  Tan Luo  Instituto Galego de Física de Altas Enerxías IGFAE,
Universidade de Santiago de Compostela, E-15782 Galicia-Spain  Xin-Nian Wang
current address<EMAIL_ADDRESS>Nuclear Science Division, Lawrence Berkeley
National Laboratory, Berkeley, CA 94720. Key Laboratory of Quark and Lepton
Physics (MOE) & Institute of Particle Physics,Central China Normal University,
Wuhan 430079, China  Physics Department, University of California, Berkeley,
California 94720, USA  Nuclear Science Division MS 70R0319, Lawrence Berkeley
National Laboratory, Berkeley, California 94720, USA

###### Abstract

Hydrodynamics and jet quenching are responsible for the elliptic flow $v_{2}$
at low transverse momentum $p_{T}$ and hadron suppression and $v_{2}$ at high
$p_{T}$, respectively, in high-energy heavy-ion collisions. A consistent
description of the hadron suppression factor $R_{AA}$ and $v_{2}$, especially
at intermediate $p_{T}$, however, remains a challenge. We solve this long-
standing $R_{AA}\otimes v_{2}$ puzzle by describing hadron production from low
to intermediate and high $p_{T}$ within the coupled linear Boltzmann
transport-hydro model that combines concurrent jet transport and medium hydro
evolution with quark coalescence in the hadronization and hadron casacde in
the final evolution of the system. We illustrate that quark coalescence and
hadron cascade are the keys to solving the puzzle. They also lead to a
splitting of $v_{2}$ for pions, kaons and protons in the intermediate $p_{T}$
region. We demonstrate for the first time that experimental data on $R_{AA}$,
$v_{2}$ and their hadron flavor dependence from low to intermediate and high
$p_{T}$ in high-energy heavy-ion collisions can be understood within this
coupled framework.

###### pacs:

25.75.Ld, 25.75.Gz, 24.10.Nz

1\. Introduction: Experimental evidences at the Relativistic Heavy-Ion
Collider (RHIC) and the Large Hadron Collider (LHC) have confirmed the
existence of a strongly coupled quark-gluon plasma (QGP) in high-energy heavy-
ion (A+A) collisions Gyulassy and McLerran (2005); Adams  _et al._ (2005a);
Adcox  _et al._ (2005); Jacobs and Wang (2005); Muller  _et al._ (2012). These
include strong anisotropic flow at low transverse momentum ($p_{T}$) Ackermann
_et al._ (2001); Adcox  _et al._ (2002a); Aamodt  _et al._ (2010); Aad  _et
al._ (2012); Chatrchyan  _et al._ (2012) and suppression of hadrons at high
$p_{T}$ Adcox  _et al._ (2002b); Adler  _et al._ (2002); Wang (2005); Aamodt
_et al._ (2011); Majumder and Van Leeuwen (2011). The spectra and flow
patterns of bulk hadrons at low $p_{T}\lesssim 2$ GeV/$c$ are well described
by the hydrodynamic expansion of the QGP as a strongly-coupled fluid
Romatschke and Romatschke (2007); Song  _et al._ (2011); Gale  _et al._
(2013a); Huovinen (2013); Heinz and Snellings (2013); Gale  _et al._ (2013b);
Niemi  _et al._ (2016); Bernhard  _et al._ (2016); McDonald  _et al._ (2017);
Zhao  _et al._ (2017). At high $p_{T}\gtrsim 10$ GeV/$c$, the hadron
suppression factor ($R_{AA}$) and the azimuthal anisotropy ($v_{2}$) Adams
_et al._ (2004); Sirunyan  _et al._ (2018); Aaboud  _et al._ (2018); Acharya
_et al._ (2018a) can be quantitatively understood in terms of jet quenching
caused by parton energy loss as hard partons propagate through the QGP medium
Wang and Gyulassy (1992); Wang (2000); Vitev and Gyulassy (2002); Wang and
Wang (2002); Eskola  _et al._ (2005); Qin  _et al._ (2008); Schenke  _et al._
(2009); Chen  _et al._ (2011); Majumder and Shen (2012); Buzzatti and Gyulassy
(2012); Zapp  _et al._ (2013); Gyulassy  _et al._ (2001); Kumar  _et al._
(2017); Zigic  _et al._ (2020); Mehtar-Tani  _et al._ (2013); Qin and Wang
(2015); Blaizot and Mehtar-Tani (2015). In the intermediate $p_{T}\sim 2-10$
GeV/$c$ region where soft and hard physics interface it remains, however, a
challenge to describe the hadron spectra and the azimuthal anisotropy
consistently. There is a longstanding puzzle that parton energy loss models
that are adjusted to describe hadron $R_{AA}$ under-predict the azimuthal
anisotropy $v_{2}$ Molnar and Sun (2013); Noronha-Hostler  _et al._ (2016);
Zhang and Liao (2013); Liao and Shuryak (2009); Kopeliovich  _et al._ (2012);
Cao  _et al._ (2017); Andres  _et al._ (2020) at intermediate $p_{T}$. Parton
energy loss alone also cannot describe the constituent quark number (NCQ)
dependence of hadron $R_{AA}$ and $v_{2}$.

Many attempts tried to solve this puzzle, from drastically increasing the jet-
medium coupling near the pseudo-critical temperature $T_{c}$ Renk (2014); Xu
_et al._ (2015); Shi  _et al._ (2018, 2019) to taking into account of the
event-by-event fluctuation of the bulk medium Noronha-Hostler  _et al._
(2016), with respect to which one measures high $p_{T}$ anisotropies, and
combining hydrodynamics with jet transport as in EPOS Werner  _et al._ (2012)
and HYDJET++ model Lokhtin  _et al._ (2012). While such a drastically large
jet-medium coupling at $T_{c}$ is not needed to describe $R_{AA}$ and $v_{2}$
of full jets Aad  _et al._ (2013); Zapp (2014); Adam  _et al._ (2016); He  _et
al._ (2019) in which hadron’s flavor information is not considered, the event-
by-event fluctuation of the bulk medium is found not to significantly increase
high $p_{T}$ $v_{2}$ Betz  _et al._ (2011); Cao  _et al._ (2017); He  _et al._
(2019). Furthermore, none of these attempts can address the NCQ dependence of
$R_{AA}$ and $v_{2}$ in the intermediate $p_{T}$ region.

It is well known that quark coalescence is the key to explain the observed NCQ
scaling of hadronic anisotropies and the enhanced baryon to meson ratios in
A+A relative to proton-proton ($p+p$) collisions at intermediate $p_{T}$ Fries
_et al._ (2003a, b); Greco  _et al._ (2003a, b, 2004); Molnar and Voloshin
(2003); Hwa and Yang (2004); Adams  _et al._ (2005b, c); Adare  _et al._
(2007); Abelev  _et al._ (2007); Adare  _et al._ (2012, 2011). Furthermore, it
has been shown that hadrons, especially baryons, at intermediate $p_{T}$ are
also sensitive to rescatterings in the hadronic phase Ryu  _et al._ (2018);
Knospe  _et al._ (2021). Quark coalescence in a hydrodynamic medium combined
with parton transport, fragmentation and hadron cascade should, therefore,
have the potential to solve the $R_{AA}\otimes v_{2}$ puzzle and describe
hadron production from low to intermediate and high $p_{T}$ in high-energy
heavy-ion collisions.

In this work, we implement the quark coalescence, especially between thermal
and jet shower partons, and a hadronic afterburner in the state-of-the-art
coupled linear Boltzmann transport(CoLBT)-hydro model Chen  _et al._ (2020)
that achieves concurrent evolution of both the bulk medium and jet shower
partons, including jet-induced medium responses. We carry out a first study
that couples event-by-event hydrodynamics, jet quenching, quark coalescence
and hadron cascade. We demonstrate that this fully coupled approach can
simultaneously describe $R_{AA}$, differential $v_{2}$, and their NCQ
dependence in the full range of $p_{T}$ in high-energy heavy-ion collisions.

2. CoLBT-hydro model: The CoLBT-hydro model Chen  _et al._ (2020) is developed to simulate the concurrent evolution of jet showers and the bulk medium by coupling the (3+1)D CCNU-LBNL viscous hydrodynamic model (CLVisc) Pang  _et al._ (2012, 2018) with the linear Boltzmann transport (LBT) model Wang and Zhu (2013); Li  _et al._ (2011); He  _et al._ (2015); Cao  _et al._ (2016, 2018). The LBT model treats the propagation of jet shower and thermal recoil partons on an equal footing and includes both pQCD elastic scattering and medium-induced gluon radiation within the high-twist approach Guo and Wang (2000); Wang and Guo (2001); Zhang and Wang (2003); Schafer  _et al._ (2007). The coupling between LBT and CLVisc is through an energy-momentum source term deposited by soft partons in the hydrodynamic equation,

$\displaystyle\partial_{\mu}T_{\rm fluid}^{\mu\nu}=J^{\nu},$ (1)
$\displaystyle J^{\nu}=\sum_{i}\frac{\theta(p^{0}_{\rm cut}-p_{i}\cdot
u)p^{\nu}}{\tau(2\pi)^{3/2}\sigma^{2}_{r}\sigma_{\eta_{s}}\Delta\tau}e^{-\frac{(\vec{x}_{\perp}-\vec{x}_{\perp
i})^{2}}{2\sigma^{2}_{r}}-\frac{(\eta_{s}-\eta_{si})^{2}}{2\sigma^{2}_{\eta_{s}}}},$
(2)

where $T_{\rm fluid}^{\mu\nu}$ is the energy-momentum tensor of the bulk
medium, the summation in the Gaussian smearing in the Milne coordinates is
over both soft recoil, radiated and “negative” partons –“hole particles”
created in elastic jet-medium collisions. The concurrent and coupled evolution
of jet shower (including hard recoil) partons and the medium is achieved by
updating the source term in CLVisc and the medium in LBT during each time
interval. For studies presented in this Letter, we set $p^{0}_{\rm cut}=3.0$
GeV/$c$, $\sigma_{r}$=0.6 fm and $\sigma_{\eta_{s}}$=0.6.

We follow Refs. Cao  _et al._ (2016, 2018) to allow the strong coupling
constant in each interaction vortex run differently and the jet-medium
coupling is set to zero in hadronic matter in LBT. In the CLVisc hydrodynamic
evolution, a lattice QCD inspired equation of state Bazavov  _et al._ (2014)
is used. The TRENTo model Moreland  _et al._ (2015) with optimized parameters
[][.Initialentropynormalizationfactoris140; thereduced-
thicknessparameter$p=0$; entropydespoliationfluctuationparameteris$k=1.5$;
andthenucleonwidthparameteris$w=0.8$fm.]Bernhard:2016tnd and a longitudinal
envelope function [][Theplateauwidthis$η_flat=2.0$fmandtheGaussianfall-
offwidthis$σ_η=1.8$fm.]envelop; Pang  _et al._ (2012, 2018) is used to
generate the initial entropy density profile with event-by-event transverse
fluctuation. The specific shear viscosity $\eta/s$=0.10, the freeze-out
temperature $T_{\rm sw}=150$ MeV, the initial time $\tau_{0}=0.6$ fm/$c$ and
parameters in the initial entropy profile have been adjusted to reproduce the
charged hadron multiplicity, $p_{T}$ spectra and integrated flow harmonics
$v_{n}$ at mid-rapidity in A+A collisions. Further details about the LBT and
CLVisc model can be found in Refs. Li  _et al._ (2011); Wang and Zhu (2013);
He  _et al._ (2015); Cao  _et al._ (2016, 2018) and Refs. Pang  _et al._
(2018, 2012), respectively.

We use PYTHIA8 Sjostrand  _et al._ (2008) with EPPS16 nuclear parton
distributions Eskola  _et al._ (2017) to generate initial jet showers. A
minimum hard scale $\hat{p}_{T0}=$ 4.0 GeV/$c$ is set for jet production and
only shower partons with $p_{T}>p_{T{\rm min}}$ are included so their
contributions to the initial total entropy are small which is subtracted from
the initial condition for hydro. The average number of such jet production per
event is $\left<N_{\rm jet}\right>=\langle N_{\rm coll}\rangle P_{\rm
trigger}^{\rm jet}$, where $\langle N_{\rm coll}\rangle$ is the number of
binary nucleon-nucleon ($N+N$) collisions whose transverse distribution is
given by the nuclear overlap function. $P_{\rm trigger}^{\rm jet}$ is the
probability for a minimum-bias $N+N$ collision to have at least one pair of
jet production,

$\displaystyle P_{\rm trigger}^{\rm jet}=\frac{1}{\sigma_{NN}^{\rm inel}}\int
d^{2}b\left[1-e^{-T_{NN}(b)\sigma_{\rm
jet}(\hat{p}_{T}>\hat{p}_{T0})}\right],$ (3)

where $T_{NN}(b)$ is the nucleon overlapping function. In this Letter, we will
only show simulations of lead-lead (Pb+Pb) collisions at $\sqrt{s_{NN}}=5.02$
TeV and discuss the colliding energy and system size dependence separately. At
this colliding energy, $\sigma_{NN}^{\rm inel}=70$ mb, $\langle N_{\rm
coll}\rangle$ =1003 (184) and we set $p_{T{\rm min}}=6.0$ (5.0) GeV/$c$ for
10-20% (40-50%) centrality classes of Pb+Pb collisions.

We assume that hard shower partons free-stream during the formation time
$\tau_{f}=2z(1-z)E/k_{\perp}^{2}$ before they interact with the QGP medium
according to the CoLBT-hydro model, where $k_{\perp}$ is the transverse
momentum, $z$ the energy fraction of the shower parton after the initial
splitting from its mother parton with energy $E$.

3\. Hadronization and hadron cascade: In this study, we adapt the Hydro-Coal-
Frag hybrid model Han  _et al._ (2016); Zhao  _et al._ (2020) for
hadronization that includes hydro freeze-out at low $p_{T}$, quark coalescence
at intermediate $p_{T}$ and fragmentation at high $p_{T}$. The interplay
between hydro hadron freeze-out and parton dynamics is defined by a separation
scale $p_{Ts}=1.5$ GeV/$c$ for the effective constituent quarks, above which
viscous corrections to the equilibrium distribution become large and parton
coalescence and fragmentation become the relevant mechanisms for
hadronization. This scale corresponds to $p_{T\rm meson}<3$ GeV/$c$ and
$p_{T\rm baryon}<4.5$ GeV/$c$ for hadron production through hydro freeze-out
on the switching hyper-surface. Accordingly, thermal quarks with $p_{T}>$1.5
GeV/$c$ on the switching hyper-surface [][Inprinciple;
theenergyandmomentumofthecoalescedthermalpartonswith$p_T<$1.5GeV/$c$shouldbesubtractedfromthebulkmediumtoconservethetotalenergyandmomentum.However;
wefindtheeffectisnegligibleinthefinalhadronspectra;
whichwewillneglect.]thermal are allowed to participate in coalescence
processes for hadronization, which include thermal-thermal, thermal-shower and
shower-shower coalescence. Shower partons include both jet shower and hard
medium recoil partons passing through the isothermal hyper-surface in CoLBT-
hydro.

Shower partons that do not coalesce will hadronize through string
fragmentation using PYTHIA8 Sjostrand  _et al._ (2008) with tuned strangeness
suppression [][WetunetheLundstringparameterprobStoUD=0.4;
whichcontrolsthesuppressionof$s$quarkpairproductionrelativeto$u;
d$quarksinthestringfragmentation; toreproducethe$K/π$ratioinminimum-biasp-
pcollisionsat$\sqrt{s}=$5.02TeV~\cite[cite]{\@@bibref{AuthorsPhrase1YearPhrase2}{Acharya:2018orn}{\@@citephrase{(}}{\@@citephrase{)}}}.]strange.
Strings are constructed by connecting shower partons in a way that minimizes
all the pairwise distances $\Delta R=\sqrt{(\Delta\eta)^{2}+(\Delta\phi)^{2}}$
between partons’ pseudorapidity $\eta$ and azimuthal angle $\phi$ Kumar  _et
al._ (2020). Finally, the Ultra-relativistic Quantum Molecular Dynamics
(UrQMD) Bass  _et al._ (1998); Bleicher  _et al._ (1999) model is used to
perform hadronic rescatterings and resonance decays in the hadronic stage of
the system until kinetic freeze-out.

4\. Nuclear modification of hadron spectra: Using CoLBT-hydro with the
inclusion of quark coalescence and hadronic afterburner, we calculate hadron
spectra from low to intermediate and high $p_{T}$ in A+A collisions. Shown in
Fig. 1 are CoLBT-hydro results on the nuclear modification factor
$R_{AA}(p_{T})$ of charged hadron spectra
[][Experimentaldataonthehadronspectrainp+pcollisionat$\sqrt{s_{NN}}$=5.02TeV~\cite[cite]{\@@bibref{AuthorsPhrase1YearPhrase2}{Acharya:2018qsh}{\@@citephrase{(}}{\@@citephrase{)}}}areusedasthebaselineforthemodelcalculationsof$R_AA$.]ppdata
which describe reasonably well the experimental data Acharya  _et al._ (2018b)
for 10-20% and 40-50% Pb+Pb collisions at $\sqrt{s_{NN}}=5.02$ GeV. To
illustrate the different hadron production mechanisms underlying
$R_{AA}(p_{T})$ in different $p_{T}$ regions, we show in Fig. 2 (a) $p_{T}$
spectra of charged hadrons in 10-20% and 40-50% and (b) identified pion
($\pi$), kaon ($K$) and proton ($p$) spectra from CoLBT-hydro in 40-50% Pb+Pb
collisions as compared to the experimental data Acharya  _et al._ (2018b,
2020). Also shown are contributions from hydro (dashed), parton coalescence
(dot-dashed) and fragmentation (dotted) in CoLBT results.

Figure 1: (Color online) The nuclear modification factor $R_{\rm AA}$ of
charged hadrons in (a) 10-20% and (b) 40-50% Pb+Pb collisions at
$\sqrt{s_{NN}}=$5.02 TeV from CoLBT-hydro simulations as compared to ALICE
experimental data Acharya  _et al._ (2018b). Figure 2: (Color online) Spectra
of (a) charged hadrons (solid) in 10-20% and 40-50% and (b) identified $\pi$,
$K$ and $p$ (solid) in 40-50% Pb+Pb collisions at $\sqrt{s_{NN}}=$5.02 TeV and
contributions from hydro freeze-out (dashed), parton coalescence (dot-dashed)
and fragmentation (dotted) in CoLBT-hydro simulations as compared to ALICE
experimental data Acharya  _et al._ (2018b, 2020).  Figure 3: (Color online)
(a) $p/\pi$ and (b) $K/\pi$ ratio in 10-20% and 40-50% Pb+Pb and $p+p$
collisions at $\sqrt{s_{NN}}=$5.02 TeV from CoLBT-hydro as compared to
experimental data Acharya  _et al._ (2020).

In CoLBT-hydro, hadron spectra are dominated by the hydro contribution below
$p_{T}<2$ GeV/$c$ where $R_{AA}$ increases rapidly with $p_{T}$ due to the
strong radial flow from the hydrodynamic expansion until it peaks at
$p_{T}\approx 2$ GeV/$c$. At intermediate $2<p_{T}<6$ GeV/$c$, $R_{AA}$
decreases from its peak value due to the onset of contributions from parton
coalescence and fragmentation in which hadron spectrum are suppressed due to
parton energy loss. At large $p_{T}>8$ GeV/$c$ where fragmentation prevails,
$R_{AA}$ is determined by the energy dependence of the parton energy loss and
the initial jet spectra.

We note that the transition in the underlying hadron production mechanism,
from hydrodynamics to parton coalescence and fragmentation, occurs at higher
$p_{T}$ in more central collisions because of stronger radial flow, pushing
the realm of hydrodynamics to higher $p_{T}$. The transition also happens at
larger $p_{T}$ for baryons than mesons. This is because of the constituent-
quark-based separate scale $p_{Ts}=1.5$ GeV/$c$ that leads to a larger cut-off
$p_{T{\rm baryon}}<4.5$ GeV/$c$ for baryons than $p_{T{\rm meson}}<3$ GeV/$c$
for mesons from hydro. The hydro spectra are also mass-ordered due to radial
flow. Hydro contributions to the hadron spectra above the $p_{T}$ cut-off come
from hadron cascade within UrQMD during the hadronic evolution. The interplay
between hydro, parton energy loss, coalescence and fragmentation implemented
in CoLBT-hydro can describe the flavor dependence of hadron spectra and their
medium modification as seen in Fig. 2 (b).

To examine the flavor composition of the hadron spectra and their $p_{T}$
dependence in detail, we compare the CoLBT-hydro results on $p/\pi$ and
$K/\pi$ ratio as a function of $p_{T}$ to the experimental data in Fig. 3.
Both $p/\pi$ and $K/\pi$ ratio exhibit a steep increase from $p_{T}=0-3$
GeV/$c$ with a mass ordering induced by the radial flow, an intrinsic feature
of hydrodynamic models. For $p_{T}>3$ GeV/$c$, $p/\pi$ decreases with $p_{T}$
while $K/\pi$ remains constant as a result of the interplay between
hydrodynamic expansion, quark coalescence and fragmentation. At large
$p_{T}>8$ GeV/$c$, these hadron ratios in Pb+Pb approach the values in $p+p$
collisions when the hadronization is dominated by vacuum-like parton
fragmentation. CoLBT-hydro describes these features in the data well.

5\. Coalescence and $R_{AA}\otimes v_{2}$ puzzle: To demonstrate the
importance of parton coalescence in resolving the $R_{AA}\otimes v_{2}$
puzzle, we examine first the azimuthal anisotropy $v_{2}$ of the hadron
spectra for different hadron species. Following the experiments Sirunyan  _et
al._ (2018); Acharya  _et al._ (2018a, c), we use the scalar-product (SP)
method to compute $v_{2}$ of charged hadrons within $|\eta|<1.0$, using
reference particles in $|\eta|<1.0$ and 0.2$<p_{T}<$ 5.0 GeV/$c$.

Figure 4: (Color online) CoLBT-hydro results on $v_{2}(p_{T})$ for $\pi$, $K$
and $p$ in (a) 10-20% and (b) 40-50% Pb+Pb collisions at $\sqrt{s_{NN}}=$5.02
TeV with (solid) and without (dashed) contributions from quark coalescence
compared to ALICE data Acharya  _et al._ (2018c).  Figure 5: (Color online)
CoLBT-hydro results on $v_{2}(p_{T})$ for $\pi$, $K$ and $p$ from hydro
freeze-out (dotted), parton coalescence (solid) and fragmentation (dashed) in
40-50% Pb+Pb collisions at $\sqrt{s_{NN}}=$5.02 TeV.

Shown in Fig. 4 are $v_{2}(p_{T})$ for $\pi$, $K$ and $p$ from CoLBT-hydro
with (solid) and without (dashed) contributions from coalescence in 10-20% and
40-50% Pb+Pb collisions as compared to the experimental data Acharya  _et al._
(2018c). The interplay among hydro, jet quenching, coalescence and
fragmentation in $v_{2}(p_{T})$ is very similar to that in $R_{AA}(p_{T})$. To
help understand this interplay, we also plot in Fig. 5 $v_{2}(p_{T})$ for
$\pi$, $K$ and $p$ from hydro freeze-out (dotted), parton fragmentation
(dashed) and quark coalescence (solid) which has a strong flavor dependence.

The rapid increase and the mass ordering of $v_{2}(p_{T})$ from hydro at low
$p_{T}<2$ GeV/$c$ in Figs. 4 and 5 are characteristics of viscous
hydrodynamics with hadronic afterburner Huovinen (2013); Heinz and Snellings
(2013); Gale  _et al._ (2013b); McDonald  _et al._ (2017); Zhao  _et al._
(2017); Ryu  _et al._ (2018). The increase slows down at $p_{T}>$ 2.5 GeV/$c$,
the total $v_{2}(p_{T})$ reaches a peak value at around $p_{T{\rm meson}}\sim
3$ GeV/$c$ and $p_{T{\rm baryon}}\sim 4.5$ GeV/$c$ before decreasing at large
$p_{T}$, where the anisotropy caused by the geometric anisotropy of the QGP
fireball and the length dependence of parton energy loss is significantly
smaller than that from hydro and parton coalescence. The mass ordering is also
reversed starting at $p_{T}\approx 2.5$ GeV/$c$. The change of $p_{T}$
dependence and mass ordering of $v_{2}(p_{T})$ are caused mainly by the
interplay among hadrons from hydro, coalescence and fragmentation. Since the
transition from hydro to coalescence and fragmentation occurs at a fixed
$p_{Ts}$ for the effective constituent quarks, both the value and the peak
position of the final $v_{2}$ for $p$ is higher than $\pi$ and $K$ in the
intermediate $3<p_{T}<8$ GeV/$c$ region, resembling the approximate NCQ
scaling Fries  _et al._ (2003a, b); Greco  _et al._ (2003a, b, 2004); Molnar
and Voloshin (2003); Hwa and Yang (2004). Though more hadrons are produced
from fragmentation than from coalescence in this intermediate $p_{T}$ region
(see Fig. 2), $v_{2}$ of $p$ ($\pi,K$) from coalescence is, however, about a
factor of 4 (2, 3) larger than that from fragmentation (see Fig. 5).
Coalescence, therefore, still contributes significantly to the final $v_{2}$
of all hadrons. Without coalescence (dashed lines), CoLBT-hydro underestimates
the $v_{2}$ by up to a factor of 2 in this $p_{T}$ region. CoLBT-hydro also
predicts a slightly larger $v_{2}$ of $K$ than $\pi$ in this region because of
the enhanced thermal strangeness in parton coalescence.

Figure 6: (Color online) CoLBT-hydro results on $v_{2}(p_{T})$ of charged
hadrons in (a) 10-20% and (b) 40-50% Pb+Pb collisions at $\sqrt{s_{NN}}=$5.02
TeV compared with data Sirunyan  _et al._ (2018); Aaboud  _et al._ (2018);
Acharya  _et al._ (2018a).

Finally, we show $v_{2}(p_{T})$ of all charged hadrons in 10-20% and 40-50%
Pb+Pb collisions as compared to experimental data Sirunyan  _et al._ (2018);
Aaboud  _et al._ (2018); Acharya  _et al._ (2018a) in Fig. 6. At high
$p_{T}>10$ GeV/$c$, fragmentation dominates the hadron production, $v_{2}$ for
all light flavor hadrons becomes the same and arises from the geometric
anisotropy of the medium and length dependence of the parton energy loss.
Since the radiative parton energy loss $\Delta E$ has a logarithmic energy
dependence and given the power-law behavior $1/p_{T}^{n}$ of the initial jet
spectra, $R_{AA}\approx 1-n\Delta E/p_{T}$ eventually becomes 1 and $v_{2}$
approaches to 0 at high $p_{T}$. LBT without thermal-shower coalescence can
describe this trend well Cao  _et al._ (2017, 2018). CoLBT-hydro with the
Hydro-Coal-Frag hadronization and hadron cascade can, therefore, consistently
describe both $R_{AA}(p_{T})$ and $v_{2}(p_{T})$ of charged hadrons in the
whole $p_{T}$ range.

6\. Summary: We carried out the first study of hadron spectra in A+A
collisions that combines the state-of-the-art CoLBT-hydro, the Hydro-Coal-Frag
hybrid hadronization and hadron cascade. We demonstrated that the interplay
between hydro freeze-out at low-$p_{T}$, parton coalescence at intermediate
$p_{T}$ and fragmentation at high $p_{T}$ can simultaneously explain the
nuclear modification $R_{AA}$, elliptic anisotropy $v_{2}$ of charged and
identified hadrons and their flavor dependence in the full range of $p_{T}$.
The long-standing $R_{AA}\otimes v_{2}$ puzzle is solved for the first time
with the inclusion of quark coalescence in this coupled approach, which
significantly increases $v_{2}$ in the intermediate $p_{T}$ region. The
predicted splitting between $v_{2}$ of $K$ and $\pi$ in the intermediate
$p_{T}$ region can serve as a precision test for the coalescence mechanism.

Acknowledgements: We thank S. Cao, Y. He, C.-M. Ko, L. Pang, G. Qin, H. Song
and Z. Yang for discussions. This work was supported in part by NSFC under
grant Nos. 11935007, 11221504, 11861131009 and 11890714, US DOE under grant
No. DE-AC02-05CH11231, US NSF under grant Nos. ACI-1550300 and OAC-2004571,
the Fundamental Research Funds for Central Universities in China and the UCB-
CCNU Collaboration Grant. Computations are performed at NSC3/CCNU.

## References

  * Gyulassy and McLerran (2005) M. Gyulassy and L. McLerran, Nucl. Phys. A 750, 30 (2005), arXiv:nucl-th/0405013 .
  * Adams  _et al._ (2005a) J. Adams _et al._ (STAR), Nucl. Phys. A 757, 102 (2005a), arXiv:nucl-ex/0501009 .
  * Adcox  _et al._ (2005) K. Adcox _et al._ (PHENIX), Nucl. Phys. A 757, 184 (2005), arXiv:nucl-ex/0410003 .
  * Jacobs and Wang (2005) P. Jacobs and X.-N. Wang, Prog. Part. Nucl. Phys. 54, 443 (2005), arXiv:hep-ph/0405125 .
  * Muller  _et al._ (2012) B. Muller, J. Schukraft, and B. Wyslouch, Ann. Rev. Nucl. Part. Sci. 62, 361 (2012), arXiv:1202.3233 [hep-ex] .
  * Ackermann  _et al._ (2001) K. H. Ackermann _et al._ (STAR), Phys. Rev. Lett. 86, 402 (2001), arXiv:nucl-ex/0009011 .
  * Adcox  _et al._ (2002a) K. Adcox _et al._ (PHENIX), Phys. Rev. Lett. 89, 212301 (2002a), arXiv:nucl-ex/0204005 .
  * Aamodt  _et al._ (2010) K. Aamodt _et al._ (ALICE), Phys. Rev. Lett. 105, 252302 (2010), arXiv:1011.3914 [nucl-ex] .
  * Aad  _et al._ (2012) G. Aad _et al._ (ATLAS), Phys. Lett. B 707, 330 (2012), arXiv:1108.6018 [hep-ex] .
  * Chatrchyan  _et al._ (2012) S. Chatrchyan _et al._ (CMS), Eur. Phys. J. C 72, 2012 (2012), arXiv:1201.3158 [nucl-ex] .
  * Adcox  _et al._ (2002b) K. Adcox _et al._ (PHENIX), Phys. Rev. Lett. 88, 022301 (2002b), arXiv:nucl-ex/0109003 .
  * Adler  _et al._ (2002) C. Adler _et al._ (STAR), Phys. Rev. Lett. 89, 202301 (2002), arXiv:nucl-ex/0206011 .
  * Wang (2005) X.-N. Wang, Nucl. Phys. A 750, 98 (2005), arXiv:nucl-th/0405017 .
  * Aamodt  _et al._ (2011) K. Aamodt _et al._ (ALICE), Phys. Lett. B 696, 30 (2011), arXiv:1012.1004 [nucl-ex] .
  * Majumder and Van Leeuwen (2011) A. Majumder and M. Van Leeuwen, Prog. Part. Nucl. Phys. 66, 41 (2011), arXiv:1002.2206 [hep-ph] .
  * Romatschke and Romatschke (2007) P. Romatschke and U. Romatschke, Phys. Rev. Lett. 99, 172301 (2007), arXiv:0706.1522 [nucl-th] .
  * Song  _et al._ (2011) H. Song, S. A. Bass, U. Heinz, T. Hirano, and C. Shen, Phys. Rev. Lett. 106, 192301 (2011), [Erratum: Phys. Rev. Lett.109,139904(2012)], arXiv:1011.2783 [nucl-th] .
  * Gale  _et al._ (2013a) C. Gale, S. Jeon, B. Schenke, P. Tribedy, and R. Venugopalan, Phys. Rev. Lett. 110, 012302 (2013a), arXiv:1209.6330 [nucl-th] .
  * Huovinen (2013) P. Huovinen, Int. J. Mod. Phys. E22, 1330029 (2013), arXiv:1311.1849 [nucl-th] .
  * Heinz and Snellings (2013) U. Heinz and R. Snellings, Ann. Rev. Nucl. Part. Sci. 63, 123 (2013), arXiv:1301.2826 [nucl-th] .
  * Gale  _et al._ (2013b) C. Gale, S. Jeon, and B. Schenke, Int. J. Mod. Phys. A28, 1340011 (2013b), arXiv:1301.5893 [nucl-th] .
  * Niemi  _et al._ (2016) H. Niemi, K. J. Eskola, and R. Paatelainen, Phys. Rev. C93, 024907 (2016), arXiv:1505.02677 [hep-ph] .
  * Bernhard  _et al._ (2016) J. E. Bernhard, J. S. Moreland, S. A. Bass, J. Liu, and U. Heinz, Phys. Rev. C94, 024907 (2016), arXiv:1605.03954 [nucl-th] .
  * McDonald  _et al._ (2017) S. McDonald, C. Shen, F. Fillion-Gourdeau, S. Jeon, and C. Gale, Phys. Rev. C95, 064913 (2017), arXiv:1609.02958 [hep-ph] .
  * Zhao  _et al._ (2017) W. Zhao, H.-j. Xu, and H. Song, Eur. Phys. J. C77, 645 (2017), arXiv:1703.10792 [nucl-th] .
  * Adams  _et al._ (2004) J. Adams _et al._ (STAR), Phys. Rev. Lett. 92, 052302 (2004), arXiv:nucl-ex/0306007 .
  * Sirunyan  _et al._ (2018) A. M. Sirunyan _et al._ (CMS), Phys. Lett. B776, 195 (2018), arXiv:1702.00630 [hep-ex] .
  * Aaboud  _et al._ (2018) M. Aaboud _et al._ (ATLAS), Eur. Phys. J. C78, 997 (2018), arXiv:1808.03951 [nucl-ex] .
  * Acharya  _et al._ (2018a) S. Acharya _et al._ (ALICE), JHEP 07, 103 (2018a), arXiv:1804.02944 [nucl-ex] .
  * Wang and Gyulassy (1992) X.-N. Wang and M. Gyulassy, Phys. Rev. Lett. 68, 1480 (1992).
  * Wang (2000) X.-N. Wang, Phys. Rev. C 61, 064910 (2000), arXiv:nucl-th/9812021 .
  * Vitev and Gyulassy (2002) I. Vitev and M. Gyulassy, Phys. Rev. Lett. 89, 252301 (2002), arXiv:hep-ph/0209161 .
  * Wang and Wang (2002) E. Wang and X.-N. Wang, Phys. Rev. Lett. 89, 162301 (2002), arXiv:hep-ph/0202105 .
  * Eskola  _et al._ (2005) K. J. Eskola, H. Honkanen, C. A. Salgado, and U. A. Wiedemann, Nucl. Phys. A 747, 511 (2005), arXiv:hep-ph/0406319 .
  * Qin  _et al._ (2008) G.-Y. Qin, J. Ruppert, C. Gale, S. Jeon, G. D. Moore, and M. G. Mustafa, Phys. Rev. Lett. 100, 072301 (2008), arXiv:0710.0605 [hep-ph] .
  * Schenke  _et al._ (2009) B. Schenke, C. Gale, and S. Jeon, Phys. Rev. C80, 054913 (2009), arXiv:0909.2037 [hep-ph] .
  * Chen  _et al._ (2011) X.-F. Chen, T. Hirano, E. Wang, X.-N. Wang, and H. Zhang, Phys. Rev. C 84, 034902 (2011), arXiv:1102.5614 [nucl-th] .
  * Majumder and Shen (2012) A. Majumder and C. Shen, Phys. Rev. Lett. 109, 202301 (2012), arXiv:1103.0809 [hep-ph] .
  * Buzzatti and Gyulassy (2012) A. Buzzatti and M. Gyulassy, Phys. Rev. Lett. 108, 022301 (2012), arXiv:1106.3061 [hep-ph] .
  * Zapp  _et al._ (2013) K. C. Zapp, F. Krauss, and U. A. Wiedemann, JHEP 03, 080 (2013), arXiv:1212.1599 [hep-ph] .
  * Gyulassy  _et al._ (2001) M. Gyulassy, I. Vitev, and X. N. Wang, Phys. Rev. Lett. 86, 2537 (2001), arXiv:nucl-th/0012092 .
  * Kumar  _et al._ (2017) A. Kumar, E. Bianchi, J. Elledge, A. Majumder, G.-Y. Qin, and C. Shen, Nucl. Phys. A 967, 536 (2017), arXiv:1706.07547 [nucl-th] .
  * Zigic  _et al._ (2020) D. Zigic, B. Ilic, M. Djordjevic, and M. Djordjevic, Phys. Rev. C 101, 064909 (2020), arXiv:1908.11866 [hep-ph] .
  * Mehtar-Tani  _et al._ (2013) Y. Mehtar-Tani, J. G. Milhano, and K. Tywoniuk, Int. J. Mod. Phys. A28, 1340013 (2013), arXiv:1302.2579 [hep-ph] .
  * Qin and Wang (2015) G.-Y. Qin and X.-N. Wang, Int. J. Mod. Phys. E24, 1530014 (2015), arXiv:1511.00790 [hep-ph] .
  * Blaizot and Mehtar-Tani (2015) J.-P. Blaizot and Y. Mehtar-Tani, Int. J. Mod. Phys. E24, 1530012 (2015), arXiv:1503.05958 [hep-ph] .
  * Molnar and Sun (2013) D. Molnar and D. Sun, (2013), arXiv:1305.1046 [nucl-th] .
  * Noronha-Hostler  _et al._ (2016) J. Noronha-Hostler, B. Betz, J. Noronha, and M. Gyulassy, Phys. Rev. Lett. 116, 252301 (2016), arXiv:1602.03788 [nucl-th] .
  * Zhang and Liao (2013) X. Zhang and J. Liao, (2013), arXiv:1311.5463 [nucl-th] .
  * Liao and Shuryak (2009) J. Liao and E. Shuryak, Phys. Rev. Lett. 102, 202302 (2009), arXiv:0810.4116 [nucl-th] .
  * Kopeliovich  _et al._ (2012) B. Z. Kopeliovich, J. Nemchik, I. K. Potashnikova, and I. Schmidt, Phys. Rev. C86, 054904 (2012), arXiv:1208.4951 [hep-ph] .
  * Cao  _et al._ (2017) S. Cao, L.-G. Pang, T. Luo, Y. He, G.-Y. Qin, and X.-N. Wang,  _Proceedings, 8th International Conference on Hard and Electromagnetic Probes of High-energy Nuclear Collisions: Hard Probes 2016 (HP2016): Wuhan, Hubei, China, September 23-27, 2016_ , Nucl. Part. Phys. Proc. 289-290, 217 (2017).
  * Andres  _et al._ (2020) C. Andres, N. Armesto, H. Niemi, R. Paatelainen, and C. A. Salgado, Phys. Lett. B 803, 135318 (2020), arXiv:1902.03231 [hep-ph] .
  * Renk (2014) T. Renk, Phys. Rev. C89, 067901 (2014), arXiv:1402.5798 [hep-ph] .
  * Xu  _et al._ (2015) J. Xu, J. Liao, and M. Gyulassy, Chin. Phys. Lett. 32, 092501 (2015), arXiv:1411.3673 [hep-ph] .
  * Shi  _et al._ (2018) S. Shi, J. Liao, and M. Gyulassy, Chin. Phys. C 42, 104104 (2018), arXiv:1804.01915 [hep-ph] .
  * Shi  _et al._ (2019) S. Shi, J. Liao, and M. Gyulassy, Chin. Phys. C 43, 044101 (2019), arXiv:1808.05461 [hep-ph] .
  * Werner  _et al._ (2012) K. Werner, I. Karpenko, M. Bleicher, T. Pierog, and S. Porteboeuf-Houssais, Phys. Rev. C 85, 064907 (2012), arXiv:1203.5704 [nucl-th] .
  * Lokhtin  _et al._ (2012) I. P. Lokhtin, A. V. Belyaev, L. V. Malinina, S. V. Petrushanko, E. P. Rogochaya, and A. M. Snigirev, Eur. Phys. J. C 72, 2045 (2012), arXiv:1204.4820 [hep-ph] .
  * Aad  _et al._ (2013) G. Aad _et al._ (ATLAS), Phys. Rev. Lett. 111, 152301 (2013), arXiv:1306.6469 [hep-ex] .
  * Zapp (2014) K. C. Zapp, Phys. Lett. B 735, 157 (2014), arXiv:1312.5536 [hep-ph] .
  * Adam  _et al._ (2016) J. Adam _et al._ (ALICE), Phys. Lett. B 753, 511 (2016), arXiv:1509.07334 [nucl-ex] .
  * He  _et al._ (2019) Y. He, S. Cao, W. Chen, T. Luo, L.-G. Pang, and X.-N. Wang, PoS HardProbes2018, 100 (2019).
  * Betz  _et al._ (2011) B. Betz, M. Gyulassy, and G. Torrieri, Phys. Rev. C 84, 024913 (2011), arXiv:1102.5416 [nucl-th] .
  * Fries  _et al._ (2003a) R. J. Fries, B. Muller, C. Nonaka, and S. A. Bass, Phys. Rev. Lett. 90, 202303 (2003a), arXiv:nucl-th/0301087 .
  * Fries  _et al._ (2003b) R. J. Fries, B. Muller, C. Nonaka, and S. A. Bass, Phys. Rev. C68, 044902 (2003b), arXiv:nucl-th/0306027 [nucl-th] .
  * Greco  _et al._ (2003a) V. Greco, C. M. Ko, and P. Levai, Phys. Rev. Lett. 90, 202302 (2003a), arXiv:nucl-th/0301093 .
  * Greco  _et al._ (2003b) V. Greco, C. M. Ko, and P. Levai, Phys. Rev. C68, 034904 (2003b), arXiv:nucl-th/0305024 [nucl-th] .
  * Greco  _et al._ (2004) V. Greco, C. M. Ko, and R. Rapp, Phys. Lett. B595, 202 (2004), arXiv:nucl-th/0312100 [nucl-th] .
  * Molnar and Voloshin (2003) D. Molnar and S. A. Voloshin, Phys. Rev. Lett. 91, 092301 (2003), arXiv:nucl-th/0302014 .
  * Hwa and Yang (2004) R. C. Hwa and C. B. Yang, Phys. Rev. C70, 024905 (2004), arXiv:nucl-th/0401001 [nucl-th] .
  * Adams  _et al._ (2005b) J. Adams _et al._ (STAR), Phys. Rev. C72, 014904 (2005b), arXiv:nucl-ex/0409033 [nucl-ex] .
  * Adams  _et al._ (2005c) J. Adams _et al._ (STAR), Phys. Rev. Lett. 95, 122301 (2005c), arXiv:nucl-ex/0504022 [nucl-ex] .
  * Adare  _et al._ (2007) A. Adare _et al._ (PHENIX), Phys. Rev. Lett. 98, 162301 (2007), arXiv:nucl-ex/0608033 [nucl-ex] .
  * Abelev  _et al._ (2007) B. I. Abelev _et al._ (STAR), Phys. Lett. B655, 104 (2007), arXiv:nucl-ex/0703040 [nucl-ex] .
  * Adare  _et al._ (2012) A. Adare _et al._ (PHENIX), Phys. Rev. C85, 064914 (2012), arXiv:1203.2644 [nucl-ex] .
  * Adare  _et al._ (2011) A. Adare _et al._ (PHENIX), Phys. Rev. C83, 064903 (2011), arXiv:1102.0753 [nucl-ex] .
  * Ryu  _et al._ (2018) S. Ryu, J.-F. Paquet, C. Shen, G. Denicol, B. Schenke, S. Jeon, and C. Gale, Phys. Rev. C97, 034910 (2018), arXiv:1704.04216 [nucl-th] .
  * Knospe  _et al._ (2021) A. G. Knospe, C. Markert, K. Werner, J. Steinheimer, and M. Bleicher, (2021), arXiv:2102.06797 [nucl-th] .
  * Chen  _et al._ (2020) W. Chen, S. Cao, T. Luo, L.-G. Pang, and X.-N. Wang, Phys. Lett. B810, 135783 (2020), arXiv:2005.09678 [hep-ph] .
  * Pang  _et al._ (2012) L. Pang, Q. Wang, and X.-N. Wang, Phys. Rev. C86, 024911 (2012), arXiv:1205.5019 [nucl-th] .
  * Pang  _et al._ (2018) L.-G. Pang, H. Petersen, and X.-N. Wang, Phys. Rev. C97, 064918 (2018), arXiv:1802.04449 [nucl-th] .
  * Wang and Zhu (2013) X.-N. Wang and Y. Zhu, Phys. Rev. Lett. 111, 062301 (2013), arXiv:1302.5874 [hep-ph] .
  * Li  _et al._ (2011) H. Li, F. Liu, G.-l. Ma, X.-N. Wang, and Y. Zhu, Phys. Rev. Lett. 106, 012301 (2011), arXiv:1006.2893 [nucl-th] .
  * He  _et al._ (2015) Y. He, T. Luo, X.-N. Wang, and Y. Zhu, Phys. Rev. C91, 054908 (2015), [Erratum: Phys. Rev.C97,no.1,019902(2018)], arXiv:1503.03313 [nucl-th] .
  * Cao  _et al._ (2016) S. Cao, T. Luo, G.-Y. Qin, and X.-N. Wang, Phys. Rev. C94, 014909 (2016), arXiv:1605.06447 [nucl-th] .
  * Cao  _et al._ (2018) S. Cao, T. Luo, G.-Y. Qin, and X.-N. Wang, Phys. Lett. B777, 255 (2018), arXiv:1703.00822 [nucl-th] .
  * Guo and Wang (2000) X.-F. Guo and X.-N. Wang, Phys. Rev. Lett. 85, 3591 (2000), arXiv:hep-ph/0005044 .
  * Wang and Guo (2001) X.-N. Wang and X.-F. Guo, Nucl. Phys. A696, 788 (2001), arXiv:hep-ph/0102230 .
  * Zhang and Wang (2003) B.-W. Zhang and X.-N. Wang, Nucl. Phys. A720, 429 (2003), arXiv:hep-ph/0301195 .
  * Schafer  _et al._ (2007) A. Schafer, X.-N. Wang, and B.-W. Zhang, Nucl. Phys. A793, 128 (2007), arXiv:0704.0106 [hep-ph] .
  * Bazavov  _et al._ (2014) A. Bazavov _et al._ (HotQCD), Phys. Rev. D90, 094503 (2014), arXiv:1407.6387 [hep-lat] .
  * Moreland  _et al._ (2015) J. S. Moreland, J. E. Bernhard, and S. A. Bass, Phys. Rev. C92, 011901 (2015), arXiv:1412.4708 [nucl-th] .
  * (94) .
  * Sjostrand  _et al._ (2008) T. Sjostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun. 178, 852 (2008), arXiv:0710.3820 [hep-ph] .
  * Eskola  _et al._ (2017) K. J. Eskola, P. Paakkinen, H. Paukkunen, and C. A. Salgado, Eur. Phys. J. C77, 163 (2017), arXiv:1612.05741 [hep-ph] .
  * Han  _et al._ (2016) K. C. Han, R. J. Fries, and C. M. Ko, Phys. Rev. C93, 045207 (2016), arXiv:1601.00708 [nucl-th] .
  * Zhao  _et al._ (2020) W. Zhao, C. M. Ko, Y.-X. Liu, G.-Y. Qin, and H. Song, Phys. Rev. Lett. 125, 072301 (2020), arXiv:1911.00826 [nucl-th] .
  * (99) .
  * (100) .
  * Kumar  _et al._ (2020) A. Kumar _et al._ (JETSCAPE), Phys. Rev. C102, 054906 (2020), arXiv:1910.05481 [nucl-th] .
  * Bass  _et al._ (1998) S. A. Bass _et al._ , Prog. Part. Nucl. Phys. 41, 255 (1998), [Prog. Part. Nucl. Phys.41,225(1998)], arXiv:nucl-th/9803035 [nucl-th] .
  * Bleicher  _et al._ (1999) M. Bleicher _et al._ , J. Phys. G25, 1859 (1999), arXiv:hep-ph/9909407 [hep-ph] .
  * (104) .
  * Acharya  _et al._ (2018b) S. Acharya _et al._ (ALICE), JHEP 11, 013 (2018b), arXiv:1802.09145 [nucl-ex] .
  * Acharya  _et al._ (2020) S. Acharya _et al._ (ALICE), Phys. Rev. C101, 044907 (2020), arXiv:1910.07678 [nucl-ex] .
  * Acharya  _et al._ (2018c) S. Acharya _et al._ (ALICE), JHEP 09, 006 (2018c), arXiv:1805.04390 [nucl-ex] .


# r-BTN: Cross-domain Face Composite and Synthesis from Limited Facial Patches

Yang Song, Zhifei Zhang, Hairong Qi
Department of Electrical Engineering and Computer Science
University of Tennessee, Knoxville, TN 37996, USA
{ysong18, zzhang61<EMAIL_ADDRESS>

###### Abstract

Recent face composite and synthesis related works have shown promising results
in generating realistic face images from deep convolutional networks. However,
these works either do not generate consistent results when the constituent
patches contain large domain variations (i.e., from face and sketch domains)
or cannot generate high-resolution images with limited facial patches (e.g.,
the inpainting approach tends to blur the generated region when the missing
area is more than 50%). Motivated by the mental imagery and simulation in
human cognition, we exploit the potential of deep learning networks in filling
large missing region (e.g., as high as 95% missing) and generating realistic
faces with high-fidelity in cross domains. We propose the recursive generation
by bidirectional transformation networks (r-BTN) that recursively generates a
whole face/sketch from a small sketch/face patch. The large missing area and
domain variations make it difficult to generate satisfactory results using a
unidirectional cross-domain learning structure. We explore that the
bidirectional transformation network can lead to the consistent result by
minimizing the forward and backward errors in the cross-domain scenario. On
the other hand, a forward and backward bidirectional learning between the face
and sketch domains would enable recursive estimation of the missing region in
an incremental manner to yield appealing results. r-BTN also adopts an
adversarial constraint to encourage the generation of realistic
faces/sketches. Extensive experiments have been conducted to demonstrate the
superior performance from r-BTN as compared to existing potential solutions.

## Introduction

We start by asking an interesting yet challenging question, “If provided with
limited facial patches from sketch/face domains where human beings may be able
to generate a real face image in brain (?) as shown in Fig. 1, can advanced
computer vision techniques generate the whole face image?” Recently, several
face synthesis methods built on neural networks have emerged (?; ?). These
methods can generate face/sketch images based on whole face information from
one domain. However, how to generate realistic faces/sketches that are
consistent to the given sketch/face patches is still a challenging task
because large missing area could lead to blurry generated images. In addition,
some existing methods (e.g., Photofit (?)) synthesize faces by stitching
patches from cross domains which deteriorates the consistency and photo-
reality. It is still unclear how to preserve the color/domain consistency
between patches with large domain variations.

In this paper, we study the above-mentioned problems which would play a key
role in many applications, such as face image stitching, face blending, face
editing, etc. To the best of our knowledge, this work represents the first
attempt to cross-filling large missing area in both face and sketch domains.
Existing works that may potentially address this problem are mainly in the
perspectives of face/sketch synthesis/transformation and image inpainting.

Figure 1: Illustration of face composite based on cross-domain patches and
face synthesis from limited facial patch.

The face/sketch synthesis works (?; ?; ?; ?) synthesize target faces from the
source domain through patch-wise searching of similar patches in the training
set. Without the generative capability, these methods fail to render
reasonable pixels for large missing areas. The rapid development of generative
adversarial networks (GANs) (?) has shown impressive performance in face
generation (?; ?), domain transformation (?; ?), and inpainting (?; ?).
However, generating faces from small patches in either single or cross domains
has not been explored. Intuitively, combining domain transformation and
inpainting works could be a potential solution. However, with large missing
area, the generated results tend to be blurred and may look unrealistic.

In this paper, we investigate the problem of cross-domain face/sketch
generation conditioned on a given small patch of sketch/face. We assume that
faces and sketches lie on high-dimensional manifolds $\mathcal{I}$ and
$\mathcal{S}$, respectively, as shown in Fig. 2 (right). The given small
sketch/face patch will initially deviate from the corresponding manifold due
to large amount of missing data. With the learned bidirectional transformation
network (BTN), i.e., $f$ and $F$, the given patch will be recursively mapped
forward and backward between $\mathcal{I}$ and $\mathcal{S}$. Each mapping
will yield a result progressively closing in onto either the face or sketch
manifold, and eventually approaching the real whole face/sketch images as
shown in Fig. 2 (middle). An adversarial network is imposed on both $f$ and
$F$, forcing more photo-realistic faces/sketches. The rationale and benefit of
the proposed r-BTN will be further discussed.

This paper makes the following contributions: 1) We tackle the challenging
problem of face/sketch generation from small patches, estimating large missing
area based on limited information while alleviating the blur effect suffered
by existing works. 2) We propose the recursive generation by bidirectional
transformation networks (r-BTN), which learns both a forward and backward
mapping function between cross domains to enable a recursive update of the
generated faces/sketches for more consistent and high-fidelity results even
with large portions of missing data. 3) We further exploit the capacity of
r-BTN in fusing multiple patches from multiple domains and multiple people
(i.e., face composite) to output a realistic and consistent face in a
generative manner.

## Related Works

We will discuss related works from three closely related areas, namely,
face/sketch synthesis/transformation, image inpainting, and face manipulation.

Face/Sketch Synthesis/Transformation related works mainly fall into two
categories: matching-based and generation-based methods. Most face/sketch
synthesis works (?; ?; ?) are matching-based, which synthesize faces from best
matched patches by searching from the training dataset. For example, (?)
divided a given face/sketch image into patches, each of which was matched to a
series of similar patches from the training dataset. Then, the patches in the
target domain corresponding to the matched patches were stitched via Markov
random field to synthesize a transformed face. The matching-based methods have
two drawbacks: 1) The matching procedure is time-consuming for a large
training dataset, and 2) they cannot effectively estimate the patch content
from missing area. The generation-based methods (?; ?) are mainly developed
from encoder-decoder networks and adversarial generative networks. For
example, (?; ?) proposed a general domain transformation method through
conditional generative adversarial network. It could also be utilized for
face/sketch transformation. However, it is not trained for the purpose of
estimating missing areas. Moreover, to achieve bidirectional face/sketch
transformation, two transformation networks (i.e., face to sketch and sketch
to face) need to be learned independently.

Figure 2: Examples of recursive generation from small patches by the
bidirectional transformation network. Left: Original face/sketch and the
corresponding input patches extracted from them. Inside of the dashed box
demonstrates the generated face/sketch at different iteration steps. Right:
Illustration of transformation between the face and sketch manifolds
$\mathcal{I}$ and $\mathcal{S}$, respectively. The green dot denotes a given
face patch. The red and blue arrows are the learned mapping $f$ and $F$,
respectively. The red and blue dots are generated sketches and faces through
corresponding mapping.

Image Inpainting aims to fill in unwanted or missing part of an image. Most
inpainting methods (?; ?; ?) estimate the missing part based on surrounding
pixels, and therefore are not suitable for filling in large missing areas.
Although some recent works (?; ?) claimed the ability of filling in up to 80%
missing regions, they tend to generate blurred results, which may be with
visible inconsistency between the given and estimated areas. In addition,
inpainting related methods train on randomly masked inputs and perform filling
in a single domain, while the proposed work uses the whole face/sketch pairs
in training and perform cross-domain filling.

Face Manipulation works (?; ?) could be a potential solution to the proposed
task because they can generate faces by manipulating the latent variables.
Given a small patch, they may search the latent space for a best matched face.
Thus, the generative model performs like matching-based methods which may be
time-consuming. A more efficient way is to minimize the error between the
generated face and the given patch. However, it cannot ensure consistent
results because only the patch location (where the error comes from) will be
updated regardless of its surroundings.

## The Bidirectional Transformation Network

In this section, we first elaborate on the benefit of the proposed BTN through
a comparison with unidirectional transformations. This is followed by a
detailed description of the training and testing stages of the proposed r-BTN.
The training stage learns the bidirectional transformation between the face
and sketch domains using whole face/sketch pairs. The testing stage
recursively generates the whole face/sketch from given small sketch/face
patches.

### The Bidirectional Network Structure

Assume a training set in $\mathcal{I}\times\mathcal{S}$, where $\mathcal{I}$
and $\mathcal{S}$ denote the face and sketch domains, respectively. The
unidirectional transformation, e.g., (?), learns a mapping
$f:\mathcal{I}\rightarrow\mathcal{S}$ which could be implemented by encoder-
decoder networks, as shown in Fig. 3 (left). The BTN, on the other hand,
simultaneously involves the forward mapping $f$ and backward mapping
$F:\mathcal{S}\rightarrow\mathcal{I}$, as shown in Fig. 3 (right). The
bidirectional transformation forms a closed loop where the output of $f$
serves as the input to $F$, and the output of $F$ serves as the input to $f$
in the next iteration. The forward transformation $f$ may discard information
in general due to the domain difference (e.g., color information will be
discarded from $\mathcal{I}$ to $\mathcal{S}$), but the backward
transformation $F$ closes the loop by connecting the output from $f$ in the
$\mathcal{S}$ domain and the original input in the $\mathcal{I}$ domain and
generates an intermediate result in $\mathcal{I}$ where additional face
information (e.g., facial outline) has been estimated and the discarded
information (e.g., color) restored. The bidirectional network structure
enables the recursive update of the face (from $F$) and sketch (from $f$),
taking advantage of the progressively learned knowledge in both domains and
generate full face/sketch with high fidelity.

Figure 3: Comparison of unidirectional and bidirectional transformations
between $\mathcal{I}$ and $\mathcal{S}$ domains. E and D are the encoder-
decoder networks. The patch (eyes) generates the sketch, and then the sketch
is transformed back where facial outline has been estimated.

The effectiveness of the recursive bidirectional transformation between face
and sketch domains is well demonstrated in Fig. 2. In general, the missing
area is roughly filled at the beginning (iteration 1 and 2) although it is
blurred. Then, facial details are progressively enhanced (iteration 10) and
sharpened (iteration 20). Finally, a realistic face/sketch, including
reasonable hair style, is generated. Because of the very limited information
provided in the input patch, it is difficult to generate a face/sketch exactly
the same as the original. However, the generated face/sketch still preserves
the pixel-level content of the given patch.

### Training Stage

Fig. 4 illustrates the details of the BTN structure where the mapping
functions, $f$ and $F$, are learned in a bidirectional fashion instead of the
commonly used unidirectional mapping.

Figure 4: Training flow of the bidirectional transformation network.
$x_{\mathcal{I}}$ and $x_{\mathcal{S}}$ are the real face/sketch pair. Red and
blue arrows denote the transformation paths of $x_{\mathcal{I}}$ and
$x_{\mathcal{S}}$, respectively. The transformation functions $f$ and $F$
could be encoder-decoder networks. $Loss$ denotes the $\ell_{1}$-norm. The
discriminator $D$ is trained on real and generated (fake) face/sketch pairs.

Given the original face/sketch pair $x_{\mathcal{I}}$ and $x_{\mathcal{S}}$,
the following transformations are performed,

$\displaystyle x_{\mathcal{S}}^{0}=f(x_{\mathcal{I}}),\;$ $\displaystyle
x_{\mathcal{I}}^{1}=F(x_{\mathcal{S}}^{0})=F(f(x_{\mathcal{I}})),$
$\displaystyle x_{\mathcal{I}}^{0}=F(x_{\mathcal{S}}),\;$ $\displaystyle
x_{\mathcal{S}}^{1}=f(x_{\mathcal{I}}^{0})=f(F(x_{\mathcal{S}})).$

The objective is to learn the bidirectional transformations between
$\mathcal{I}$ and $\mathcal{S}$, so that any face/sketch pair could be
uniquely mapped forward and backward into another domain. To achieve
invertible transformation, i.e., preserving the identity of face and sketch
during transformations, we minimize the reconstruction error
$\mathcal{L}_{rec}$ between real and generated faces or sketches as Eq. 1.

$\mathcal{L}_{rec}=\sum_{i=0}^{1}\left(\|x_{\mathcal{I}}-x_{\mathcal{I}}^{i}\|_{1}+\|x_{\mathcal{S}}-x_{\mathcal{S}}^{i}\|_{1}\right),$
(1)

where the $\ell_{1}$-norm instead of the $\ell_{2}$-norm is used to avoid
blurry results. Besides $\mathcal{L}_{rec}$, an adversarial constraint is
employed to encourage photo-realistic face/sketch pairs. The discrimination
loss can be written as

$\mathcal{L}_{adv}=\mathbb{E}_{\omega\in{\Omega}}\left[\log
D(\omega)\right]-\mathbb{E}_{\begin{subarray}{c}x_{\mathcal{I}}\in\mathcal{I}\\\
x_{\mathcal{S}}\in\mathcal{S}\end{subarray}}[\log
D(x_{\mathcal{I}},x_{\mathcal{S}})],$ (2)

where

$\displaystyle\Omega=$
$\displaystyle\left\\{(x_{\mathcal{I}},x_{\mathcal{S}}^{0})_{j},(x_{\mathcal{I}}^{1},x_{\mathcal{S}}^{0})_{j},(x_{\mathcal{I}}^{0},x_{\mathcal{S}})_{j},(x_{\mathcal{I}}^{0},x_{\mathcal{S}}^{1})_{j}\right\\}$
$\displaystyle=$
$\displaystyle\left\\{(x_{\mathcal{I}},f(x_{\mathcal{I}}))_{j},(F(f(x_{\mathcal{I}}),f(x_{\mathcal{I}})))_{j},\right.$
$\displaystyle\;\;\left.(F(x_{\mathcal{S}}),x_{\mathcal{S}})_{j},(F(x_{\mathcal{S}}),f(F(x_{\mathcal{S}})))_{j}\right\\}$

indicates the fake face/sketch pairs, and $j$ indexes the fake pairs generated
from the $j$th real pair in a mini-batch. Note that only
$\left(x_{\mathcal{I}},x_{\mathcal{S}}\right)$ is the real pair. Combining
Eqs. 1 and 2, the objective function is

$\min_{f,F,D}\;\mathcal{L}_{adv}+\lambda\mathcal{L}_{rec},$ (3)

where $\lambda$ balances the adversarial loss and reconstruction loss. In
optimization, $f$, $F$, and $D$ are updated alternatively. The discriminator
$D$ is updated by minimizing $\mathcal{L}_{adv}$. The update of $f$ and $F$ is
performed by

$\displaystyle\min_{f}\;$
$\displaystyle\mathbb{E}_{\omega\in\Omega_{f}}\left[\log
D(\omega)\right]+\lambda\sum_{i=0}^{1}\|x_{\mathcal{S}}-x_{\mathcal{S}}^{i}\|_{1},$
(4) $\displaystyle\min_{F}\;$
$\displaystyle\mathbb{E}_{\omega\in\Omega_{F}}\left[\log
D(\omega)\right]+\lambda\sum_{i=0}^{1}\|x_{\mathcal{I}}-x_{\mathcal{I}}^{i}\|_{1},$
(5)

where

$\displaystyle\Omega_{f}=$
$\displaystyle\left\\{(x_{\mathcal{I}},x_{\mathcal{S}}^{0})_{j},(x_{\mathcal{I}}^{0},x_{\mathcal{S}}^{1})_{j}\right\\}$
$\displaystyle=$
$\displaystyle\left\\{(x_{\mathcal{I}},f(x_{\mathcal{I}}))_{j},(x_{\mathcal{I}}^{0},f(x_{\mathcal{I}}^{0}))_{j}\right\\},$
$\displaystyle\Omega_{F}=$
$\displaystyle\left\\{(x_{\mathcal{I}}^{0},x_{\mathcal{S}})_{j},(x_{\mathcal{I}}^{1},x_{\mathcal{S}}^{0})_{j}\right\\}$
$\displaystyle=$
$\displaystyle\left\\{(F(x_{\mathcal{S}}),x_{\mathcal{S}})_{j},(F(x_{\mathcal{S}}^{0}),x_{\mathcal{S}}^{0})_{j}\right\\},$

and $\Omega=\Omega_{f}\cup\Omega_{F}$. Here, $j$ is again the index of
training samples in a mini-batch.

### Testing Stage

During testing, given an arbitrary patch from either domain, a whole face from
the other domain could be generated in a recursive manner through the
bidirectional transformation. The testing flow is shown in Fig. 5,

Figure 5: Testing flow of r-BTN, assuming a face patch $p_{\mathcal{I}}$ as
the input. At step $k$, the generated face is $x_{\mathcal{I}}^{k}$. Replacing
the corresponding area of $x_{\mathcal{I}}^{k}$ by the patch $p_{\mathcal{I}}$
and transforming $x_{\mathcal{I}}^{k}$ to $x_{\mathcal{S}}^{k}$, we get a
face/sketch pair $\left(x_{\mathcal{I}}^{k},x_{\mathcal{S}}^{k}\right)$. Then,
this pair is adjusted by the error back propagated from $D$ as comparing to
the output of real pairs. Finally, $x_{\mathcal{S}}^{k}$ is transformed back
to the face domain, generating $x_{\mathcal{I}}^{k+1}$.

which demonstrates the case of given a face patch $p_{\mathcal{I}}$.
Similarly, if a sketch patch $p_{\mathcal{S}}$ is given, it will be fed to
$x_{\mathcal{S}}$ and similar testing flow can be carried out to generate a
whole face image. In this paper, a patch is created through multiplying a
whole face/sketch by a mask $M$, e.g., $p_{\mathcal{I}}=x_{\mathcal{I}}\odot
M$ where $\odot$ denotes the element-wise multiplication.

The bidirectional transformation network structure enables a recursive
generation between sketches and faces. Given the current result
$x_{\mathcal{I}}^{k}$, the next generation $x_{\mathcal{I}}^{k+1}$ can be
obtained by

$\displaystyle{x}_{\mathcal{I}}^{k}$ $\displaystyle\leftarrow
x_{\mathcal{I}}^{k}\odot(1-M)+p_{\mathcal{I}},$ (6)
$\displaystyle{x}_{\mathcal{S}}^{k}$ $\displaystyle\leftarrow
f\left({x}_{\mathcal{I}}^{k}\right),$ (7) $\displaystyle{x}_{\mathcal{S}}^{k}$
$\displaystyle\leftarrow{x}_{\mathcal{S}}^{k}-\frac{\partial
D(x_{\mathcal{I}}^{k},x_{\mathcal{S}}^{k})}{\partial x_{\mathcal{S}}^{k}},$
(8) $\displaystyle{x}_{\mathcal{I}}^{k+1}$ $\displaystyle\leftarrow
F\left({x}_{\mathcal{S}}^{k}\right).$ (9)

In order to generate photo-realistic faces/sketches such that the given patch
and the estimated complement blend together in a consistent fashion, we have
applied two constraints during the recursive generation process. First, we
keep the given patch, $p_{\mathcal{I}}$, as the anchor that remains the same
across different iterations. In other words, $p_{\mathcal{I}}$ directly covers
the corresponding area of the newly generated face to explicitly preserve the
given content (Eq. 6). Then, $x_{\mathcal{I}}^{k}$ is transformed to the
sketch domain by $f$ (Eq. 7). Unlike most GANs related works which utilize $D$
only in the training stage, we utilize $D$ as a second constraint in the
testing process to ensure realistic faces/sketches generation in each
iteration such that small deviations get to be corrected instead of
accumulated through iterations.

Given a small patch, the testing stage needs multiple iterations to gradually
generate a whole face/sketch, as illustrated previously in Fig. 2. In each
iteration, backpropagating the loss of $D$ will enforce the photo-reality
during the recursive generation. In the case of Fig. 5, the backpropagation
error is used to adjust the generated sketch $x_{\mathcal{S}}^{k}$ as shown in
Eq. 8. Finally, $x_{\mathcal{S}}^{k}$ is mapped back to the face domain (Eq.
9), generating $x_{\mathcal{I}}^{k+1}$ as an improved version of
$x_{\mathcal{I}}^{k}$ with more details. Repeating this procedure, the large
missing area can be filled up gradually.

To illustrate the effect of the two constraints, i.e., the given patch and the
adversarial constraints, applied during the testing stage, Fig. 6 shows the
generated results with/without the constraints. The given patch and the
adversarial constraints are denoted as “Patch” and “Adv”, respectively. It is
interesting to observe that the generated face/sketch without “Patch” (the
second and third columns) cannot preserve the identity of the input patch, and
those without “Adv” (the second and forth columns) tend to yield unrealistic
face/sketch (e.g., the left ear location) or hair style (e.g., the extra hair
below the left ear in the fourth column). The results with both constraints
obviously outperform the others.

Figure 6: Comparison of generated results with/without the given patch (Patch)
and adversarial (Adv) constraints.

## Experiment and Results

### Data Collection

We collect 1,577 face/sketch pairs from the datasets CUHK (?), CUFSF (?), AR
(?), FERET (?), and IIIT-D (?). Because the dataset with face/sketch pairs is
limited, we train a face to sketch transformation network based on Pix2Pix (?)
to generate sketches from faces. We collect frontal face images with uniform
background and controlled illumination from datasets CFD (?), SiblingsDB (?),
and PUT (?), as well as from searching engines by keywords like “XXX
University faculty profile”. Finally, we obtain 3,126 face/sketch pairs, from
which 300 pairs are randomly selected as the testing dataset.

### Implementation Details

All the face/sketch images are cropped and well-aligned based on the eye
locations, and preprocessed to be uniform white background. The
transformations $f$ and $F$ are implemented by the Conv-Deconv network. The
discriminator $D$ is implemented by the Conv network but adding a fully-
connected layer of single output with the sigmoid activation function. In
addition, the input layer is modified to be $256^{2}\times 6$ because the
inputs to $D$ are image pairs. Inspired by (?), each Conv layer is
concatenated to its symmetrically corresponding Deconv layer, thus more
details bypass the bottleneck. In the training, we adopt ADAM (?)
($\alpha=0.0002,\;\beta=0.5$). Because we utilize $D$ to enforce realistic
generations during testing, an approximately optimal $D$ is preferred.
Therefore, we update $D$ three times for each update of $f$ and $F$. The
parameter $\lambda$ in Eq. 3 is set to be 100. Details are shown in
supplementary materials. After 100 epochs, we could achieve the results as
shown in this paper.

During testing, given a small patch from either the face or the sketch domain,
it will be transformed recursively as discussed in testing stage. Empirically,
the generated images will have most facial features filled quickly at the
first five to ten iterations and then tend to converge after 50 iterations.
The results shown in this paper are mostly obtained at the 100th iteration.

### Qualitative Evaluation

#### Face Composite

We explore the r-BTN to generate consistent and realistic faces from multiple
patches that may be from two domains and multiple people. Examples generated
from multiple patches are shown in Fig. 7, demonstrating the great versatility
of r-BTN. We again observe the strong consistency and fidelity between the
generated face/sketch pairs.

Figure 7: Examples of generated faces/sketches from multiple patches, which
are from different people and/or different domains. Four examples are
displayed in a 2-by-2 matrix. In each cell, the original faces and sketches
are given on the left. The patches are extracted from where indicated by the
arrows. The right are generated face/sketch pairs.

#### Face Synthesis from Limited Facial Patches

The results generated by proposed r-BTN with respect to different missing
percentage are shown in Fig. 8. From the result, it demonstrated that the
proposed method cannot preserve the identity when the missing percentage is
more than 70%. This phenomenon is consistent with human cognitive. For human
beginnings, if only providing limited information, it is still hard to imagine
a unique result.

Figure 8:  Comparison in generating faces/sketches from patches with different
missing percentage. The red boxes indicate the given face/sketch patches. The
rest rows are correspondingly generated sketches/faces.

We also compare the proposed r-BTN with Pix2Pix (?) and image inpainting (?).
The inpainting method compared in this paper is modified from (?) to achieve
cross-domain inpainting. Specifically, the inputs are faces/sketches with
random mask (20%$\sim$80% masked), and the outputs are the whole sketch/face.
Pix2Pix and r-BTN are trained with the whole face/sketch pairs. All methods
are trained on the same training dataset with the same parameter setting. The
comparison results are shown in Fig. 9. The Pix2Pix and inpainting methods
train face-sketch and sketch-face transformation networks independently, so
the identity between generated sketches and faces cannot be preserved. For
example, comparing the two rows labeled with “inpainting”, especially the
4th-6th columns, the sketches seem female while the faces appear like male. In
addition, the inpainting results present apparent discontinuity between the
given patch and the estimated area. On the other hand, the results from r-BTN
demonstrate higher fidelity, better consistency to given patches, and better
identity preservation. More results are shown in supplementary materials.

Figure 9: Comparison with other potential methods for filling large missing
areas. The first row shows the input patches, and the rest rows display the
results from different methods. The percentage indicates missing proportion
(missing area over image area). Because Pix2Pix is for domain transfer rather
than missing area filling, its results cannot compete with inpainting or
r-BTN. We show them here to provide the baseline of domain transfer methods in
filling large missing areas.

### Quantitative Analysis

#### Evaluation Metrics

To numerically evaluate the quality of generated faces, we design the metric
named “face recognition rate (FRR)”. It evaluates whether the generated images
present facial elements and geometric structure, i.e., reasonable position of
eyebrows, eyes, nose, lips, and chin. We adopt the off-the-shelf face landmark
detection method (?) to detect and localize those facial elements. An
unsuccessful detection indicates a failure of face generation. Therefore, FRR
is the ratio between the numbers of successfully detected and total generated
faces. Fig. 10 (left) shows FRR of each method, computed from 300 generated
faces using patches with different missing percentages. We observe that when
the missing percentage is larger than 50%, Pix2Pix fails to generate
reasonable faces while inpainting and r-BTN maintain high and similar FRR.
However, we recall from Fig. 9 that inpainting results are not photo-realistic
as r-BTN although they are both capable of preserving the facial structure.

#### Convergence of Recursive Generation

Will the generated faces/sketches converge to a certain point? How many
iterations are sufficient to achieve a photo-realistic result? This section
mainly answers these two questions.

We first define the residual in the face domain between subsequent iterations
as $r^{k+1}=\left(x_{\mathcal{I}}^{k+1}-x_{\mathcal{I}}^{k}\right)$, where
$x_{\mathcal{I}}^{k}$ and $x_{\mathcal{I}}^{k+1}$ denote the $k$th and $k$+1th
generated results. The convergence is mainly evaluated by calculating the
averaged residual on testing samples (i.e., 300 samples generated with
different missing percentage) with respect to $k$ as shown in Fig. 10
(middle). However, the average residual is not sufficient to demonstrate the
convergence because some pixels may significantly increase while the other
decrease with the same level. In this case, we calculate the averaged absolute
residual which illustrate the changing amplitude as shown in Fig. 10 (right).

Figure 10: Left:Comparison of different methods on the proposed metrics: FRR.
Middle and Right: Convergence evaluation of the proposed r-BTN. Averaged
absolute (middle) and average (right) of residual with respect to iteration
$k$ are shown at missing percentage of 95%, 80%, 60%, 40%, and 20%,
respectively.

With more iterations, the averaged residual approaches zero while the averaged
absolute residual stabilizes at a small value. This well demonstrates that the
generated faces are stable. In addition, from the experiments (e.g., Fig. 2
and 8), the generated faces/sketches will not significantly change after 20
iterations. Therefore, we could empirically conclude that the recursive
generation will converge to certain face/sketch for a given patch.

#### Similarity/Diversity Evaluation

Intuitively speaking, the generated faces from the patches of the same person
should be similar. By contrast, patches from different persons are supposed to
yield diverse faces. To verify this property, we collect 50 faces and pick
patches of different size around the eyes, the nose, and the mouth. The
proposed r-BTN is then applied to generate full faces from those patches. To
measure the similarity/diversity between generated faces, we utilize the pre-
trained VGG-Face (?) model to extract high-level features and compute their
Euclidean distance. We perform two comparisons: 1) self comparison
(similarity) and 2) mutual comparison (diversity), conducting on faces
generated from patches of the same and different persons, respectively. Fig.
11 (left) shows the averaged distance and standard deviation with respect to
missing percentage. The blue circles shows the results of self comparison, and
the red triangles denote mutual comparison.

With lower missing percentage, e.g., 0.1 to 0.6, the generated faces preserve
relatively high intra-class (same person) similarity and inter-class
(different persons) diversity. As the missing percentage increases, the two
curves eventually intersect, indicating the generated faces from very small
patches (e.g., 95% missing) have lost the identity of the original face.
Interestingly, we discover that the generated faces from either the left or
right eye of the same person still tend to be more similar as compared to
those generated from nose/mouth as illustrated in Fig. 11 (right). This
discovery is well in line with the quality of different biometrics where
studies have shown eyes to carry more valuable cues than nose or mouth in face
recognition tasks. This finding, from another perspective, demonstrates the
high effectiveness of r-BTN in generating high-fidelity and realistic
faces/sketches.

Figure 11: Left: Evaluation of similarity/diversity with increasing missing
percentage. The bars indicate corresponding standard deviation. Middle and
Right: High-level feature of generated faces at missing percentage of 10% and
95%, respectively. There are three same markers for type (person), denoting
the generated faces from patches around left eye, right eye, and mouth. Solid
lines connect the faces generated from eyes, and the dashed lines connect to
the faces generated from mouth.

## Discussion and Future works

In this paper, we proposed and solved the challenging task of cross-domain
face generation with large missing area. A novel recursive generation method
by bidirectional transformation networks (r-BTN) was proposed to achieve high-
fidelity and consistent face/sketch even with as large as $95\%$ missing area.
We demonstrated the effectiveness of r-BTN by comparing to some potential
solutions like pix2pix and inpainting. However, r-BTN requires well-aligned
faces/sketches. Otherwise, the generated results may not be visually pleasing
because the network would fail to localize facial components and thus missing
their geometric structure. In the future, we plan to improve the proposed
r-BTN from four perspectives: 1) concatenating a face calibration mechanism to
r-BTN to battle against the alignment problem, 2) extending this work to be
unsupervised like (?; ?) to alleviate the requirement for paired dataset, 3)
generalizing r-BTN as a framework for cross-domain transformation, especially
with large missing area, and further evaluating the performance on other
datasets (?), and 4) adpting this for mobile network application (?).

## References

  * [Bhatt et al. 2012] Bhatt, H. S.; Bharadwaj, S.; Singh, R.; and Vatsa, M.  2012\.  Memetically optimized MCWLD for matching sketches with digital face images.  IEEE Transactions on Information Forensics and Security 7(5):1522–1535.
  * [Criminisi, Pérez, and Toyama 2004] Criminisi, A.; Pérez, P.; and Toyama, K.  2004\.  Region filling and object removal by exemplar-based image inpainting.  IEEE Transactions on Image Processing 13(9):1200–1212.
  * [Du, Abdalmageed, and Doermann 2013] Du, X.; Abdalmageed, W.; and Doermann, D.  2013\.  Large-scale signature matching using multi-stage hashing.  In IEEE Document Analysis and Recognition (ICDAR), 2013 12th International Conference on, 976–980.
  * [Efros and Leung 1999] Efros, A. A., and Leung, T. K.  1999\.  Texture synthesis by non-parametric sampling.  In IEEE International Conference on Computer Vision, volume 2, 1033–1038.
  * [Goodfellow et al. 2014] Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y.  2014\.  Generative adversarial nets.  In Advances in Neural Information Processing Systems, 2672–2680.
  * [Isola et al. 2016] Isola, P.; Zhu, J.-Y.; Zhou, T.; and Efros, A. A.  2016\.  Image-to-image translation with conditional adversarial networks.  arXiv preprint arXiv:1611.07004.
  * [Kasinski, Florek, and Schmidt 2008] Kasinski, A.; Florek, A.; and Schmidt, A.  2008\.  The PUT face database.  Image Processing and Communications 13(3-4):59–64.
  * [Kazemi and Sullivan 2014] Kazemi, V., and Sullivan, J.  2014\.  One millisecond face alignment with an ensemble of regression trees.  In IEEE Conference on Computer Vision and Pattern Recognition, 1867–1874.
  * [Kingma and Ba 2014] Kingma, D., and Ba, J.  2014\.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * [Kosslyn, Thompson, and Ganis 2006] Kosslyn, S. M.; Thompson, W. L.; and Ganis, G.  2006\.  The case for mental imagery.  Oxford University Press.
  * [Li et al. 2016] Li, D.; Salonidis, T.; Desai, N. V.; and Chuah, M. C.  2016\.  Deepcham: Collaborative edge-mediated adaptive deep learning for mobile object recognition.  In IEEE Edge Computing (SEC), IEEE/ACM Symposium on, 64–76.
  * [Ma, Correll, and Wittenbrink 2015] Ma, D. S.; Correll, J.; and Wittenbrink, B.  2015\.  The Chicago face database: A free stimulus set of faces and norming data.  Behavior Research Methods 47(4):1122–1135.
  * [Martinez and Benavente 2007] Martinez, A., and Benavente, R.  2007\.  The AR face database.  Computer Vision Center, Technical Report 3\.
  * [Parkhi, Vedaldi, and Zisserman 2015] Parkhi, O. M.; Vedaldi, A.; and Zisserman, A.  2015\.  Deep face recognition.  In British Machine Vision Conference, 6.
  * [Pathak et al. 2016] Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; and Efros, A. A.  2016\.  Context encoders: Feature learning by inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition, 2536–2544.
  * [Phillips et al. 2000] Phillips, P. J.; Moon, H.; Rizvi, S. A.; and Rauss, P. J.  2000\.  The FERET evaluation methodology for face-recognition algorithms.  IEEE Transactions on Pattern Analysis and Machine Intelligence 22(10):1090–1104.
  * [Photofit ] Photofit.  http://www.open.edu/openlearn/body-mind/photofit-me.  [Online].
  * [Radford, Metz, and Chintala 2015] Radford, A.; Metz, L.; and Chintala, S.  2015\.  Unsupervised representation learning with deep convolutional generative adversarial networks.  arXiv preprint arXiv:1511.06434.
  * [Sangkloy et al. 2016] Sangkloy, P.; Lu, J.; Fang, C.; Yu, F.; and Hays, J.  2016\.  Scribbler: Controlling deep image synthesis with sketch and color.  arXiv preprint arXiv:1612.00835.
  * [Shen and Chan 2002] Shen, J., and Chan, T. F.  2002\.  Mathematical models for local nontexture inpaintings.  SIAM Journal on Applied Mathematics 62(3):1019–1043.
  * [Song et al. 2014] Song, Y.; Bao, L.; Yang, Q.; and Yang, M.-H.  2014\.  Real-time exemplar-based face sketch synthesis.  In European Conference on Computer Vision, 800–813.  Springer.
  * [Taigman, Polyak, and Wolf 2016] Taigman, Y.; Polyak, A.; and Wolf, L.  2016\.  Unsupervised cross-domain image generation.  arXiv preprint arXiv:1611.02200.
  * [Tang and Wang 2003] Tang, X., and Wang, X.  2003\.  Face sketch synthesis and recognition.  In IEEE International Conference on Computer Vision, 687–694.
  * [Vieira et al. 2014] Vieira, T. F.; Bottino, A.; Laurentini, A.; and De Simone, M.  2014\.  Detecting siblings in image pairs.  The Visual Computer 30(12):1333–1345.
  * [Wang and Tang 2009] Wang, X., and Tang, X.  2009\.  Face photo-sketch synthesis and recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11):1955–1967.
  * [Yan et al. 2016] Yan, X.; Yang, J.; Sohn, K.; and Lee, H.  2016\.  Attribute2image: Conditional image generation from visual attributes.  In European Conference on Computer Vision, 776–791.  Springer.
  * [Yeh et al. 2016] Yeh, R.; Chen, C.; Lim, T. Y.; Hasegawa-Johnson, M.; and Do, M. N.  2016\.  Semantic image inpainting with perceptual and contextual losses.  arXiv preprint arXiv:1607.07539.
  * [Zhang, Song, and Qi 2017a] Zhang, Z.; Song, Y.; and Qi, H.  2017a.  Age progression/regression by conditional adversarial autoencoder.  In IEEE Conference on Computer Vision and Pattern Recognition.
  * [Zhang, Song, and Qi 2017b] Zhang, Z.; Song, Y.; and Qi, H.  2017b.  Stabilizing the conditional adversarial network by decoupled learning.  In ICML Workshop on Implicit Models.
  * [Zhang, Wang, and Tang 2010] Zhang, W.; Wang, X.; and Tang, X.  2010\.  Lighting and pose robust face sketch synthesis.  In European Conference on Computer Vision, 420–433.  Springer.
  * [Zhang, Wang, and Tang 2011] Zhang, W.; Wang, X.; and Tang, X.  2011\.  Coupled information-theoretic encoding for face photo-sketch recognition.  In IEEE Conference on Computer Vision and Pattern Recognition, 513–520.
  * [Zhou, Kuang, and Wong 2012] Zhou, H.; Kuang, Z.; and Wong, K.-Y. K.  2012\.  Markov weight fields for face sketch synthesis.  In IEEE Conference on Computer Vision and Pattern Recognition, 1091–1097.
  * [Zhu et al. 2016] Zhu, J.-Y.; Krähenbühl, P.; Shechtman, E.; and Efros, A. A.  2016\.  Generative visual manipulation on the natural image manifold.  In European Conference on Computer Vision, 597–613.  Springer.
  * [Zhu et al. 2017] Zhu, J.-Y.; Park, T.; Isola, P.; and Efros, A. A.  2017\.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  arXiv preprint arXiv:1703.10593.