#  $p_{T}$ dispersion of inclusive jets in high-energy nuclear collisions

Shi-Yong Chen  Huanggang Normal University, Huanggang 438000, China  Key
Laboratory of Quark & Lepton Physics (MOE) and Institute of Particle Physics,
Central China Normal University, Wuhan 430079, China  Jun Yan  Key Laboratory
of Quark & Lepton Physics (MOE) and Institute of Particle Physics, Central
China Normal University, Wuhan 430079, China  Wei Dai  School of Mathematics
and Physics, China University of Geosciences, Wuhan 430074, China  Ben-Wei
<EMAIL_ADDRESS>Key Laboratory of Quark & Lepton Physics
(MOE) and Institute of Particle Physics, Central China Normal University,
Wuhan 430079, China  Guangdong Provincial Key Laboratory of Nuclear
Science,Institute of Quantum Matter, South China Normal University, Guangzhou
510006, China  Enke Wang  Guangdong Provincial Key Laboratory of Nuclear
Science,Institute of Quantum Matter, South China Normal University, Guangzhou
510006, China  Key Laboratory of Quark & Lepton Physics (MOE) and Institute of
Particle Physics, Central China Normal University, Wuhan 430079, China

( )

###### Abstract

In this study, we investigate the impact of jet quenching on the $p_{T}$
dispersion($p_{T}D$) of inclusive jets ($R=0.2$) in central Pb+Pb (0-10%)
collisions at $\sqrt{s}=2.76$ TeV. The partonic spectrum in the initial hard
scattering of elementary collisions is obtained by an event generator
POWHEG+PYTHIA, which matches the next-to-leading order (NLO) matrix elements
with parton showering, and the energy loss of a fast parton traversing through
hot/dense QCD medium is calculated using Monte Carlo simulation within the
Higher-Twist formalism of jet quenching in heavy-ion collisions. We present
model calculations of the normalized $p_{T}D$ distributions of inclusive jets
in p+p and central Pb+Pb collisions at $\sqrt{s}=2.76$ TeV, which offer good
descriptions of ALICE measurements. It is shown that the $p_{T}D$
distributions of inclusive jets in central Pb+Pb collisions shift
significantly to a higher $p_{T}D$ region relative to those in p+p collisions.
Thus the nuclear modification ratio of the $p_{T}D$ distributions of inclusive
jets is smaller than unity in the small $p_{T}D$ region, and larger than one
in the large $p_{T}D$ region. This behavior is caused by a more uneven $p_{T}$
distribution of jet constituents as well as the fraction alteration of
quark/gluon initiated jets in heavy-ion collisions. The difference in $p_{T}D$
distribution between groomed and ungroomed jets in Pb+Pb collisions is also
discussed.

###### pacs:

13.87.-a; 12.38.Mh; 25.75.-q

##  I Introduction

Energetic partons created in the early stage of heavy-ion collisions (HIC) may
suffer energy loss owing to their interacting with the quark-gluon plasma
(QGP), a novel state of matter with deconfined quarks and gluons under extreme
high temperature and energy density. This phenomenon is referred as jet
quenching Wang:1991xy ; Gyulassy:2003mc ; Qin:2015srf , which could provide
powerful tools to study the creation and properties of the QCD medium. In the
last decade the investigations of jet quenching have been extended from the
leading hadron productions suppression Khachatryan:2016odn ; Acharya:2018qsh ;
Aad:2015wga ; Burke:2013yra ; Chen:2010te ; Chen:2011vt ; Liu:2015vna ;
Dai:2015dxa ; Dai:2017piq ; Dai:2017tuy ; Ma:2018swx ; Xie:2019oxg ;
Zhang:2022fau  to medium modifications of a wealth of reconstructed jet
observables, such as inclusive jets productions, di-jets asymmetry,
correlations of gauge bosons and jets , as well as heavy flavor jets
Vitev:2008rz ; Vitev:2009rd ; Aad:2010bu ; Chatrchyan:2011sx ;
Chatrchyan:2012gt ; Aad:2014bxa ; Chatrchyan:2012gw ; Chatrchyan:2013kwa ;
Aad:2014wha ; Sirunyan:2017jic ; CasalderreySolana:2010eh ; Sirunyan:2018ncy ;
Young:2011qx ; He:2011pd ; ColemanSmith:2012vr ; Neufeld:2010fj ; Zapp:2012ak
; Dai:2012am ; Ma:2013pha ; Senzel:2013dta ; Casalderrey-Solana:2014bpa ;
Milhano:2015mng ; Chang:2016gjp ; Majumder:2014gda ; Chen:2016cof ;
Chien:2016led ; Apolinario:2017qay ; Connors:2017ptx ; Zhang:2018urd ;
Dai:2018mhw ; Luo:2018pto ; Chang:2019sae ; Wang:2019xey ; Chen:2019gqo ;
Chen:2020kex ; Wang:2020qwe ; Yan:2020zrz ; Wang:2020ukj ; Zhang:2021sua . A
full reconstructed jet is a collimated spray of hadrons created in
$e^{+}e^{-}$ collisions, p+p reactions as well as nucleus-nucleus collisions
with large momentum transfer, and the existence of QCD medium should naturally
modify the yields and the internal structures of full jets, and thus the
medium modifications of jet observables could be used to tomography of QGP
formed in HIC.

The nuclear modifications of jet substructure have received a growing
attention in the heavy-ion community. One interesting jet substructure is jet
$p_{T}$ dispersion($p_{T}D$), which characterizes the fragmentation of a jet
Giele:1997hd ; Acharya:2018uvf ; KunnawalkamElayavalli:2017hxo ;
Agafonova:2019tqe ; Wan:2018zpq . The nuclear modification of jet $p_{T}D$
distribution may improve our understanding of jet-medium interaction, and put
new insight on how jet substructure is resolved by QCD medium. Recently, ALICE
Collaboration has measured $p_{T}D$ distributions for small-radius ($R=0.2$)
jets in heavy-ion collisions Acharya:2018uvf , which further facilitates the
studies of $p_{T}D$ in distributions HIC, since the theoretical calculations
could be confronted with the data directly to infer some crucial information
of jet propagation in the QCD medium.

In this paper, we present our study on the normalized $p_{T}D$ distributions
for inclusive jets with jet radius $R=0.2$ both in p+p and central Pb+Pb
collisions at $\sqrt{s_{NN}}=2.76$ TeV. We employ POWHEG+PYTHIA Alioli:2010xa
; Alioli:2010qp ; Buckley:2016bhy , a Monte Carlo model matching NLO matrix
elements with parton shower (PS), including hadronization process to obtain
the solid baseline results of jet $p_{T}D$ in p+p collisions, which are then
served as input to simulate parton energy loss within the higher-twist
approach Guo:2000nz ; Zhang:2003yn ; Zhang:2003wk ; Majumder:2009ge  to
compute the $p_{T}D$ distribution in heavy-ion collisions. Our model
calculations of $p_{T}D$ distribution for inclusive jets could provide
satisfactory descriptions of ALICE data both in p+p and Pb+Pb collisions,
where we observe a shift of $p_{T}D$ distribution toward higher values in
Pb+Pb collisions relative to that in p+p. We further make a comprehensive
understanding of the distinct feature between quark and gluon initiated jets,
and the nuclear modification ratio $p_{T}D$ distribution. We find $p_{T}D$ can
be analytical expressed as the standard deviation and the multiplicity of jet
constitutes. After jet quenching, more jet constituents lied further from the
mean value of $p_{T}$.

The remainder of this paper is organized as follows. In Sec. II we will
introduce the framework used to calculate the normalized $p_{T}D$
distributions in both p+p and central Pb+Pb collisions. Our numerical results
and detailed discussions of the medium modifications of $p_{T}D$ distributions
for groomed and ungroomed jets are presented in Section II. In Sec. IV we will
give a summary.

##  II Analysis framework

We study a jet substructure observable, the $p_{T}D$, which characterizes the
second moment of the constituent $p_{T}$ distribution inside a jet
Giele:1997hd ; Acharya:2018uvf , and is defined as:

$\displaystyle p_{T}D=\frac{\sqrt{\sum_{i}p^{2}_{T,i}}}{p_{T,{\rm jet}}}$ (1)

where $p_{T,i}$ represents the transverse momentum of $i$th jet ingredient
inside the jet with transverse momentum $p_{T,{\rm jet}}$. $p_{T}D$ is
connected to how hard or soft the jet fragmentation is, and whether $p_{T,i}$
distribution is uniform or not. For example, in the extreme case of very few
constituents carrying a lion’s share of the jet momentum, $p_{T}D$ will be
close to unity; while in the case of jets containing a large number of
constituents with soft momentum, $p_{T}D$ may approach to zero. It is noted
that jet dispersion is one of a class of jet substructure observables, the
generalized jet angularities Larkoski:2014pca ; ALICE:2021njq , which are
defined as
$\lambda^{\kappa}_{\beta}=\Sigma_{i}z^{\kappa}_{i}\theta^{\beta}_{i}$, where
$z_{i}=p_{T,i}/p_{T,jet}$ is the momentum fraction of jet constituents,
$\theta_{i}=\Delta{R_{i}}/R$, $\Delta{R_{i}}$ is the opening angle from
constituent to jet axis, $\kappa$ and $\beta$ are free parameters. One can see
that $(p_{T}D)^{2}$ is equal to $\lambda^{\kappa}_{\beta}(\kappa=2,\beta=0)$.

In this work, we use a Monte Carlo model POWHEG+PYTHIA, which performs next-
to-leading order (NLO) matrix elements matched with parton showering
Alioli:2010xa ; Alioli:2010qp ; Buckley:2016bhy , to generate jet productions
in p+p collisions. In our simulation the POWHEG BOX code is utilized
Alioli:2010xa ; Alioli:2010qp , which provides a computer framework for
performing NLO calculations in parton shower Monte Carlo programs in
accordance with the POWHEG method Frixione:2007vw . Previous studies have
shown that, POWHEG BOX Monte Carlo program matched with parton showering could
give nice description of productions and correlations for a variety of
processes in p+p collisions, such as di-jet, gauge boson tagged jets, heavy
flavour jets etc. powheg-box . We generate the NLO matrix elements for QCD
dijet events with POWHEG BOX, and then matched with PYTHIA6 by Perugia 2011
tunes Skands:2010ak  to perform parton showering and hadronization
Sjostrand:2006za . After that, Fastjet package Cacciari:2008gp  is employed to
reconstruct final state hadrons into full jets.

Figure 1: Normalized $p_{T}D$ distribution of inclusive jets in p+p collisions
at $\sqrt{s}=7$ TeV from POWHEG+PYTHIA calculation as compared with ALICE data
Acharya:2018uvf .

In order to compare with the available experimental data, we selected events
according to the same kinematic cuts as adopted by the experimental
measurements. In ALICE Collaboration data, jets are reconstructed using
anti-$k_{T}$ algorithm with radius parameter $R=0.2$ from charged hadrons
which are required to have $p_{T}>0.15$ GeV. Those reconstructed jets are
accepted in transverse momentum range of $40\ {\rm GeV}<p_{T,\rm jet}<60$ GeV
and rapidity range of $\left|\eta_{\rm jet}\right|<0.7$. Our numerical results
of normalized distributions of $p_{T}D$ in p+p collisions at $\sqrt{s}=7$ TeV
and their comparison with ALICE data are shown in Fig. 1. We can observe that,
POWHEG+PYTHIA calculations show well agreements with experimental measurements
in p+p collisions in the overall $p_{T}D$ region, which will be served as
input for the subsequent study of nuclear modification in HIC. The $p_{T}D$
distributions of quark- and gluon-initiated jets are also plotted in Fig. 1,
respectively. We find that at the same jet $p_{T}$, the peak of $p_{T}D$
distribution for gluon jets is located at smaller region relative to that for
quark jets. It implies as compared to quark jets, gluon jets favor harder
radiation, on average. To make a comprehensive understanding of the distinct
feature between quark and gluon jets, we derived the $p_{T}D$ with standard
deviation(labeled as $\sigma$) and the multiplicity of jet constitutes(labeled
as $n$) in the following.

Figure 2: Top panel: normalized $\sigma^{2}$ distribution of quark- and gluon-
initiated jets in p+p collisions at $\sqrt{s}=7$ TeV from POWHEG+PYTHIA
calculation; Bottom panel: ratio of normalized $\sigma^{2}$ distribution of
gluon and quark jets.

The standard deviation could describe the average degree of a dataset. It
tells us, on average, how far each value lies away from the mean value. A high
standard deviation signifies that values are generally far from the mean
value, while a low standard deviation means that values are clustered near to
the mean value. Aa a result of that, in our study, the standard deviation of
transverse momentum of jet constituents can be written as:

$\displaystyle\sigma=\frac{\sqrt{\sum_{i}(p_{T,i}-\langle
p_{T,i}\rangle)^{2}}}{p_{T,{\rm jet}}}$ (2)

Then $\sigma^{2}$ can be expressed as:

$\displaystyle\sigma^{2}$ $\displaystyle=$
$\displaystyle\frac{\sum_{i}(p_{T,i}-\langle
p_{T,i}\rangle)^{2}}{(n\cdot\langle p_{T,i}\rangle)^{2}}$ $\displaystyle=$
$\displaystyle\frac{\sum_{i}(p^{2}_{T,i}-2p_{T,i}\langle
p_{T,i}\rangle+\langle p_{T,i}\rangle^{2})}{(n\cdot\langle
p_{T,i}\rangle)^{2}}$ $\displaystyle=$ $\displaystyle(p_{T}D)^{2}-1/n$ (3)

Conversely, we have

$(p_{T}D)^{2}=\sigma^{2}+1/n\,.$

Shown in Fig. 2 are the normalized $\sigma^{2}$ distributions (top) of quark
and gluon jets in p+p collisions at $\sqrt{s}=7$ TeV, as well as their
gluon/quark ratios (bottom). We observed more gluon jets distributed in lower
$p_{T}D$ region compared to quark jets. It is because gluon jets contain more
fragment ingredients. Thus, at the same energy, the value of standard
deviation for gluon jets is smaller than that for quark jets.

In heavy-ion collisions, fast partons produced from hard scattering will
interact with medium partons and lose their energy. In our calculations, the
initial jet shower partons are generated by POWHEG+PYTHIA, then they are
arranged to have initial positions which are sampled from Glauber model
Alver:2008aq . We assume all the partons move in QGP in the same way as
classical particles do. The probability for gluon radiation happens in QGP
during each time step $\Delta t$ can be expressed as He:2015pra ; Cao:2016gvr
; Cao:2017hhk ; Wang:2019xey :

$\displaystyle P_{rad}(t,\Delta t)=1-e^{-\left\langle N(t,\Delta
t)\right\rangle}\,.$ (4)

Here $\left\langle N(t,\Delta t)\right\rangle$ is the averaged number of
emitted gluons, which can be integrated from the medium induced radiated gluon
spectrum within Higher-Twist(HT) method Guo:2000nz ; Zhang:2003yn ;
Zhang:2003wk ; Majumder:2009ge :

$\displaystyle\frac{dN}{dxdk^{2}_{\perp}dt}=\frac{2\alpha_{s}C_{s}P(x)\hat{q}}{\pi
k^{4}_{\perp}}\sin^{2}(\frac{t-t_{i}}{2\tau_{f}})(\frac{k^{2}_{\perp}}{k^{2}_{\perp}+x^{2}M^{2}})^{4}$
(5)

Here $\alpha_{s}$ denotes the strong coupling constant, $x$ is the energy
fraction of the radiated gluon, $M$ is the mass of parent parton, and
$k_{\perp}$ is the $p_{T}$ of the radiated gluon. A lower $p_{T}$ cut-off with
$x_{min}=\mu_{D}/E$ of the emitted gluon is applied in our calculations, and
$\mu_{D}$ is the Debye screening mass. $P(x)$ is the parton splitting function
in vacuum, $C_{s}$ is the Casimir factor for gluons ($C_{A}$) and quarks
($C_{F}$). The formation time of the radiated gluons is
$\tau_{f}=2Ex(1-x)/(k^{2}_{\perp}+x^{2}M^{2})$. $\hat{q}$ is jet transport
parameter, which is proportional to the local parton distribution density in
the QCD medium, The jet transport parameter $\hat{q}$ is proportional to the
local parton density distribution in the QCD medium, and related to the space
and time evolution of the medium relative to its initial value $\hat{q}_{0}$
in the central region when QGP formed, which controls the magnitude of energy
loss due to jet-medium interaction.

The number of emitted gluons is sampled from a Poisson distribution during
each time step:

$\displaystyle P(n_{g},t,\Delta t)=\frac{\left\langle N(t,\Delta
t)\right\rangle^{n_{g}}}{n_{g}!}e^{-\left\langle N(t,\Delta
t)\right\rangle}\,.$ (6)

In our calculation, $P_{rad}(t,\Delta t)$ would be firstly evaluated to
determine whether the radiation happen during $\Delta t$. If accepted, the
Possion distribution $P(n_{g},t,\Delta t)$ is used to sample the number of
radiated gluon. At last, the energy fraction ($x$) and transverse momentum
($k_{\perp}$) of the radiated gluon could be sampled based on the spectrum
shown in Eq.5.

To calculate the collisional energy loss of these showered partonsDai:2018mhw
; Wang:2019xey , a Hard Thermal Loop (HTL) formula has been adopted in this
work Neufeld:2010xi :
$\frac{dE^{coll}}{dt}=\frac{\alpha_{s}C_{s}\mu^{2}_{D}}{2}ln\frac{\sqrt{ET}}{\mu_{D}}$.
The space time evolution of bulk medium is given by the smooth iEBE-VISHNU
hydrodynamical code Shen:2014vra . Jet partons stop their propagation in QGP
medium when local temperature fall below $T_{c}=165$ MeV. After all the
partons escape from QGP, PYQUEN method is used to carry out the hadronization
process Lokhtin:2000wm ; Lokhtin:2005px . In the model, the radiated gluons
are rearranged in the same string as their parent partons, and these partons
could fragment into hadrons by standard PYTHIA hadronization procedure.

Figure 3: Normalized $p_{T}D$ distributions of inclusive jets in p+p and 0-10%
Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ TeV as compared with ALICE data
Acharya:2018uvf .

##  III Results and discussion

###  III.1 $p_{T}D$ distributions of ungroomed jets in Pb+Pb collisions

We now could calculate the jet number normalized $p_{T}D$ distributions in
Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ TeV. We use the same jet selection
criterion as we did in p+p collisions in Sec. II. Our numerical results of jet
number normalized $p_{T}D$ distributions for inclusive jets in p+p and Pb+Pb
collisions are shown in Fig. 3, which are confronted with the existing
experimental data in Pb+Pb by ALICE Collaboration Acharya:2018uvf . We find
our theoretical calculations could provide quite decent descriptions of
experimental measurements. Relative to that in p+p collisions, the observed
normalized $p_{T}D$ distribution in Pb+Pb is shifted towards higher values.
That is to say, $p_{T}D$ distribution of inclusive jets in Pb+Pb collisions is
shifted towards quark jets after jet quenching. It indicate that a jet in
Pb+Pb collisions may have more softer constituents than that in p+p.

Figure 4: Nuclear modification ratio of $p_{T}D$ distribution of inclusive
jets as well as quark and gluon jets. The ratio from ALICE Collaboration is
preformed by Pb+Pb measurements scaled by MC simulation in p+p Acharya:2018uvf
.

To investigate the deviation of jet $p_{T}D$ distributions in HIC to those in
p+p in a more straightforward way, it is essential and helpful to define the
nuclear modification ratio($R_{AA}^{\rm p_{T}D}$) of $p_{T}D$ distributions
as:

$\displaystyle R_{AA}^{\rm
p_{T}D}={\dfrac{1}{N_{AA}}\dfrac{dN_{AA}}{dp_{T}D}}/{\dfrac{1}{N_{pp}}\dfrac{dN_{pp}}{dp_{T}D}}\,\,.$
(7)

Shown in Fig. 4 are $R_{AA}^{\rm p_{T}D}$ of normalized $p_{T}D$ distributions
for inclusive jets, as compared with ALICE ratio Acharya:2018uvf . Our
calculated results could provide nice description of ALICE ratio in the
overall $p_{T}D$ region. We note that, the ALICE ratio is preformed form Pb+Pb
measurements scaled by MC simulation in p+p due to the lack of corresponding
measurement in p+p. It is a normal practice to utilize MC simulations in p+p
as reference in the studying of nuclear modification ratios when the
corresponding p+p baseline is not available ALICE:2017nij ; ALICE:2018lyv ;
ALICE:2018hbc ; ALICE:2021vrw . Of course, truly measured results of $R_{AA}$
in experiments are more favorable, and then we could confront our simulations
directly with the experimental data. The $R_{AA}^{\rm p_{T}D}$ of quark jets
and gluon jets as the components of inclusive jets are also plotted in Fig. 4.
One can observe there is a suppression of $p_{T}D$ distribution for both quark
jets and gluon jets at low $p_{T}D$ region, while an enhancement at high
$p_{T}D$ region. The nuclear corrections of gluon jets $p_{T}D$ distribution
is much stronger than that for quark jets. The curve of $R_{AA}^{\rm p_{T}D}$
for inclusive jets goes between the curves of $R_{AA}^{\rm p_{T}D}$ for quark
jets and gluon jets, since inclusive jets are the combinations of quark jets
and gluon jets.

To further understand the nuclear modification mechanism of $p_{T}D$
distribution for inclusive jets, we start with the modifications of the
relative fraction of quark and gluon jets due to jet quenching. In our
calculations, as most of other jet quenching models, gluons may lose more
energy than quarks in QGP with their larger color charge. Therefore, generally
we should see an enhancement of contribution fraction of quark jets in A+A
collisions relative to that in p+p. Qualitatively such enhancement will
naturally lead $p_{T}D$ distributions for inclusive jets to larger value
region, since quark jets peak at larger value of $p_{T}D$ than gluon jets.
This is illustrated in Fig. 4, where the curve labelled “inclusive jet (q/g)”
represents our numerical result of $R_{AA}^{\rm p_{T}D}$ by only considering
the effect of quark/gluon jets fraction alterations due to jet quenching while
assuming there are no medium modifications for $p_{T}D$ distributions of pure
quark and gluon jets in heavy-ion collisions.

Figure 5: Top: normalized $\sigma^{2}$ distribution of inclusive jets in 0-10%
Pb+Pb and p+p collisions at $\sqrt{s_{NN}}=2.76$ TeV; Bottom: ratio of
normalized $\sigma^{2}$ distribution in central Pb+Pb (0-10%) and p+p
collisions.
Figure 6: Nuclear modification ratio of z distributions for inclusive jets at
$\sqrt{s_{NN}}=2.76$ TeV as compared with ATLAS data Aad:2014wha .

To explore why for normalized jet $p_{T}D$ distributions $R_{AA}^{\rm
p_{T}D}<1$ at small $p_{T}D$ region, and $R_{AA}^{\rm p_{T}D}>1$ at large
$p_{T}D$ region, as shown in Fig. 4. Eq.(3) indicate the nuclear modification
of $p_{T}D$ distributions has a very strong correlation with the nuclear
correction of standard deviation of $p_{T,i}$. Thus we turn to investigate the
standard deviation of $p_{T,i}$ for jets in p+p and 0-10% Pb+Pb collisions.
Presented in Fig. 5 are the normalized distribution of variance ($\sigma^{2}$)
of $p_{T,i}$ in p+p and 0-10% Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ TeV. We
can observed the distributions of jet variance is also shifted to higher value
region in heavy-ion collisions compared with that in p+p. Which indicates
after jet quenching, the value of $p_{T,i}$ lies further from the mean value
in heavy-ion collisions relative to p+p. It is noted that the changes of mean
multiplicities of Pb+Pb relative to p+p is rather small, and the estimated
mean values of jet constituents number ($\bar{n}$) in p+p and Pb+Pb collisions
are $\bar{n}_{pp}=6.72$ and $\bar{n}_{PbPb}=6.54$ respectively.

To see the point more clearly, we also plotted the nuclear modification ratio
of momentum fraction for jet constituents($z=p_{T,i}/p_{T,\rm jet}$) in Fig.
6. One can see, our model calculations could provide nice description of ATLAS
data for jets with $p_{T}>100$ GeV Aad:2014wha . For jets with $40<p_{T}<60$
GeV, the nuclear modification ratios of charged-particle transverse momentum
distributions in Pb+Pb collisions to those measured in p+p exhibit an
enhancement in fragment yield in central collisions for $0.02<z<0.05$, a
reduction in fragment yields for $0.05<z<0.3$, and an enhancement in the
fragment yield for $0.3<z<1$. Which means the number of jet constituents with
lower and higher value of $p_{T}$ are enhanced, more constituents lied further
from the mean value. Therefore, the jet standard deviation is shifted to
higher region in heavy-ion collisions.

###  III.2 $p_{T}D$ distributions of groomed jets in Pb+Pb collisions

In this section, we will study $p_{T}D$ distributions of groomed jets in
central Pb+Pb collisions. Jet grooming techniques have seen a particularly
great of interest from both experimental and theoretical side Larkoski:2014wba
; Dasgupta:2013ihk . It is designed to remove soft wide-angle radiation from
the jet, allowing for a more direct comparison between experimental data and
purely perturbative QCD calculations, since hadronization and underlying event
contributions are significantly reduced during grooming procedure. A full jet
constructed using radius $R$ via the anti-$k_{T}$ algorithm is first re-
clustered using the Cambridge-Aachen (C/A) algorithm Dokshitzer:1997in ;
Wobisch:1998wt  until two hard branches are found to satisfy the following
condition:

$\displaystyle\frac{min(p_{T1},p_{T2})}{p_{T1}+p_{T2}}\equiv{z_{g}}>z_{cut}\left(\frac{\Delta
R}{R}\right)^{\beta}$ (8)

where $\left(\frac{\Delta R}{R}\right)$ is an additional parameter of the
relative angular distance between the two sub-jets, $z_{cut}$ and $\beta$ are
free parameters which can be used to control how strict the soft drop
condition is. For the heavy-ion studies conducted so far, $z_{cut}$ has been
set to $0.1$ and $\beta$ has been set to zero.

Figure 7: Top: nuclear modification factor of $p_{T}D$ distribution for
groomed and ungroomed jets; Bottom: ratio of $p_{T}D$ distribution for groomed
and ungroomed jets in p+p and 0-10% Pb+Pb collisions.

In the top panel of Fig. 7, we plot the nuclear modification ratio of $p_{T}D$
distributions for groomed and ungroomed jets respectively. One can observe
that the nuclear modification pattern of $p_{T}D$ distributions for groomed
jets are similar with that for ungroomed jets, and the $p_{T}D$ distribution
for groomed jets is also shifted to higher $p_{T}D$ region. Besides, compared
with ungroomed jets, the nuclear modification of groomed jets becomes weaker.
It implies grooming procedure could not only remove soft radiation from the
jet in QCD vacuum, but also reduce soft radiation in QCD medium. In the bottom
panel of Fig. 7 we present the ratios of $p_{T}D$ distributions for the
groomed jet to that for the ungroomed both in p+p and Pb+Pb collisions. It is
shown that these ratios are below to unity at small $p_{T}D$, whereas larger
than one at large $p_{T}D$. To understand deeper the alteration of $p_{T}D$
distributions originate from jet grooming procedure, in the following we may
investigate the difference of standard deviation of jet constituents $p_{T,i}$
and jet constituents number between groomed and ungroomed jets in Pb+Pb
collisions.

Figure 8: Top: normalized $\sigma^{2}$ distribution of groomed and ungroomed
jets in 0-10% Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ TeV; Bottom: ratio of
normalized $\sigma^{2}$ distribution for groomed and ungroomed jets in 0-10%
Pb+Pb collisions.

Firstly, in Fig. 8 we plotted the variance ($\sigma^{2}$) distributions of jet
constituents $p_{T,i}$ for groomed and ungroomed jets in central Pb+Pb
collisions. The distributions of jet $\sigma^{2}$ are shifted to lower region
for groomed jets compared with that for ungroomed jets, which is in contrast
to the alteration of $p_{T}D$ distributions. It indicates that after jet soft-
drop procedure, the values of $p_{T,i}$ in groomed jets lies closer to the
mean value relative to those in ungroomed jets. That is because during the
grooming, some particles with low $p_{T,i}$ are dropped from the jet
constituents.

Figure 9: Top: normalized $(1/n)$ distribution of groomed and ungroomed jets
in 0-10% Pb+Pb collisions at $\sqrt{s_{NN}}=2.76$ TeV; Bottom: ratio of
normalized $(1/n)$ distribution for groomed and ungroomed jets in 0-10% Pb+Pb
collisions.

Secondly, the number of jets constituents are modified during soft drop
grooming process, which will be contribute to the correction of $p_{T}D$
distributions. Presented in Fig. 9 are the number of jets constituents
distributions of groomed and ungroomed jets in p+p and central Pb+Pb
collisions. One cam see the value of $1/n$ is enhanced after grooming process.
As we shown in Eq. 3, the value of $p_{T}D$ is equivalent to the standard
deviation added by $1/n$. Therefore, even though the grooming process will
lead to lower value of standard deviation for jet constituents transverse
momenta, it will enhance the value of $1/n$ meanwhile. Though these two
effects offset with each other, the correction of $1/n$ are more pronounced,
which results in the increase of the ratio of $p_{T}D$ distributions for
groomed jets to that for ungroomed jets with $p_{T}D$ (as shown in the bottom
panel of Fig. 7), a trend similar to the increase of ratio of normalized
($1/n$) distribution for groomed to ungroomed jets with $1/n$ (as demonstrated
in the bottom panel of Fig. 9).

##  IV Summary

In this paper, by using a NLO$+$PS event generator POWHEG$+$PYTHIA for p+p
baseline and HT parton energy loss approach for jet quenching, we have studied
the nuclear modifications of $p_{T}D$ distributions for inclusive jets with
small radius $R=0.2$ in 0-10% Pb+Pb at $\sqrt{s_{NN}}=2.76$ TeV. Our simulated
results of inclusive jets could provide decent description of ALICE
measurements. The $p_{T}D$ distributions for inclusive jets are shifted toward
higher $p_{T}D$ region in central Pb+Pb collisions compared to those in p+p,
and similar trends have also been found for quark and gluon jets. We further
find two elements could contribute to the nuclear modifications of $p_{T}D$
distributions: more uneven $p_{T}$ of jet constituents, and the enhanced
fraction of quark-initiated jets after jet-medium interaction in HIC. The
observed nuclear modifications of $p_{T}D$ distributions for gluon jets are
stronger than that for quark jets in HIC since gluons may lose more energy
than quarks in our model. Additionally, we also investigate the medium
modifications of $p_{T}D$ distributions for groomed jets in central Pb+Pb
collisions. We observe weaker nuclear modifications of $p_{T}D$ distributions
for groomed jets compare to that for ungroomed jets.

Acknowledgments: The authors would like to thank P Ru, S L Zhang, S Wang and Q
Zhang for helpful discussions.This research was supported in part by Guangdong
Major Project of Basic and Applied Basic Research No. 2020B0301030008, by
Natural Science Foundation of China (NSFC) under Project Nos. 11935007 and and
12035007. S.-Y. Chen is supported by the MOE Key Laboratory of Quark and
Lepton Physics (CCNU) under Project No. QLPL2020P01.

## References

  * (1) X. N. Wang and M. Gyulassy, Phys. Rev. Lett. 68, 1480 (1992).
  * (2) M. Gyulassy, I. Vitev, X. N. Wang and B. W. Zhang, In *Hwa, R.C. (ed.) et al.: Quark gluon plasma* 123-191 [nucl-th/0302077].
  * (3) G. Y. Qin and X. N. Wang, Int. J. Mod. Phys. E 24, no. 11, 1530014 (2015).
  * (4) CMS, V. Khachatryan et al.,  JHEP 04, 039 (2017), arXiv:1611.01664.
  * (5) ALICE, S. Acharya et al.,  JHEP 11, 013 (2018), arXiv:1802.09145.
  * (6) ATLAS, G. Aad et al.,  JHEP 09, 050 (2015), arXiv:1504.04337.
  * (7) JET, K. M. Burke et al.,  Phys. Rev. C90, 014909 (2014), arXiv:1312.5003.
  * (8) X. Chen, C. Greiner, E. Wang, X. N. Wang and Z. Xu, Phys. Rev. C 81, 064908 (2010).
  * (9) X. Chen, T. Hirano, E. Wang, X. N. Wang and H. Zhang, Phys. Rev. C 84, 034902 (2011).
  * (10) Z. Liu, H. Zhang, B. Zhang and E. Wang, Eur. Phys. J. C 76, no.1, 20 (2016) [arXiv:1506.02840 [nucl-th]].
  * (11) W. Dai, X. Chen, B. Zhang and E. Wang, Phys. Lett. B 750, 390-395 (2015) [arXiv:1506.00838 [nucl-th]].
  * (12) W. Dai, X. Chen, B. Zhang, H. Zhang and E. Wang, Eur. Phys. J. C 77, no.8, 571 (2017) [arXiv:1702.01614 [nucl-th]].
  * (13) W. Dai, B. W. Zhang and E. Wang, Phys. Rev. C 98, 024901 (2018).
  * (14) G. Y. Ma, W. Dai, B. W. Zhang and E. K. Wang, Eur. Phys. J. C 79, no. 6, 518 (2019).
  * (15) M. Xie, S. Y. Wei, G. Y. Qin and H. Z. Zhang, Eur. Phys. J. C 79, no. 7, 589 (2019).
  * (16) Q. Zhang, W. Dai, L. Wang, B. W. Zhang and E. Wang, [arXiv:2203.10742 [hep-ph]].
  * (17) Atlas Collaboration, G. Aad et al.,  Phys.Rev.Lett. 105, 252303 (2010), arXiv:1011.6182.
  * (18) CMS, S. Chatrchyan et al.,  Phys. Rev. C84, 024906 (2011), arXiv:1102.1957.
  * (19) CMS, S. Chatrchyan et al.,  Phys. Lett. B718, 773 (2013), arXiv:1205.0206.
  * (20) ATLAS, G. Aad et al.,  Phys. Rev. Lett. 114, 072302 (2015), arXiv:1411.2357.
  * (21) CMS Collaboration, S. Chatrchyan et al.,  JHEP 1210, 087 (2012), arXiv:1205.5872.
  * (22) CMS Collaboration, S. Chatrchyan et al.,  Phys.Lett. B730, 243 (2014), arXiv:1310.0878.
  * (23) ATLAS, G. Aad et al.,  Phys. Lett. B739, 320 (2014), arXiv:1406.2979.
  * (24) A. M. Sirunyan et al. [CMS], Phys. Rev. Lett. 119, no.8, 082301 (2017) [arXiv:1702.01060 [nucl-ex]].
  * (25) A. M. Sirunyan et al. [CMS], Phys. Rev. Lett. 122, no.15, 152001 (2019) [arXiv:1809.08602 [hep-ex]].
  * (26) I. Vitev, S. Wicks and B. W. Zhang, JHEP 0811, 093 (2008).
  * (27) I. Vitev and B. W. Zhang, Phys. Rev. Lett. 104, 132001 (2010).
  * (28) J. Casalderrey-Solana, J. G. Milhano and U. A. Wiedemann, J. Phys. G 38, 035006 (2011).
  * (29) C. Young, B. Schenke, S. Jeon and C. Gale, Phys. Rev. C 84, 024907 (2011).
  * (30) Y. He, I. Vitev and B. W. Zhang, Phys. Lett. B 713, 224 (2012).
  * (31) C. E. Coleman-Smith and B. Muller, Phys. Rev. C 86, 054901 (2012).
  * (32) R. B. Neufeld, I. Vitev and B.-W. Zhang, Phys. Rev. C 83, 034902 (2011).
  * (33) K. C. Zapp, F. Krauss and U. A. Wiedemann, JHEP 1303, 080 (2013).
  * (34) W. Dai, I. Vitev and B. W. Zhang, Phys. Rev. Lett. 110, 142001 (2013).
  * (35) G. L. Ma, Phys. Rev. C 87, no. 6, 064901 (2013).
  * (36) F. Senzel, O. Fochler, J. Uphoff, Z. Xu and C. Greiner, J. Phys. G 42, no. 11, 115104 (2015).
  * (37) J. Casalderrey-Solana, D. C. Gulhan, J. G. Milhano, D. Pablos, and K. Rajagopal,  JHEP 10, 019 (2014), arXiv:1405.3864,  [Erratum: JHEP09,175(2015)].
  * (38) J. G. Milhano and K. C. Zapp, Eur. Phys. J. C 76, no. 5, 288 (2016).
  * (39) N. B. Chang and G. Y. Qin, Phys. Rev. C 94, no. 2, 024902 (2016).
  * (40) A. Majumder and J. Putschke, Phys. Rev. C 93, no. 5, 054909 (2016).
  * (41) L. Chen, G. Y. Qin, S. Y. Wei, B. W. Xiao and H. Z. Zhang, Phys. Lett. B 782, 773 (2018).
  * (42) Y. T. Chien and I. Vitev, Phys. Rev. Lett. 119, no. 11, 112301 (2017).
  * (43) L. Apolinario, J. G. Milhano, M. Ploskon and X. Zhang, Eur. Phys. J. C 78, no. 6, 529 (2018)
  * (44) M. Connors, C. Nattrass, R. Reed and S. Salur, Rev. Mod. Phys. 90, 025005 (2018)
  * (45) S. L. Zhang, T. Luo, X. N. Wang and B. W. Zhang, Phys. Rev. C 98, 021901 (2018).
  * (46) W. Dai, S. Wang, S. L. Zhang, B. W. Zhang and E. Wang, Chin. Phys. C 44, 104105 (2020) [arXiv:1806.06332 [nucl-th]].
  * (47) T. Luo, S. Cao, Y. He and X. Wang, Phys. Lett. B 782, 707-716 (2018) [arXiv:1803.06785 [hep-ph]].
  * (48) N. Chang, Y. Tachibana and G. Qin, Phys. Lett. B 801, 135181 (2020) [arXiv:1906.09562 [nucl-th]].
  * (49) S. Wang, W. Dai, B. W. Zhang and E. Wang, Eur. Phys. J. C 79, no.9, 789 (2019) [arXiv:1906.01499 [nucl-th]].
  * (50) S. Chen, B. W. Zhang and E. Wang, Chin. Phys. C 44, no.2, 024103 (2020) [arXiv:1908.01518 [nucl-th]].
  * (51) L. Chen, S. Wei and H. Zhang, [arXiv:2001.07606 [hep-ph]].
  * (52) S. Wang, W. Dai, B. W. Zhang and E. Wang, [arXiv:2005.07018 [hep-ph]].
  * (53) J. Yan, S. Y. Chen, W. Dai, B. W. Zhang and E. Wang, Chin. Phys. C 45, no.2, 024102 (2021) [arXiv:2005.01093 [hep-ph]].
  * (54) S. Wang, W. Dai, B. W. Zhang and E. Wang, Chin. Phys. C 45, no.6, 064105 (2021) [arXiv:2012.13935 [nucl-th]].
  * (55) S. L. Zhang, M. Q. Yang and B. W. Zhang, [arXiv:2105.04955 [hep-ph]].
  * (56) W. T. Giele, E. W. N. Glover and D. A. Kosower, Phys. Rev. D 57, 1878 (1998) [hep-ph/9706210].
  * (57) S. Acharya et al. [ALICE Collaboration], JHEP 1810, 139 (2018) [arXiv:1807.06854 [nucl-ex]].
  * (58) R. Kunnawalkam Elayavalli and K. C. Zapp, JHEP 07, 141 (2017) [arXiv:1707.01539 [hep-ph]].
  * (59) V. Agafonova, Universe 5, no. 5, 114 (2019).
  * (60) R. Z. Wan, L. Ding, X. Gui, F. Yang, S. Li and D. C. Zhou, Chin. Phys. C 43, no. 5, 054110 (2019) [arXiv:1812.10062 [hep-ph]].
  * (61) A. Buckley and D. Bakshi Gupta, arXiv:1608.03577 [hep-ph].
  * (62) S. Alioli, P. Nason, C. Oleari and E. Re, JHEP 01, 095 (2011) [arXiv:1009.5594 [hep-ph]].
  * (63) S. Alioli, K. Hamilton, P. Nason, C. Oleari and E. Re, JHEP 04, 081 (2011) [arXiv:1012.3380 [hep-ph]].
  * (64) X. F. Guo and X. N. Wang, Phys. Rev. Lett. 85 (2000) 3591 [hep-ph/0005044].
  * (65) B. W. Zhang and X. N. Wang, Nucl. Phys. A 720, 429 (2003).
  * (66) B. W. Zhang, E. Wang and X. N. Wang, Phys. Rev. Lett. 93 (2004) 072301 [nucl-th/0309040].
  * (67) A. Majumder, Phys. Rev. D 85 (2012) 014023
  * (68) A. J. Larkoski, J. Thaler and W. J. Waalewijn, JHEP 11, 129 (2014) [arXiv:1408.3122 [hep-ph]].
  * (69) S. Acharya et al. [ALICE], [arXiv:2107.11303 [nucl-ex]].
  * (70) S. Frixione, P. Nason and C. Oleari, JHEP 0711, 070 (2007) [arXiv:0709.2092 [hep-ph]].
  * (71) For more processes, please check the website: http://powhegbox.mib.infn.it
  * (72) P. Z. Skands, Phys. Rev. D 82, 074018 (2010) [arXiv:1005.3457 [hep-ph]].
  * (73) T. Sjostrand, S. Mrenna and P. Z. Skands, JHEP 05, 026 (2006) [arXiv:hep-ph/0603175 [hep-ph]].
  * (74) M. Cacciari, G. P. Salam and G. Soyez, JHEP 0804, 063 (2008).
  * (75) B. Alver, M. Baker, C. Loizides and P. Steinberg, [arXiv:0805.4411 [nucl-ex]].
  * (76) Y. He, T. Luo, X. Wang and Y. Zhu, Phys. Rev. C 91, 054908 (2015) [arXiv:1503.03313 [nucl-th]].
  * (77) S. Cao, T. Luo, G. Qin and X. Wang, Phys. Rev. C 94, no.1, 014909 (2016) [arXiv:1605.06447 [nucl-th]].
  * (78) S. Cao, T. Luo, G. Qin and X. Wang, Phys. Lett. B 777, 255-259 (2018) [arXiv:1703.00822 [nucl-th]].
  * (79) R. B. Neufeld, Phys. Rev. D 83 (2011) 065012
  * (80) C. Shen, Z. Qiu, H. Song, J. Bernhard, S. Bass and U. Heinz, Comput. Phys. Commun. 199 (2016) 61
  * (81) I. Lokhtin and A. Snigirev, Eur. Phys. J. C 16, 527-536 (2000) [arXiv:hep-ph/0004176 [hep-ph]].
  * (82) I. Lokhtin and A. Snigirev, Eur. Phys. J. C 45, 211-217 (2006) [arXiv:hep-ph/0506189 [hep-ph]].
  * (83) S. Acharya et al. [ALICE], Phys. Lett. B 776, 249-264 (2018) [arXiv:1702.00804 [nucl-ex]].
  * (84) S. Acharya et al. [ALICE], JHEP 10, 174 (2018) [arXiv:1804.09083 [nucl-ex]].
  * (85) S. Acharya et al. [ALICE], Phys. Lett. B 793, 212-223 (2019) [arXiv:1809.10922 [nucl-ex]].
  * (86) S. Acharya et al. [ALICE], JHEP 10, 003 (2021) [arXiv:2105.04936 [nucl-ex]].
  * (87) A. J. Larkoski, S. Marzani, G. Soyez and J. Thaler, JHEP 05, 146 (2014) [arXiv:1402.2657 [hep-ph]].
  * (88) M. Dasgupta, A. Fregoso, S. Marzani and G. P. Salam, JHEP 09, 029 (2013) [arXiv:1307.0007 [hep-ph]].
  * (89) Y. L. Dokshitzer, G. D. Leder, S. Moretti and B. R. Webber, JHEP 08, 001 (1997) [arXiv:hep-ph/9707323 [hep-ph]].
  * (90) M. Wobisch and T. Wengler, [arXiv:hep-ph/9907280 [hep-ph]].


# Face Hallucination via Split-Attention in Split-Attention Network

Tao Lu  Hubei Key Laboratory of Intelligent Robot, Wuhan Institute of
Technology, Wuhan, China  ,  Yuanzhi Wang∗ Hubei Key Laboratory of Intelligent
Robot, Wuhan Institute of Technology, Wuhan, China ,  Yanduo Zhang  Hubei Key
Laboratory of Intelligent Robot, Wuhan Institute of Technology, Wuhan, China ,
Yu Wang  Hubei Key Laboratory of Intelligent Robot, Wuhan Institute of
Technology, Wuhan, China ,  Wei Liu  Hubei Key Laboratory of Intelligent
Robot, Wuhan Institute of Technology, Wuhan, China ,  Zhongyuan Wang  School
of Computer Science, Wuhan University, Wuhan, China and  Junjun Jiang  School
of Computer Science and Technology, Harbin Institute of Technology, Harbin,
China

###### Abstract.

Recently, convolutional neural networks (CNNs) have been widely employed to
promote the face hallucination due to the ability to predict high-frequency
details from a large number of samples. However, most of them fail to take
into account the overall facial profile and fine texture details
simultaneously, resulting in reduced naturalness and fidelity of the
reconstructed face, and further impairing the performance of downstream tasks
(e.g., face detection, facial recognition). To tackle this issue, we propose a
novel external-internal split attention group (ESAG), which encompasses two
paths responsible for facial structure information and facial texture details,
respectively. By fusing the features from these two paths, the consistency of
facial structure and the fidelity of facial details are strengthened at the
same time. Then, we propose a split-attention in split-attention network
(SISN) to reconstruct photorealistic high-resolution facial images by
cascading several ESAGs. Experimental results on face hallucination and face
recognition unveil that the proposed method not only significantly improves
the clarity of hallucinated faces, but also encourages the subsequent face
recognition performance substantially. Codes have been released at
https://github.com/mdswyz/SISN-Face-Hallucination

face hallucination, facial structure, facial texture detail, split-attention
in split-attention network

$*$ is the corresponding author<EMAIL_ADDRESS>
††ccs: Computing methodologies Image processing††ccs: Computing methodologies
Image representations††ccs: Computing methodologies Appearance and texture
representations

##  1\. Introduction

Face hallucination is a domain-specific super-resolution (SR) for enhancing
quality and resolution of low-resolution (LR) facial images. Using face
hallucination for restoring latent high-resolution (HR) facial images from the
observed LR ones is helpful for various downstream tasks. Thus this task plays
an important role in many real-world applications, such as face detection,
verification, and analysis (Zhang et al., 2016; Chen et al., 2018a; Lu et al.,
2018, 2021).

Recently, convolutional neural networks (CNNs) based face hallucination
methods have achieved significant improvements over conventional face
hallucination methods because of its powerful feature representation capacity.
Compared with the general image, face image is a highly structured object
(Jiang et al., 2021), which is composed of facial shape model and facial
shape-free model (facial texture model) (Cootes et al., 1998; Liu, 2015). From
this point, the CNNs based face hallucination can be divided into two
categories: texture-oriented (Yang et al., 2018; Wang et al., 2020a; Jiang et
al., 2020; Lu et al., 2019) and shape-oriented methods (Chen et al., 2018b;
Kalarot et al., 2020; Bulat and Tzimiropoulos, 2018; Zhang et al., 2018c).
Texture-oriented methods aim to restore fine-grained facial texture details
through deep semantic features extracted by CNN. For example, Zhou _et al._
(Zhou et al., 2015a) firstly introduced CNNs into face hallucination and
designed bi-channel convolutional neural network to learn a mapping function
from LR face images to HR face images. Lu _et al._ (Lu et al., 2020) proposed
a global-local fusion network to refine high-frequency information, thereby
recovering fine facial texture details.

Despite the great success for texture-oriented methods, they ignore facial
structure information, generating face images with blurry facial structure.
Therefore, some researchers aim to develop shape-oriented methods that utilize
extra prior information (e.g., facial parsing maps, facial landmarks) to
provide facial structure information, thereby helping to restore facial shape,
and further use these shapes to predict missing high-frequency information.
For example, Song _et al._ (Song et al., 2017) used SRCNN (Dong et al., 2014)
to extract facial landmarks to divide facial components from LR face image and
feed the five components into different branches to reconstruct corresponding
components, thereby generating HR face image. Chen _et al._ (Chen et al.,
2018b) proposed a FSRNet, which utilized facial landmarks and parsing maps to
super-resolve high-fidelity facial image. There is a noteworthy problem that
extra prior information needs extra model parameters to predict, which
increase the cost and difficulty of training undoubtedly, and even make it
difficult for the model to fit and predict unrealistic prior information,
resulting in negative influence on the subsequent reconstruction process of
the shape-oriented methods. Therefore, the shape-oriented methods fail to
maintain the fidelity of facial detail restoration and consistency of facial
structure simultaneously, and even corrupt texture detail due to incorrect
prior information.

To address the aforementioned problems, we propose a simple but efficient end-
to-end method to reconstruct high-quality and high-fidelity HR face images.
Specifically, we first propose an internal-feature split attention (ISA) to
improve internal correlation of feature by splitting and classifying the
channel dimension of feature, which enables the network focuses on
information-rich regions and pay less attention to less information-rich
regions. ISA can be defined as internal-feature attention because it improves
internal correlation of features. Then, we design an internal-feature split
attention block (ISAB) to focus on restoration of facial texture details by
using proposed ISA. An external-internal split attention group (ESAG) is
proposed to simultaneously maintain the consistency of facial structure and
the fidelity of facial detail restoration by fusing multi-path features, where
one path cascades several ISABs to focus on facial texture details and the
other path focuses on the facial structure information through ISA. The
process of multi-path feature fusion can be defined as external-feature
attention because of improving external correlation between multi-path
features. Finally, we propose a split-attention in split-attention network
(SISN) to cascade several ESAGs for reconstructing photorealistic HR face
images. The contributions of our work can be summarized as follows:

Figure 1. Network architecture of SISN, which consists of four parts: coarse
feature extraction layer, split-attention in split-attention deep feature
extraction, upscale module, and reconstruction module.

(i) We propose an ISA to enable network to focus on information-rich regions
and ignores information-poor regions by splitting and classifying the channel
dimension of feature, and design an ISAB to focus on restoration of facial
texture details by ISA.

(ii) To fully maintain the consistency of facial structure and the fidelity of
facial detail restoration, an ESAG is proposed to fuse multi-path features
responsible for facial texture details and facial structure information,
respectively.

(iii) We design a SISN to reconstruct the high-fidelity and high-quality HR
face image from the input LR image by cascading several ESAGs, and demonstrate
that the proposed method can effectively maintain the fidelity of facial
detail restoration and consistency of facial structure through face
reconstruction experiments under two public face datasets and face recognition
experiments under a real-world surveillance scenarios dataset.

##  2\. Related work

Face hallucination, also known as face super-resolution, was first proposed by
Baker and Kanade (Baker and Kanade, 2000), which is the pioneer of face
hallucination technique. Since then many methods were proposed to improve the
performance of hallucinating face. Some early techniques have been proposed to
super-resolve LR face image by exploring and exploiting the prior of global or
local facial structures. Therefore, they can be roughly classified into global
based methods (Wang and Tang, 2005; Zhou et al., 2015b) and local based
methods (Chang et al., 2004; Yang et al., 2008; Jiang et al., 2013). Global
based methods super-resolve the whole image together to maintain the global
structure information. Zhou _et al._ (Zhou et al., 2015b) used orthogonal
canonical correlation analysis to achieve global face reconstruction. Local
based methods divide image into several overlapped small patches for better
local representation ability. Chang _et al._ (Chang et al., 2004) designed
locally linear representation for exploring the complex relationship between
LR and HR images. These methods fail to produce satisfactory results when
faced with high upscale factor because of weak feature representation
capability.

Recently, CNNs has been widely used in various upstream and downstream visual
tasks (Zhang et al., 2018b; Liu et al., 2020b, a; Wang et al., 2020b, 2021)
because of its powerful feature representation capabilities. Naturally, CNNs
based face hallucination methods (Yang et al., 2018; Chen et al., 2018b; Song
et al., 2017; Lu et al., 2019; Wang et al., 2020a; Jiang et al., 2020; Lu et
al., 2020) have achieved satisfactory performance, and have been further
driven to explore higher upscale factors. In the latest research, attention
mechanism is introduced into the CNNs based methods allowing deep neural
networks to pay more attention to useful information (Xin et al., 2019;
Kalarot et al., 2020; Wang et al., 2020a; Zhao and Zhang, 2020; Wang et al.,
2020c; Chen et al., 2020). For example, Xin _et al._ (Xin et al., 2019)
designed a attribute attention network to extract and exploit latent
representation of face attributes, thereby enhancing reconstruction
performance for face image. Kalarot _et al._ (Kalarot et al., 2020) proposed
facial component-wise attention maps, which utilized extra facial prior
information to allow network focusing on face feature inherent patterns. Chen
_et al._ (Chen et al., 2020) used spatial attention mechanism to focus on
different face structures, and produced high-quality face image.

However, the above attention based methods have a serious problem. They used
attention mechanism to either focus on the facial texture information or on
the facial structure information to use the structural information to
constrain texture restoration. Therefore, these methods fail to focus on
facial texture details and facial structure information at the same time,
resulting in unclear and unnatural face images.

##  3\. Split-Attention in Split-Attention Network for Face hallucination

###  3.1. Network Architecture

In this subsection, we describe the network architecture of SISN in detail.
Figure. 1 shows the architecture of SISN, which mainly consists four parts:
coarse feature extraction, split-attention in split-attention deep feature
extraction, upscale module, and reconstruction module. Let’s denote $I_{LR}$,
$I_{SR}$ as the input and output of SISN, and we use only one convolutional
layer with a kernel size of 3$\times$3 to extract the coarse feature $F_{CF}$
from the input images.

(1) $F_{CF}=H_{CF}(I_{LR}),$

where $H_{CF}(.)$ denotes the coarse feature extracting operation with one
convolutional layer. $F_{CF}$ is then used for split-attention in split-
attention deep feature extraction. So we can further have

(2) $F_{DF}=H_{DF}(F_{CF}),$

where $H_{DF}(.)$ denotes the deep feature extraction structure, which
contains $G$ ESAGs. After the $F_{CF}$ passes through deep feature extraction
and the network gets the deep feature being denoted $F_{DF}$, the upscaling
operation needs to be performed by the upscale module.

(3) $F_{UP}=H_{UP}(F_{DF}),$

where $F_{UP}$ and $H_{UP}(.)$ denote the upscaled feature and a upscale
module respectively. Finally, the upscaled feature is then reconstructed via a
reconstruction module. The reconstructed $I_{SR}$ is formulated as:

(4) $I_{SR}=H_{Recon}(F_{UP})=H_{SISN}(I_{LR}),$

where $H_{Recon}(.)$ and $H_{SISN}(.)$ denote the reconstruction module
composed of a convolutional layer with a kernel size of 3$\times$3 and the
function of our SISN, respectively.

Then SISN is optimized with loss function. To trade off the PSNR and image
construction quality, the proposed method utilizes Mean Absolute Error (MAE),
namely $L_{1}$ loss function, to define our loss function. Given a training
set $\left\\{I_{LR}^{i},I_{HR}^{i}\right\\}_{i=1}^{N}$, which contains $N$ LR
inputs and their HR counterparts. The $L_{1}$ loss function of SISN can be
represented as:

(5)
$L\left(\theta\right)=~{}\frac{1}{N}{\sum\limits_{i=1}^{N}\left\|{H_{SISN}(I_{LR}^{i})-I_{HR}^{i}}\right\|_{1}},$

where $\theta$ denotes the parameter set of our network.

###  3.2. Internal-feature Split Attention Block with Internal-feature
Attention

Previous attention based methods (Chen et al., 2020; Zhao and Zhang, 2020;
Wang et al., 2020a) used large-scale residual blocks to extract fine-grained
features, which made the these methods getting better performance. But the
limited receptive-field size and lacking of cross-channel interaction for
these residual blocks also reduce the internal correlation of features.
Inspired by previous work (Zhang et al., 2020), we propose the ISA to improve
internal correlation of features by splitting and classifying the channel
dimension of feature, which enable the network focuses on information-rich
regions and ignore information-poor regions. Then, we design an ISAB as the
basic block of deep feature extraction to focus on restoration of facial
texture details. The detailed structure of ISA and ISAB are shown in the
Figure. 1.

For the ISA, the input feature is first divided into $r$ splits along the
channel axis, and the number of feature channels is defined as $C$, thus the
number of channels per split is $\frac{C}{r}$. Then these splits are fused via
an element-wise summation across multiple splits.

(6)
$F_{fusion}=H_{Sum}({Split_{1}^{1}},{Split_{1}^{2}},\cdots,Split_{1}^{r}),$

where $H_{Sum}(.)$ denotes an element-wise summation operation,
$Split_{1}^{r}$ denotes the $r^{th}$ split by the first division, and
$F_{fusion}$ denotes the output by $H_{Sum}(.)$. The $F_{fusion}$ is then
passed through an adaptive average pooling layer and two 1$\times$1
convolutional layers, and is classified into $r$ splits again by a $r-softmax$
function according to the information richness of each previous split. Each of
the current $r$ splits is multiplied by each of the previous $r$ splits using
an element-wise product operation respectively, thereby improving the internal
correlation of features in channel-level. Finally, we use an element-wise
summation to fuse these splits as the output of ISA, which is denoted as
$F_{ISA}$. $F_{ISA}$ is formulated as:

(7)
$\begin{split}F_{ISA}=H_{Sum}(H_{MP}({Split}_{2}^{1},{Split}_{1}^{1}),H_{MP}({Split}_{2}^{2},\\\
{Split}_{1}^{2}),\cdots,H_{MP}({Split}_{2}^{r},{Split}_{1}^{r}))=H_{ISA}(F_{input}^{ISA}),\end{split}$

where $H_{MP}(.)$ denotes an element-wise product operation, the $H_{ISA}(.)$
represents the function of ISA, and $F_{input}^{ISA}$ denotes the input of
ISA.

Figure 2. Comparison with different combination of ESAGs and ISABs for
4$\times$ SR on FFHQ testing dataset. The best result is available when ESAGs
is 10 and ISABs is 10.

###  3.3. External-internal Split Attention Group with Internal- and External-
feature Attention

We now describe the proposed ESAG, which contains two paths. The first path
consists of $I$ ISABs and a convolution layer, which is mainly focus on
restoring fine facial details because it produces rich semantic information.
The output $F_{path}^{1}$ from this path is formulated as:

(8) $F_{path}^{1}=H_{Conv}(H_{ISAB}^{I}(H_{ISAB}^{I-1}(\cdots
H_{ISAB}^{1}(F_{input}^{ESAG})\cdots))),$

where $H_{Conv}(.)$ and $H_{ISAB}^{I}(.)$ denote a convolutional layer with
kernel size of 3$\times$3 and $I^{th}$ ISAB, respectively. $F_{input}^{ESAG}$
denotes the input of ESAG.

Considering the shallower features can provide more precise facial structure
information, for an ESAG, the input feature $F_{input}^{ESAG}$ can be used as
a shallow feature. Therefore, the second path uses $F_{input}^{ESAG}$ as input
and utilizes ISA to focus on facial structure information. The output
$F_{path}^{2}$ from the second path is formulated as:

(9) $F_{path}^{2}=H_{ISA}(F_{input}^{ESAG}).$

Finally, we use a multi-path features fusion module to fuse the features from
two paths to improve the external correlation of features, thereby restoring
fine facial details while maintaining the consistency of the facial structure.
The fused feature $F_{ESAG}$ is represented as:

(10) $F_{ESAG}=H_{MFM}(F_{path}^{1},F_{path}^{2})=H_{ESAG}(F_{input}^{ESAG}),$

where $H_{MFM}(.)$ denotes a multi-path features fusion module using element-
wise summation operation, and $H_{ESAG}(.)$ represents the function of ESAG.

Figure 3. Visual comparison for 8$\times$ SR with different methods. Three
images from the FFHQ testing set (HR size is 256$\times$256 pixels) are
selected as the test samples to show the reconstruction results of face
images.

##  4\. EXPERIMENTS

###  4.1. Dataset and Implementation Details

In this paper, two publicly available face datasets, namely FFHQ (Karras et
al., 2019) and CelebA (Liu et al., 2015), are used for extensive
reconstruction experiments. For FFHQ dataset, as a novel face dataset, it
consists of 70,000 high-quality images with considerable variation in terms of
facial attributes such as age and ethnicity as well as image background. We
randomly select 850 images as the training dataset, 100 images as the
validation dataset and 50 images as the testing dataset. The HR image size is
1024$\times$1024 pixels, and the downsampling factor are the 4 and 8, so that
the LR image size are 256$\times$256 and 128$\times$128 pixels respectively.
For CelebA dataset, as a large-scale face attributes dataset, it is consist of
more than 200k face images. We randomly select 2,000 images as the training
dataset, 140 images as the validation dataset and 60 images as the testing
dataset. All HR images are resized to 216$\times$176 pixels.

Data augmentation is performed on the all training images, which are randomly
rotated by 90°, 180°, 270°, and flipped horizontally. All experiments are
trained by Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and
$\epsilon=10^{-8}$. The initial leaning rate is set to $10^{-4}$, and then
decreases to half every 50 epochs. We use PyTorch to implement our models with
a RTX 2070 Super GPU.

###  4.2. Evaluation Metrics

The SR results are evaluated with four evaluation metrics: Peak Signal to
Noise Ratio (PSNR), Structural Similarity (SSIM) (Wang et al., 2004), Learned
Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018a), and Mean
Perceptual Score (MPS) (El Helou et al., 2020). PSNR and SSIM are standard
evaluation metrics, which are widely used in low-level visual tasks. LPIPS and
MPS are the novel perceptual metrics to measure the perceptive quality of
image, where the smaller value of LPIPS means more perceptual similarity. MPS
is the average of the SSIM and LPIPS, which is formulated as:

(11) $MPS=0.5\times(SSIM+(1-LPIPS)).$

Table 1. Verifying the effect of the two paths in ESAG. ”SISN w/o structure” indicates that the path of focus on facial structure is removed. ”SISN w/o detail” indicates that the ISA from all ISABs are removed. Methods | PSNR | SSIM | LPIPS | MPS
---|---|---|---|---
SISN w/o structure | 36.70 | 0.9260 | 0.2282 | 0.8489
SISN w/o detail | 36.59 | 0.9248 | 0.2296 | 0.8476
SISN | 36.86 | 0.9278 | 0.2212 | 0.8533

###  4.3. Ablation Experiments

####  4.3.1. Verifying the effect of the two paths in ESAG

To demonstrate the role of the two paths in ESAG, we set up a group of
experiments to verify. First, we remove the path that focuses on facial
structure based on SISN. Second ,we remove the ISA from all ISABs based on
SISN (equivalent to removing the core part of ISABs). The results of this
group of ablation experiments for 4$\times$ SR on the FFHQ face testing
dataset (HR images are 1024$\times$1024 pixels) are shown in Table 1 , where
we can conclude that when one of the paths is removed, there is a considerable
decrease in performance, which proves that the two paths have a key role in
the improvement of the reconstruction performance.

####  4.3.2. Exploring the best combination of ESAGs and ISABs

In this section, we conduct the ablation studies on the FFHQ face testing
dataset (HR images are 1024$\times$1024 pixels) to explore the most effective
combination of ESAGs and ISABs. We design two sets of experiments under
different constraints for 4$\times$ SR. Inspired by previous work (Zhang et
al., 2018b), in the first set of experiments, we constrain the number of ESAGs
to 10, and test the reconstruction performance of SISN when the number of
ISABs is from 5 to 20 and step is 1. In the second set of experiments, we
constrain the number of ISABs to 10, and test the reconstruction performance
of SISN when the number of ESAGs is from 5 to 20 and step is 1. The total
number of the above two sets of experiments is 31.

Figure. 2 shows the experiment results of SISN with above two sets of
experiments, we can find that when ESAGs is 10 and ISABs is 10, the
performance reaches the best. Meanwhile, the fluctuation range of PSNR (dB)
and SSIM in these experiment results are small, which also proves that the
proposed method does not rely on the deepening of the network to obtain better
performance.

Table 2. Comparison for 4$\times$ and 8$\times$ SR with the state-of-the-arts on FFHQ dataset, and perform experiments on 1024$\times$1024 images. The bold denotes the best results.  Methods | Scales | PSNR | SSIM | LPIPS | MPS
---|---|---|---|---|---
Bicubic | $\times 4$ | 34.80 | 0.9037 | 0.3104 | 0.7967
LCGE (Song et al., 2017) | 35.26 | 0.9042 | 0.2370 | 0.8336
RCAN (Zhang et al., 2018b) | 36.59 | 0.9250 | 0.2309 | 0.8471
SRFBN (Li et al., 2019) | 36.53 | 0.9243 | 0.2298 | 0.8473
MTC (Wang et al., 2020a) | 36.62 | 0.9259 | 0.2249 | 0.8505
SPARNet (Chen et al., 2020) | 36.37 | 0.9240 | 0.2323 | 0.8459
SISN(Ours) | 36.86 | 0.9278 | 0.2212 | 0.8533
Bicubic | $\times 8$ | 31.34 | 0.8495 | 0.4918 | 0.6789
LCGE (Song et al., 2017) | 31.18 | 0.8403 | 0.4040 | 0.7182
RCAN (Zhang et al., 2018b) | 33.15 | 0.8723 | 0.3776 | 0.7474
SRFBN (Li et al., 2019) | 32.99 | 0.8701 | 0.3810 | 0.7446
MTC (Wang et al., 2020a) | 32.94 | 0.8695 | 0.3794 | 0.7451
SPARNet (Chen et al., 2020) | 32.57 | 0.8684 | 0.3785 | 0.7450
SISN(Ours) | 33.25 | 0.8738 | 0.3674 | 0.7532
Table 3. Comparison for 4$\times$ and 8$\times$ SR with the state-of-the-arts on FFHQ dataset, and perform experiments on 256$\times$256 images. The bold denotes the best results.  Methods | Scales | PSNR | SSIM | LPIPS | MPS
---|---|---|---|---|---
Bicubic | $\times 4$ | 29.82 | 0.8459 | 0.3361 | 0.7549
LCGE (Song et al., 2017) | 31.09 | 0.8674 | 0.2189 | 0.8243
RCAN (Zhang et al., 2018b) | 32.65 | 0.8980 | 0.1720 | 0.8630
SRFBN (Li et al., 2019) | 32.40 | 0.8947 | 0.1742 | 0.8603
MTC (Wang et al., 2020a) | 31.99 | 0.8888 | 0.1628 | 0.8630
SPARNet (Chen et al., 2020) | 32.36 | 0.8933 | 0.1878 | 0.8528
SISN(Ours) | 32.83 | 0.9011 | 0.1664 | 0.8674
Bicubic | $\times 8$ | 25.99 | 0.7313 | 0.5594 | 0.5860
LCGE (Song et al., 2017) | 26.13 | 0.7326 | 0.4159 | 0.6584
RCAN (Zhang et al., 2018b) | 28.17 | 0.7932 | 0.3329 | 0.7302
SRFBN (Li et al., 2019) | 28.07 | 0.7906 | 0.3388 | 0.7259
MTC (Wang et al., 2020a) | 27.57 | 0.7767 | 0.3537 | 0.7115
SPARNet (Chen et al., 2020) | 28.20 | 0.7965 | 0.3355 | 0.7305
SISN(Ours) | 28.37 | 0.7999 | 0.3230 | 0.7385
Table 4. Comparison for 4$\times$ and 8$\times$ SR with the state-of-the-arts on CelebA dataset. The bold denotes the best results.  Methods | Scales | PSNR | SSIM | LPIPS | MPS
---|---|---|---|---|---
Bicubic | $\times 4$ | 28.82 | 0.8330 | 0.3589 | 0.7371
LCGE (Song et al., 2017) | 27.22 | 0.6818 | 0.5379 | 0.5720
RCAN (Zhang et al., 2018b) | 32.19 | 0.8993 | 0.1876 | 0.8559
SRFBN (Li et al., 2019) | 32.12 | 0.8981 | 0.1895 | 0.8543
MTC (Wang et al., 2020a) | 31.92 | 0.8947 | 0.1910 | 0.8519
SPARNet (Chen et al., 2020) | 31.99 | 0.8962 | 0.1965 | 0.8499
SISN(Ours) | 32.34 | 0.9009 | 0.1852 | 0.8579
Bicubic | $\times 8$ | 24.74 | 0.6969 | 0.6290 | 0.5340
LCGE (Song et al., 2017) | 23.73 | 0.5620 | 0.8356 | 0.3632
RCAN (Zhang et al., 2018b) | 27.32 | 0.7834 | 0.3842 | 0.6996
SRFBN (Li et al., 2019) | 27.22 | 0.7810 | 0.3534 | 0.7138
MTC (Wang et al., 2020a) | 26.91 | 0.7712 | 0.3566 | 0.7073
SPARNet (Chen et al., 2020) | 27.29 | 0.7884 | 0.3573 | 0.7156
SISN(Ours) | 27.59 | 0.7912 | 0.3496 | 0.7208
Figure 4. Visual comparison for 8$\times$ SR with different methods. Three
images from the CelebA testing set are selected as the test samples to show
the reconstruction results of face images.

###  4.4. Comparison with State-of-the-Arts

In this section, we select some excellent face hallucination or general SR
methods as: Bicubic, LCGE (Song et al., 2017), RCAN (Zhang et al., 2018b),
SRFBN (Li et al., 2019), MTC (Wang et al., 2020a), and SPARNet (Chen et al.,
2020). Bicubic is a classic image interpolation algorithm; LCGE is a face
hallucination using facial landmarks; RCAN is a deep residual channel
attention network based SR method; SRFBN is a feedback network based SR
method; MTC is a novel face hallucination that uses the texture-attention
mechanism; SPARNet is the latest face hallucination based on spatial attention
mechanism.

####  4.4.1. Comparing on FFHQ dataset (HR images are 1024$\times$1024 pixels)

Table 2 lists the quantitative experimental results of different state-of-the-
arts and proposed method on 1024$\times$1024 FFHQ testing set for 4$\times$
and 8$\times$ SR, it is obviously that our SISN outperforms state-of-the-arts
in PSNR, SSIM, LPIPS, and MPS significantly. These results prove that SISN can
improve the reconstruction performance of face images effectively by improving
the consistency of facial structure and the fidelity of facial details.

For the visual performance, each algorithm has a little gap in visual
reconstruction results since the size of HR face images are large. To further
prove the effectiveness of the proposed method, we will discuss experiments on
small-scale face images in the next part.

Figure 5. Visual reconstruction results on real-world surveillance scenarios
for 4$\times$ SR. Intuitively, the results of Bicubic, LCGE, and SRFBN are
very vague. There are texture distortion and unrealistic artifacts around the
red mark. In contrast, our method produces cleaner face and maintains more
facial structure information.

####  4.4.2. Comparing on FFHQ dataset (HR images are 256$\times$256 pixels)

In the real-world surveillance scenarios, the resolution of the face image
captured by the imaging sensor is often lower than the size of 100$\times$100,
or even lower than 50$\times$50\. At this resolution, the various downstream
tasks related to face will be almost collapsed. Therefore, the performance of
face hallucination on small-scale face images is important particularly. To
verify the performance of the proposed method on small-size face images, we
resize the original HR image to 256$\times$256 as the groundtruth, the
corresponding LR image size for 4$\times$ and 8$\times$ SR are 64$\times$64
and 32$\times$32 pixels, respectively.

Table 3 summarizes the quantitative results of different methods on the
256$\times$256 FFHQ testing set for 4$\times$ and 8$\times$ SR, it is no
surprise that SISN still surpasses these state-of-the-arts, which prove that
the proposed method has high generalization ability in small-scale face
images.

In terms of visual performance, Figure. 3 shows the qualitative results of
different methods for 8$\times$ SR on 256$\times$256 FFHQ testing set, we
select three representative samples to compare. (a) to (f) are the results
from six selected state-of-the-art algorithms, and (h) is the HR groundtruth
as the benchmark. From these visual results, we can observe that the results
of LCGE is almost same as Bicubic, and the facial texture details are almost
lost. For the sample (1), we discover intuitively that all of these state-of-
the-arts cause severe structural and textural distortions in eye region,
resulting in abnormal human eyes. RCAN, SRFBN, and SPARNet lost some details
in the left eye region of sample (2), while MTC causes the distortion of
structural information in the eye and mouth regions. In contrast, our method
achieves the high visual performance and maintains the satisfactory facial
structure.

####  4.4.3. Comparing on CelebA dataset

On the FFHQ dataset, our method shows high reconstruction performance.
Compared with the FFHQ dataset, the original images of the CelebA dataset are
blurred and contain some noise, which are closer to the real-world scenarios.
Therefore, in this section, we use this challenging dataset for experiments.
We resize the original image to 176$\times$216 as the groundtruth, the
corresponding LR image size for 4$\times$ and 8$\times$ SR are only
44$\times$54 and 22$\times$27 pixels, respectively.

Table 4 shows the quantitative results of different methods on CelebA dataset,
the proposed SISN has achieved an overwhelming victory in all evaluation
metrics. Figure 4 shows the qualitative results of different methods for
8$\times$ SR on CelebA testing set. We can conclude that compared with other
methods, our SISN can still restore fine facial details while maintaining
better facial structure consistency.

It is worth noting that both quantitative and qualitative experiments of LCGE
have caused extremely poor results, and we analyze the potential reason that
LCGE can not adapt to these images containing noise and blur, thereby
resulting in the reconstructed image containing a large number of unrealistic
artifacts, which can be seen in (b) of Figure 4.

###  4.5. Face Reconstruction and Recognition on Real-world Surveillance
scenarios

####  4.5.1. Face Reconstruction

All experiments in the previous section are in the simulation scenarios, which
can not simulate the real-world scenarios well. For face hallucination, real-
world surveillance scenarios are a challenging environment. Compared with the
simulation scenarios, the surveillance scenarios face two serious problems:
(1) there is no corresponding HR images; (2) the face image captured by the
surveillance camera often contains a lot of noise and serious distortion of
color and texture. Therefore, in this scenarios, it is a severe challenge to
face hallucination and requires the algorithm to reconstruct the HR face image
with good facial structure and texture details from the distorted LR image as
much as possible to improve the performance of downstream tasks such as face
detection and recognition.

To verify the reconstruction performance of our method in real-world
surveillance scenarios, we select a low-quality face image (image size is
42$\times$56 pixels) from SCface dataset (Grgic et al., 2011) which contains a
large number of face images captured by surveillance cameras to perform face
reconstruction experiments. Visual result of different methods (4$\times$ SR)
is shown in Figure. 5. We can conclude that compared to these state-of-the-
arts, our SISN produces cleaner face image and maintains more facial structure
information, which can improve the performance of downstream tasks effectively
and will be proved in the next part.

Figure 6. A sample pair of Candidate 1, where (A) denotes a high-definition frontal face image captured by a SLR camera as a face feature sample of Candidate 1, and (B) denotes a LR face image captured from a surveillance camera as a target image of the face recognizer. Table 5. Comparison results for face similarity of face images reconstructed by different methods. We randomly select high-definition frontal face images of six test candidates captured by SLR camera in the SCface dataset as source face feature samples, and select the LR face images captured by the same surveillance camera corresponding to the six test candidates as the target image of the face recognizer. The bold denotes the best similarity, and the ”-” means that the face of candidate is not detected. Methods | Candidate 1 | Candidate 2 | Candidate 3 | Candidate 4 | Candidate 5 | Candidate 6 | Average Similarity
---|---|---|---|---|---|---|---
LR image | - | - | - | - | - | 0.449837 | 0.074973
LCGE (Song et al., 2017) | 0.468556 | 0.222741 | 0.269097 | 0.106820 | - | 0.607645 | 0.279143
RCAN (Zhang et al., 2018b) | 0.356093 | 0.590624 | 0.509804 | 0.512736 | 0.347659 | 0.612440 | 0.488226
SRFBN (Li et al., 2019) | 0.397624 | 0.627165 | 0.519975 | 0.537452 | 0.359543 | 0.577195 | 0.503159
MTC (Wang et al., 2020a) | 0.430793 | 0.523213 | 0.583992 | 0.534206 | 0.404109 | 0.556816 | 0.505522
SPARNet (Chen et al., 2020) | 0.444318 | 0.483088 | 0.442757 | 0.543900 | 0.343557 | 0.607939 | 0.477593
SISN(Ours) | 0.475734 | 0.599383 | 0.502253 | 0.548835 | 0.490079 | 0.624352 | 0.540106

####  4.5.2. Face Recognition

As a low-level visual task, the ultimate goal of face hallucination is to
improve the performance of high-level visual tasks. Among the high-level
vision tasks related to faces, face recognition tasks are widely used in
various fields. Therefore, in this part, we use MTCNN (Zhang et al., 2016) as
the face detector and MobileFacenets (Chen et al., 2018a) as the face
recognizer to compare the impact of the reconstructed results of different
methods on face recognition performance in real-world surveillance scenarios.

First, we randomly select high-definition frontal face images of six test
candidates (identities) captured by SLR (Single Lens Reflex) camera in the
SCface dataset as source face feature samples. Then, we select the LR face
images captured by the same surveillance camera corresponding to the six test
candidates as the target image of the face recognizer. To visualize the
selected frontal high-definition face image and the face image captured by the
surveillance camera, Figure 6 shows a sample pair of face images for Candidate
1, where (A) denotes a source face feature sample of Candidate 1, and (B)
denotes a target image of the face recognizer.

To take the Candidate 1 for example, the complete face recognition experiment
consists of the following steps. First, the high-definition frontal face of
Candidate 1 is detected by face detector from (A), and face features are
extracted by face recognizer from (A). Then the same method is used to perform
face detection and feature extraction on (B). Finally the face recognizer
calculates the similarity based on the two face features to get the normalized
similarity (similarity range is 0-1), the higher the value means the face
captured by the surveillance camera is more similar to the face of Candidate
1, and further improves the performance of face recognition. To compare the
effect of different face hallucination methods on face recognition performance
objectively, we use the LR face image as a baseline and calculate the
similarity between the face images reconstructed by different methods
(4$\times$ SR) and the high-definition frontal face images.

Table 5 shows the face similarity of each LR face images and each HR face
images reconstructed by different methods for selected six test candidates
after face recognition in real-world surveillance scenarios. From these
quantitative results, we can draw two conclusions: (1) only one face is
detected among the six original LR face images; (2) the reconstructed face
images after face hallucination can significantly improve the performance of
face recognition due to the improved face similarity. The above two points
prove the challenge of the selected test samples and the importance of face
hallucination. From the data in Table 5, we can further conclude that the
proposed method has the highest score on the four test candidates (Candidate
1, 4, 5, and 6) and achieves the best in average similarity, which proves
convincingly that the proposed method can maintain the fidelity of facial
detail restoration and consistency of facial structure simultaneously.

##  5\. CONCLUSION

In this paper, a novel face hallucination method is proposed, which can
reconstruct high-quality and high-confidence face images consistently proven
by extensive face reconstruction and face recognition experiments. We design a
SISN to maintain the fidelity of facial detail restoration and consistency of
facial structure by fusing multi-path features responsible for facial texture
details and facial structure information, respectively. In the future, we
believe that the proposed method can be easily applied to other image
restoration problems, such as denoising, deblurring and general image SR.

###### Acknowledgements.

This work is supported by the National Natural Science Foundation of China
(62072350, 61771353, 62001334), Hubei Technology Innovation Project
(2019AAA045), the Central Government Guides Local Science and Technology
Development Special Projects (2018ZYYD059), 2020 Hubei Province High-value
Intellectual Property Cultivation Project, the Wuhan Enterprise Technology
Innovation Project (202001602011971), the Graduate Innovation Fund of Wuhan
Institute of Technology (CX2020223).

## References

  * (1)
  * Baker and Kanade (2000) Simon Baker and Takeo Kanade. 2000.  Hallucinating faces. In _Proceedings Fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580)_. IEEE, Washington, DC, USA, 83–88.
  * Bulat and Tzimiropoulos (2018) Adrian Bulat and Georgios Tzimiropoulos. 2018.  Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, USA, 109–117.
  * Chang et al. (2004) Hong Chang, Dit-Yan Yeung, and Yimin Xiong. 2004\.  Super-resolution through neighbor embedding. In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , Vol. 1. IEEE, Washington, DC, USA, 275–282.
  * Chen et al. (2020) Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, and Kwan-Yee K Wong. 2020.  Learning Spatial Attention for Face Super-Resolution.  _IEEE Transactions on Image Processing_ 30 (2020), 1219–1231.
  * Chen et al. (2018a) Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. 2018a.  Mobilefacenets: Efficient cnns for accurate real-time face verification on mobile devices. In _Chinese Conference on Biometric Recognition_. Springer, Urumqi, China, 428–438.
  * Chen et al. (2018b) Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. 2018b.  Fsrnet: End-to-end learning face super-resolution with facial priors. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Salt Lake City, UT, USA, 2492–2501.
  * Cootes et al. (1998) Timothy F Cootes, Gareth J Edwards, and Christopher J Taylor. 1998\.  Active appearance models. In _European conference on computer vision_. Springer, Freiburg, Germany, 484–498.
  * Dong et al. (2014) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2014\.  Learning a deep convolutional network for image super-resolution. In _European conference on computer vision_. Springer, Zurich, Switzerland, 184–199.
  * El Helou et al. (2020) Majed El Helou, Ruofan Zhou, Sabine Süsstrunk, Radu Timofte, et al. 2020\.  AIM 2020: Scene Relighting and Illumination Estimation Challenge. In _Proceedings of the European Conference on Computer Vision Workshops (ECCVW)_. Springer, Glasgow, UK, 499–518.
  * Grgic et al. (2011) Mislav Grgic, Kresimir Delac, and Sonja Grgic. 2011\.  SCface–surveillance cameras face database.  _Multimedia tools and applications_ 51, 3 (2011), 863–879.
  * Jiang et al. (2013) Junjun Jiang, Ruimin Hu, Zhen Han, Zhongyuan Wang, Tao Lu, and Jun Chen. 2013\.  Locality-constraint iterative neighbor embedding for face hallucination. In _2013 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, San Jose, USA, 1–6.
  * Jiang et al. (2021) Junjun Jiang, Chenyang Wang, Xianming Liu, and Jiayi Ma. 2021\.  Deep Learning-based Face Super-resolution: A Survey.  _arXiv preprint arXiv:2101.03749_ (2021).
  * Jiang et al. (2020) Kui Jiang, Zhongyuan Wang, Peng Yi, Tao Lu, Junjun Jiang, and Zixiang Xiong. 2020\.  Dual-Path Deep Fusion Network for Face Image Hallucination.  _IEEE Transactions on Neural Networks and Learning Systems_ (2020).
  * Kalarot et al. (2020) Ratheesh Kalarot, Tao Li, and Fatih Porikli. 2020.  Component attention guided face super-resolution network: Cagface. In _The IEEE Winter Conference on Applications of Computer Vision_. IEEE, Snowmass, CO, USA, 370–380.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. IEEE, Long Beach, CA, USA, 4401–4410.
  * Li et al. (2019) Zhen Li, Jinglei Yang, Zheng Liu, Xiaomin Yang, Gwanggil Jeon, and Wei Wu. 2019\.  Feedback network for image super-resolution. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, Long Beach, CA, USA, 3867–3876.
  * Liu (2015) Haowei Liu. 2015\.  Face Technologies on Mobile Devices.  In _Facial Detection and Recognition on Mobile Devices_ , Haowei Liu (Ed.). Morgan Kaufmann, Edinburgh, 11–38.
  * Liu et al. (2020a) Wei Liu, Xianxu Hou, Jiang Duan, and Guoping Qiu. 2020a.  End-to-end single image fog removal using enhanced cycle consistent adversarial networks.  _IEEE Transactions on Image Processing_ 29 (2020), 7819–7833.
  * Liu et al. (2020b) Wei Liu, Fei Zhou, Tao Lu, Jiang Duan, and Guoping Qiu. 2020b.  Image Defogging Quality Assessment: Real-World Database and Method.  _IEEE Transactions on Image Processing_ 30 (2020), 176–190.
  * Liu et al. (2015) Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015\.  Deep learning face attributes in the wild. In _Proceedings of the IEEE international conference on computer vision_. IEEE, Santiago, Chile, 3730–3738.
  * Lu et al. (2018) Tao Lu, Yingjie Guan, Yanduo Zhang, Shenming Qu, and Zixiang Xiong. 2018.  Robust and efficient face recognition via low-rank supported extreme learning machine.  _Multimedia Tools and Applications_ 77, 9 (2018), 11219–11240.
  * Lu et al. (2019) Tao Lu, Xiaohui Hao, Yanduo Zhang, Kai Liu, and Zixiang Xiong. 2019.  Parallel region-based deep residual networks for face hallucination.  _IEEE Access_ 7 (2019), 81266–81278.
  * Lu et al. (2020) Tao Lu, Jiaming Wang, Junjun Jiang, and Yanduo Zhang. 2020\.  Global-local fusion network for face super-resolution.  _Neurocomputing_ 387 (2020), 309–320.
  * Lu et al. (2021) Tao Lu, Qiang Zhou, Wenhua Fang, and Yanduo Zhang. 2021\.  Discriminative metric learning for face verification using enhanced Siamese neural network.  _Multimedia Tools and Applications_ 80, 6 (2021), 8563–8580.
  * Song et al. (2017) Yibing Song, Jiawei Zhang, Shengfeng He, Linchao Bao, and Qingxiong Yang. 2017.  Learning to Hallucinate Face Images via Component Generation and Enhancement. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence_. Melbourne, Australia, 4537–4543.
  * Wang et al. (2020c) Chenyang Wang, Zhiwei Zhong, Junjun Jiang, Deming Zhai, and Xianming Liu. 2020c.  Parsing Map Guided Multi-Scale Attention Network For Face Hallucination. In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, Barcelona, Spain, 2518–2522.
  * Wang and Tang (2005) Xiaogang Wang and Xiaoou Tang. 2005.  Hallucinating face by eigentransformation.  _IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews)_ 35, 3 (2005), 425–434.
  * Wang et al. (2020a) Yu Wang, Tao Lu, Ruobo Xu, and Yanduo Zhang. 2020a.  Face Super-Resolution by Learning Multi-view Texture Compensation. In _International Conference on Multimedia Modeling_. Springer, Daejeon, Korea, 350–360.
  * Wang et al. (2020b) Yuanzhi Wang, Tao Lu, Tao Zhang, and Yuntao Wu. 2020b.  Seeing Pedestrian in the Dark via Multi-Task Feature Fusing-Sharing Learning for Imaging Sensors.  _Sensors_ 20, 20 (2020), 5852.
  * Wang et al. (2021) Yuanzhi Wang, Tao Lu, Yanduo Zhang, and Yuntao Wu. 2021\.  Multi-scale Self-calibrated Network for Image Light Source Transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 252–259.
  * Wang et al. (2004) Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ 13, 4 (2004), 600–612.
  * Xin et al. (2019) Jingwei Xin, Nannan Wang, Xinbo Gao, and Jie Li. 2019\.  Residual attribute attention network for face image super-resolution. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. Hawaii, USA, 9054–9061.
  * Yang et al. (2008) Jianchao Yang, John Wright, Thomas Huang, and Yi Ma. 2008\.  Image super-resolution as sparse representation of raw image patches. In _2008 IEEE conference on computer vision and pattern recognition_. IEEE, Anchorage, Alaska, USA, 1–8.
  * Yang et al. (2018) Xi Yang, Tao Lu, Jiaming Wang, Yanduo Zhang, Yuntao Wu, Zhongyuan Wang, and Zixiang Xiong. 2018.  Enhanced discriminative generative adversarial network for face super-resolution. In _Pacific Rim Conference on Multimedia_. Springer, Hefei, China, 441–452.
  * Zhang et al. (2020) Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Zhi Zhang, Haibin Lin, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. 2020\.  Resnest: Split-attention networks.  _arXiv preprint arXiv:2004.08955_ (2020).
  * Zhang et al. (2018c) Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H Hsu, Yu Qiao, Wei Liu, and Tong Zhang. 2018c.  Super-identity convolutional neural network for face hallucination. In _Proceedings of the European conference on computer vision (ECCV)_. Springer, Munich, Germany, 183–198.
  * Zhang et al. (2016) Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016\.  Joint face detection and alignment using multitask cascaded convolutional networks.  _IEEE Signal Processing Letters_ 23, 10 (2016), 1499–1503.
  * Zhang et al. (2018a) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. 2018a.  The unreasonable effectiveness of deep features as a perceptual metric. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. IEEE, Salt Lake City, UT, USA, 586–595.
  * Zhang et al. (2018b) Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. 2018b.  Image super-resolution using very deep residual channel attention networks. In _Proceedings of the European Conference on Computer Vision (ECCV)_. Springer, Munich, Germany, 286–301.
  * Zhao and Zhang (2020) Tianyu Zhao and Changqing Zhang. 2020.  SAAN: Semantic Attention Adaptation Network for Face Super-Resolution. In _2020 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, London, UK, 1–6.
  * Zhou et al. (2015a) Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. 2015a.  Learning face hallucination in the wild. In _Proceedings of the AAAI Conference on Artificial Intelligence_. Austin, Texas, USA.
  * Zhou et al. (2015b) Huiling Zhou, Jiwei Hu, and Kin-Man Lam. 2015b.  Global face reconstruction for face hallucination using orthogonal canonical correlation analysis. In _2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA)_. IEEE, Hong Kong, China, 537–542.