# Subtracted Cumulants: Mitigating Large Background in Jet Substructure

Yang-Ting Chien<EMAIL_ADDRESS>Center for Theoretical Physics, Massachusetts
Institute of Technology, Cambridge, MA 02139, USA  Daekyoung Kang
<EMAIL_ADDRESS>Key Laboratory of Nuclear Physics and Ion-beam Application
(MOE) and Institute of Modern Physics, Fudan University, Shanghai, China
200433  Kyle Lee<EMAIL_ADDRESS>C.N. Yang Institute for Theoretical
Physics, Stony Brook University, Stony Brook, NY 11794, USA  Department of
Physics and Astronomy, Stony Brook University, Stony Brook, NY 11794, USA
Yiannis Makris<EMAIL_ADDRESS>Theoretical Division T-2, Los Alamos National
Laboratory, Los Alamos, NM, 87545, USA

###### Abstract

We introduce a new approach for jet physics studies using subtracted cumulants
of jet substructure observables, which are shown to be insensitive to
contributions from soft-particle emissions uncorrelated with the hard process.
Therefore subtracted cumulants allow comparisons between theoretical
calculations and experimental measurements without the complication of large
background contaminations such as underlying and pileup events in hadron
collisions. We test our method using subtracted jet mass cumulants by
comparing Monte Carlo simulations to analytic calculations performed using
soft-collinear effective theory. We find that, for proton-proton collisions,
the method efficiently eliminates contributions from multipart on interactions
and pileup events. We also find within theoretical uncertainty that our
analytic calculations are in good agreement with the subtracted cumulants
calculated by using ATLAS jet mass measurements.

††preprint: LA-UR-18-31092††preprint: MIT-CTP 5088

##  I Introduction

Jets have become essential objects of study at high energy colliders. They are
produced ubiquitously in hard scattering processes as well as hadronic decays
of heavy particles. Tremendous progress has been made to understand and use
jets for testing the standard model and searching for new physics, with
reliable Monte Carlo simulations Sjostrand  _et al._ (2008); Bellm  _et al._
(2016) of high-energy collisions as well as useful jet substructure analysis
tools Larkoski  _et al._ (2017); Andrews  _et al._ (2018). However,
theoretical precision is often limited by the need to model soft radiation
such as from multiparton interactions (MPI) in hadron collisions, which are
insensitive to the hard process in a wide range of energies. Other
contributions to the underlying event (UE) are either calculable (e.g. initial
state radiation (ISR)) or relatively small (e.g. hadronization). In high-
luminosity (HL) collisions there can also be a large number of uncorrelated
pileup (PU) events producing a significant background of soft particles. With
accurate vertex determination using charged-particle tracks, one can remove PU
charged particles but still not PU neutral particles. Also, heavy-ion
collisions (HIC) can produce a large number of soft particles
($\sim\mathcal{O}(10^{5})$) through the interactions among a large number of
nucleons in the nuclei. The soft particles are observed to be mostly
uncorrelated with the hard process of interest and exhibit novel collective
behaviors Voloshin and Zhang (1996); Poskanzer and Voloshin (1998). However,
correlated effects certainly exist and significantly quench the jets Bjorken ;
Connors  _et al._ (2018).

It is clear that jet observables are affected dramatically by an uncorrelated
large background, which can overwhelm the correlated effects one wants to
probe such as jet modifications and medium responses in HIC. Many subtraction
techniques Cacciari and Salam (2008); Feige  _et al._ (2012); Soyez  _et al._
(2013); Larkoski and Thaler (2014); Krohn  _et al._ (2014); Cacciari  _et al._
(2015); Bertolini  _et al._ (2014a); Berta  _et al._ (2014); Larkoski  _et
al._ (2014); Komiske  _et al._ (2017); Soyez (2019) have been developed in
order to correct jets back to their true compositions. Ideally one would like
the subtraction to work for each jet, which is impossible due to the intrinsic
ambiguity between signal and background particles. One could then hope to
remove the background and correct jet observable distributions statistically.
The precision of the background subtraction will then have to be quantified
when comparing measurements to analytic calculations.

In this paper, instead of relying on algorithms to remove soft particles out
of jets, we provide an alternative approach for comparing theoretical
calculations directly to experimental measurements without the complication of
modeling soft uncorrelated emissions (SUEs). Specifically, we define
subtracted cumulants which cancel the contributions from SUEs. This approach
was first introduced in the context of the transverse energy of Drell-Yan
processes Kang  _et al._ (2018a). Here we extend its use to jet substructure
observables to which SUEs additively contribute. Additive contributions from
uncorrelated emissions can be easily removed, and the resulting subtracted
cumulant is thus useful for precision jet physics studies. The jet mass,
$m_{J}$ is a classic observable which receives additive contributions from
SUEs. In this paper, as a proof of concept, we focus only on the first
cumulant of jet invariant mass in proton-proton collisions. It is
straightforward to extend our study to other jet substructure observables and
higher cumulants.

The rest of the paper is organized as follows. We first give the definition of
subtracted jet mass cumulant $\Delta$, and we demonstrate its robustness
against SUEs using Pythia Monte Carlo simulations. Since jet substructure
observables highly depend on the jet-initiating partons we also show the
sensitivity of $\Delta$ to quark-gluon jet fractions. Finally, we show that
the comparisons of our theoretical predictions performed using soft-collinear
effective theory (SCET) Bauer  _et al._ (2000, 2001); Bauer and Stewart
(2001); Bauer  _et al._ (2002a, b); Beneke  _et al._ (2002) to the results
computed from ATLAS measurements are in a good agreement.

##  II Definition of the observable

For a jet substructure observable, $e$, which receives additive contributions
from individual particles within a jet, we have

$e=\sum_{i\in\text{jet}}\hat{e}^{(i)}\;,$ (1)

where $\hat{e}^{(i)}$ depends on the four-momenta of the $i$-th particle in
the jet. In the presence of SUEs, because of the additivity, the observable
can be decomposed into two terms,

$e=\sum_{i\in\text{signal}}\hat{e}^{(i)}+\sum_{i\in\text{SUEs}}\hat{e}^{(i)}\equiv
e_{\text{S}}+e_{\text{B}}\;,$ (2)

where “S” refers to the signal contributions which are correlated with the
hard process and “B” to the background from SUEs. Here $e_{\text{B}}$ is the
background contribution statistically independent of $e_{\text{S}}$. Its
probability density does not depend on the kinematics and details of the hard
process, such as the jet energy, angular direction, and the flavor of the
initiating parton. Let $P_{\text{S}}(e_{\text{S}})$,
$P_{\text{B}}(e_{\text{B}})$, and $P(e)$ denote probability densities of the
observables $e_{\text{S}}$, $e_{\text{B}}$, and $e$, respectively. Since SUEs
are uncorrelated with the signal, the probability density at the values
$e_{\text{S}}$ and $e_{\text{B}}$ is simply a product of uncorrelated
distributions
$P(e_{\text{S}},e_{\text{B}})=P_{\text{S}}(e_{\text{S}})P_{\text{B}}(e_{\text{B}})$.
Then, $P(e)$ is given by

$\displaystyle P(e)$ $\displaystyle=$ $\displaystyle\int
de_{\text{S}}\;de_{\text{B}}\;\delta(e-e_{\text{S}}-e_{\text{B}})P(e_{\text{S}},e_{\text{B}})$
(3) $\displaystyle=$ $\displaystyle\int
de_{\text{B}}\;P_{\text{S}}(e-e_{\text{B}})P_{\text{B}}(e_{\text{B}})\;,$

which has a convolution form. The cumulants $\kappa_{n}(e)$ are defined using
the cumulant-generating function $K(t)$,

$K(t)=\sum_{\rm n=1}^{\infty}\kappa_{n}\frac{t^{n}}{n!}=\log\langle{\rm
exp}(te)\rangle\;,$ (4)

where $\langle\cdots\rangle$ denotes the expectation value. Note the
additivity of cumulants:
$\kappa_{n}(e)=\kappa_{n}(e_{\text{S}})+\kappa_{n}(e_{\text{B}})$ which will
allow us to cancel uncorrelated contributions in the subtraction between
cumulants. Also, cumulants are in one-to-one correspondence with moments
$\langle e^{n}\rangle$: $\kappa_{1}(e)=\langle e\rangle$,
$\kappa_{2}(e)=\langle e^{2}\rangle-\langle e\rangle^{2}$,
$\kappa_{3}(e)=\langle e^{3}\rangle-3\langle e^{2}\rangle\langle
e\rangle+2\langle e\rangle^{3}$, etc.

Although Eq.(2) is by definition true for all additive observables, jet
substructure observables defined with the standard jet axis determined via
E-scheme is subject to an axis recoil effect due to soft radiation, which
correlates the soft background and signal radiation. However, such effects are
shown to be power-suppressed Ellis  _et al._ (2010) for many of the additive
observables, including the jet mass we study here. Also, as we show later
using simulation data, this effect is expected to be small for subtracted
cumulants due to the uniformity of soft radiation. Alternatively one may use a
recoil free axis such as the winner-take-all axis Bertolini  _et al._ (2014b).
There are also many additive jet observables which are insensitive to jet axis
direction which would not be subject to such recoil effects Banfi  _et al._
(2005).

We define the jet substructure observable $\hat{\tau}$ which is closely
related to the jet invariant mass and receives additive contributions from
signal and background,

$\hat{\tau}=2\cosh(\eta)\sum_{i\in\text{jet}}p_{i}^{+}=\hat{\tau}_{S}+\hat{\tau}_{B}=\frac{m_{J}^{2}}{p_{T}}\left[1+\mathcal{O}\Big{(}\frac{m_{J}^{2}}{p_{T}^{2}}\Big{)}\right]\;,$
(5)

where $p_{T}$ and $\eta$ are the jet transverse momentum and pseudorapidity
with respect to the beam axis, and $p^{+}=p^{0}-\vec{n}\cdot\vec{p}$ is the
small light-cone component of the constituent’s momentum with respect to the
jet axis $\vec{n}$. Then for the dimensionless observable
$\tau=\hat{\tau}/p_{T}$, up to corrections suppressed by $(p^{\rm
SUE}_{T}R^{2}/p_{T})^{2}$, where $R$ is the jet radius, Eq.(3) becomes

$\frac{d\sigma}{d\tau dp_{T}d\eta}(\tau,p_{T})=\int
d\hat{\tau}_{\text{B}}\frac{d{\sigma}_{\text{corr}}}{d\tau
dp_{T}d\eta}\Big{(}\tau-\frac{\hat{\tau}_{\text{B}}}{p_{T}},p_{T}\Big{)}f(\hat{\tau}_{\text{B}})\;,$
(6)

where the function $f$ is the normalized probability distribution of the SUE
contribution to the observable $\hat{\tau}$. It was shown that this
convolutional expression with a simple model for $f$ well describes the MPI
contribution in Monte Carlo simulations Stewart  _et al._ (2015); Hoang  _et
al._ (2017); Kang  _et al._ (2018a) and experimental measurements Kang  _et
al._ (2018b). Note that the expression in Eq. (6) resembles the factorization
of hadronization contributions derived using operator product expansion Lee
and Sterman (2007), and hadronization is correlated with the hard process.
Here, $f$ only includes SUEs and is observed to be independent of the jet
pseudorapidity in the plateau region. Due to the similar convolution structure
for hadronization effects, hadronization effects are also largely removed in
the subtracted cumulant we define below for proton-proton collisions.

For the first $\tau$ cumulant (the first moment, equivalently), which is
denoted by $\langle\tau\rangle$ and is a function of jet $p_{T}$ and $\eta$,

$\langle\tau\rangle=\left(\frac{d\sigma}{dp_{T}d\eta}\right)^{-1}\int
d\tau\,\tau\frac{d\sigma}{d\tau
dp_{T}d\eta}=\langle{\tau}_{\text{corr}}\rangle+\frac{\Omega_{f}}{p_{T}}\,,$
(7)

where $\langle{\tau}_{\text{corr}}\rangle$ is the first cumulant in the
absence of SUEs and $\Omega_{f}=\int
d\hat{\tau}_{\text{B}}\,\hat{\tau}_{\text{B}}f(\hat{\tau}_{\text{B}})$ is
independent of hard scale $p_{T}$. Therefore one can define SUE-independent
observable by taking the derivative of the $p_{T}$-weighted cumulant:

$\frac{d}{dp_{T}}p_{T}\langle\tau\rangle=\frac{d}{dp_{T}}p_{T}\langle{\tau}_{\text{corr}}\rangle\;.\,$
(8)

For a binned cross section $\sigma^{[i,j]}$ of the $i$-th bin in $\tau$ and
$j$-th bin in jet $p_{T}$, the $\tau$ cumulant of the $j$-th $p_{T}$ bin is
the following:

$\langle\tau\rangle^{[j]}=\frac{\sum_{i}\tau^{[i]}\sigma^{[i,j]}}{\sum_{i}\sigma^{[i,j]}}=\langle{\tau}_{\text{corr}}\rangle^{[j]}+\Omega_{f}\langle
p_{T}^{-1}\rangle^{[j]}\,,$ (9)

where $\tau^{[i]}$ is the central value of the $i$-th $\tau$ bin and $\langle
p_{T}^{-1}\rangle^{[j]}=(\int_{{\rm
bin}~{}j}p_{T}^{-1}d\sigma)/\sum_{i}\sigma^{[i,j]}$. We then define the
subtracted cumulant with the same mass dimension as $\tau$,

$\Delta_{\tau}^{jk}=\langle\tau\rangle^{[j]}-\langle\tau\rangle^{[k]}\frac{\langle
p_{T}^{-1}\rangle^{[j]}}{\langle
p_{T}^{-1}\rangle^{[k]}}=\langle{\tau}_{\text{corr}}\rangle^{[j]}-\langle{\tau}_{\text{corr}}\rangle^{[k]}\frac{\langle
p_{T}^{-1}\rangle^{[j]}}{\langle p_{T}^{-1}\rangle^{[k]}}\,.$ (10)

The model function dependence vanishes and we are left with purely signal-
correlated contributions. Note that we do not have to assume any specific form
for the model function, $f(\hat{\tau}_{\text{B}})$.

Figure 1: Jet invariant mass distributions simulated in Pythia at parton level
(red), hadron level with underlying events (black), pileup with $\langle
N_{\text{PU}}\rangle=7.5$ (green) and 50 (magenta).

##  III Removal of soft uncorrelated emissions

We discuss and demonstrate using Pythia simulations that subtracted cumulants
are indeed insensitive to MPI and PU contributions in proton-proton
collisions. We compare the results with the perturbative calculation performed
in Kang  _et al._ (2018b) at next-to-leading logarithmic and next-to-leading
order accuracy (NLL′+NLO) using SCET. (See also Dasgupta  _et al._ (2012);
Chien  _et al._ (2013); Jouttenus  _et al._ (2013); Liu  _et al._ (2015);
Hornig  _et al._ (2017) for previous jet mass calculations.) Within
theoretical uncertainties, the calculation agrees well with the simulations
even for a large number of PU events.

The NLL′+NLO result is obtained by matching the resummed and fixed order
results,

$\frac{d\sigma^{\text{NLL${}^{\prime}$+NLO}}}{d\tau
dp_{T}d\eta}=\frac{d\sigma^{\text{NLL${}^{\prime}$}}}{d\tau
dp_{T}d\eta}+\frac{d\sigma^{\text{NLO}}}{d\tau
dp_{T}d\eta}-\frac{d\sigma^{\text{NLO-sing.}}}{d\tau dp_{T}d\eta}\,,$ (11)

where $d\sigma^{\text{NLL${}^{\prime}$}}$, $d\sigma^{\text{NLO}}$, and
$d\sigma^{\text{NLO-sing.}}$ are the resummed, fixed-order, and fixed-order
singular cross sections, respectively. The NLO result111 In this paper, ‘NLO’
means the fixed-order calculation at $\mathcal{O}(\alpha_{s})$ accuracy. Since
$\mathcal{O}(\alpha_{s})$ is the first order at which the mass is nonzero,
$\mathcal{O}(\alpha_{s})$ is sometimes referred to as the ‘LO’ contribution
Salam (2010). is obtained using MadGraph 5 Alwall  _et al._ (2014). For the
simulation, we use Pythia 8 Sjostrand  _et al._ (2006, 2008) with the
ATLAS-A14-variation-2$+$ tune. We study the effect of MPI on subtracted
cumulants by switching on and off its contribution in Pythia. PU events are
simulated by soft QCD processes and added on top of signal events, and the PU
event number follows a Poisson distribution with the mean $\langle N_{\rm
PU}\rangle$. Here we present results for $\langle N_{\rm PU}\rangle=$ 7.5 and
50. Jets are reconstructed using the anti-$k_{t}$ algorithm Cacciari  _et al._
(2008) implemented in FastJet Cacciari  _et al._ (2012).

We first show in FIG. 1 the contributions of MPI and PU to the $m_{J}/p_{T}$
distributions. Both MPI and PU affect the peak position of the distribution
significantly, especially for lower $p_{T}$ jets with a large $\langle N_{\rm
PU}\rangle$. FIG. 2 then shows the results of subtracted jet mass cumulants.
The blue band is the theoretical uncertainty of the NLL′+NLO calculation
estimated by varying characteristic energy scales with a factor of two.
Remarkably, the simulation results from Pythia for different cases with and
without MPI or PU contributions all agree with the analytic calculation of the
signal distribution within theoretical uncertainty. This clearly demonstrates
that the proposed subtracted cumulants largely mitigate contributions from UE
and PU.

Figure 2: Subtracted jet mass cumulants $\Delta_{\tau}^{jk}$ from perturbative
calculation (blue band) and Pythia simulations without MPI (red), as well as
with MPI (black) and pileup with $\langle N_{\text{PU}}\rangle=7.5$ (green)
and 50 (magenta). The following transverse momentum bins are used:
$p_{T}\in\left\\{[126,158],[158,199],[199,258],[258,316],[316,398],[398,500]\right\\}$.

##  IV Modification for high luminosity collisions

For the situation with large background contamination from PU at HL-LHC or UE
in HIC, the jet $p_{T}$ is significantly altered by SUEs and the jet mass is
no longer an additive observable from jet constituents. Therefore, we instead
consider the observable, $\hat{\tau}$, defined in Eq.(5) which is explicitly
additive. Note that the jet direction $\vec{n}$ is assumed to be only mildly
affected by a large but approximately uniform background, or one can use a
recoil-free axis Bertolini  _et al._ (2014b). On the other hand, since SUE
contamination alters the value of jet $p_{T}$ significantly, in order to
compare subtracted cumulants between experiment and theory we need to correct
for the jet $p_{T}$ bin migration. This can be effectively achieved using the
area subtraction method Cacciari and Salam (2008); Cacciari  _et al._ (2010),
and we refer to the corrected $p_{T}$ as $\hat{p}_{T}$. The subtracted
cumulants for $\hat{\tau}$ are defined as follows,

$\Delta^{jk}_{\hat{\tau}}=\langle\hat{\tau}\rangle^{[j]}-\langle\hat{\tau}\rangle^{[k]}=\langle{\hat{\tau}}_{\text{corr}}\rangle^{[j]}-\langle{\hat{\tau}}_{\text{corr}}\rangle^{[k]}\,,$
(12)

where the indices $j,k$ label the $\hat{p}_{T}$ bins. Note that the subtracted
cumulant of $\hat{\tau}$ above is different from Eq. (10) in $p_{T}$-weighting
factor.

Figure 3: Subtracted cumulants $\Delta_{\hat{\tau}}$ obtained using Pythia
simulations at the parton level (red) and hadron level with large PU
contaminations of $\langle N_{\text{PU}}\rangle=200$ (green).

In FIG. 3 we demonstrate the robustness of $\Delta^{jk}_{\hat{\tau}}$ against
large SUEs by comparing the Pythia partonic result to the one including MPI
and PU with $\langle N_{\text{PU}}\rangle=200$, which is typical at HL-LHC and
can give an indication of how this observable removes SUEs in HIC. Note the
remarkable agreement between the two results. In practice, we use the
approximation $\hat{\tau}\simeq m_{J}^{2}/p_{T}$ which is in terms of the
well-studied invariant mass. For this reason and in contrast to the previous
plots, we choose to subtract the highest, instead of the lowest, $\hat{p}_{T}$
bin where this approximation is more accurate.

##  V Sensitivity to quark/gluon jet fraction

We discuss the sensitivity of subtracted cumulants to quark and gluon jet
fractions, $f_{q}=1-f_{g}$ and $f_{g}$, respectively. Assuming that the
fractions vary slowly within each $p_{T}$ bin $j$, the $\tau$ distribution is
a weighted sum of the corresponding quark and gluon distributions,

$\frac{d\sigma^{[j]}}{d\tau}=f_{g}^{[j]}\;\frac{d\sigma^{[j]}_{g}}{d\tau}+(1-f_{g}^{[j]})\;\frac{d\sigma^{[j]}_{q}}{d\tau},$
(13)

and similarly for $\langle\tau\rangle$,

$\langle\tau\rangle^{[j]}=f_{g}^{[j]}\;\langle\tau\rangle^{[j]}_{g}+(1-f_{g}^{[j]})\;\langle\tau\rangle^{[j]}_{q},$
(14)

Since $0<f_{g}<1$ and
$\langle\tau\rangle^{[j]}_{g}>\langle\tau\rangle^{[j]}_{q}$, we have
$\langle\tau\rangle^{[j]}_{q}<\langle\tau\rangle^{[j]}<\langle\tau\rangle^{[j]}_{g}$.
The subtracted cumulants are

$\displaystyle\Delta_{\tau}^{jk}$ $\displaystyle=$
$\displaystyle\Delta^{jk}_{\tau,q}+\Big{[}f_{g}^{[j]}(\langle\tau\rangle_{g}-\langle\tau\rangle_{q})^{[j]}$
(15) $\displaystyle\qquad\qquad-
f_{g}^{[k]}(\langle\tau\rangle_{g}-\langle\tau\rangle_{q})^{[k]}\frac{\langle
p_{T}^{-1}\rangle^{[j]}}{\langle p_{T}^{-1}\rangle^{[k]}}\Big{]}\;.$

We use Pythia to simulate pure quark and gluon jets, and we mix the samples
manually using the parametrized function $f_{g}(p_{T};a,b)$ (see Appendix for
details). Within the $p_{T}$ range of interest we examine two scenarios in
which the gluon jet fraction is larger (model 1) or smaller (model 2) than the
expected value in $pp$ collisions.

Figure 4: Top: Gluon jet fractions for the two models in this analysis (see
text and Appendix A). Bottom: Results of subtracted cumulants from analytic
calculation (blue band) and Pythia simulations with the gluon fractions from
models 1 and 2, as well as pure quark and gluon jets.

FIG. 4 shows the gluon jet fraction and subtracted cumulant as a function of
jet $p_{T}$ for model 1 and model 2, as well as theoretical predictions at
NLL′ accuracy for $pp$ collisions. We find that a change of quark-gluon jet
fraction can induce a significant change of the subtracted cumulant
distinguishable with the theoretical precision. Precise measurements of
subtracted cumulants of inclusive jets (gluon-enriched) and photon-tagged jets
(quark-enriched) will then give useful information about the different quark-
gluon jet fractions as well as subtracted cumulants of pure quark and gluon
jet samples. Since quark and gluon jets are initiated by partons with
different color charges, one expects that the two are quenched differently and
thus their fractions may change from proton-proton to HIC Spousta and Cole
(2016); Chien and Vitev (2016). The fraction change can induce modifications
of jet substructure which should be disentangled from the jet-by-jet
modification, for which subtracted cumulants can be very useful.

Figure 5: Comparison of the subtracted cumulant results from NLL′+NLO
calculation (blue band) with Pythia simulations with (black) and without (red)
MPI and hadronization, as well as subtracted cumulants calculated from the
experimental data measured by the ATLAS Aad  _et al._ (2012); The ATLAS
collaboration (2018). The top and bottom panels correspond to the collisional
center of mass energy at $\sqrt{s}=7$ TeV and$\sqrt{s}=5.02$ TeV,
respectively.

##  VI Comparison with experimental data

We compare our analytic calculation and simulation to subtracted cumulants
calculated from the experimental data measured by the ATLAS Collaboration at
the LHC with the collisional center of mass energies 7 TeV Aad  _et al._
(2012) and 5.02 TeV The ATLAS collaboration (2018).

FIG. 5 shows the results for the NLL′+NLO calculation (blue band) and Pythia
simulations with (black) or without (red) MPI effect and hadronization. The
data points are calculated from ATLAS measurements of jet mass distributions.
The error bars include only the statistical uncertainty and are calculated
from the variance of $\langle\tau\rangle^{[j]}$:
$\sqrt{\text{Var}[\tau]^{[j]}/N^{[j]}}$, where $\text{Var}[\tau]^{[j]}$ is the
variance of the $\tau$ distribution and $N^{[j]}$ is the total number of jets
estimated from the integrated luminosity: $\mathcal{L}_{\text{int.}}\times
d\sigma^{[j]}$. The statistical error in these experiments is small resulting
in the small error bars in the plots. Including the systematic uncertainty
requires experiment details and is beyond the scope of this work. For the 7
TeV case, only the differential distributions in jet mass are available rather
than $\tau=m_{J}^{2}/p_{T}^{2}$ thus we redefine $\Delta$ in terms of the
cumulant of $s=m_{J}^{2}$ as follows,

$\Delta^{jk}_{s}=\langle s\rangle^{[j]}-\langle s\rangle^{[k]}\frac{\langle
p_{T}\rangle^{[j]}}{\langle
p_{T}\rangle^{[k]}}=\langle{s}_{\text{corr}}\rangle^{[j]}-\langle{s}_{\text{corr}}\rangle^{[k]}\frac{\langle
p_{T}\rangle^{[j]}}{\langle p_{T}\rangle^{[k]}}\,.$ (16)

This redefinition is only necessary due to the large $p_{T}$ bin sizes in the
experiment. The average values $\langle p_{T}\rangle^{[j]}$ are not given in
Aad  _et al._ (2012) and we use the ones generated by Pythia including
hadronization and underlying event contributions since these quantities are
well described by simulations. For both the 7 and 5.02 TeV cases, we find that
the results of analytic calculations and simulations are in good agreement
with the experimental data.

A fixed-order expression for mean square jet mass $\langle m^{2}\rangle$ for
quark or gluon jets in small radius limit ($R\ll 1$) is given in Ref. Salam
(2010): $\langle\tau\rangle_{i}\simeq\langle m^{2}\rangle_{i}/p_{T}^{2}\simeq
C_{i}\frac{\alpha_{s}}{\pi}R^{2}\,,$ where $i=q,g$ and
$C_{q}=\tfrac{3}{8}C_{F}$ and
$C_{g}=\tfrac{7}{20}C_{A}+\tfrac{1}{20}n_{f}T_{R}$. By inserting it into Eq.
(15), the fixed-order subtracted cumulant is given by

$\displaystyle\Delta^{jk}_{\tau}\simeq\frac{\alpha_{s}}{\pi}R^{2}$
$\displaystyle\left[C_{q}\frac{p_{T}^{[j]}-p_{T}^{[k]}}{p_{T}^{[j]}}\right.$
(17)
$\displaystyle\quad\left.+(C_{g}-C_{q})\frac{p_{T}^{[j]}\,f_{g}^{[j]}-p_{T}^{[k]}\,f_{g}^{[k]}}{p_{T}^{[j]}}\right]\,,$

We apply Eq. (17) to the case $R=0.4$ in the lower panel of FIG. 5 and we
obtained the NLO curve (red dotted) by using the gluon fraction in FIG. 4. The
result underestimates the data and the calculation including resummation and
this suggests that resummation plays an important role. This can be understood
from the fact that the jet mass has a maximum allowed value
$m_{J}^{\text{max}}\simeq p_{T}^{J}\tan(R/2)\simeq 0.2\,p_{T}^{J}$ which is
not sufficiently far from the location of the peak in FIG. 1 where resummation
is needed. The cumulants of $e^{+}e^{-}$ thrust Abbate  _et al._ (2012) also
show a significant difference between the fixed-order result at
$\mathcal{O}(\alpha_{s})$ and the resummed result at NLL.

##  VII Conclusions

Figure 6: Cumulants (left panel) and subtracted cumulants (right panel)
corresponding to different quark and gluon jet fractions: pure quark (dashed),
pure gluon (solid) and two interpolations between pure quark or gluon across
jet $p_{T}$ (red and blue), as well as generic cases A and B (dotted) which
lie between the pure quark and gluon cases.

In this paper, we extend the work in Kang  _et al._ (2018a) to jet
substructure observables and introduce the new method of comparing theoretical
calculations of jet substructure observables to data using subtracted
cumulants. An advantage of our observable, compared to event by event
subtraction techniques, is that our approach can be applied to existing data
for which only the jet four-momenta are available. Furthermore it does not
require any tunes or Monte Carlo input since it does not depend on any
parameters or models. The method makes the comparison insensitive to soft
uncorrelated emissions such as multiparton interactions and pileup without
using background subtraction algorithms to correct each jet or having to model
uncorrelated effects. Our theoretical prediction at NLL′+NLO accuracy using
SCET shows an excellent agreement with the subtracted cumulants calculated
from two independent ATLAS jet mass measurements and those from Pythia
simulations. We also demonstrate that subtracted jet substructure cumulants
remove large background contaminations up to 200 pileup events. Its robustness
makes subtracted cumulants useful for jet studies at the high-luminosity LHC
and in the heavy-ion collisions, where the identification of signal jets is
challenged by a large background. We also show that subtracted cumulants are
sensitive to the change of quark-gluon jet fraction. This could allow for
precise determination of the fraction and its modification in heavy-ion
collisions, which will be useful for discriminating possible medium effects
and contributions. For example, in addition to the p-p jet mass measurements,
in Ref. The ATLAS collaboration (2018), exist also preliminary Pb-Pb data for
the same transverse momentum. It will be very interesting to see a
comprehensive analysis and a comparison of the subtracted jet mass cumulants
from the two measurements and what that reveals for the medium induced
effects. The mitigation of UE with flow modulation in HIC will be studied in
future work.

###### Acknowledgements.

The authors would like to thank Yongsun Kim, Yen-Jie Lee, Christopher Lee,
Duff Neill, Felix Ringer, Iain Stewart, Jesse Thaler and Ivan Vitev for useful
conversations during the completion of this work. Y.T.C. is supported by the
LHC Theory Initiative Postdoctoral Fellowship under the National Science
Foundation grant PHY-1419008. D.K. is supported by the National Natural
Science Foundation of China under Grant No. 11875112. K.L. is supported by the
National Science Foundation under Grants No. PHY-1316617 and No. PHY-1620628.
Y.M. is supported by the DOE Office of Science under Contract No. DE-
AC52-06NA25396, the Early Career Program (Christopher Lee, P.I.) and the LDRD
Program at LANL.

##  Appendix A Dependence of subtracted cumulants on quark and gluon jet
fraction

We give the details of the parametrization used in FIG. 4 for the two models
with different quark and gluon jet fractions. The gluon fraction, $f_{g}$, has
the following power-law modification form:

$f_{g}(p_{T};a,b)=f^{\text{NLL}}_{g}(p_{T})\Big{(}\frac{p_{T}}{a}\Big{)}^{b}\;,$
(18)

where $a$ and $b$ can be varied. The function $f_{g}^{\text{NLL}}$ is the
analytic result extracted from the NLL′ calculation of the inclusive cross
section Kang  _et al._ (2016) and tt is the gluon fraction at the hard scale
$\mu_{H}=p_{T}$. We check using Pythia simulations and find that the
distribution formed by weighing the pure quark and gluon distributions from
$gg\to q\bar{q}$ and $gg\to gg$ processes with the fraction
$f_{i}^{\text{NLL}}$ agrees well with the full simulation. For models 1 and 2
in FIG. 4 we choose the following parameters:

model 1: $\displaystyle a=120\;\text{GeV},\;b=+1/3$ model 2: $\displaystyle
a=120\;\text{GeV},\;b=-1/2.$ (19)

Also, we demonstrate that different quark and gluon jet fractions can give the
same subtracted cumulants, as shown in FIG. 6. For simplicity, we assume that
the cumulants $\langle\tau\rangle$ of pure quark and pure gluon jets depend
linearly on jet $p_{T}$, and the subtracted cumulants $\Delta_{\tau}$ are
defined in Eq. (15). The left panel shows the cumulants corresponding to
different quark and gluon jet fractions: pure quark, pure gluon and two
interpolations between pure quark or gluon across jet $p_{T}$, as well as
cases A and B that sit between the pure quark and gluon cases. The right panel
shows the subtracted cumulants. We can clearly see that cases A and B give the
same subtracted cumulant, and that the cases of pure quark and gluon jets do
not represent extreme values of subtracted cumulants.

## References

  * Sjostrand  _et al._ (2008) T. Sjostrand, S. Mrenna, and P. Z. Skands, Comput. Phys. Commun. 178, 852 (2008), arXiv:0710.3820 [hep-ph] .
  * Bellm  _et al._ (2016) J. Bellm _et al._ , Eur. Phys. J. C76, 196 (2016), arXiv:1512.01178 [hep-ph] .
  * Larkoski  _et al._ (2017) A. J. Larkoski, I. Moult, and B. Nachman, (2017), arXiv:1709.04464 [hep-ph] .
  * Andrews  _et al._ (2018) H. A. Andrews _et al._ , “Novel tools and observables for jet physics in heavy-ion collisions,” (2018), arXiv:1808.03689 [hep-ph] .
  * Voloshin and Zhang (1996) S. Voloshin and Y. Zhang, Z. Phys. C70, 665 (1996), arXiv:hep-ph/9407282 [hep-ph] .
  * Poskanzer and Voloshin (1998) A. M. Poskanzer and S. A. Voloshin, Phys. Rev. C58, 1671 (1998), arXiv:nucl-ex/9805001 [nucl-ex] .
  * (7) J. D. Bjorken, FERMILAB-PUB-82-059-THY, FERMILAB-PUB-82-059-T .
  * Connors  _et al._ (2018) M. Connors, C. Nattrass, R. Reed, and S. Salur, Rev. Mod. Phys. 90, 025005 (2018), arXiv:1705.01974 [nucl-ex] .
  * Cacciari and Salam (2008) M. Cacciari and G. P. Salam, Phys. Lett. B659, 119 (2008), arXiv:0707.1378 [hep-ph] .
  * Feige  _et al._ (2012) I. Feige, M. D. Schwartz, I. W. Stewart, and J. Thaler, Phys. Rev. Lett. 109, 092001 (2012), arXiv:1204.3898 [hep-ph] .
  * Soyez  _et al._ (2013) G. Soyez, G. P. Salam, J. H. Kim, S. Dutta, and M. Cacciari, Phys. Rev. Lett. 110, 162001 (2013), arXiv:1211.2811 [hep-ph] .
  * Larkoski and Thaler (2014) A. J. Larkoski and J. Thaler, Phys. Rev. D90, 034010 (2014), arXiv:1406.7011 [hep-ph] .
  * Krohn  _et al._ (2014) D. Krohn, M. D. Schwartz, M. Low, and L.-T. Wang, Phys. Rev. D90, 065020 (2014), arXiv:1309.4777 [hep-ph] .
  * Cacciari  _et al._ (2015) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C75, 59 (2015), arXiv:1407.0408 [hep-ph] .
  * Bertolini  _et al._ (2014a) D. Bertolini, P. Harris, M. Low, and N. Tran, JHEP 10, 059 (2014a), arXiv:1407.6013 [hep-ph] .
  * Berta  _et al._ (2014) P. Berta, M. Spousta, D. W. Miller, and R. Leitner, JHEP 06, 092 (2014), arXiv:1403.3108 [hep-ex] .
  * Larkoski  _et al._ (2014) A. J. Larkoski, S. Marzani, G. Soyez, and J. Thaler, JHEP 05, 146 (2014), arXiv:1402.2657 [hep-ph] .
  * Komiske  _et al._ (2017) P. T. Komiske, E. M. Metodiev, B. Nachman, and M. D. Schwartz, JHEP 12, 051 (2017), arXiv:1707.08600 [hep-ph] .
  * Soyez (2019) G. Soyez, Phys. Rept. 803, 1 (2019), arXiv:1801.09721 [hep-ph] .
  * Kang  _et al._ (2018a) D. Kang, Y. Makris, and T. Mehen, JHEP 09, 055 (2018a), arXiv:1803.04413 [hep-ph] .
  * Bauer  _et al._ (2000) C. W. Bauer, S. Fleming, and M. E. Luke, Phys.Rev. D63, 014006 (2000), arXiv:hep-ph/0005275 [hep-ph] .
  * Bauer  _et al._ (2001) C. W. Bauer, S. Fleming, D. Pirjol, and I. W. Stewart, Phys.Rev. D63, 114020 (2001), arXiv:hep-ph/0011336 [hep-ph] .
  * Bauer and Stewart (2001) C. W. Bauer and I. W. Stewart, Phys.Lett. B516, 134 (2001), arXiv:hep-ph/0107001 [hep-ph] .
  * Bauer  _et al._ (2002a) C. W. Bauer, D. Pirjol, and I. W. Stewart, Phys.Rev. D65, 054022 (2002a), arXiv:hep-ph/0109045 [hep-ph] .
  * Bauer  _et al._ (2002b) C. W. Bauer, S. Fleming, D. Pirjol, I. Z. Rothstein, and I. W. Stewart, Phys.Rev. D66, 014017 (2002b), arXiv:hep-ph/0202088 [hep-ph] .
  * Beneke  _et al._ (2002) M. Beneke, A. P. Chapovsky, M. Diehl, and T. Feldmann, Nucl. Phys. B643, 431 (2002), arXiv:hep-ph/0206152 [hep-ph] .
  * Ellis  _et al._ (2010) S. D. Ellis, C. K. Vermilion, J. R. Walsh, A. Hornig, and C. Lee, JHEP 11, 101 (2010), arXiv:1001.0014 [hep-ph] .
  * Bertolini  _et al._ (2014b) D. Bertolini, T. Chan, and J. Thaler, JHEP 04, 013 (2014b), arXiv:1310.7584 [hep-ph] .
  * Banfi  _et al._ (2005) A. Banfi, G. P. Salam, and G. Zanderighi, JHEP 03, 073 (2005), arXiv:hep-ph/0407286 [hep-ph] .
  * Stewart  _et al._ (2015) I. W. Stewart, F. J. Tackmann, and W. J. Waalewijn, Phys. Rev. Lett. 114, 092001 (2015), arXiv:1405.6722 [hep-ph] .
  * Hoang  _et al._ (2017) A. H. Hoang, S. Mantry, A. Pathak, and I. W. Stewart, (2017), arXiv:1708.02586 [hep-ph] .
  * Kang  _et al._ (2018b) Z.-B. Kang, K. Lee, X. Liu, and F. Ringer, JHEP 10, 137 (2018b), arXiv:1803.03645 [hep-ph] .
  * Lee and Sterman (2007) C. Lee and G. F. Sterman, Phys. Rev. D75, 014022 (2007), arXiv:hep-ph/0611061 [hep-ph] .
  * Dasgupta  _et al._ (2012) M. Dasgupta, K. Khelifa-Kerfa, S. Marzani, and M. Spannowsky, JHEP 10, 126 (2012), arXiv:1207.1640 [hep-ph] .
  * Chien  _et al._ (2013) Y.-T. Chien, R. Kelley, M. D. Schwartz, and H. X. Zhu, Phys. Rev. D87, 014010 (2013), arXiv:1208.0010 [hep-ph] .
  * Jouttenus  _et al._ (2013) T. T. Jouttenus, I. W. Stewart, F. J. Tackmann, and W. J. Waalewijn, Phys. Rev. D88, 054031 (2013), arXiv:1302.0846 [hep-ph] .
  * Liu  _et al._ (2015) Z. L. Liu, C. S. Li, J. Wang, and Y. Wang, JHEP 04, 005 (2015), arXiv:1412.1337 [hep-ph] .
  * Hornig  _et al._ (2017) A. Hornig, D. Kang, Y. Makris, and T. Mehen, JHEP 12, 043 (2017), arXiv:1708.08467 [hep-ph] .
  * Salam (2010) G. P. Salam,  _2008 CTEQ-MCnet Summer School on QCD Phenomenology and Monte Carlo Event Generators (MCnet 08) (CTEQ 08) Debrecen, Hungary, August 8-16, 2008_ , Eur. Phys. J. C67, 637 (2010), arXiv:0906.1833 [hep-ph] .
  * Alwall  _et al._ (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), arXiv:1405.0301 [hep-ph] .
  * Sjostrand  _et al._ (2006) T. Sjostrand, S. Mrenna, and P. Z. Skands, JHEP 05, 026 (2006), arXiv:hep-ph/0603175 [hep-ph] .
  * Cacciari  _et al._ (2008) M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008), arXiv:0802.1189 [hep-ph] .
  * Cacciari  _et al._ (2012) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C72, 1896 (2012), arXiv:1111.6097 [hep-ph] .
  * Cacciari  _et al._ (2010) M. Cacciari, G. P. Salam, and S. Sapeta, JHEP 04, 065 (2010), arXiv:0912.4926 [hep-ph] .
  * Spousta and Cole (2016) M. Spousta and B. Cole, Eur. Phys. J. C76, 50 (2016), arXiv:1504.05169 [hep-ph] .
  * Chien and Vitev (2016) Y.-T. Chien and I. Vitev, JHEP 05, 023 (2016), arXiv:1509.07257 [hep-ph] .
  * Aad  _et al._ (2012) G. Aad _et al._ (ATLAS), JHEP 05, 128 (2012), arXiv:1203.4606 [hep-ex] .
  * The ATLAS collaboration (2018) The ATLAS collaboration (ATLAS), ATLAS-CONF-2018-014 (2018).
  * Abbate  _et al._ (2012) R. Abbate, M. Fickinger, A. H. Hoang, V. Mateu, and I. W. Stewart, Phys. Rev. D86, 094002 (2012), arXiv:1204.5746 [hep-ph] .
  * Kang  _et al._ (2016) Z.-B. Kang, F. Ringer, and I. Vitev, JHEP 10, 125 (2016), arXiv:1606.06732 [hep-ph] .


# Analysis of face detection, face landmarking, and face recognition
performance with masked face images

Ožbej Golob

###### Abstract

Abstract Face recognition has become an essential task in our lives. However,
the current COVID-19 pandemic has led to the widespread use of face masks. The
effect of wearing face masks is currently an understudied issue. The aim of
this paper is to analyze face detection, face landmarking, and face
recognition performance with masked face images. HOG and CNN face detectors
are used for face detection in combination with 5-point and 68-point face
landmark predictors and VGG16 face recognition model is used for face
recognition on masked and unmasked images. We found that the performance of
face detection, face landmarking, and face recognition is negatively impacted
by face masks.

##  1 Introduction

Face recognition has become an essential task in our daily life. The wide
availability of powerful and low-cost computing systems has popularized face
recognition for a variety of applications, including biometric authentication,
surveillance, human-computer interaction, and multimedia management [1]. Many
highly accurate face recognition models were developed. Some of those models
are FaceNet [2], DeepFace [3], and DeepID3 [4]. However, face occlusion can
negatively impact the accuracy of face recognition models. This problem has
already been addressed in the scope of face detection by Opitz et al. [5].
Some research has also been carried out by Song et al. [6] to develop
occlusion invariant face recognition solutions. However, this research focuses
on typical in-the-wild occlusion scenarios such as wearing sunglasses.

The current COVID-19 pandemic has led to the widespread use of face masks to
prevent the spread of the disease. For this reason, it is essential to
research the specific effect of wearing face masks on the performance of face
recognition models.

Face landmarking represents one of the first steps in a standard face
recognition pipeline. During landmarking, the location of certain facial
features (eye corners, the tip of the nose, etc.) is identified in the face
images and used to align faces prior to feature extraction. If the landmarks
are not detected properly, the alignment procedure will fail and result in
poorly aligned or partial facial areas that will ultimately affect face
recognition performance. In this paper, we explore how face detection, face
landmarking, and face recognition work with masked faces.

##  2 Related work

Damer et al. [7] study the effect of masked faces on the behavior of three
top-performing face recognition systems. Two of these algorithms are academic
approaches, namely the ArcFace [8] and SphereFace [9]. The third algorithm is
a commercial off-the-shelf (COTS) from the vendor Neurotechnology [10].
Authors evaluate the face verification performance without masks and compare
results to the face verification performance with masks. The verification
performance of the ArcFace and SphereFace is negatively affected when the
faces are masked, while the COTS is not significantly affected by masked
faces.

Wang et al. [11] propose three publicly available masked face datasets: Masked
Face Detection Dataset (MFDD), Real-world Masked Face Recognition Dataset
(RMFRD), and Simulated Masked Face Recognition Dataset (SMFRD). Authors
propose a face-eye-based multi-granularity recognition model where they apply
different attention weights to key features visible in masked faces (face
contour, ocular and periocular details, forehead, etc.). The model improves
the recognition accuracy of masked faces from the initial 50% to 95%.

##  3 Methods

###  3.1 Face detection

We implemented two face detectors: (i) Histogram of Oriented Gradients (HOG)
feature combined with a linear classifier, an image pyramid, and sliding
window detection scheme, and (ii) Max-Margin Convolutional Neural Network
(CNN) face detector. The HOG detector is accurate and computationally
efficient while the CNN detector is accurate and robust, capable of detecting
faces from varying viewing angles, lighting conditions, and occlusion. Both
face detectors are implemented in Dlib [12].

###  3.2 Face landmarking

We implemented two landmark predictors: (i) 68-point landmark predictor (see
Figure 1), and (ii) 5-point landmark predictor (see Figure 1, points 34, 37,
40, 43, and 46). Both landmark predictors are Dlib’s implementation of [13].

Figure 1: 68-point landmarks.

Face landmarking performance was evaluated with the normalized root mean
square error (NRMSE). The normalization is done with respect to the inter-
ocular distance (IOD), which is the distance between the two eye centers.
Normalizing landmark localization errors with the IOD makes the performance
evaluation independent of the face size or the camera zoom factor.

The normalized distance $\delta$ is computed as the Euclidean distance
$d(.,.)$ between the ground-truth landmark coordinates $(x,y)$ and the
predicted landmark coordinates $(\tilde{x},\tilde{y})$, normalized by the IOD.
Equation 1 shows the formula for the normalized distance of each landmark,
where subscript $k$ indicates one of the landmarks.

$\delta_{k}=\frac{d\\{(x_{k},y_{k}),(\tilde{x_{k}},\tilde{y_{k}})\\}}{IOD}$
(1)

NRMSE of each image ($NRMSE_{local}$) is calculated by the formula shown in
Equation 2, where $n$ is the number of landmarks.

$NRMSE_{local}=\sqrt{\frac{\displaystyle\sum_{k=1}^{n}\leavevmode\nobreak\
\delta_{k}^{2}}{n}}$ (2)

###  3.3 Face recognition

VGG16 architecture which is pre-trained on a huge ImageNet database with more
than 1 million images belonging to 1000 different categories is used to train
the input face images. The Softmax layer is removed in order to get image
feature vectors. The extracted features are fed as input to the Fully
Connected Layer and Softmax activation. Figure 2 shows the VGG16 architecture.

Figure 2: VGG16 architecture.

##  4 Experiments

###  4.1 Datasets

To evaluate face landmarking performance, we identified a couple of datasets
with annotated facial landmarks. These datasets are Labeled Face Parts in the
Wild (LFPW) [14], Annotated Faces in the Wild (AFW) [15], HELEN [16], and IBUG
[17]. All used datasets are collected in the wild. The LFPW dataset consists
of 1,432 faces, where 1,132 images are a part of the train set and 300 images
are a part of the test set. The HELEN dataset is composed of 2330 face images,
where 2000 images are a part of the train set and 330 images are a part of the
test set. The AFW dataset contains 205 images with 468 faces. The IBUG dataset
includes 135 images with extreme poses and expressions. Evaluation on LFPW and
HELEN was performed on the test set only while evaluation on AFW and IBUG was
performed on the whole dataset. All used datasets were re-annotated with 68
landmarks as a part of 300 Faces In-the-Wild Challenge (300-W) [18] using the
mark-up of Figure 1.

To evaluate face landmarking performance on masked faces, we generated masked
versions of LFPW, AFW, HELEN, and IBUG datasets with the help of the
MaskTheFace tool [19].

To evaluate face recognition performace, we used a subset of CASIA dataset.
960 train images and 417 test images, belonging to 10 identities, were used
for evaluation. To evaluate face recognition performance on masked faces, we
generated masked images with the help of the MaskTheFace tool.

###  4.2 Experimentation details

Face detection results were measured as a percentage of annotated faces that
the detector was able to detect. Face landmarking results were measured as
NRMSE. Figure 3 shows positions of ground truth and predicted landmarks. The
distance between ground truth and predicted landmarks (see Equation 1) is
calculated for each landmark and NRMSE (see Equation 2) is calculated for each
image. NRMSE is then averaged for the whole dataset.

Figure 3: Face landmarks (red dots represent ground truth landmarks, green
dots represent predicted landmarks).

We ignored images where zero faces were detected and images where the
annotated face was not detected. Images with errors were ignored because they
increased the NRMSE significantly.

VGG16 model for face recognition was initialized with weights learned from a
pre-trained model. VGG16 model was additionaly trained on unmasked images and
evaluated on unmasked and masked images. Model was then re-trained and
evaluated on masked images.

##  5 Results

###  5.1 Face detection

Table 1 shows the HOG face detection accuracy on original and masked datasets.

Table 1: HOG face detection accuracy. Dataset | Original | Masked
---|---|---
HELEN | 0.967 | 0.735
LFPW | 0.987 | 0.801
AFW | 0.896 | 0.565
IBUG | 0.711 | 0.487
Average | 0.890 | 0.647

Table 2 shows the CNN face detection accuracy on original and masked datasets.

Table 2: CNN face detection accuracy. Dataset | Original | Masked
---|---|---
HELEN | 1.000 | 0.892
LFPW | 1.000 | 0.878
AFW | 1.000 | 0.708
IBUG | 0.941 | 0.635
Average | 0.985 | 0.778

###  5.2 Face landmarking

Table 3 shows the Dlib 68-point face landmarking NRMSE with HOG face detector
on original and masked datasets.

Table 3: Dlib 68-point face landmarking NRMSE with HOG face detector. Dataset | Original | Masked
---|---|---
HELEN | 0.044 | 0.153
LFPW | 0.054 | 0.159
AFW | 0.063 | 0.169
IBUG | 0.102 | 0.204
Average | 0.066 | 0.171

Table 4 shows the Dlib 68-point face landmarking NRMSE with CNN face detector
on original and masked datasets.

Table 4: Dlib 68-point face landmarking NRMSE with CNN face detector. Dataset | Original | Masked
---|---|---
HELEN | 0.065 | 0.187
LFPW | 0.073 | 0.189
AFW | 0.114 | 0.214
IBUG | 0.209 | 0.266
Average | 0.115 | 0.214

Table 5 shows the Dlib 5-point face landmarking
NRMSE with HOG face detector on original and masked datasets.

Table 5: Dlib 5-point face landmarking NRMSE with HOG face detector. Dataset | Original | Masked
---|---|---
HELEN | 0.034 | 0.105
LFPW | 0.050 | 0.113
AFW | 0.062 | 0.123
IBUG | 0.092 | 0.172
Average | 0.060 | 0.128

Table 6 shows the Dlib 5-point face landmarking
NRMSE with CNN face detector on original and masked datasets.

Table 6: Dlib 5-point face landmarking NRMSE with CNN face detector. Dataset | Original | Masked
---|---|---
HELEN | 0.037 | 0.114
LFPW | 0.052 | 0.115
AFW | 0.070 | 0.136
IBUG | 0.107 | 0.152
Average | 0.067 | 0.129

###  5.3 Face recognition

VGG16 model trained on unmasked images achieved test accuracy of 0.966 on
unmasked images and 0.867 on masked images. After the model was re-trained on
masked images, the model achieved 0.952 accuracy on masked images.

##  6 Conclussion

In the scope of this paper, we analyzed face detection, face landmarking, and
face recognition performance with masked face images. We observed a negative
impact of face masks on face detection. Face detection accuracy dropped by
0.243 on average for the HOG face detector and by 0.207 on average for the CNN
face detector for masked images. Face landmarking performance was also
negatively impacted by face masks. NRMSE of the 68-point face landmarking
predictor increased nearly 3-times with the HOG face detector and nearly
2-times with the CNN face detector for masked images. NRMSE of the 5-point
face landmarking predictor increased by approximately 2-times for HOG and CNN
face detector for masked images. Face recognition performance was also
negatively impacted by face masks, lowering the accuracy by 0.099 on masked
images. We were able to improve the accuracy to 0.952 by additionally training
the model on masked images.

We conclude that face masks have a negative impact on face detection, face
landmarking, and face recognition. This implies that facial areas beneath face
masks (mouth, nose, etc.) hold significant information for face detection,
face landmarking, and face recognition. We found that certain models can be
improved by additionally training the model on masked images.

## References

  * [1] Anil K Jain and Stan Z Li.  Handbook of face recognition, volume 1.  Springer, 2011.
  * [2] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [3] Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [4] Yi Sun, Ding Liang, Xiaogang Wang, and Xiaoou Tang.  Deepid3: Face recognition with very deep neural networks.  arXiv preprint arXiv:1502.00873, 2015.
  * [5] Michael Opitz, Georg Waltner, Georg Poier, Horst Possegger, and Horst Bischof.  Grid loss: Detecting occluded faces.  In European conference on computer vision, pages 386–402. Springer, 2016.
  * [6] Lingxue Song, Dihong Gong, Zhifeng Li, Changsong Liu, and Wei Liu.  Occlusion robust face recognition based on mask learning with pairwise differential siamese network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [7] Naser Damer, Jonas Henry Grebe, Cong Chen, Fadi Boutros, Florian Kirchbuchner, and Arjan Kuijper.  The effect of wearing a mask on face recognition performance: an exploratory study.  In 2020 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1–6. IEEE, 2020.
  * [8] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [9] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 212–220, 2017.
  * [10] Neurotechnology MegaMatcher 11.2 SDK.  https://www.neurotechnology.com/.  Accessed: 2022-02-01.
  * [11] Zhongyuan Wang, Guangcheng Wang, Baojin Huang, Zhangyang Xiong, Qi Hong, Hao Wu, Peng Yi, Kui Jiang, Nanxi Wang, Yingjiao Pei, et al.  Masked face recognition dataset and application.  arXiv preprint arXiv:2003.09093, 2020.
  * [12] Davis E. King.  Dlib-ml: A machine learning toolkit.  Journal of Machine Learning Research, 10:1755–1758, 2009.
  * [13] Vahid Kazemi and Josephine Sullivan.  One millisecond face alignment with an ensemble of regression trees.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014.
  * [14] Peter N Belhumeur, David W Jacobs, David J Kriegman, and Neeraj Kumar.  Localizing parts of faces using a consensus of exemplars.  IEEE transactions on pattern analysis and machine intelligence, 35(12):2930–2940, 2013.
  * [15] Xiangxin Zhu and Deva Ramanan.  Face detection, pose estimation, and landmark localization in the wild.  In 2012 IEEE conference on computer vision and pattern recognition, pages 2879–2886. IEEE, 2012.
  * [16] Vuong Le, Jonathan Brandt, Zhe Lin, Lubomir Bourdev, and Thomas S Huang.  Interactive facial feature localization.  In European conference on computer vision, pages 679–692. Springer, 2012.
  * [17] Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic.  300 faces in-the-wild challenge: The first facial landmark localization challenge.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 397–403, 2013.
  * [18] 300 Faces In-the-Wild Challenge (300-W), ICCV 2013.  https://ibug.doc.ic.ac.uk/resources/300-W/.  Accessed: 2022-02-01.
  * [19] Aqeel Anwar and Arijit Raychowdhury.  Masked face recognition for secure authentication.  arXiv preprint arXiv:2008.11104, 2020.