###### Abstract

Image-based jet analysis is built upon the jet image representation of jets
that enables a direct connection between high energy physics and the fields of
computer vision and deep learning. Through this connection, a wide array of
new jet analysis techniques have emerged. In this text, we survey jet image
based classification models, built primarily on the use of convolutional
neural networks, examine the methods to understand what these models have
learned and what is their sensitivity to uncertainties, and review the recent
successes in moving these models from phenomenological studies to real world
application on experiments at the LHC. Beyond jet classification, several
other applications of jet image based techniques, including energy estimation,
pileup noise reduction, data generation, and anomaly detection, are discussed.

##  Chapter 0 Image-Based Jet Analysis

###### Contents

  1. 0 Image-Based Jet Analysis
    1. 1 Introduction
      1. 1 Notations and Definitions
    2. 2 Jets and Jet Physics Challenges
    3. 3 Jet Images and Preprocessing
      1. 1 Preprocessing
    4. 4 Computer Vision and Convolutional Neural Networks
    5. 5 Jet tagging
      1. 1 Jet Tagging on Single Channel Jet Images
      2. 2 Multi-Channel Jet Tagging with CNNs
      3. 3 Sensitivity to Theory Uncertainties
      4. 4 Jet images in LHC Experiments
    6. 6 Understanding jet Image based tagging
      1. 1 Probing CNNs
    7. 7 Other Applications of Jet images
      1. 1 Jet Energy Regression and Pileup removal
      2. 2 Generative Models with Jet Images
      3. 3 Anomaly Detection
    8. 8 Conclusion

###  1 Introduction

The jet image [1] approach to jet tagging is built upon the rapidly developing
field of Computer Vision (CV) in Machine Learning (ML). Jets [2, 3] are
collimated streams of particles produced by the fragmentation and hadronizaton
of high energy quarks and gluons. The particles are subsequently measured by
particle detectors and clustered with jet clustering algorithms to define the
jets. Jet images view the energy depositions of the stream of particles
comprising a jet within a fixed geometric region of a detector as an image,
thereby connecting particle detector measurements with an image representation
and allowing the application of image analysis techniques from CV. In this
way, models built upon advancements in deep convolutional neural networks
(CNN) can be trained for jet classification, energy determination through
regression, and the reduction of noise e.g. from simultaneous background
interactions at a high intensity hadron collider such as the Large Hadron
Collider (LHC). Throughout this text, the focus will be on the use of jet
image techniques studied within the context of hadron colliders like the LHC
[4].

Jet images form a representation of jets highly connected with the detector;
one can look at segmented detectors as imaging devices and interpret the
measurements as an image. In contrast, other representations of jets exist
that are built more closely from the physics of jet formation, such as viewing
jets as sequences [5, 6] or trees [7] formed through a sequential emission
process, or viewing jets as sets, graphs, or point clouds [8, 9] with the
geometric relationship between constituents of the jet encoded in the
adjacency matrix and node properties. There are overlaps in these approaches,
for instance a graph can be defined over detector energy measurements, but
these approaches will not be discussed in detail in this chapter. The
utilization of an image-based approach comes with the major advantage that CV
is a highly developed field of ML with some of the most advanced models
available for application to jet analysis with jet images. From the
experimental viewpoint, the detector measurements are fundamental to any
subsequent analysis, and the detailed knowledge of the detector and its
systematic uncertainties can be highly advantageous for analysis of LHC data.

Among the earliest use of jet images was for the classification of the parent
particle inducing the jet [1], and relied on utilizing linear discriminants
trained on image representations of jets for this task. While the remainder of
this text will focus on deep learning approaches to jet images, even this
early work saw interesting discrimination power for this task. By utilizing
the detector measurements directly, rather than relying on jet features
developed using physics domain knowledge, additional discrimination power
could be extracted. Deep learning approaches surpass such linear methods, but
build on this notion of learning discriminating information from detector
observables rather than engineered features.

Figure 1: An example jet image of a Lorentz boosted top quark jet after
preprocessing has been applied [10].

While designed to take advantage of advances in computer vision, jet images
have notable differences with respect to typical natural images in CV. Jet
images are sparse, with most pixels in the image having zero content. This is
markedly different from natural images that tend to have all pixels containing
content. Moreover, jet images tend to have multiple localized regions of high
density in addition to diffusely located pixels throughout the image, as
opposed to the smooth structures typically found in natural images. An example
top quark jet image illustrating these features can be seen in Figure 1. These
differences can lead to notable challenges, for instance the number of
parameters used in jet image models (and consequently the training time) tend
to be large to account for the size of the image, even though most pixels
carry no information. Some techniques exist for sparse-image computer vision
approaches [11], but have not been explored in depth within the jet image
community.

This text will first discuss jets and typical jet physics in Section 2. The
formation of jet images and the jet image preprocessing steps before
classification are discussed in Section 3. A brief introduction to Computer
Vision is found in Section 4. The application of jet images in various jet
classification problems is then discussed in Section 5, followed by a
discussion on the interpretation of information learned by jet image based
classifiers in Section 6. Some recent applications of jet images beyond
classification are discussed in Section 7. A brief note on the notation used
throughout the text follows below (Section 1).

It should be noted that the majority of the studies presented in this text
relate to phenomenological work using simplified setting that often do not
include a realistic modeling of a detector’s impact on observables. These will
frequently be denoted as phenomenological studies, in contrast to the studies
using realistic detector simulations or using real experiment data that are
discussed mainly in Section 4.

####  1 Notations and Definitions

As we will focus on studies of jet images in the LHC setting, we will
primarily utilize the hadron collider coordinate system notation. The beam-
line defines the $z$-axis, $\phi$ indicates the azimuthal angle,
$\eta=-\log\tan\frac{\theta}{2}$ is the pseudo-rapidity which is a
transformation of the polar angle $\theta$. The rapidity is defined as
$y=\frac{1}{2}\log\Big{[}\frac{E+p_{z}}{E-p_{z}}\Big{]}$ is frequently used
for the polar measurement of massive particles, such as jets, as differences
in rapidity are invariant with respect to Lorentz boosts along the beam
direction. The angular separation of particles is defined as $\Delta
R(p_{1},p_{2})=\sqrt{(y_{1}-y_{2})^{2}+(\phi_{1}-\phi_{2})^{2}}$. The
transverse momentum $p_{T}=\sqrt{p_{x}^{2}+p_{y}^{2}}$, is frequently used as
it is invariant with respect to Lorentz boosts along the beam direction. The
transverse energy is defined as $E_{T}=E\sin(\theta)$.

###  2 Jets and Jet Physics Challenges

Jets are collimated streams of particles produced by the fragmentation and
hadronizaton of high energy quarks and gluons. Jet clustering algorithms are
used to combine particles into clusters that define the jets (See references
[2, 3] for recent reviews). At the LHC, jet algorithms typically rely on
sequential reclustering algorithms which, given a definition of distance,
iteratively combine the closest two constituents (either particles or
previously combined sets of particles denoted proto-jets) until a stopping
condition is met. Different distance metrics define different jet algorithms
and perhaps the most commonly used algorithm at the LHC is the anti-$k_{T}$
algrithm [12] in which the distance between particle i and particle j is
defined as $d_{ij}=\min\\{k_{T,i}^{-2}\
,k_{T,j}^{-2}\\}\Delta_{ij}^{2}/R^{2}$. Here,
$\Delta_{ij}^{2}=(y_{i}-y_{j})^{2}+(\phi_{i}-\phi_{j})^{2}$ and $y$, $\phi$,
and $k_{T}$ are the particle rapidity, azimuth, and transverse momentum,
respectively. The parameter $R$ of the jet algorithm has the effect of
defining the spatial span, or approximate “radius” (though the jet is not
necessarily circular), of the jet. Jets and jet algorithms are required to be
IRC safe, i.e. insensitive to additional infrared radiation or collinear
splittings of particles, in order for the jet properties to be calculable in
fixed-order perturbation theory. This allows comparison between jets clustered
on partons from the hard scattering process, referred to as parton jets, on
final state particles after showering and hadronization simulation, referred
to as particle jets, and on reconstructed particles in detectors, referred to
as reconstructed jets.

Most of the work presented in this text are phenomenological studies outside
the context of any individual experiment. These studies primarily utilize
particle level simulation after fragmentation and hadronization and thus study
particle jets defined after clustering the final state particles. These
studies typically do not use a simulation of a detector and its impact on
particle kinematic measurements. Studies of jets and jet images after real
detector simulation or in real detector data are discussed in Section 4. In
the detector setting, various inputs to jet algorithms can be used to define
jets: (i) towers refer to a fixed spatial extent in $\eta$ and $\phi$ in which
all energy within the longitudinal depth of the calorimeter is summed, (2)
topological clusters [13] are used to cluster together energy depositions in
nearby calorimeter cells, (3) tracks, or charged particle trajectories,
measured using tracking detectors. The Particle Flow (PF) algorithm [14] is
used by the CMS collaboration to match charged particles with energy in the
calorimeter in order to utilize both measurements to define PF candidates that
can be used as inputs to jet algorithms.

The $R$-parameter of the jet is used to define the spatial extent to which
particles are clustered into the jet. When studying quark and gluon jets,
$R=0.4$ is frequently used. When studying the decay of Lorentz boosted heavy
particles, in which multiple partons may be spatially collimated, large-$R$
jets are often used which have a larger $R=1.0$ or $R=1.2$. Subjets, defined
by running a jet clustering algorithm with smaller radius on the constituents
of a jet, are frequently used to study the internal properties of a jet. More
broadly, jet substructure refers to the study of the internal structure of
jets and the development of theoretically motivated jet features which are
useful for discrimination and inference tasks (See references [15, 16] for
recent reviews).

One particularly important feature of a jet is the jet mass, computed as:
$m^{2}=\big{(}\sum_{i\in\textrm{jet}}p_{i}\big{)}^{2}$. The sum of four-
vectors runs over all the constituents $i$ clustered into the jet. As
different heavy resonances have different masses, this feature can be a strong
discriminant between jet types. Note that any operation performed on a jet
which alter the constituents, such as the pileup mitigation discussed in the
next paragraph, may alter the jet mass.

It is important to note that additional proton-proton interactions within a
bunch crossing, or pileup, creates additional particles present within an
event that can impact jet clustering and the estimation of jet properties.
This is especially important for large-$R$ jets which cover large spatial
extents. Dedicated pileup removal algorithms are used to mitigate the impact
of pileup [17]. Jet trimming [18] is a jet grooming technique used to remove
soft and wide angle radiation from jets, in which the constituents within the
jet using a jet algorithm with a smaller radius to define subjets and remove
subjets carrying a fraction of the jet energy below a threshold, and is
frequently used on ATLAS to aid in pileup mitigation. The pileup per particle
identification algorithm (PUPPI) [19] is frequently used by CMS, in which for
each particle a local shape parameter, which probes the collinear versus soft
diffuse structure in the neighborhood of the particle, is calculated. The
distribution of this shape parameter per event is used to calculate per
particle weights that describe the degree to which particles are pileup-like.
Particle four-momenta are then weighted and thus the impact of (down-weighted)
pileup particles on jet clustering is reduce [19]. Pileup mitigation can
greatly improve the estimation of the jet mass, energy and momentum by
removing / downweighting the pileup particles clustered into a jet that only
serve as noise in the jet properties estimation.

Jet identification, energy estimation, and pileup estimation / reduction are
among the primary challenges for which the jet images approach has been
employed: (i) Jet identification refers to the classification of the parent
particle type that gave rise to the jet, and is needed to determine the
particle content of a collision event. (ii) Jet energy estimation refers to
the regression of the true jet total energy from the noisy detector
observations, and is needed to determine the kinematic properties of an event.
(iii) Jet pileup estimation and reduction refers to the determination of the
stochastic contributions to detector observations arising from incident
particles produced in proton-proton collisions that are not the primary hard
scattering. This form of denoising is required to improve the energy and
momentum resolutions of measurements of jets.

Among the primary physics settings in which jet images have been used are in
studies of jets produced by Lorentz boosted heavy particles, such as a $W$ or
$Z$ boson, Higgs boson ($h$), top quark ($t$), or a hypothetical new beyond
the Standard Model particle. When a heavy short-lived particle is produced
with a momentum on the order of twice its mass or more, the quark decay
products of such a heavy particle have a high likelihood of a collimated
emergence in which the subsequent hadronic showers produced by the quarks
overlap. Jet clustering algorithms can capture the entirety of the heavy
particle decay within one large-$R$ jet with an $R$-parameter typically
between 0.8 and 1.0, though in some cases larger $R$ parameters have been
used. The internal structure of such a boosted jet can be highly non-trivial
and significantly different than a typical jet produced by a single quark or
gluon. However, the production of quarks and gluons is ubiquitous at hadron
colliders, and thus powerful discrimination methods, or taggers, are needed to
identify relatively clean samples of heavy-particle-induced boosted jets.
Moreover, the mass scale of heavy hadronically decaying particles in the
Standard Model is similar, from the $W$ boson mass of $\sim 80$ GeV [20] up to
the top quark mass of $\sim 173$ GeV [20]. Typical discrimination tasks thus
include discriminating boosted $W$-, $Z$-, $h$-, or $t$-jets from quarks and
gluons, but also in discriminating between boosted heavy particle jets.

Jet images have also been employed for studying jets from individual quarks
and gluons. This includes discriminating between quark and gluon jets, and
between jets produced by quarks of different flavour. In these cases, smaller
jets typically with $R=0.4$ are used.

###  3 Jet Images and Preprocessing

Jet images are built using a fixed grid, or pixelation, of the spatial
distribution of energy within a jet. Early instances of such pixelation relied
on energy depositions in calorimeter detectors, wherein the angular
segmentation of the detector cells was used to define the “pixels” of the jet
image and the pixel “intensity” was defined with the transverse energy in a
cell. More recently, high resolution measurements of charged particles from
tracking detectors have also been used to form images, wherein the transverse
momentum of all particles found within the spatial extent of a jet image pixel
are summed to define the pixel intensity. While calorimeter and tracking
detectors typically span a large angular acceptance, a typical jet has limited
angular span. The angular span of a jet is related to the $R$-parameter of the
jet clustering algorithm. Jet images are thus designed to cover the catchment
area of the jet [21]. In many cases, the jet image is first defined to be
slightly larger than the expected jet catchment area, to ensure that
preprocessing steps (discussed in Section 1) do not disrupt peripheral pixel
estimates, and then after pre-processing are cropped. Nonetheless, only a
slice of the angular space of the detector is used to define the jet image,
with the image centered on the direction of the jet and the image size chosen
to capture the extend of a jet with a given $R$ parameter. If depth
segmentation is present in a calorimeter, the energy is often summed in depth.
From this vantage point, a jet image can be viewed as a grey-scale image
comprising the energy measurements encapsulated by the angular span of the
jet. In some cases energy depositions from hadronic and electromagnetic
calorimeters will be separated into different images, or separate images will
be formed from both calorimeter cell measurements and the spatially pixelated
charged particle measurement. In these cases, the set of jet images, each
defining a view of the jet from a different set of measurements, can be seen
as color channels of jet image.

It should be noted that jet pileup mitigation, such as the aforementioned
trimming or PUPPI algorithms, is vital to reduce the impact of pileup on
downstream jet image prediction tasks. While not explicitly discussed as a
part of the jet image preprocessing, this step is almost always performed
prior to jet image formation using the jet constituents, especially in the
case of studying large-$R$ jets.

####  1 Preprocessing

An important consideration in the training of a classifier is how to process
data before feeding it to the classifier such that the classifier can learn
most efficiently. For instance, a common preprocessing step in ML is to
standardize inputs by scaling and mean shifting each input feature such that
each feature has zero mean and unit variance. In this case, standardization
helps to ensure that features have similar range and magnitude so that no
single feature dominates gradient updates. In general, data preprocessing can
help to stabilize the optimization process and can help remove redundancy in
the data features to ease the learning of useful representations and improve
the learning sample efficiency. However, data preprocessing may come at a cost
if the preprocessing step requires approximations that lead to distortion of
the information in the data. The primary jet preprocessing steps include:

Translation: An important consideration when preparing inputs to a classifier
are the symmetries of data and transformations of inputs that should not
affect the classifier prediction. In the case of jet images, these symmetries
are related to the physical symmetries of the system. At a particle collider,
there is no preferred direction transverse to the beam line, and the physics
should be invariant to azimuthal rotations in the transverse plane. In terms
of jet images, given a fixed parent particle, the distribution of jet images
at a given azimuthal coordinate $\phi=\phi_{a}$ should not differ from the
distribution at a different $\phi=\phi_{b}$. As such, an important
preprocessing step is to translate all jet images to be “centered” at
$\phi=0$. This is often performed by translating the highest $p_{T}$ subjet
(formed by clustering the jet constituents with a small $R$-parameter jet
algorithms), or the jet energy centroid, to be located at $\phi=0$. The same
invariance is not generically true for changes in $\eta$, as translations in
$\eta$ correspond to Lorentz boosts along the beam direction which could alter
the jet properties if not handled carefully. When energy is used for jet image
pixel intensities, a translation in $\eta$ while keeping pixel intensities
fixed will lead to a change in the jet mass. However, when the transverse
momentum, which is invariant to boosts along the beam direction, is used to
define pixel intensities, a translation in $\eta$ can be performed without
altering the jet mass distribution. With this definition of pixel intensities,
jet images are typically translated such that the leading subjet is located at
$\eta=0$. By centering the jet on the leading $p_{T}$ subjet, the classifier
can focus on learning the relative variations of a jet, which are key for
classification.

Rotation: The radiation within a jet is also approximately symmetric about the
jet axis in the $\eta-\phi$ plane. A common preprocessing step is thus to
rotate jet images, after centering the image on the leading $p_{T}$ subjet,
such that the second leading $p_{T}$ subjet or the first principle axis of
spatial distribution of $p_{T}$ in the image is aligned along the $y$ axis of
the image. However, there are challenges with rotations. First, rotations in
the $\eta-\phi$ plane can alter the jet mass, thus potentially impacting the
classification performance222Alternative definitions of rotations have been
proposed that preserve jet mass [22] but may alter other key jet properties..
Second, as jet images are discretized along the spatial dimensions, rotations
by angles other than factors of $\pi/2$ can not be performed exactly. One
approach is to perform a spline interpolation of the $p_{T}$ distribution
within a jet image, apply a rotation to this spline function, and then impose
an image grid to discretize the spline back to an image. The interpolation and
the post-rotation discretization can spatially smear information in the jet
and lead to aliasing. As such, there is varying use of rotation preprocessing
in jet image research.

Flipping: A transformation $\phi\to-\phi$ should not affect the physics of the
jet, and this transformation can be performed to ensure that positive $\phi$
contains the half of the jet with more energy, for instance due to radiation
emission.

Normalization: A step often found in image preprocessing for computer vision
tasks is image normalization, typically through taking an $L^{2}$ norm of the
image such that $x_{i}\to x_{i}/\sum_{j}x_{j}^{2}$ where $x_{i}$ is a pixel
intensity and the sum runs over all pixels in an image. However, in the case
of jet images, such a normalization may be destructive, as it does not
preserve the total mass of the jet (as computed from the pixels) and can
deteriorate discrimination performance due to this loss of information [23].
As such, there is varying usage of image normalization in jet image research.

The impact on the jet mass, as computed from the pixels of jet images, for $W$
boson jets within a fixed $p_{T}$ range and within a fixed pre-pixelation mass
range can be found in Figure 2. The distortion on the jet mass from
pixelation, rotations for images with energies as pixel intensities, and from
$L^{2}$ normalization, can be seen clearly, whilst translation and flipping do
not show distortions of the jet image mass. As expected, mild distortion of
the mass can be seen when rotations are performed on jet images with
transverse energy used for pixel intensities. These distortions may or may not
be impactful on downstream tasks, depending on if the jet mass is a key
learned feature for the downstream model.

Figure 2: The impact of various preprocessing steps on the distribution of
estimated jet mass for boosted $W$ boson jet images [23].

###  4 Computer Vision and Convolutional Neural Networks

Object classification in Computer Vision (CV) tasks served as a primary
setting where Deep Learning had major early successes [24], quickly surpassing
then state of the art approaches and serving as one of the drivers for a deep
learning revolution. While much of the work in CV has focused on understanding
natural images, data collected by physics experiments come from heterogenous
detectors, tend to be sparse, and do not have a clear similarity to natural
images. Nonetheless, the success of DL in CV inspired a parallel effort in the
collider physics community to explore applications of such techniques to HEP
data. Below we present a brief introduction to convolutional neural networks
(CNNs) [25] and some of the state-of-the-art architecture variants in order to
provide some background for the models used in jet tagging and other
applications. For a more in depth pedagogical introduction to this material
see for instance [26].

Most of the models discussed in this text rely on the use of convolutional
layers. However, it should be noted that some models make use of locally-
connected layers [25, 27, 28], in which a given neuron only has access to a
small patch of an input but, unlike convolutional layers that rely on weight
sharing (as discussed below), the neuron processing each image patch is
associated with a different set of weights.

Convolutional neural networks rely on neuron local spatial connectivity and
weight sharing to produce translationally equivariant models that are well
adapted to image analysis. A typical CNN is built by stacking one or more
convolutional and non-linear activation layers often followed by a pooling
layer. This structure is repeated several times. Fully connected layers, with
full connections from all inputs to activations, are used to perform the final
classification or regression prediction. Images processed by CNNs are
represented as 3D tensors with dimensions width $\times$ height $\times$ depth
and are often referred to as the image volume. The height and width dimensions
correspond to the spatial extend of the image while the depth is typically the
color channel.

Convolutional layers are composed of a set of filters, where each filter
applies an inner product between a set of weights and small patch of an input
image. The filter is scanned, or convolved, across the height and width of the
image to produce a 2D map, often referred to as a response map or convolved
image, that gives the response of applying the filter at each position of the
image. The response at each position becomes large when the filter and the
image patch match, i.e. when their inner product is large. The filters will
thus learn to recognize visual features such as edges, textures, and shapes,
and produce large responses when such visual features are present in a patch
of an image. The spatial extent of the input patch is known as the receptive
field or filter size, and the filters extend to the full depth of the image
volume. Several filters are learned simultaneously to respond to different
visual features. The response map of the filters are then stacked in depth,
producing an output convolved image volume. Finally, the response maps are
passed through point-wise (i.e. per pixel) non-linear activations to produce
an activation map.

By sharing weights between neurons, i.e. by scanning and applying the same
filter at each image location, it is implicitly assumed that it is useful to
apply the same set of weights to different image locations. This assumption is
reasonable, as a visual feature may be present at any location in an image and
the filter is thus testing for that feature across the image. This results in
the convolutional layers being translationally equivariant, in that if a
visual feature is shifted in an image, the response to that feature will be
shifted in the activation map. In addition, parameter sharing results in
dramatic reduction in the number of free parameters in the network relative to
a fully connected network of the same number of neurons.

Pooling layers reduce the spatial extent of the image volume while leaving the
depth of the volume unchanged [29]. This further reduces the number of
parameters needed by the network and helps control for overfitting. Pooling is
often performed with a max operation wherein only the largest activation in a
region, typically $2\times 2$, is kept for subsequent processing.

Normalization layers may be used to adjust activation outputs, typically to
control the mean and variance of the activation distribution. This helps
ensure the that a neuron does not produce extremely large or small activations
relative to other neurons, which can aid in gradient-based optimization and in
mitigating exploding / vanishing gradients. Batch normalization [30] is a
common normalization method in which, for each mini-batch, the mean and
variance within the mini-batch of each activation dimension are used to
normalize the activation to have approximately zero and unit variance. A
linear transformation of the normalized activation, with learnable scale and
offset parameters, is then applied.

Fully connected layers are applied at the end of the network after flattening
the image volume into a vector in order to perform classification or
regression predictions. At this stage, auxiliary information, potentially
processed by a separate set of neural network layers, may be merged with the
information gleaned from the processing by convolutional layers in order to
ensure that certain features are provided for classification. Within the jet
tagging context, such information may correspond to information about the jet,
such as its mass, or global event information such as the number of
interactions in a given collision.

Residual Connections: While CNNs encode powerful structural information into
the model, such as translation equivariance, it has been noted that scaling up
such models by stacking large numbers of convolutional layers can lead to
large challenges in training [31]. In order to train large models using the
backpropagation algorithm, the chain rule is used to compute the gradient from
the model output back to the relevant weight. In early layers, the
multiplication of many gradients can lead to vanishingly small or exploding
gradients, thus resulting in unhelpful gradient updates. To overcome this
challenge, the residual block [32] was proposed, and has led to the
development of residual networks. While a typical neural network layer passes
input $z$ through a nonlinear function $f(\cdot)$ to produce an output
$z^{\prime}=f(z)$, a residual block also uses a “skip connection” to pass the
input to the output in the form $z^{\prime}_{\textrm{res}}=W_{s}z+f(z)$ where
the weights $W_{s}$ can be used to project the channels $z$ to have the same
dimension as the function $f(z)$. In this way, the function $f(\cdot)$ is
tasked with learning the relative change to the input. Moreover, the skip
connection provides a path to efficiently pass gradients backwards to earlier
layers of the network without distortion through the nonlinearities, making
gradient descent much easier and thus enabling the training of significantly
deeper models. Note that the function $f(\cdot)$ can contain several layers of
convolutions, nonlinearities, and normalization before being recombined with
the input.

Training in supervised learning tasks is performed by minimizing an
appropriate loss function that compares the CNN prediction with a true label.
The loss is typically the cross-entropy in the case of binary classification,
and the mean squared error in the case of regression. Minimization is
performed using stochastic gradient descent (SGD) [33], or one of its variants
such as ADAM [34] designed to improve the convergence of the learning
algorithm.

Evaluation Metrics Receiver operating characteristic (ROC) curves are
frequently used to examine and compare the performance of binary
classification algorithms. Given a model which produces a classification
prediction $c(x)$, where $x$ is the input features and $c(\cdot)\in[0,1]$, the
model is applied to a set of inputs thus yielding a distribution of
predictions. A threshold $\tau$ on the prediction is scanned from 0 to 1, and
the fraction of inputs for each the signal and background classes above this
threshold, i.e. the signal efficiency ($\epsilon_{S}$) and background
efficiency ($\epsilon_{B}$) for surviving this threshold, defines a point on
the ROC curve for each $\tau$ value. ROC curves thus display the background
efficiency (or background rejection defined as 1 divided by the background
efficiency) versus the signal efficiency. When the ROC curve is defined as the
background efficiency versus the signal efficiency, a metric commonly used to
evaluate the overall model performance is the ROC integral, also known as the
area under the curve (AUC).

Significance improvement characteristic (SIC) curves [35] are closely related
to ROC curves, but display $\epsilon_{S}/\sqrt{\epsilon_{B}}$ as a function of
the signal efficiency $\epsilon_{S}$. This curve targets displaying the
potential improvement in statistical significance when applying a given
discriminant threshold relative to not applying such a threshold.

###  5 Jet tagging

Jet tagging refers to the classification of the parent particle which gave
rise to a jet. Linear discriminant methods were first applied to jet images
defined using a single channel, or “grey-scale”, with pixel intensities
defined as the calorimeter cell $p_{T}$ [1]. Subsequently, CNN based
classifiers trained on single channel images were developed for discriminating
between $W/Z$ jets and quark / gluon jets [23, 27], between top jets and quark
/ gluon jets [36, 10, 37, 38], and for discriminating between quarks and
gluons [39]. Quark / gluon discrimination with single-channel jet images has
also been explored for use in heavy ion collisions [40]. The extension to
utilizing jet images “in color” with multiple channels, defined for instance
using charged particle information, has shown promising performance
improvements over single channel approaches in many of these tasks [39, 41,
42, 10, 43, 44, 45], and has been explored in realistic experimental settings
by the ATLAS and CMS collaborations [46, 47].

####  1 Jet Tagging on Single Channel Jet Images

#####  $W/Z$ Tagging:

The discrimination of boosted $W$ and $Z$ vector boson initiated jets from
quark / gluon jets has served as a benchmark task in boosted jet tagging. The
color singlet nature of electroweak bosons decaying to quark pairs leads to an
internal structure of boosted $W/Z$ jets in which there are typically two high
energy clusters, or subjets, and additional (dipole) radiation tends to appear
in the region between such subjets. The Higgs boson, also a color singlet with
decays to quark pairs, has a similar substructure, although the decays of
heavy flavour bottom and charm quark pairs can lead to some structural
differences owing to the long lifetime of such quarks and their harder
fragmentation than lighter quarks. In contrast, single quarks and gluons tend
to produce jets with a high energy core, lower energy secondary subjets
created through radiative processes, as well as diffuse wide angle radiation
further from the core of the jet. These features can be seen clearly in Figure
3, which shows the average $W$ boson jet image and average quark / gluon jet
image after preprocessing.

Figure 3: Average $W$ boson jet image (left) and average quark / gluon jet
image (right) after preprocessing [23].

Building ML models applied to jet images for this discrimination task avoids
the explicit design of physics-inspired features, and rather focuses on the
learning task of identifying differences in the jet image spatial energy
distributions. In phenomenological studies, both fully convolutional [23] and
models with locally connected layers [27] have been examined for
discriminating jet images of boosted $W$ and $Z$ vector boson initiated jets
from quark / gluon jets. The CNN models were examined in simulated samples of
jets without pileup. The locally connected models were examined in events both
with and without pileup, thus enabling the examination of the impact of pileup
noise on jet image based tagging.

Within both the studies on convolutional [23] and locally connected [27]
models, hyperparameter scans were performed to find model parameters that
maximized performance333In the case of CNNs the AUC was maximized whilst the
Spearmint Bayesian Optimization package [48] was used to optimize the model
with locally connected layers. The hyperparameters that were considered in the
scans included the number of convolutional / locally connected layers, the
number of hidden units per layer, and the number of fully connected layers.
The resulting optimized models were similar, containing 3 to 4 convolutional
or locally connected layers, as well as 2 to 4 fully connected layers with
approximately 300 to 400 hidden units at the end of the network. In the CNN,
32 filters were used in each convolutional layer, as well as $(2\times 2)$ or
$(3\times 3)$ downsampling after each convolutional layer. One notable
additional optimization performed for the CNN models was the size of the
convolution filters in the first layer. While filter sizes are typically
$(3\times 3)$ or $(4\times 4)$ in standard CV applications, in the case of
application to jet images it would found that a larger $(11\times 11)$ filter
in the first convolutional layer (with later layers using standard $(3\times
3)$ filter sizes) resulted in the best performance. It was hypothesized that
the such large filters were beneficial when applied to sparse images [23], in
order to ensure that some non-zero pixels are likely to be found within the
image patch supporting the filter application.

The ROC curves indicating the performance of the CNN model and locally
connected model (applied to jets with pileup included) are shown in Figure 4.
It should be noted that the jets in these figures correspond to different
$p_{T}$ ranges, with jets of $p_{T}\in(250,300)$ GeV for the CNN model and of
$p_{T}\in[300,400]$ GeV for the locally connected model, and thus are not
directly comparable. Also shown are combinations of common physics expert
engineered jet substructure features, such as the jet mass, the distance
between the two leading $p_{T}$ subjets, the $\tau_{21}$ n-subjetiness [49],
and the energy correlation function $D_{2}^{\beta=2}$ [50]. Two variable
combinations were computed using 2D binned likelihood ratios. Both the CNN and
locally connected model significantly outperform the 2D jet substructure
feature combinations. It can also be seen that the jet image approach is not
overly sensitive to the effects of pileup as the large performance gain over
jet substructure features persists both with and without the presence of
pileup, owing to the use of jet trimming to reduce the impact of pileup noise
in the jets. In addition, a Boosted Decision Tree (BDT) classifier [51]
combining six substructure features was compared with the locally connected
model and found to have similar performance. While these early jet image based
models did not significantly outperform combinations of several jet
substructure features, this may be due to their relatively small model
structure. As will be seen, more complex architectures and the use of multi-
channel jet images can lead to large gains over combinations of jet
substructure features.

One can also see the effect of $L^{2}$ image normalization on CNN models,
which appears to improve performance over unnormalized images. This effect was
found to occur because the CNN model output was observed to have only small
correlation with the jet mass, and thus was not learning to be heavily reliant
on the jet mass information that is distorted by normalization. As a result,
the regulation of the image variations due to normalization was found to be
beneficial enough to overcome the induced distortion of the jet mass. With
more powerful models that learn representations more correlated to the jet
mass, this balance may not occur.

(a)

(b)

Figure 4: ROC curves for quark/gluon background rejection versus boosted $W$
boson tagging efficiency for (a) events without pileup [23], and (b) events
with pileup [27]. The jet image based CNN taggers are seen to outperform
combinations of jet substructure features, and to be stable with respect to
the addition of pileup.

##### Top Tagging:

The discrimination between boosted top quark jets and quark / gluon jets using
CNNs applied to jet images has also been examined both in phenomenological
studies [36] and in realistic simulations by the CMS experiment [47]. Top
quark jet images are structurally more complex than $W/Z/h$ jet images as
hadronic decays of top quarks contain three quarks. This can have implications
on both the preprocessing and the tagging performance. That is, some of the
pre-processing steps previously defined will lead to uniformity among jet
images for two quark systems, such as the rotation step which aligns the
leading two subjets, but may not lead to the same level of uniformity for
three quark systems.

The DeepTop [36] model is a CNN applied to single channel jet images after the
preprocessing described above, including image normalization. Hyperparameter
optimization yielded a model with four convolutional layers, each with 8
filters of size $(4\times 4)$, MaxPooling for image downsampling after the
second convolutional layer, and three dense layers of 64 hidden units each for
classification. For these phenomenological studies, the model was trained with
approximately 150k jets using the mean squared error (MSE) loss. While
structurally similar to the single channel CNN used for $W/Z$ tagging in
reference [23] there are some notable differences such as the use of fewer
numbers of filters (8 rather than 32) and the smaller filter size in the first
layer of convolution. The reason for these difference may be due to (a) the
presence of three quarks in the top quark decay leads to more pixel-populated
images and thus allowed for the use of smaller initial filter sizes, or (b)
the global nature of the hyperparameter scan wherein the number of filters and
the size of the filters was fixed to be the same across all convolutional
layers.

(a)

(b)

Figure 5: ROC curves for quark / gluon jet rejection versus boosted top
efficiency for (a) the DeepTop model [36], and (b) the updated DeepTop model
from reference [42]. In both cases, the CNN based DeepTop models outperform
individual and BDT combinations of substructure features, while the updated
model in (b) is also seen to significantly improve the DeepTop performance.

The performance of the DeepTop model can be found in Figure 5(a) in terms of
the ROC curve comparing the quark / gluon rejection versus the boosted top jet
tagging efficiency for jets with $p_{T}\in[350,450]$ GeV. In this momentum
range, the decay products of the top quark may not be contained in a single
jet, and such a containment was not required for the jets under study. DeepTop
was compared with a combination of mass and $n$-subjettiness, as well and a
BDT, denoted MotherOfTaggers, combining several jet substructure features. The
jet image based DeepTop algorithm showed clear performance gains over
substructure approaches across most of the signal efficiency range. As
previously mentioned, pre-processing steps have the potential to be beneficial
for the learning process by producing more uniform images, but may also lead
to performance degradation. This was studied within the scope of the DeepTop
algorithm, by examining the tagging performance using full preprocessing and a
minimal preprocessing that only performed centering but not the rotation or
the flipping. This can be seen in Figure 5(a), where a clear performance
benefit was observed when utilizing only minimal pre-processing. While the
full pre-processing may be beneficial for small sample sizes, with sufficient
sample sizes and model complexity the CNN models appear able to learn well all
the variations in jet images. In this case, the approximations introduced by
pre-processing steps appear to be more detrimental than the benefits from
uniformization of the jet image distributions.

Building upon the DeepTop design, developments in architecture design, jet
image preprocessing, and optimization were introduced in the phenomenological
study of reference [42]. These developments include: (i) the cross entropy
loss function, rather than the mean squared error loss, was used as it is more
suitable to binary classification problems, (ii) a learning rate adaptive
optimizer, AdaDelta [52], and small mini-batch sizes of 128 was used rather
than vanilla stochastic gradient descent and large mini-batches of 1000, (iii)
larger numbers of filters per convolutional layer, between 64 and 128 rather
than 8, and 256 neurons in the dense layers instead of 64, (iv) preprocessing
is performed before pixelation under the assumption that one would have access
to high resolution particle momentum measurements, for instance using Particle
Flow [14] approaches to jet reconstruction, and (v) the training set size was
increased by nearly a factor of 10. While the individual effects of these
developments will be examined further in Section 2 when discussing top tagging
on multi-channel jet images, the combination of these developments can be seen
to provide large performance improvements over DeepTop of nearly a factor of
two in background rejection at fixed signal efficiencies in Figure 5(b).

In terms of more complex architectures, the ResNeXt-50 architecture [53] was
adapted to boosted top jet tagging task using single channel jet images in the
phenomenological studies in reference [10]. ResNeXt-50 utilizes blocks
containing parallel convolutional layers that are aggregated and merged also
with a residual connection at the end of the block. As the jet images
typically have fewer pixels than natural images, the architecture was adapted
to the top tagging dataset by reducing the number of filters by a factor of
four in all but the first convolutional layer, and dropout was added before
the fully connected layer. In addition, smaller pixel sizes in the jet images
were utilized in this model, with a granularity of 0.025 radians in
$\eta-\phi$ space (whereas the jet image granularities typically used in other
models is 0.1 radians in $\eta-\phi$ space).

Figure 6: ROC curve comparisons of various boosted top tagging models is shown
[10]. Both ResNeXt and CNN curves are jet image based taggers using CNN based
architectures.

The ROC curve comparing the ResNeXt-50 model to a CNN based on references [36,
42], and comparing to several other neural network models with varying
architectures can be found in Figure 6. The ResNeXt-50 model provides
approximately 20% improvement in background rejection for fixed signal
efficiency over the CNN model, and is among the most performant algorithms
explored. This is notable as many of the other neural network models utilize
particle 4-vectors as inputs, rather than aggregated particle information with
a pixel cell, and make use of particle charge information, while the ResNeXt
model only utilizes the distribution of energy within the jet. However, the
ResNeXt-50 model contains nearly 1.5 million parameters, which is far more
than other models such as the CNN which contains $\approx$610k parameters and
the tree structured neural network (TreeNiN) which contains $\approx$34k
parameters. Thus powerful information for discrimination can be extracted with
jet image based models even from single channel images, but it may come with
the price of models with large parameter counts.

This model comparison study has been performed in a phenomenological setting
on particle level simulations, and the ultimate question remains as to the
suitability for using these models in real experiment settings. In
experimental settings, realistic detector noise, detection efficiency,
detector heterogeneity, and data taking conditions such as pileup, underlying
event, and beam evolution will impact the model performance. Powerful models,
including the large ResNext and CNN models, will likely have sufficient
flexibility to learn powerful discriminators even in these more challenging
settings. However, in general it remains to be seen if these models can be
accurate whilst maintaining a low calibration error (where calibration in this
context refers to the criteria that the predicted class probabilities
correspond to the true probabilities of a given data input having a given
label) [54], or if additional care is needed to ensure calibration. Moreover,
applications in real experimental settings must consider systematic
uncertainties associated with training ML models in (high fidelity) simulation
but applying them in real data with potentially different feature
distributions. The relationship between model complexity and sensitivity to
systematic uncertainties in real experiment settings still remains to be
thoroughly explored. The potential benefits in terms of sensitivity to
systematic uncertainties when using neural networks with different structural
assumptions, such as convolutional versus graph models, also requires further
study and will likely depend on the details of how a given systematic
uncertainty effects the feature distributions. Some exploration of these
challenges can be found in Section 3 examining model sensitivity to
theoretical uncertainties and in Section 4 examining applications of these
models in HEP experiments. Nonetheless, these remain important and exciting
avenues of future work.

##### Decorrelated tagging with Jet Images:

A common strategy in HEP to search for a particle is the so-called bump hunt
in which the particle would give rise to a localized excess on top of a
smoothly falling background in the distribution of the mass of reconstructed
particle candidates. For instance, one may aim to identify the $W$-boson mass
peak over the quark and gluon background from the distribution of jet mass. In
addition to the particle mass being localization, a key to this strategy is
that the smoothly falling background mass distribution can typically be
characterized with simple parametric functions, thus facilitating a fit of the
data to identify the excess above this background. Jet classification methods
can cause challenges in the aforementioned strategy, as the classifier may
preferentially select for jets with a specific mass, thereby sculpting the
selected jet mass distribution of the background and rendering the search
strategy unusable. As a result, one line of work has focused on de-correlating
classifiers from a sensitive feature (e.g. mass) such that the sensitive
feature is not sculpted by the application of the tagger. Such methods tend to
rely on data augmentation or regularization, and overviews of these methods
can be found for instance in references [55, 56]. Two recent regularization
techniques that have seen strong de-correlation capability include (i)
adversarial techniques [57, 58], wherein a second neural network is trained
simultaneously with the jet classifier to penalize the jet classifier when the
value of the sensitive feature can be predicted from the classifier’s output
or its hidden representations, and (ii) distance correlation regularizers
[59], wherein the jet classifier loss is augmented with an additional
regularization which explicitly computes the correlation between the
classifier predictions and the sensitive feature. In both cases, the amount of
penalization from the regularization can be varied through a hyperparameter
scaling the relative size of the regulation term to the classification loss.

De-correlation for $W$-boson jet tagging with jet images using CNNs was
examined in phenomenological studies in reference [59], using a CNN
architecture similar to the model described in [42]. The quark and gluon
background jet mass distribution before and after applying a threshold on the
output of a CNN can be seen in Figure 7(a), showing a clear sculpting of the
mass distribution. However, when the distance correlation regularization, or
Disco, is used during training, the mass distribution remains largely
unsculpted after applying a classification threshold. The level of de-
correlation can be estimated by examining the agreement between the mass
distribution before and after applying a classifier threshold, for instance
using the Jensen-Shannon divergence (JSD) computed between the binned mass
distributions. For classifier thresholds fixed to 50% signal efficiency,
Figure 7(b) shows the JSD as a function of the background rejection where the
curves are produced through training with varying sizes of the regularization
hyperparameter. The CNN models are compared with neural networks trained on
substructure features and other classifiers with de-correlation methods
applied. The CNN models, both the adversarial and distance correlation
regularization, are seen to typically provide the highest background rejection
for a given level of de-correlation compared to other models.

(a)

(b)

Figure 7: (a) For boosted $W$ boson tagging, the jet mass is shown before
applying a threshold on a trained CNN tagger and after applying a threshold on
a standard and mass decorrelated tagger [59]. A clear reduction in mass
sculpting is observed. (b) The rejection at 50% signal efficiency versus one
over the Jensen-Shannon divergence, computed on the binned jet mass
distribution before and after tagging, is shown for various taggers [59]. Jet
image based CNN taggers are seen to outperform other methods, either using
adversarial or distance-correlation based mass decorrelation.

####  2 Multi-Channel Jet Tagging with CNNs

Recent work on jet image based tagging has shown performance gains through the
use of multi-channel images. While single channel jet images have provided
gains in classification performance over individual, or pairings of,
engineered substructure features, the performance benefits were typically
smaller when compared to ML models trained on larger groups of substructure
features (except when very large models were used, as in [10]). Multi-channel
jet images use calorimeter images as only a single input image channel, with
additional channels computed from charged particle features such as the
momentum, multiplicity, or charge. There is a significant amount of freedom in
choosing the definition of the additional image channels, allowing for a
flexibility in the choice of inductive bias to deliver relevant information to
the CNN.

One challenge in combining charged particle trajectory information and
calorimeter images is the mismatch in resolution; charged particle
trajectories tend to have a significantly finer spatial resolution than
calorimeters, thus leading to the questions of how to combine such
information. As charged particles are not measured on a regular grid, often
the same spatial grid for the calorimeter component is used for the charged
particle image and the energy of the constituents is summed within each pixel.
Alternatively, separate CNN blocks (or upsampling procedures) can be used to
process charged and calorimeter images separately into a latent representation
of equal size such that they can be merged for further processing. Note that
when Particle Flow objects are used, and thus both neutral and charged
particle measurements do no necessarily fall on a grid, a fine grid can be
used to exploit the better charged particle momentum resolution. It should
also be noted that while phenomenological studies at particle-level often use
fixed grids to emulate the discretization of real detectors, different inputs
(i.e. charge vs neutral) in real detector settings have different resolutions
which may be difficult to account for in simple discretization approaches.

Multi-Channel jet image based tagging was introduced in phenomenological
studies of discriminating between quark initiated and gluon initiated jets
[39, 41] and has since been explored within the quark vs. gluon context on the
ATLAS experiment [46], in CMS Open Data [60, 61], and for tagging in heavy ion
collision environments [40]. More broadly, multi-channel jet image tagging has
lead to improved performance in phenomenological studies of boosted top quark
jet tagging [42, 37, 10], as well as in boosted $W/Z$ jet tagging [43] and in
boosted Higgs boson tagging [44, 45]. Notably, multi-channel jet image based
boosted top tagging has been explored on the CMS experiment [47] including the
comparison and calibration of this discriminant with respect to CMS collision
data, thus adding additional insights into the usability of such models within
LHC data analysis.

The use of multi-channel jet images built from charged particle momentum and
multiplicity information within the context of discriminating between quarks
and gluons is natural, as the number of charged particles within such a jet is
known to be a powerful discriminant for this challenging task [62]. As such,
in the phenomenological studies of reference [39] three jet image channels
were defined: (1) the transverse momentum of charged particles within each
pixel, (2) the transverse momentum of neutral particles within each pixel, and
(3) the charged particle multiplicity within each pixel. The same pixel size
was used in each image, thus facilitating the direct application of multi-
channel CNNs. This approach thus relies on the ability to separate the charged
and neutral components of a jet; while the charged component is measured using
tracking detectors, the unique identification of the neutral component of a
jet is significantly more challenging task. However, advancements in Particle
Flow [14] aid in such a separation, albeit not perfectly and with differing
resolutions between charged and neutral measurements.

The benefit of the multi-channel approach for quark vs. gluon discrimination
can be seen in the ROC and SIC curves in Figure 8. Both the calorimeter only
approach, denoted Deep CNN grayscale, as well as the multi-channel approach,
denoted Deep CNN w/ color, outperform single features engineered for this
task, BDTs trained using five of such features, and a linear discriminant
trained on the greyscale jet images. In addition, the multi-channel model is
seen to dominate over the single channel model in both the ROC curve for jets
with a momentum of $p_{T}\approx 1000$ GeV and in the SIC curve across a range
of jet momentum. The multi-channel approach is found to be especially
beneficial at higher momentum where the jets have a large charged particle
multiplicity.

(a)

(b)

Figure 8: ROC curve (a) and SIC curve (b) for quark versus gluon tagging using
multi-channel jet images [39]. Comparisons with jet substructure based
discriminants is shown in (a), while comparison between single channel and
multi-channel jet image based tagging with CNNs is shown in (b).

This multi-channel approach using charged, neutral, and multiplicity channels
was also found to be powerful in phenomenological studies of discriminating
between boosted Higgs boson jets and a background of gluon splitting to
$b\bar{b}$ jets in multi-jet events [44]. In addition to a CNN focused on
discrimination based on jet images, this work also explored simultaneously
processing an event image, defined using the aforementioned three channels
over the entire calorimeter, through a separate set of convolutional layers
and combining with the output of the convolutional processing of jet image
before discrimination. By including such an event image, one may explore the
potential benefits of event topology information outside of the jet image for
discrimination. The SIC curve for this discrimination task can be seen in
Figure 9, where the CNN approaches were seen to significantly outperform
single engineered features. CNNs using only the jet image, event image, or
both (denoted “Full CNN Architecture” in Figure 9) were compared, showing that
much of the discrimination power rests in the jet image whilst the event image
may provide some modest improvements. In addition, the jet image
discrimination without the neutral particle channel was also found to be
comparable to one using the neutral channel, indicating that much of the
discrimination power lies in the charged particle information within the jet.

Figure 9: SIC curve for boosted Higgs to $b\bar{b}$ tagging using multi-
channel jet images [44]. Models using only jet images, and models using both
jet images and “event images” are shown.

While a clear approach to extending jet images to contain multiple channels is
to sum the momentum of the charged particles or compute multiplicities in each
pixel to form an image channel, the high resolution of the charged particle
information allows for the introduction of additional inductive bias. More
specifically, given the set of charged particles contained in the region of a
pixel, one may compute pixel-level features that may be more amenable to a
given discrimination task. This approach was followed for building CNNs to
discriminate between (a) up and down type quarks, and (b) quarks and gluons
[41]. In these phenomenological studies, knowledge of the utility of the jet
charge feature [63, 64, 65] for discriminating jets of different parent
particle charge inspired the development of the jet image channel computed per
pixel as the $p_{T}$ weighted charge
$Q_{\kappa}=\frac{1}{(\sum_{j}p_{T}^{(j)})^{\kappa}}\sum_{j}Q^{(j)}(p_{T}^{(j)})^{\kappa}$.
The SIC curve showing the performance of the CNN trained on the two channel
jet images, one channel for $p_{T}$ and one for $Q_{\kappa}$ per pixel, is
shown in Figure 10(a). The two channel CNN significantly outperformed the
total jet charge and classifiers trained on engineered features, and is
comparable to other deep architectures trained for this task.

(a)

(b)

Figure 10: SIC curves for (a) discriminating down quarks from up quarks [41],
and (b) discriminating between $W^{+}$ and $W^{-}$ bosons [43]. The CNN
$\kappa=0.1$ model in the left figure and the CNN models of the right Figure
utilize the per pixel $p_{T}$ weighted charge image.

A similar jet charge based multi-channel CNN was explored for discriminating
between boosted $W^{+}/W^{-}/Z$ boson jets in the phenomenological studies of
reference [43]. The per pixel charge image averaged over the test set for
$W^{+}$, $W^{-}$, and $Z$ jet images is shown in Figure 11. The geometry of
all three images is similar, but the average per pixel charge differs
significantly as expected, with the typical $W^{+}$ image carrying a positive
pixel value, the typical $W^{-}$ image carrying a negative pixel value, and
the typical $Z$ image having charge close to zero. The SIC curve for
discriminating between $W^{+}$ and $W^{-}$ jets can be seen in Figure 10(b).
Two CNNs were explored in this work, one denoted CNN in which both a $p_{T}$
and $Q_{\kappa}$ image were processed together (i.e. as a single multi-channel
image processed by convolutional layers) and one denoted CNN2 in which each
channel is processed by a separate stack of convolutional layers and then
combined before the classification layers. Both CNNs significantly outperform
methods based on engineered features.

Figure 11: Average image of the per pixel $p_{T}$ weighted charge $Q_{\kappa}$
is shown for $W^{+}$ (left), $W^{-}$ (middle), and $Z$ bosons (right) [43].

Multi channel jet images were explored for top tagging in the phenomenological
studies of reference [42], using four channel jet images defined with the
neutral jet component as measured by the calorimeter, the charged particle sum
$p_{T}$ per pixel, the charged particle multiplicity per pixel, and the muon
multiplicity per pixel. The architecture is discussed in Section 1 within the
context of single channel jet images. The inclusion of the muon image channel
targets the identification of $b$ quark initiated subjets within the top jet
as muons can be produced in $b$-hadron decays. As noted in Section 1, several
changes to the model architecture, preprocessing, and training procedure
relative to the first proposed DeepTop model [36] were included in this work.
The impact of these individual changes can be see in Figure 12 wherein
developments on top of the first proposed DeepTop model [36] are sequentially
added to the model and the resulting ROC curve is shown. The inclusion of
multiple “color” channels was only seen to provide modest performance gains
over single channel jet images. Notable among changes that led to the largest
improvements were changing the optimization objective to be more suitable for
classification tasks and changing the optimizer to ADAM (denoted training in
the figure), increasing the model size (denoted architecture in the figure),
and increasing the sample size. In agreement with these results, recent CNN
models built for processing jet images have also tended to focus on larger
models with large samples for training.

Figure 12: ROC curve of boosted top jet tagging efficiency versus background
quark and gluon rejection for the minimal DeepTop model [36] compared with
models sequentially including the changes proposed in [42].

####  3 Sensitivity to Theory Uncertainties

While matrix element and parton shower Monte Carlo generators often provide
high fidelity predictions of the data generation process, they provide only
approximations to the scattering and showering processes and empirical models
of the hadronization process. As such, uncertainties in the theoretical
predictions of these generators must be propagated to downstream analyses. One
mechanism for doing this is to compare an observable computed with samples
from different Monte Carlo generators. While not a precise estimation of
theoretical uncertainty, this comparison can provide a test of whether an
observable is potentially sensitive to the different approximations of the
different generators.

This sensitivity has been examined for CNN-based taggers operating on jet
images in several works, and we focus here on $W$-tagging in a
phenomenological study [66] and on quark / gluon tagging using ATLAS
simulation [46]. As the CNN + jet image approaches utilize the distribution of
energy throughout a jet image to discriminate, one concern is that the
differences in modeling of the jet formation process by different generators
may lead to large performance variations. To study this, reference [46]
trained a CNN model on boosted $W$ boson jet images generated by Pythia [67,
68] and applied this trained model on samples of boosted $W$ boson jet images
generated by different Monte Carlo generators. The ROC curves of the
performance can be see in Figure 13(a), wherein, at the same signal
efficiency, reductions of background rejection of up to 50% can be seen when
this tagger is applied to different generators. While such a variation is not
ideal, it should be noted that similar variations were seen when a tagger of
only substructure features, a binned two dimensional signal over background
likelihood ratio of the distribution of jet mass and $\tau_{21}$, is applied
for the same tagging task. Similar levels of performance variation are also
seen in the ROC curves built for quark vs gluon tagging in ATLAS simulation
with a CNN trained on Pythia jet images applied to Herwig [69] generated jet
images, as seen in Figure 13(b). Interestingly, when the test is reversed and
the CNN is trained on jet images from Herwig and applied to jet images from
Pythia, the tagging performance is similar to the CNN trained and applied to
Pythia jet images. This suggests that the CNNs in both cases are learning
similar representations of information useful for quark vs gluon tagging, but
the amount this information is expressed in the jet images varies between
generators [46]. Thus while these studies show that CNNs applied to different
samples may vary in performance, there may be an underlying robustness to the
information learned by CNNs for jet tagging.

(a)

(b)

Figure 13: (a) ROC curves for boosted $W$ tagging for a jet image based CNN
tagger trained on Pythia generated jet images and applied to jet images from
various generators are shown [66]. (b) In events with the ATLAS full detector
simulation, ROC curve for quark versus gluon tagging for jet image based CNN
tagger trained and applied on Pythia and Herwig based jet images are shown
[46].

Beyond the potential tagging variations due to generator uncertainties, a key
question when developing a jet observable of any kind is whether such an
observable is theoretically sound and calculable. This is often expressed as
whether the observable is infrared and collinear (IRC) safe. IRC safety for
jet image based tagging of boosted top jets with CNNs has been examined
empirically in the phenomenological studies of reference [70]. In this work,
within the context of boosted top jet tagging using a jet image based CNN, a
feature denoted $\Delta_{NN}$ is studied which explores the impact of merging
soft/collinear radiation with nearby partons. $\Delta_{NN}$ is constructed as
follows: (a) a CNN was trained on particle level jet images for boosted top
tagging, (b) parton level jet images are generated for boosted top decays
without (unmerged) and with (merged) adding the closest gluon to a top quark
parton together before forming the image, (c) the difference in CNN output
between unmerged and merged jet images is defined as $\Delta_{NN}$. By
examining the distribution of $\Delta_{NN}$ and its variations with features
that explore soft or collinear effects, the sensitivity of the CNN tagger to
IRC effects can be studied empirically. This can be seen in Figure 14, where
the 2D distribution of $\Delta_{NN}$ and the gluon relative transverse
momentum, and the $\Delta R$ to the parton, are shown. As either the gluon
relative momentum or the $\Delta R$ tend to zero, the $\Delta_{NN}$
distribution tends towards a sharp peak at 0, which would be indicative of the
CNN being insensitive to IRC perturbations.

(a)

(b)

Figure 14: $\Delta_{NN}$ versus (a) gluon relative transverse momentum and (b)
$\Delta R$ between the gluon on the nearest top decay parton. $\Delta_{NN}$ is
the difference between particle jet trained CNN output applied on parton level
jet images with and without merging the closest gluon with a top decay parton
[70]. Red points denote the point at which 90% of events within a vertical
slice of the distribution are contained.

####  4 Jet images in LHC Experiments

The ultimate tests for the efficacy of jet image based tagging approaches are
that the performance observed in phenomenological studies is also observed in
realistic high fidelity simulations and that their performance generalizes to
real data without large systematic uncertainties. With that in mind, jet image
base tagging approaches have been examined for quark vs gluon tagging in ATLAS
simulations [46] and in CMS Open Data [60] simulated samples [61], and for
boosted top quark jet tagging in CMS simulation and real data [47].

The ATLAS quark vs gluon jet image based CNN tagger [46] was trained using
fully simulated ATLAS events [71, 72]. Multi-channel jet images were used,
with one channel containing an image of the sum of measured charged particle
track $p_{T}$ per pixel. A second image for calorimeter measurements was
examined in two forms, a jet image containing either the transverse energy
measured in calorimeter towers of size $\Delta\eta\times\Delta\phi=0.1\times
0.1$ or a jet image containing a projection onto a fixed grid of topologically
clustered calorimeter cells (topo-clusters) [13]. Translation, rotation, and
normalization pre-processing was performed. A three layer CNN with filter
sizes of $5\times 5$, $5\times 5$, and $3\times 3$, respectively, and max
pooling after each convolutional layer was used. As can be seen in the ROC
curve in Figure 15(a), the CNN processing the track + tower jet images
outperforms other standard taggers for quark vs gluon tagging. Interestingly,
the standard tagger based on the combination of two jet substructure features
(number of charged particles and the jet width) outperforms the CNN approach
at low quark efficiency. This is likely due to the track image discretization
that may result in multiple tracks falling in the same pixel. As track
multiplicity is not stored in the images, this useful discriminating
information is lost for the CNN. In Figure 15(b), the impact on performance of
utilizing different jet image channels was examined, wherein utilizing only
calorimeter based jet images provides significantly less performance than
tagging using track and calorimeter images. In addition, topo-cluster based
images, which are formed by projecting the continuous topo-cluster direction
estimates into a discrete grid, are seen to have lower performance than tower
based images. This is likely due to the projection onto a fixed grid for use
in a CNN, as this may cause a loss of information about the spatial
distribution of energy within a topo-cluster and may result in the overlap of
several clusters in the same pixel. Moreover, it can be seen that the track +
calorimeter image approach does not reach the performance found when a CNN is
trained on a jet image formed from truth particles (i.e. without the impact of
detector smearing). It was noted in [46] that when comparing the performance
of a CNN trained on only track images to a CNN trained on only charged truth
particles, the observed performances were extremely similar. This similarity
is driven by the excellent charged particle track resolution, and further
indicates the difference between the track + calorimeter jet image based CNN
tagger and the truth particle based CNN tagger is driven by the low
resolution, and thus loss of information, of the calorimeter.

(a)

(b)

Figure 15: ROC curves for quark jet efficiency versus gluon jet rejection in
ATLAS fully simulated datasets showing comparisons of (a) jet image based CNN
taggers against jet width and number-of-tracks discriminants, and (b) of jet
image basd CNN taggers trained with different input images [46].

The CMS boosted top jet image based CNN tagger [47], denoted ImageTop, was
trained on fully simulated CMS events [71]. Multi-channel jet images with six
channels were built using particle flow (PF) objects found within an $R=0.8$
jet. Before pixelation, particle flow objects within the jet are pre-processed
using translation, rotation, flipping, and normalization. The six channels
were defined as the sum of PF candidate $p_{T}$ per pixel with one channel
containing all PF candidates, and one channel each for PF candidate flavor,
i.e. charged, neutral, photon, electron, and muon candidates. ImageTop was
based on the multi-Channel DeepTop algorithm [42], and comprises four
convolutional layers each using $4\times 4$ filter sizes and max pooling after
two consecutive convolutional layers, followed by four dense layers before
classification. To aid the classification of top quark decays containing
$b$-quarks, a $b$-tagging identification score [73] evaluated on subjets of
the large jet was also fed as input to the dense layers of the tagger before
classification. In addition to a baseline ImageTop, a mass decorrelated
version denoted ImageTop-MD was also trained, wherein the mass decorrelation
was performed by down-sampling the background quark and gluon jet samples to
have the same mass distribution as the sample of boosted top jets used for
training. In this way, the discriminating information from the jet mass is
removed to first order 444As the authors note, though this method is not
guaranteed to remove tagger mass dependence, it was found to work sufficiently
well in this case as the baseline tagger inputs were not observed to have a
strong correlation to mass..

The ROC curve showing the performance of the ImageTop model is seen in Figure
16(a). Several algorithms were compared to ImageTop, including several jet
substructure feature based taggers and a deep neural network, denoted DeepAK8,
based on processing PF candidates. ImageTop is seen to outperform all other
algorithms except DeepAK8, and generally the deep network based taggers are
found to significantly outperform other algorithms. Moreover, once mass
decorrelation is included, the ImageTop-MD is found to be the highest
performing mass-deccorrelated model. The smaller change in performance due to
mass decorrelation of ImageTop relative to other algorithms such as DeepAK8
may be due the the image preprocessing; images are both normalized and
“zoomed” using a Lorentz boost determined by the jet $p_{T}$ to increase
uniformity of jet images over the $p_{T}$ range. These steps can result in a
reduction of mass information in the images and thus a reduction of the
learned dependence of ImageTop on the mass. The mass spectrum for background
quark and gluon jets before (in grey) and after applying a 30% signal
efficiency tagging threshold for ImageTop and ImageTop-MD (in green) can be
seen in Figure 16(b). The decorrelation method greatly helped to preserve the
mass distribution and was not seen to significantly degrade performance, as
seen in the ROC curves of Figure 16(a).

(a)

(b)

Figure 16: Examination of the CMS ImageTop tagger [47] trained on fully
simulated CMS events in (a) ROC curves of the quark / gluon jet efficiency
versus boosted top jet tagging efficiency comparing several taggers and
showing the dominant performance of the deep neural net based taggers, and (b)
the impact of applying a threshold on tagger outputs to the background jet
mass distribution wherein the mass decorrelated taggers show significantly
less sculpting.

As noted early, one concern with jet image based approaches to jet tagging is
their potential dependence on pileup conditions. For a fixed ImageTop tagging
threshold giving an inclusive 30% top jet tagging efficiency, the variations
of the top jet tagging efficiency as a function of the number of primary
vertices in the event can be seen in Figure 17. Efficiency variations for both
ImageTop and ImageTop-MD were found to be small, at the level of less than 1%,
across the values of number of primary vertices. A similar level of stability
was observed for the background mis-identification rate. This stability draws
largely from the pileup mitigation applied to the jet before creating the jet
images, and this stability is not disturbed by the CNN discriminant.

Figure 17: Variations as a function of the number of reconstructed vertices in
an event of boosted top jet tagging efficiency after applying a fixed tagger
output threshold on the CMS ImageTop tagger [47], as well as several other
taggers, trained on fully simulated CMS events.

While the simulation based training of classifiers can lead to powerful
discriminants, differences in feature distributions between data and
simulation could cause the tagger to have differing performance between data
and simulation. As such, the discriminant is typically calibrated before
application in data. Calibration entails defining control samples of jets in
data where the tagging efficiency and mis-identification rate can be measured
in data and simulation. The efficiency of the tagger as a function of jet
$p_{T}$ is evaluated in data and simulation, and a $p_{T}$ dependent ratio of
efficiencies known as a Scale Factor (SF) is derived. This SF can then be used
to weight events such that the simulation trained tagger efficiency matches
the data. The SFs for the ImageTop signal efficiency were estimated in a
sample of single muon events selected to have a high purity of top-pair events
in the 1-lepton decay channel, while quark and gluon background mis-
identification rates were estimated in dijet samples and samples of photons
recoiling off of jets. Systematic uncertainties were evaluated on the data
based estimation of the tagging efficiency and propagated to SF uncertainties.
These systematic uncertainties included theory uncertainties in the parton
showering model, renormalization and factorization scales, parton distribution
functions, as well as experimental uncertainties on the jet energy scale and
resolution, $p_{T}^{\textrm{miss}}$ unclustered energy, trigger and lepton
identification, pileup modeling, and integrated luminosity, as well as
statistical uncertainties of simulated samples.

(a)

(b)

Figure 18: Calibration scale factors as a function of jet $p_{T}$ for (a) the
top jet tagging efficiency in single muon events, and (b) the quark / gluon
jet mistag efficiency in dijet events, for the CMS ImageTop tagger [47]
trained on fully simulated CMS events and calibrated to data.

The scale factors for ImageTop and ImageTop-MD for both the top tagging
efficiency and the background mis-identification rate can be found in Figure
18. The signal efficiency scale factors were largest at low momentum, showing
a departure from unity of around 10%, but were significantly closer to unity
in essentially all other $p_{T}$ ranges. The systematic uncertainties ranged
from approximately 5-10%, with the largest uncertainties at low $p_{T}$. The
scale factors for the mis-identification rate tended to be larger, up to a 20%
departure from unity in dijet samples but with smaller scale factors in the
photon+jet samples. These calibrations indicate that while some departures
from unity of the scale factors are observed, they are largely consistent with
observations from other taggers. The situation is similar in terms of the
scale factor uncertainties. As such, the jet image and CNN based tagging
approach can be seen to work well in data, without extremely large
calibrations and uncertainties, thus indicating its viability for use in
analysis.

###  6 Understanding jet Image based tagging

Interpretability and explainability are vital when applying ML methods to
physics analysis in order to ensure that (i) reasonable and physical
information is being used for discrimination rather than spurious features of
the data, and (ii) when training models on simulation, models are not highly
reliant on information that may be mismodeled with respect to real data.
Interpretability and explainability of deep neural networks is highly
challenging and is an active area of research within the ML community [74].
While a large number of techniques exist for examining CNNs, a subset of the
techniques from the ML community have been applied within the study of jet
images. A benefit of the computer vision approach to jet analysis is that
while the data input to ML models may be high dimensional, in this case with a
large number of pixels, they can be visualized on the image grid for
inspection and interpretation. Thus the tools for interpreting CNN models
applied to jet images tend to center on this aspect with tools such as pixel-
discriminant correlation maps, filter examination, and finding images that
maximally activate neurons.

Given a jet image $x$ with pixel values $\\{x_{ij}\\}$ and discriminant
$c(x)$, one can examine how changes to the input may effect the discriminant
prediction. Correlation maps examine Pearson correlation coefficients between
each pixel and the discriminant prediction, thus probing how each input
feature is correlated with increases and decreases in prediction over a sample
of inputs. For a sample of $N$ inputs, the correlation map is computed as
$\rho_{ij}=\frac{1}{\sigma_{x_{ij}}\sigma_{c}}\sum_{k=1}^{N}(x_{ij}^{(k)}-\bar{x}_{ij})(c(x^{(k)})-\bar{c})$,
where $\bar{x}_{ij}=\frac{1}{N}\sum_{k=1}^{N}x_{ij}^{(k)}$ and
$\bar{c}=\frac{1}{N}\sum_{k=1}^{N}c(x^{(k)})$ are the mean feature and
prediction values, while
$\sigma_{x_{ij}}=\frac{1}{N}\sum_{k=1}^{N}(x_{ij}^{(k)}-\bar{x}_{ij})^{2}$ and
$\sigma_{c}=\frac{1}{N}\sum_{k=1}^{N}(c(x^{(k)})-\bar{c})^{2}$ are the
variances of the feature and prediction values.

The filters of a CNN perform local feature matching and are applied directly
to the pixels of the image (or convolved image), and thus one may plot each
filter as an image and examine what features each filter is targeting. As
there can be a large number of filters at each CNN layer as well as a large
number of channels in layers deep within a CNN, this approach tends to be
easiest at the first layers of the CNN. In addition, rather than examining the
filters themselves, after processing an image by a CNN model, one may examine
the output of any given filter. This will produce a convolved image in which
the local feature matching has been applied at each position of the image and
will highlight the location of the image in which a given filter has become
active. In order to highlight difference in convolved images between classes,
the difference between average convolved image between two classes can
highlight relative differences in the spatial location of information relevant
for discrimination.

Maximally activating images or image patches correspond to applying a CNN
model on a large set of images and finding the images, or image patches, that
cause a given neuron to output a large activation. In the case of neurons in
the fully connected layers at the end of the network, this corresponds to full
images, whilst for neurons in convolutional layers this corresponds to image
patches in which the neuron is most active.

####  1 Probing CNNs

In Figure 19, the filters in the CNNs for $W$ tagging [23] and top tagging
[36] with jet images are examined. Several filters from the first
convolutional layer of the CNN for $W$ tagging are shown in the top row of
Figure 19(a), and the bottom row shows the corresponding difference between
the average convolved image resulting from applying each filter. While the
filters are not easy to interpret, one can see dark regions of the filters
corresponding to relative locations of large energy depositions in the jet
image as well as some intensity gradients that help identify regions where
additional radiation may be expected. After applying the filters to sets of
signal and background images and taking the difference of the average
convolutions to each sample, one can explore how each filter is finding
different information in signal and background-like images. The more signal-
like regions are shown in red while the more background-like regions are shown
in blue. The blue region at the centers identifies wider energy depositions at
the center of the jet image, whilst the signal-like regions at the bottom of
such images identify common locations for the subleading energy deposition.
There is a strong focus on identifying signal-like radiation between the
leading two energy depositions. Similarly for the DeepTop model, in Figure
19(b) one can see the convolved average image difference for several filters
at each layer of the model, where the rows correspond to layer depth from top
to bottom. We again see the tendency for the central region to be background-
like, whilst the signal-like regions correspond to different locations of the
subleading subjet and radiation between the two leading subjets. One can also
see broader radiation patterns which vary depending on the location of the
subleading subjet and attempt to identify likely locations of additional
subjets in the image.

(a)

(b)

Figure 19: (a) Filters from the first convolutional layer for boosted $W$
tagging with a jet image based CNN tagger [23] are shown in the top row, while
the bottom row shows the average difference between signal and background
convolved images from the corresponding filter in the top row. (b) The average
difference between signal and background convolved images for several filters
of the DeepTop jet image based CNN tagger for boosted top jets [36].

Figure 20 examines the average of the 500 images that lead to the highest
activation for each of several neurons in the last (dense) layer of the CNN
for $W$ tagging [23]. The fraction of signal jet images in this sample is also
noted, and the images are ordered left to right in terms of this signal
fraction. The neuron that activates predominantly on signal jet images has a
clear two prong structure and a tight core between these two prongs where
radiation is expected. The neuron activating predominantly on background jet
images shows a very different pattern, with a much broader central region
where energy may be found and a broad ring around the central region where
additional wide angle radiation may be present. These features are in
agreement with the known physics of such jets.

Figure 20: The average jet image which most activates a neural in the final
layer of a jet image based CNN for boosted $W$ taggeing [23]. The fraction of
signal events for each neuron is noted, thus indicating if the neuron was most
activated by signal or background-like images.

For a more global view of what the discriminant has learned, one can examine
the correlation maps for the CNN $W$ tagging [23] and the DeepTop model [36]
using full preprocessing in Figure 21. Structurally they are quite
similar555The relative location of the second subjet was rotated to be below
the leading subjet in the case of $W$-tagging and above the leading subjet for
DeepTop which leads to the apparent flip in the correlation images over the
horizontal axis., however the the regions of signal (red) and background
(blue) correlation appear inverted. For $W$ tagging, the location of the
subleading subjet at the bottom of the image is a strong indicator of signal
owing to the fact that $W$ jets have a two particle decay structure which
strongly restricts the relative location of the two subjets for a fixed jet
$p_{T}$. This relative location is not as strict in quark/gluon jets and may
vary due to additional radiation. The region around the central core of the
jet is correlated with background-like images where additional radiation may
be found. For top tagging, a strong energy deposition above the central
leading energy deposition as well as addition energy depositions, i.e. the
third expected subjet in a top quark decay, are correlated with signal-like
images. This correlation pattern indicates that the discriminant relies
heavily on the identification of the third subjet, as would be expected in a
top quark decay.

(a)

(b)

Figure 21: Correlation images showing the Pearson linear correlation
coefficient per pixel between jet image pixels and a jet image based CNN
tagger output for (a) boosted $W$ tagging [23], and (b) boosted top tagging
with DeepTop [36].

###  7 Other Applications of Jet images

In addition to classification tasks, the approach of using jet images and
convolutional layers for processing have also been explored in several other
data analysis challenges. We briefly examine some of these applications,
showing how this computer vision to jet analysis can be powerful in a variety
of settings.

####  1 Jet Energy Regression and Pileup removal

Among the major challenges facing analyses utilizing jets at high luminosity
hadron colliders is the presence of pileup, or interactions occurring in the
same bunch crossing as the primary hard scattering. Pileup interactions lead
to additional particles which may fall within the catchment area of a jet and
thus are effectively “noise” in the estimation of jet properties. A variety of
techniques have been proposed for pileup mitigation in jets [17] ranging from
subtracting an average pileup energy density from a jet to techniques
targeting the classification of each particle in a jet as pileup or from the
hard scatter.

Within the paradigm of jet images, one approach to pileup mitigation is to
predict the per pixel pileup contributions, as is done in the PUMML method
[75]. In this technique, a jet can be considered as composed of four
components, the charged and neutral hard scatter contributions and the charged
and neutral pileup contributions. While the charged components of the hard
scatter and pileup are known from charged particle tracking measurements, the
neutral hard scatter and pileup components are only observed together in
calorimeter measurements. PUMML performs a per pixel regression of the neutral
component of the hard scatter contributions to the jet. A multi-channel jet
image was used as input, with one channel for each the hard scatter and pileup
charged components of the jet, and one channel for the combined neutral
component. As the charged contribution measured by tracking detectors has
significantly better resolution than the neutral component, a significantly
smaller pixel size of $\Delta\eta\times\Delta\phi=0.025\times 0.025$ was used
for the charged images than the $\Delta\eta\times\Delta\phi=0.1\times 0.1$
pixels sizes used for the calorimeter images. Upsampling was then used to
create a finer pixel image that matches the resolution of the charged
component. These three channel images were then processed by a three layer CNN
with a per pixel output prediction of the neutral hard scatter component of
the jet.

The hard scatter neutral component prediction was combined with the known
charged component to estimate jet properties and examine the efficacy of the
method. In phenomenological studies using simulated dijet events produced from
the decay of a hypothetical new resonance and with an average of 140
additional pileup interactions, the distributions of jet momentum and mass
before and after pileup mitigation from PUMML and other methods were compared,
as shown in Figure 22. In terms of momentum prediction, comparing the pileup
corrected distributions to the true distribution showed that all methods
produced predictions of similar quality, though PUMMPL was seen to have lower
per-jet reconstruction error. In terms of jet mass distribution prediction,
PUMML was seen to better replicate the underlying true jet mass distribution
over other techniques. While not yet applied in an experiment setting, similar
ideas applied to pileup reduction for missing energy estimation have been
explored on ATLAS in fully simulated events [76] and have shown promising
initial results.

(a)

(b)

Figure 22: The impact of pileup mitigation on (a) jet $p_{T}$, and (b) jet
mass, for various mitigation techniques including the jet image based PUMML
algorithm [75].

####  2 Generative Models with Jet Images

Among the earliest work applying deep generative models as approximations for
HEP high fidelity simulators made use of jet images as the data representation
[28]. The aim of this work was to learn the structure of jet images as they
may appear in a calorimeter and subsequently draw sample jets from the learned
generative model. As a neural generative model can be significantly faster
than running a high fidelity simulator, such approaches have the potential to
significantly reduce the large simulation times in HEP. In the
phenomenological studies of reference [28], a generative adversarial network
(GAN) setup was used to train a generative model to transform samples from a
standard normal distribution into samples of jet images, whilst a second
discriminator network was used to penalize the generative model if it could
discriminate between real and generated jet images. Locally connected layers
as well as convolutional layers were investigated for use in the networks. The
distribution of $p_{T}$ for $W$ boson jets and of quark/gluon jets were
compared between the Pythia simulator [67, 68] and the GAN generated images,
as seen in Figure 23(a). Figure 23(b) shows a set of Pythia simulated jet
images in the top row and their nearest neighbor GAN generated jet images in
the bottom row. Both the distribution of jet properties and the general
structure of jet images were reasonably well produced by the GAN approach.
While not yet reaching the fidelity of HEP simulators, this early work in HEP
data generation showed the potential utility in examining fast approximation
simulators from deep generative models for HEP.

(a)

(b)

Figure 23: (a) The jet image $p_{T}$ for $W$ boson jet and quark/gluon jets
comparing the GAN generated distribution to the Pythia simulated distribution
[28]. (b) A visual comparison of Pythia simulated (top) and the nearest GAN
generated (bottom) jet images.

####  3 Anomaly Detection

The use of CNNs to process jet images provides a powerful scheme to learn
useful representations of the information contained within a jet. In typical
classification tasks, these representations are used for discriminating
classes of jets. However, when searching for signs of new physics, one may not
know _a priori_ the properties of such a new signal but only that such a
signal would have properties that deviate from known Standard Model processes.
Such anomaly detection tasks are challenging due to the lack of signal
knowledge and thus the inability to use standard classifiers for this task.
Within the context of a search for jets produced by new particles, recent work
has combined the power of CNN representation learning on jet images with
autoencoder network architectures [77, 78] to search for anomalous jets [79,
80].

Autoencoder models are designed to map an input to a compressed latent
representation through an “encoder”, and then decompress the latent
representation back to original input via a “decoder”. Such models are trained
to minimize the “reconstruction error” computed as the MSE between the
original input and the autoencoder output. The reconstruction error can be
used to identify inputs that are not well adapted for the compression and
reconstruction scheme learned by the autoencoder. When used for anomaly
detection, autoencoders are trained to compress and reconstruct one class of
events. Under the assumption that this compression and reconstruction scheme
would not be well adapted for inputs from classes different from the training
sample, the reconstruction for inputs from new classes is expected to perform
poorly and thus lead to a large reconstruction error.

When applied to searches for anomalous jets in phenomenological studies, jet
images have been examined as the data representation, and convolutional layers
combined with max pooling and with upsampling have been used for the encoder
and decoder respectively. In this case, the autoencoder is trained on a
background sample of standard quark and gluon jets, and the ability to
identify different signal jets was examined. The reconstruction error was used
directly to search for excesses of events, as seen in Figure 24 where the
signal was either a sample of top quark jets or jets from a hypothetical new
gluino particle. The distribution of the reconstruction error shows a large
separation from the background, denoted QCD, and the potential signal jets.
The ROC curve for identifying top jets, produced by scanning a threshold on
the reconstruction error, is also shown and compares the CNN based autoencoder
with dense architecture based autoencoder applied to a flattened vector of
pixel $p_{T}$’s (denoted Dense), principle components analysis, and applying a
threshold only on the jet mass. The jet image+CNN architecture approach
dominated the other methods. However, it should be noted that this domination
was not seen for gluino jets.

(a)

(b)

Figure 24: (a) The distribution of autoencoder reconstruction error trained on
quarks / gluon jets, showing potential top or gluino signal distributions. (b)
ROC curves for identifying boosted top jets from quarks / gluon jets using
autoencoders with various architectures, wherein the jet image based CNN is
shown to outperformance other methods [80].

One challenge with autoencoder approaches for anomalous jet searches is the
possibility that the autoencoder reconstruction quality is dependent on the
jet mass. In this case the signal identification efficiency could be mass
dependent. Moreover, if a bump hunt analysis in the jet mass spectrum is
subsequently performed, such a reconstruction error correlation with mass
could disturb the jet mass distribution and render the bump hunt strategy
infeasible. To overcome such a challenge, an adversarial approach was
investigated in reference [79], wherein a second network is simultaneously
trained with the autoencoder to predict the jet mass from the autoencoder
output whilst the autoencoder is penalized during training if the second
network is successful. The resulting adversarial autoencoder performance for
identifying a top jet signal can be see in Figure 25. With the adversary in
use, the jet mass distribution was kept relatively stable even when applying a
threshold on the reconstruction error which only permits a 5% background jet
false positive rate. However, as seen in the ROC curve, increasing the
strength $\lambda$ of the adversarial penalty on the autoencoder could
significantly decrease the top jet signal sensitivity.

(a)

(b)

Figure 25: (a) The jet mass distribution after applying a threshold allowing
3% or 5% quark / gluon jet background efficiency on the jet image based
adversarial autoencoder. The background is largely unsculpted and the top jet
peaks can be clearly seen [79]. (b) ROC curves for quark / gluon jet rejection
versus top jet efficiency for jet image based adversarial autoencoders with
varying strength of adversarial penalty during training [79].

###  8 Conclusion

The representation of jets as images has proven highly useful for connecting
the fields of high energy physics and machine learning. Through this
connection, advanced methods in deep learning and computer vision, primarily
with convolutional neural network architectures, have been applied to the
challenges of jet physics and have shown promising performance both in
phenomenological studies and in experiments at the LHC. Jet images have seen a
broad set of use cases, not only for jet classification but also for energy
regression, pileup noise removal, data generation, and anomaly detection.
Image based jet tagging remains an active area of research and broad classes
of state of the art deep neural network architectures for computer vision are
being explored within the field of high energy physics.

While much of the work presented in this text has been in phenomenological
studies using particle level simulations, there remains open question on the
applicability of these methods on high-fidelity simulated data and in real
experimental data. In more realistic settings, the complexity of the detector
and the data-taking conditions, and the challenges of the differences between
simulated and real data will be key challenges for understanding and
optimizing these models. Understanding the relationships in realistic data
between model accuracy and calibration error, and model complexity /
structural assumptions and sensitivity to systematic uncertainties, will be
important for the long-term efficacy of these image-based methods.
Nonetheless, initial results from both ATLAS and CMS have shown promise,
pointing towards the exciting potential for jet imaging in the future.

### Acknowledgments

This work was supported by the US Department of Energy (DOE) under grant DE-
AC02-76SF00515, and by the SLAC Panofsky Fellowship.

## References

  * [1] J. Cogan, M. Kagan, E. A. Strauss, and A. Schwarztman, Jet-images: computer vision inspired techniques for jet tagging, Journal of High Energy Physics 2015 (2015) 1.
  * [2] S. Marzani, G. Soyez, and M. Spannowsky, Looking inside jets: an introduction to jet substructure and boosted-object phenomenology.  Springer, Cham, Switzerland, 2019.
  * [3] J. Shelton, Jet Substructure, pp. 303–340.  World Scientific, 2013.
  * [4] L. Evans and P. Bryant, LHC Machine, Journal of Instrumentation 3 (aug, 2008) S08001.
  * [5] A. Andreassen, I. Feige, C. Frye, and M. D. Schwartz, JUNIPR: a framework for unsupervised machine learning in particle physics, The European Physical Journal C 79 (2019) 2, 102.
  * [6] ATLAS Collaboration, Deep Sets based Neural Networks for Impact Parameter Flavour Tagging in ATLAS, Tech. Rep. ATL-PHYS-PUB-2020-014, CERN, Geneva, May, 2020.  https://cds.cern.ch/record/2718948.
  * [7] G. Louppe, K. Cho, C. Becot, and K. Cranmer, QCD-aware recursive neural networks for jet physics, Journal of High Energy Physics 2019 (2019) 1, 57.
  * [8] P. T. Komiske, E. M. Metodiev, and J. Thaler, Energy flow networks: deep sets for particle jets, Journal of High Energy Physics 2019 (2019) 1, 121.
  * [9] H. Qu and L. Gouskos, Jet tagging via particle clouds, Physical Review D 101 (Mar, 2020) 056019.
  * [10] G. Kasieczka, T. Plehn, A. Butter, K. Cranmer, D. Debnath, B. M. Dillon, M. Fairbairn, D. A. Faroughy, W. Fedorko, C. Gay, and et al., The Machine Learning landscape of top taggers, SciPost Physics 7 (Jul, 2019) .
  * [11] B. Graham and L. van der Maaten, Submanifold Sparse Convolutional Networks, arXiv:1706.01307 [cs.NE].
  * [12] M. Cacciari, G. P. Salam, and G. Soyez, The anti-$k_{t}$ jet clustering algorithm, Journal of High Energy Physics 2008 (Apr, 2008) 063.
  * [13] ATLAS Collaboration, Topological cell clustering in the ATLAS calorimeters and its performance in LHC Run 1, The European Physical Journal C 77 (2017) 7, 490.
  * [14] CMS Collaboration, Particle-flow reconstruction and global event description with the CMS detector, Journal of Instrumentation 12 (Oct., 2017) P10003, arXiv:1706.04965 [physics.ins-det].
  * [15] R. Kogler et al., Jet Substructure at the Large Hadron Collider: Experimental Review, Reviews of Modern Physics 91 (2019) 4, 045003, arXiv:1803.06991 [hep-ex].
  * [16] A. J. Larkoski, I. Moult, and B. Nachman, Jet Substructure at the Large Hadron Collider: A Review of Recent Advances in Theory and Machine Learning, Physics Reports 841 (2020) 1, arXiv:1709.04464 [hep-ph].
  * [17] G. Soyez, Pileup mitigation at the LHC: A theorist’s view, Physics Reports 803 (Apr, 2019) 1.
  * [18] D. Krohn, J. Thaler, and L.-T. Wang, Jet trimming, Journal of High Energy Physics 2010 (Feb, 2010) .
  * [19] D. Bertolini, P. Harris, M. Low, and N. Tran, Pileup per particle identification, Journal of High Energy Physics 2014 (Oct, 2014) .
  * [20] Particle Data Group, Review of Particle Physics, Progress of Theoretical and Experimental Physics 2020 (08, 2020) .
  * [21] M. Cacciari, G. P. Salam, and G. Soyez, The catchment area of jets, Journal of High Energy Physics 2008 (Apr, 2008) .
  * [22] J. Pearkes, W. Fedorko, A. Lister, and C. Gay, Jet Constituents for Deep Neural Network Based Top Quark Tagging, arXiv:1704.02124 [hep-ex].
  * [23] L. de Oliveira, M. Kagan, L. Mackey, B. Nachman, and A. G. Schwartzman, Jet-images – deep learning edition, Journal of High Energy Physics 2016 (2016) 1.
  * [24] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, Communications of the ACM 60 (May, 2017) 84.
  * [25] Yann LeCun, Generalization and network design strategies, Tech. Rep. CRG-TR-89-4, University of Toronto, 1989.  http://yann.lecun.com/exdb/publis/pdf/lecun-89.pdf.
  * [26] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [27] P. Baldi, K. Bauer, C. Eng, P. Sadowski, and D. Whiteson, Jet substructure classification in high-energy physics with deep neural networks, Physical Review D 93 (May, 2016) 094034.
  * [28] L. de Oliveira, M. Paganini, and B. Nachman, Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis, Computing and Software for Big Science 1 (2017) 1.
  * [29] D. Scherer, A. Müller, and S. Behnke, Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition,  in Artificial Neural Networks – ICANN 2010, K. Diamantaras, W. Duch, and L. S. Iliadis, eds.  Springer Berlin Heidelberg, Berlin, Heidelberg, 2010.
  * [30] S. Ioffe and C. Szegedy, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, in Proceedings of the 32nd International Conference on International Conference on Machine Learning, F. Bach and D. Blei, eds.  Proceedings of Machine Learning Research, Lille, France, 07–09 Jul, 2015\.  http://proceedings.mlr.press/v37/ioffe15.html.
  * [31] X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, in Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Y. W. Teh and M. Titterington, eds.  Journal of Machine Learning Research Workshop and Conference Proceedings, Chia Laguna Resort, Sardinia, Italy, 13–15 May, 2010.  http://proceedings.mlr.press/v9/glorot10a.html.
  * [32] K. He, X. Zhang, S. Ren, and J. Sun, Deep Residual Learning for Image Recognition,  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).  2016\.
  * [33] L. Bottou, F. E. Curtis, and J. Nocedal, Optimization Methods for Large-Scale Machine Learning, SIAM Review 60 (2016) 2, 223, arXiv:1606.04838 [stat.ML].
  * [34] D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, in International Conference for Learning Representations.  2015\.  arXiv:1412.6980 [cs.LG].
  * [35] J. Gallicchio, J. Huth, M. Kagan, M. D. Schwartz, K. Black, and B. Tweedie, Multivariate discrimination and the Higgs+W/Z search, Journal of High Energy Physics 2011 (Apr., 2011) 69, arXiv:1010.3698 [hep-ph].
  * [36] G. Kasieczka, T. Plehn, M. C. Russell, and T. Schell, Deep-learning top taggers or the end of QCD?, Journal of High Energy Physics 2017 (2017) 1.
  * [37] S. Diefenbacher, H. Frost, G. Kasieczka, T. Plehn, and J. M. Thompson, CapsNets Continuing the Convolutional Quest, arXiv:1906.11265 [hep-ph].
  * [38] S. Bollweg, M. Haussmann, G. Kasieczka, M. Luchmann, T. Plehn, and J. Thompson, Deep-Learning Jets with Uncertainties and More, SciPost Physics 8 (2020) 6.
  * [39] P. T. Komiske, E. M. Metodiev, and M. D. Schwartz, Deep learning in color: towards automated quark/gluon jet discrimination, Journal of High Energy Physics 2017 (2017) 1.
  * [40] Y.-T. Chien and R. Kunnawalkam Elayavalli, Probing heavy ion collisions using quark and gluon jet substructure, arXiv:1803.03589 [hep-ph].
  * [41] K. Fraser and M. D. Schwartz, Jet charge and machine learning, Journal of High Energy Physics 2018 (2018) 10, 93.
  * [42] S. Macaluso and D. Shih, Pulling out all the tops with computer vision and deep learning, Journal of High Energy Physics (2018) 10, 121.
  * [43] Y.-C. J. Chen, C.-W. Chiang, G. Cottin, and D. Shih, Boosted $w$ and $z$ tagging with jet charge and deep learning, Phys. Rev. D 101 (Mar, 2020) 053001. https://link.aps.org/doi/10.1103/PhysRevD.101.053001.
  * [44] J. Lin, M. Freytsis, I. Moult, and B. Nachman, Boosting H$\to b\bar{b}$ with machine learning, Journal of High Energy Physics 2018 (Oct, 2018) 101, arXiv:1807.10768 [hep-ph].
  * [45] J. H. Kim, M. Kim, K. Kong, K. T. Matchev, and M. Park, Portraying double Higgs at the Large Hadron Collider, Journal of High Energy Physics 2019 (Sep, 2019) 47, arXiv:1904.08549 [hep-ph].
  * [46] ATLAS Collaboration, Quark versus Gluon Jet Tagging Using Jet Images with the ATLAS Detector, Tech. Rep. ATL-PHYS-PUB-2017-017, CERN, Geneva, Jul, 2017.  https://cds.cern.ch/record/2275641.
  * [47] CMS Collaboration, Identification of heavy, energetic, hadronically decaying particles using machine-learning techniques, Journal of Instrumentation 15 (2020) 06, P06005, arXiv:2004.08262 [hep-ex].
  * [48] J. Snoek, H. Larochelle, and R. P. Adams, Practical Bayesian Optimization of Machine Learning Algorithms, in Proceedings of the 25th International Conference on Neural Information Processing Systems.  2012\.  {https://papers.nips.cc/paper/2012/hash/05311655a15b75fab86956663e1819cd-Abstract.html}.
  * [49] J. Thaler and K. Van Tilburg, Identifying boosted objects with N-subjettiness, Journal of High Energy Physics 2011 (Mar, 2011) .
  * [50] A. J. Larkoski, I. Moult, and D. Neill, Power counting to better jet observables, Journal of High Energy Physics 2014 (Dec, 2014) .
  * [51] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction.  Springer-Verlag, New York, NY, 2 ed., 2009.
  * [52] M. D. Zeiler, ADADELTA: An Adaptive Learning Rate Method, arXiv:1212.5701 [cs.LG].
  * [53] S. Xie, R. B. Girshick, P. Dollár, Z. Tu, and K. He, Aggregated Residual Transformations for Deep Neural Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) 5987.
  * [54] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, On Calibration of Modern Neural Networks, in Proceedings of the 34th International Conference on Machine Learning, D. Precup and Y. W. Teh, eds.  Proceedings of Machine Learning Research, International Convention Centre, Sydney, Australia, 06–11 Aug, 2017.  http://proceedings.mlr.press/v70/guo17a.html.
  * [55] ATLAS Collaboration, Performance of mass-decorrelated jet substructure observables for hadronic two-body decay tagging in ATLAS, Tech. Rep. ATL-PHYS-PUB-2018-014, CERN, Geneva, Jul, 2018.  https://cds.cern.ch/record/2630973.
  * [56] Bradshaw, Layne and Mishra, Rashmish K. and Mitridate, Andrea and Ostdiek, Bryan, Mass Agnostic Jet Taggers, SciPost Physics 8 (2020) 1, 011, arXiv:1908.08959 [hep-ph].
  * [57] G. Louppe, M. Kagan, and K. Cranmer, Learning to Pivot with Adversarial Networks, in Proceedings of the 31st International Conference on Neural Information Processing Systems.  2017\.  {https://papers.nips.cc/paper/2017/hash/48ab2f9b45957ab574cf005eb8a76760-Abstract.html}.
  * [58] Shimmin, Chase and Sadowski, Peter and Baldi, Pierre and Weik, Edison and Whiteson, Daniel and Goul, Edward and Søgaard, Andreas, Decorrelated Jet Substructure Tagging using Adversarial Neural Networks, Physical Review D 96 (2017) 7, 074034, arXiv:1703.03507 [hep-ex].
  * [59] G. Kasieczka and D. Shih, DisCo Fever: Robust Networks Through Distance Correlation, arXiv:2001.05310 [hep-ph].
  * [60] CERN, “CERN Open Data Portal.” Http://opendata.cern.ch/, 2017.
  * [61] M. Andrews, J. Alison, S. An, B. Burkle, S. Gleyzer, M. Narain, M. Paulini, B. Poczos, and E. Usai, End-to-end jet classification of quarks and gluons with the CMS Open Data, Nuclear Instruments and Methods in Physics Research A 977 (Oct, 2020) 164304.
  * [62] J. Gallicchio and M. D. Schwartz, Quark and gluon jet substructure, Journal of High Energy Physics 2013 (2013) 1.
  * [63] R. Field and R. Feynman, A parametrization of the properties of quark jets, Nuclear Physics B 136 (1978) 1, 1 .
  * [64] D. Krohn, M. D. Schwartz, T. Lin, and W. J. Waalewijn, Jet Charge at the LHC, Physical Review Letters 110 (May, 2013) 212001.
  * [65] W. J. Waalewijn, Calculating the charge of a jet, Physical Review D 86 (Nov, 2012) 094030.
  * [66] J. Barnard, E. N. Dawe, M. J. Dolan, and N. Rajcic, Parton shower uncertainties in jet substructure analyses with deep neural networks, Physical Review D 95 (Jan, 2017) 014018.
  * [67] T. Sjöstrand, S. Mrenna, and P. Skands, A brief introduction to PYTHIA 8.1, Computer Physics Communications 178 (2008) 11, 852 .
  * [68] T. Sjöstrand, S. Mrenna, and P. Skands, PYTHIA 6.4 physics and manual, Journal of High Energy Physics 2006 (May, 2006) 026.
  * [69] J. Bellm, S. Gieseke, D. Grellscheid, A. Papaefstathiou, S. Platzer, P. Richardson, C. Rohr, T. Schuh, M. H. Seymour, A. Siodmok, A. Wilcock, and B. Zimmermann, Herwig++ 2.7 Release Note, arXiv:1310.6877 [hep-ph].
  * [70] S. Choi, S. J. Lee, and M. Perelstein, Infrared safety of a neural-net top tagging algorithm, Journal of High Energy Physics 2019 (2018) 1.
  * [71] Geant4 Collaboration, Geant4 simulation toolkit, Nuclear Instruments and Methods in Physics Research A 506 (2003) 3, 250.
  * [72] ATLAS Collaboration, The ATLAS Simulation Infrastructure, The European Physical Journal C 70 (2010) 3, 823.
  * [73] CMS Collaboration, Performance of $b$-tagging algorithms in proton-proton collisions at 13 TeV with Phase 1 CMS detector, Tech. Rep. CMS-DP-2018-033, CERN, Jun, 2018.  https://cds.cern.ch/record/2627468.
  * [74] U. Bhatt, A. Xiang, S. Sharma, A. Weller, A. Taly, Y. Jia, J. Ghosh, R. Puri, J. M. F. Moura, and P. Eckersley, Explainable machine learning in deployment, Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (2020) .
  * [75] P. T. Komiske, E. M. Metodiev, B. Nachman, and M. D. Schwartz, Pileup Mitigation with Machine Learning (PUMML), Journal of High Energy Physics 2017 (2017) 1.
  * [76] ATLAS Collaboration, A. Collaboration, Convolutional Neural Networks with Event Images for Pileup Mitigation with the ATLAS Detector, Tech. Rep. ATL-PHYS-PUB-2019-028, CERN, Geneva, Jul, 2019.  https://cds.cern.ch/record/2684070.
  * [77] P. Baldi and K. Hornik, Neural networks and principal component analysis: Learning from examples without local minima, Neural Networks 2 (1989) 1, 53 .
  * [78] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, Extracting and Composing Robust Features with Denoising Autoencoders,  in Proceedings of the 25th International Conference on Machine Learning, International Conference on Machine Learning ’08.  Association for Computing Machinery, New York, NY, USA, 2008.
  * [79] T. Heimel, G. Kasieczka, T. Plehn, and J. Thompson, QCD or what?, SciPost Physics 6 (Mar, 2019) .
  * [80] M. Farina, Y. Nakai, and D. Shih, Searching for new physics with deep autoencoders, Physical Review D 101 (Apr, 2020) 075021.