#  VBF vs. GGF Higgs with Full-Event Deep Learning:
Towards a Decay-Agnostic Tagger

Cheng-Wei Chiang<EMAIL_ADDRESS>Department of Physics, National
Taiwan University, Taipei, Taiwan 10617, ROC  Physics Division, National
Center for Theoretical Sciences, Taipei, Taiwan 10617, ROC  David Shih
<EMAIL_ADDRESS>NHETC, Department of Physics and Astronomy, Rutgers
University, NJ 08854, USA  Shang-Fu Wei<EMAIL_ADDRESS>Department of
Physics, National Taiwan University, Taipei, Taiwan 10617, ROC

###### Abstract

We study the benefits of jet- and event-level deep learning methods in
distinguishing vector boson fusion (VBF) from gluon-gluon fusion (GGF) Higgs
production at the LHC. We show that a variety of classifiers (CNNs, attention-
based networks) trained on the complete low-level inputs of the full event
achieve significant performance gains over shallow machine learning methods
(BDTs) trained on jet kinematics and jet shapes, and we elucidate the reasons
for these performance gains. Finally, we take initial steps towards the
possibility of a VBF vs. GGF tagger that is agnostic to the Higgs decay mode,
by demonstrating that the performance of our event-level CNN does not change
when the Higgs decay products are removed. These results highlight the
potentially powerful benefits of event-level deep learning at the LHC.

##  I Introduction

The discovery Aad  _et al._ (2012); Chatrchyan  _et al._ (2012) of the Higgs
boson in 2012 was a monumental occasion, providing a capstone to decades of
experimental and theoretical works in particle physics, and confirming the
final missing piece of the Standard Model (SM).

Since the original discovery, much effort Cepeda  _et al._ (2019); Grazzini
(2019) has been devoted to measuring ever more precisely the couplings of the
Higgs boson to other SM particles. Since the Higgs has numerous production
modes and decay modes, measurements in many different final states are
necessary to disentangle all the various effects and pin down the Higgs
couplings to all the SM fields ATL (2020a); Aad  _et al._ (2020a); ATL (2020b,
2021); Sirunyan  _et al._ (2021a); CMS (2022a); Sirunyan  _et al._ (2021b). A
key component of this program is distinguishing the vector boson fusion (VBF)
production mode from other production modes, most predominantly gluon-gluon
fusion (GGF). VBF is essential for measuring the Higgs couplings to the SM
$W/Z$ gauge bosons, thereby testing the most essential property of the Higgs,
namely its role in electroweak symmetry breaking (EWSB).

Previous works Chan  _et al._ (2017); Chung  _et al._ (2020) have studied the
question of VBF vs GGF classification with machine learning methods. The main
thing that distinguishes VBF from GGF events is that VBF events come with two
forward quark-initiated jets from the hard process, while GGF jets are going
to be from ISR and will tend to be gluon-initiated. In Ref. Chan  _et al._
(2017), boosted decision trees (BDTs) trained on high-level physics variables
such as invariant mass and rapidity difference of the leading jets, sum of
transverse momenta of the Higgs decay products, and various jet shape
variables were brought to bear on the question of VBF vs. GGF classification,
in the context of $H\to\gamma\gamma$ and $H\to WW^{*}$ final states
specifically. Meanwhile, Ref. Chung  _et al._ (2020) studied the multiclass
classification of multiple Higgs production modes (including VBF and GGF) in
the boosted $H\to bb$ regime, considering BDTs trained on high-level features,
as well as a specialized two-stream convolutional neural network (CNN), which
was previously developed for boosted $H\to bb$ tagging Lin  _et al._ (2018),
and was trained on event images made out of low-level inputs (the pixelated
$p_{T}$’s of all the particles in the event).

Experimental studies ATL (2021, 2020b, 2020a); Aad  _et al._ (2020a, 2021a,
2021b); Sirunyan  _et al._ (2021a); CMS (2022b, a); Sirunyan  _et al._ (2020a)
have also used BDTs or dense neural networks (DNNs) on a variety of Higgs
decay modes to discriminate VBF from GGF events, while other techniques such
as recurrent neural networks (RNNs) were also found useful in practice Aad
_et al._ (2020a). The BDTs, DNNs, RNNs used by the experimental groups take
the high-level features as input.

In this work we will revisit the question of VBF vs GGF event-level
classification, exploring the benefits that machine learning methods (both
shallow and deep) can bring to this problem. Our starting point will be a BDT
trained on high-level features (HLFs) defined from the leading two jets and
the Higgs decay products; this baseline method is designed to characterize the
previous state of the art from Chan  _et al._ (2017) and from the actual ATLAS
and CMS analyses. To go beyond, we consider the following methods:

  * •

Training a jet-level CNN to distinguish the leading two jets from VBF from
their GGF counterparts, and adding the jet-CNN scores to the inputs of the HLF
BDT.

  * •

Training an event-level CNN to distinguish full VBF events from full GGF
events; we make full-event images out of the energy deposits of all the
reconstructed particles in the event.

  * •

Training an event-level network based on the self-attention mechanism Lin  _et
al._ (2017); Vaswani  _et al._ (2017) as an interesting alternative to the
event-level CNN. In such a self-attention model, we convert the input event
into a sequence which directly records the detector-level information.

We will see that while augmenting the HLFs with the jet-CNN scores offers some
gain in classification performance, a much bigger boost comes from the event-
level classifiers trained on low-level inputs. We investigate the reasons for
the performance gains of the event-level CNN and find it is due in part to
additional hadronic activity beyond the leading two jets. Interestingly, this
includes both additional jet activity, as well as unclustered hadronic
activity in the event (i.e., hadronic activity that leads to softer jets below
the jet $p_{T}$ threshold). The pattern of soft radiation is different in VBF
vs. GGF events, again presumably due to differing quark vs. gluon content in
the initial states.

In this paper we will also highlight an added benefit of event-level
classifiers trained on low-level inputs: they can be Higgs decay mode
agnostic. Since the Higgs is a color singlet, the Higgs decay should be fairly
well factorized from the VBF or GGF initial state jets, especially when it
decays to electroweak states. Besides, the $p_{T}$-balance of the full event
ensures that the kinematics of the Higgs can be well-reconstructed from all
the other final state objects. Using the diphoton mode as an explicit example,
we will show that as long as our models take the whole event into account,
adding information from the Higgs decay does not improve the performance of
the classifier. This raises the possibility that a single VBF vs. GGF
classifier could be trained and deployed in a variety of Higgs analyses with
different final states, with no loss in performance.

Much work in the literature has focused on boosted jet classification Pumplin
(1991); Cogan  _et al._ (2015); Almeida  _et al._ (2015); de Oliveira  _et
al._ (2016); Baldi  _et al._ (2016); Komiske  _et al._ (2017); Kagan  _et al._
(2016); Guest  _et al._ (2016); Barnard  _et al._ (2017); Komiske  _et al._
(2018a); ATL (2017a); Pearkes  _et al._ (2017); Kasieczka  _et al._ (2017);
Datta and Larkoski (2017); Butter  _et al._ (2018); Datta and Larkoski (2018);
Egan  _et al._ (2017); Schramm (2018); Louppe  _et al._ (2019); Cheng (2018);
Sirunyan  _et al._ (2018); Komiske  _et al._ (2018b); Choi  _et al._ (2019);
Macaluso and Shih (2018); Komiske  _et al._ (2019); Kasieczka  _et al._
(2019); Dreyer  _et al._ (2018); Fraser and Schwartz (2018); Lin  _et al._
(2018); Chen  _et al._ (2020); Datta  _et al._ (2019); Qu and Gouskos (2020);
Chakraborty  _et al._ (2019); Lee  _et al._ (2019a, b); Diefenbacher  _et al._
(2020); Moreno  _et al._ (2020a); Andreassen  _et al._ (2019); Moreno  _et
al._ (2020b); Erdmann (2020); Li  _et al._ (2021); Bols  _et al._ (2020);
Chakraborty  _et al._ (2020); Bernreuther  _et al._ (2021); Lim and Nojiri
(2022); Guo  _et al._ (2021); Dolan and Ore (2021); Mikuni and Canelli (2020);
Li and Sun (2020); Kagan (2020); Erdmann  _et al._ (2021); Dreyer and Qu
(2021); Nakai  _et al._ (2020); Bhattacharya  _et al._ (2022); Sirunyan  _et
al._ (2020b); Andrews  _et al._ (2021); Filipek  _et al._ (2021); Mikuni and
Canelli (2021); Konar  _et al._ (2022); Shimmin (2021); Dreyer  _et al._
(2021); Aguilar-Saavedra (2021); Khosa and Marzani (2021); Gong  _et al._
(2022); Kim and Martin (2021); Qu  _et al._ (2022); ATL (2017b, 2020c), but
relatively less work has been done on event-level classification Louppe  _et
al._ (2019); Nguyen  _et al._ (2019); Andrews  _et al._ (2020); Lin  _et al._
(2018); Du  _et al._ (2020); Diefenbacher  _et al._ (2020); Chung  _et al._
(2020); Guo  _et al._ (2021); Tannenwald  _et al._ (2020). Our work
illustrates the potential benefits of full event-level classification.

For simplicity, we will not consider SM backgrounds in this work; of course,
these backgrounds are highly dependent on the Higgs final state. In certain
decay modes such as $H\to ZZ^{*}\to 4\ell$ Aad  _et al._ (2020a, b); Sirunyan
_et al._ (2021b, c), the non-Higgs background is highly suppressed, so our
work could directly apply there. For other decay modes where the SM background
is less suppressed (e.g., $H\to\gamma\gamma$), we imagine the “universal” VBF
vs. GGF classifier could be combined with a Higgs decay classifier for full
event classification including non-Higgs background rejection if necessary ATL
(2020b, a); Sirunyan  _et al._ (2020a).

An outline of our paper is as follows. In Sec. II, we describe the simulation
of our sample as well as the VBF pre-selection criteria and the numbers of
training, validation, and testing sets for the classifier. In Sec. III, we
describe the classifiers used in this study. We show the results in Sec. IV,
which is comprised of a comparison of tagger performances, a discussion about
what the event-level CNN has learned, and possible improvements of the BDT
from adding information beyond the leading two jets. In Sec. V, we examine the
$p_{T}$-balance of the full event and explore the possibility of the Higgs-
decay-mode-agnostic classifier. Finally, we conclude in Sec. VI. Appendix A
lists the structures of all the classifier models considered in this study.
Appendix B examines an extension of CNN for our classification problem
motivated by Chung  _et al._ (2020) and finds no further improvement.

##  II Sample Preparation

We use Madgraph5_aMC@NLO 2.7.3 Alwall  _et al._ (2014) with parton
distribution functions (PDFs) of CT10 Lai  _et al._ (2010) to generate Higgs
plus up to three jets events starting from $pp$ collisions at $\sqrt{s}=14$
TeV. The additional jets are matched using the MLM matching scheme with
parameters $xqcut=30~{}\text{GeV}$ and $qcut=45~{}\text{GeV}$. For VBF we just
use tree-level MG5, while for GGF we use a model generated by FeynRules 2.3.33
Alloul  _et al._ (2014) following the effective vertex method.

The samples are then showered and hadronized by Pythia 8.244 Sjöstrand  _et
al._ (2015, 2006), and finally passed through the Delphes 3.4.2 de Favereau
_et al._ (2014) fast detector simulation. The detector configuration in
Delphes is based upon the default ATLAS card, while the inputs of the jet
cluster module are EFlow objects instead of the default Tower objects. The jet
clustering is done by FastJet 3.3.2 Cacciari  _et al._ (2012) using the
anti-$k_{T}$ Cacciari  _et al._ (2008) algorithm with $R=0.4$. Jets are
required to have $p_{T}>30$ GeV.

In our sample preparation, we let the Higgs decay to two photons and use their
invariant mass cut to select the required Higgs production samples. Although
we generate samples in this particular Higgs decay mode, as discussed in the
Introduction, we will demonstrate later that the full-event classifiers
trained on low-level inputs are actually agnostic to the Higgs decay products,
in that their performance does not suffer when those decay products are
removed.

The samples used in the following analysis of this study are all extracted
from the events passing the VBF pre-selection criteria, as inspired by
experimental studies, that $N_{\gamma}\geq 2$, $120\leq M_{\gamma\gamma}\leq
130$ GeV, $N_{j}\geq 2$, and $\Delta\eta_{jj}\geq 2$. We have generated 500k
events each for the VBF and GGF samples and, after the VBF pre-selection, are
left with 175k events for VBF and 140k for GGF.

Throughout this paper, we consider VBF as the signal and GGF as the
background. For all of the event-level classifiers, the generated samples are
split into training, validation, and testing sets as indicated in Table 1.
Since for any event we take the leading two jets as the samples of the jet-
level classifier (i.e., jet-CNN), the numbers of the samples in different sets
of the jet-level classifier are twice as those in Table 1.

| training | validation | testing
---|---|---|---
VBF events | 112k | 28k | 35k
GGF events | 89k | 22k | 28k
Table 1: Numbers of training, validation, and testing sets for event-level
classifiers.

##  III Classifier models

###  III.1 BDT

Max depth |  | 3
---|---|---
Learning rate |  | 0.1
Objective |  | binary logistic
Early stop |  | 10 epochs
Evaluation metric |  | binary logistic
Table 2: Hyperparameters of the BDT

We start by considering BDT models that are implemented in XgBoost 1.5.0 Chen
and Guestrin (2016). (The hyperparameters and the details of the BDT models
are summarized in Table 2.) We train three different BDTs based on the
features summarized in Table 3 and Fig. 1. The first, “baseline”, is based on
six high level features from the study of VBF vs. GGF classification in Ref.
Chan  _et al._ (2017), which is inspired by ATLAS’s setup Aaboud  _et al._
(2018). This baseline BDT characterizes the discrimination power from the
kinematics of the photons and the jets in the event.111We have checked that a
simple DNN trained on these high-level features does not outperform the BDTs,
so we will focus on BDTs as our baseline.

Based on the experimental setup, Ref. Chan  _et al._ (2017) further considers
the jet shape variables Shelton (2013) as additional input features, such as
the girth summed over the two leading jets and the central/sided integrated
jet shape. Including these jet shape variables leads to our second BDT, which
we call “baseline + shape”.

Finally, we consider the benefits of replacing the human-engineered jet shape
variables of Shelton (2013); Chan  _et al._ (2017) with the output of a jet-
level CNN classifier trained on VBF vs GGF jets. We call this the “baseline +
jet-CNN” BDT. For more details on the jet-level CNN, see Section III.2.

baseline |  1\. $m_{jj}$, the invariant mass of $j_{1}$ and $j_{2}$
---|---
2\. $\Delta\eta_{jj}$, the absolute difference of the pseudo-rapidities of
$j_{1}$ and $j_{2}$
3\. $\phi^{*}$, defined by the $\phi$-difference between the leading di-photon
and di-jet
4\. $p_{Tt}^{\gamma\gamma}$, defined by
$\left|\left(\mathbf{p}_{T}^{\gamma_{1}}+\mathbf{p}_{T}^{\gamma_{2}}\right)\times\hat{t}\right|$,
where
$\hat{t}=\left(\mathbf{p}_{T}^{\gamma_{1}}-\mathbf{p}_{T}^{\gamma_{2}}\right)/\left|\mathbf{p}_{T}^{\gamma_{1}}-\mathbf{p}_{T}^{\gamma_{2}}\right|$
5\. $\Delta R_{\gamma j}^{\text{min}}$, defined by the minimum $\eta$-$\phi$
separation between $\gamma_{1}$/$\gamma_{2}$ and $j_{1}$/$j_{2}$
6\. $\eta^{*}$, defined by
$\left|\eta_{\gamma_{1}\gamma_{2}}-\left(\eta_{j_{1}}+\eta_{j_{2}}\right)/2\right|$,
where $\eta_{\gamma_{1}\gamma_{2}}$ is the pseudo-rapidity of the leading di-
photon
shape |  8\. the girth summed over the two leading jets $\sum_{j=1}^{2}g_{j}=\sum_{j=1}^{2}\sum_{i\in J^{j}}^{N}\ p_{T,i}^{j}r_{i}^{j}/p_{T}^{j}$
9\. the central integrated jet shape $\Psi_{c}=\sum_{j=1}^{2}\sum_{i\in
J^{j}}^{N}\ p_{T,i}^{j}(0<r_{i}^{j}<0.1)/(2p_{T}^{j})$
10\. the sided integrated jet shape $\Psi_{s}=\sum_{j=1}^{2}\sum_{i\in
J^{j}}^{N}\ p_{T,i}^{j}(0.1<r_{i}^{j}<0.2)/(2p_{T}^{j})$
jet-CNN |  11\. the jet scores of the two leading jets, output by the jet-CNN, soon to be introduced in Section III.2
Table 3: Summary of the features used in BDT. $j_{1}$ and $j_{2}$ mean
respectively the $p_{T}$-leading and -subleading jets, while $\gamma_{1}$ and
$\gamma_{2}$ mean respectively the $p_{T}$-leading and -subleading photons. In
the jet shape variables, $i$ represents the constituent of the jet and $r$ is
the distance between the constituent and the jet axis. Figure 1: Distributions
of BDT input variables. All histograms are normalized so that the area under
each curve is one.

###  III.2 Jet-CNN

In this subsection, we introduce the VBF vs. GGF jet-level CNN used in the
“baseline + jet-CNN scores” BDT described in the previous subsection. The jet-
level CNN is trained on jet images formed out of the leading two jets from the
VBF and GGF events.222Another possible labeling scheme is to identify whether
the jet is quark or gluon initiated, since VBF (GGF) events tend to contain
more quark (gluon) jets. However, our trials show that both labeling schemes
are equally useful when they are considered as features in the subsequent
event-level BDT. We will focus exclusively on the process-labeling in the
following study. Our image pre-processing, which basically follows the
procedure outlined in Ref. Macaluso and Shih (2018), contains image
centralization, rotation, and flipping, followed by pixelation from the
detector responses to the jet image. Finally, we pixelate the detector
responses into images ($10\times 10$ pixels) for each of the following four
channels: Tower $E_{T}$, Tower hits, Track $E_{T}$ and Track hits. (Following
the Delphes particle flow algorithm: “Tower” means EFlowNeutralHadron or
EFlowPhoton, and “Track” means EFlowTrack.)

Our jet-CNN model starts from a Batch Normalization Layer Ioffe and Szegedy
(2015), followed by several Convolution Layers and Average Pooling Layers,
which capture the features of the images. The sizes of the filters in
Convolution Layers and pools in Pooling Layers are all $2\times 2$. Due to the
relatively small size of the images ($10\times 10$ pixels), the neural network
(NN) does not need to be very deep. Since the image size shrinks as it passes
through a Pooling Layer, the number of Pooling Layers is restricted. After the
Convolution and Pooling Layers, the images are then flattened and fully
connected to three Dense Layers with 128 neurons respectively. The last Dense
Layer with 2 neurons, activated by the SoftMax function, represents the final
output score as probabilities. All the other Dense Layers and Convolution
Layers use the ReLU activation function Nair and Hinton (2010). The model
structure is plotted in Fig. 10.

The CNNs in this study are all implemented in TensorFlow 2.0.0 Abadi  _et al._
(2015) with Keras Chollet  _et al._ (2015) as its high-level API. We use Adam
Kingma and Ba (2014) as our optimizer during the training stage with the
categorical cross entropy loss function in all of our NN models. By monitoring
the loss of the validation set, early stopping is implemented to prevent over-
fitting in all of the NN and BDT models. The hyperparameters of the model are
summarized in Table 4.

Optimizer |  | Adam
---|---|---
Loss function |  | categorical cross entropy
Early stopping |  | 20 epochs
Batch size |  | 1024
Table 4: Hyperparameters for the jet-CNN tagger.

Our jet-CNN takes a jet image as its input and outputs a score ranging from 0
(GGF-jet) to 1 (VBF-jet). The scores of leading and subleading jets can thus
be useful features for subsequent event-by-event classification. The
distributions of the jet-CNN scores and the ROC curve for the jet-CNN are
shown in Fig. 2. The AUC of the jet-CNN is 0.711, which is less than an
efficient classifier. However, we will show that the jet-CNN scores are indeed
useful information in the subsequent event-level classification. Instead of
training and testing separate taggers for the leading and subleading jets
respectively, we utilize one tagger which is trained on mixed samples
including the leading and subleading jets. Our trial shows that doing this way
makes no loss of performance.

Figure 2: Distributions of the jet-CNN scores (left) and the ROC curve of the
jet-CNN (right). All histograms on the left are normalized so that each area
under the curve is one.

###  III.3 Event-CNN

A potentially more powerful way to perform event-level classification is to
leverage the capabilities of deep learning to predict the VBF vs. GGF label
directly from the lowest-level features of each event (in our case, the
4-vectors of all the particles in the event). In this paper we consider two
approaches to this, a CNN trained on whole-event images, to be described in
this subsection, and a self-attention model trained on sequences of the
particle 4-vectors, to be described in the next subsection.

Our whole-event images are preprocessed similarly to the jet images of the
previous subsection. However, unlike jets, the whole event is not a localized
object, nor is there an approximate boost or rotation invariance. So the
preprocessing consists of just the following steps: we first move the $\phi$
coordinate of the weighted center to the origin, and flip the image vertically
or horizontally to make the upper-right quadrant more energetic than all the
other quadrants. Finally, the detector responses are pixelated into images
with $40\times 40$ pixels for each of the six channels, which includes the
same four channels used in the jet-CNN and two additional ones recording the
hits and $E_{T}$ of the isolated photons.

An example of single event images is shown in Fig. 3. The left plot shows the
isolated photon $E_{T}$ and Tower $E_{T}$ combined with Track $p_{T}$ of an
event before the pre-processing, while the right plot is after the pre-
processing.

Figure 3: The isolated photon $E_{T}$ and Tower $E_{T}$ combined with Track
$p_{T}$ of an event without pre-processing (left) and after pre-processing
(right). The color of each pixel indicates the energy in units of GeV.

We employ a toy ResNet model He  _et al._ (2015) in our event-CNN. Two
Convolution Layers form a residual block in ResNet. There are shortcuts
connecting the residual blocks, enabling us to deepen our model without
suffering from the degradation problem. The sizes of filters in the
Convolution Layers and pools in the Pooling Layers are all $3\times 3$. The
detailed model structure of the event-CNN is shown in Fig. 11. The
hyperparameters are the same as those in Table 4.

In order to extract information from both the local jet-level and global
event-level features, Ref. Chung  _et al._ (2020) adopts a two-stream CNN
architecture, where one stream processes an image of the highest $p_{T}$ non-
Higgs jet in the event, and the other stream processes the full-event image.
Motivated by this, we further study the performance of an extension of our
full-event CNN in Appendix B, using a similar structure containing three
streams of CNN, dealing with event images and leading two jet images
respectively. However, we find no improvement from our original single-stream
event-CNN. This does not contradict the works of Ref. Chung  _et al._ (2020)
since they did not compare the performance of their two-stream CNN against a
single-stream CNN consisting of just the full-event classifier.

###  III.4 Self-attention

For comparison, we also consider another whole-event low-level-feature
classifier based on the technique of self-attention Lin  _et al._ (2017),
which is used in the famous Transformer model Vaswani  _et al._ (2017) dealing
with sequence-to-sequence tasks. The original motivation of this model is to
use the multi-head attention layers to capture the correlation among elements
in the input sequence. Inspired by this idea, instead of representing an event
as an image, we view the event as a sequence, where the elements of the
sequence are the $p_{T}$, $\eta$, $\phi$, and electric charge of the 100
highest-$p_{T}$ reconstructed particles in the event (with zero padding for
events with fewer than 100 particles). In principle, the self-attention
network could be advantageous over event-level images, because it is not
subject to the information loss induced by pixelation. Also, a nice property
of the self-attention mechanism is that it preserves permutation invariance of
the inputs (as does a CNN).

The implementation of the self-attention model is based on TensorFlow 2.5.0
and Keras. The model structure of the self-attention model is shown in Fig.
12. There are three five-head attention layers at the beginning, followed by a
Global Average Pooling (GAP) Layer, which converts the sequence of detector
responses into a single vector by taking the element-wise average. Dense
Layers are not implemented before the GAP Layer to keep permutation invariance
of the input sequence. Then the model is passed into seven Dense Layers. The
hyperparameters are listed in Table 5.

Optimizer |  | Adam
---|---|---
Loss function |  | categorical crossentropy
Early stopping |  | 50 epochs
Batch size |  | 1024
Table 5: Hyperparameters of the self-attention model.

##  IV Results

###  IV.1 Comparison of methods

Figure 4: ROC curves of several event-level classifiers. |  FPR |  AUC
---|---|---
BDT: baseline | 0.066 | 0.761
BDT: baseline + shape | 0.048 | 0.803
BDT: baseline + jet-CNN | 0.039 | 0.831
Self-attention | 0.030 | 0.834
Event-CNN | 0.017 | 0.874
Table 6: Performance comparison at TPR = 0.3.

The performance of the event-level classifiers defined in the previous section
is shown in Fig. 4. As an explicit example, Table 6 lists both false positive
rates (FPRs) and AUCs at the working point where the true positive rate (TPR)
is fixed at 0.3. From Fig. 4 and Table 6, we can easily compare the different
event-level classifiers.

First of all, “BDT: baseline” has the lowest AUC since it only considers the
high-level kinematic features in an event. Indeed, including additional
information on the jet shape variables can improve a little, but not as much
as using the jet-CNN score as an input. Notably, our jet-CNN scores serve as a
better feature than the jet shape variables, with the former reducing the FPR
from the baseline by a factor of 1.7 while the latter only by a factor of 1.4.
Therefore, despite a low AUC of the jet-CNN as shown in Fig. 2, its score
still provides valuable information. We have also checked that combining jet
shape variables and jet-CNN scores in the input features together did not
provide extra improvement in the AUC, indicating that the jet-CNN has learned
all the information contained in the human-engineered jet shape variables.

Second, we see that our self-attention model and event-CNN both perform better
than the BDTs. This is understandable because the BDTs only take into account
high-level variables or features of the two leading jets and photons only,
while the self-attention and event-CNN taggers take in the entire event and
catch more features therein.

Finally, the event-CNN is the most powerful classifier among all considered
taggers. Its inverse FPR is roughly a factor of 1.5 better than the self-
attention model for most of the TPR. Its AUC reaches 0.874 and the FPR is
reduced by a factor of 3.9 from the baseline at the assumed working point in
Table 6.

###  IV.2 Saliency maps of event-CNN

To further investigate what the event-CNN has learned, we examine its saliency
maps Simonyan  _et al._ (2014). Let the input pixel $x$ be identified as
$x_{c,h,w}$, where $c$ is the channel index, $h$ is the height index, and $w$
is the width index. The saliency is defined by the gradient of the $i$-th
class score $P^{i}$ with respect to the input pixel $x_{c,h,w}$,

$w^{i}_{c,h,w}\equiv\frac{\partial P^{i}}{\partial x_{c,h,w}}~{},$ (1)

where the gradient is calculated by back-propagation. In our case, we only
deal with binary classifiers, so it suffices to only consider the VBF class
score $P$. Putting $w_{c,h,w}$ together according to the indices, one can
obtain the saliency maps. However, what we are actually interested in is the
saliency according to the standardized pixels $y_{c,h,w}$ which have no scale
difference across channels,

$x_{c,h,w}\to y_{c,h,w}=\frac{x_{c,h,w}-\mu_{c}}{\sigma_{c}}~{},$ (2)

where ${\sigma_{c}}^{2}$ and $\mu_{c}$ are the variance and mean of the
channel $c$ in the whole sample, including the training, validation, and
testing sets. Hence, we will consider the following gradient,

$\tilde{w}_{c,h,w}\equiv\frac{\partial P}{\partial
y_{c,h,w}}=w_{c,h,w}\times\sigma_{c}~{}.$ (3)

Finally, we arrange $\tilde{w}$ according the $c,h,w$ indices and then plot
its absolute value $|\tilde{w}_{c,h,w}|$ to get the saliency maps as the lower
row in Fig. 5 and 6.

We utilize the visualization toolkit tf-keras-vis 0.8.0 Kubota (2021) to
implement the saliency maps of our event-CNN tagger. In the following, we pick
as examples a VBF event (Fig. 5) with a high CNN score (i.e., more VBF-like)
and a GGF event (Fig. 6) with a low CNN score (i.e., more GGF-like). In the
plots, the clustered jets are marked by black circles, with their sizes
indicating the jet’s ordering in $p_{T}$. The color maps of the upper row
indicate the actual value of the input, with the unit being GeV for Tower
$E_{T}$, Track $p_{T}$, and isolated photon $E_{T}$ and counts for Tower hits,
Track hits, and isolated photon hits. In contrast, the color maps of the lower
row indicate the relative saliency, i.e. the most salient pixel is scaled to
one in plotting,

$\left|\tilde{w}_{c,h,w}\right|\to\frac{\left|\tilde{w}_{c,h,w}\right|}{\displaystyle\max_{c,h,w}\left\\{\left|\tilde{w}_{c,h,w}\right|\right\\}}~{}.$
(4)

Figure 5: A VBF event with a high event-CNN score. The upper six plots show
the raw inputs of the model, while the lower counterparts are the saliency
maps calculated by the corresponding normalized channels. The black circles
show the locations of the clustered jets, with the circle size indicating the
ordering in $p_{T}$. The color maps of the upper row indicate the actual
input. The unit is GeV for Tower $E_{T}$, Track $p_{T}$, and isolated photon
$E_{T}$, and counts for Tower hits, Track hits, and isolated photon hits. The
color maps of the lower row indicate the relative saliency.  Figure 6: Same as
Fig. 5, but for a GGF event with a low event-CNN score.

From the saliency maps, we observe that the CNN model generally focuses on the
locations with more hadronic activities, as anticipated, because the jets
contain crucial information for the classification of VBF and GGF events. In
addition, the CNN is also seen to make use of lower $p_{T}$ jets and hadronic
activity that falls below the jet $p_{T}$ threshold (set to 30 GeV in this
work). This explains why the event-CNN performs better than the BDT. In our
setup of the BDTs, we do not feed the information of the third jet into the
model. Moreover, the input of the BDTs relies on our knowledge about what kind
of high-level features is beneficial and hence cannot make use of unclustered
energy in an event. Finally, we can observe that the event-CNN is much more
focused on where jets are than the locations of photons, which indicates that
the photon information is not crucial in the classification. This sheds light
on the possibility of the Higgs-decay-mode-agnostic classifier which solely
relies on the jet information. Details will be described in Section V.

###  IV.3 Improvements of BDTs

In this subsection, we investigate more about how the BDTs, which rely on
high-level kinematic variables as the features for training, can be further
improved. Based on the study of the saliency maps in the previous subsection,
we are motivated to consider information about additional hadronic activity in
the event beyond the leading two jets. So we will study the benefits of
including the 4-vector momentum of the third hardest jet, as well as inclusive
kinematic variables that take all jets into account.

  * •

4-vector momentum of the third jet in $p_{T}$ ordering, which is denoted as
“j3vec;”

  * •

$HT=\sum\limits_{j\in\text{jets}}p_{T}^{j}$, which characterizes the $p_{T}$
distribution of the jets;

  * •

$\tilde{\eta}=\sum\limits_{j\in\text{jets}}\left|\eta^{j}\right|$, which
characterizes the positional distribution of the jets; and

  * •

number of jets.

We will call the set of features including $HT$, $\tilde{\eta}$, and the
number of jets as a “jet-profile.” The normalized distributions of $HT$,
$\tilde{\eta}$, and the number of jets are already shown in the last row of
Fig. 1.

Figure 7: ROC curves of BDT trained on additional high-level features.

We are interested in how the additional information improves the best BDT we
have so far, so we will add these extra variables to “BDT: baseline + jet-
CNN.” The ROC curves of the BDTs trained with further inputs of these
additional variables are plotted in Fig. 7. From the AUCs, we can see both the
additional 4-vector momentum of the third jet and the jet-profile can improve
the performance of classification. Their improvements are comparable to each
other, as seen from the ROC curves as well as the similar AUCs. We have also
checked that adding the additional 4-vector momentum and the jet-profile
together into the BDT does not further improve the AUC, which is a piece of
direct evidence that these two sets of variables provide equivalent
information to the BDTs. The reason is that the crucial information contained
in both sets is the existence of the third jet. A characteristic of GGF events
is that they tend to have more than two jets, which can be seen in the
distribution of the number of jets in Fig. 1. By examining the actual trees in
the BDTs trained by the 4-vector momentum of the third jet and the jet-
profile, respectively, we indeed find that the existence of the third jet
provides a clear separation between VBF and GGF events and therefore plays an
important role in both cases.

Finally, in “BDT: all variables”, we consider all the high-level features,
including event-related characteristics (i.e., $m_{jj}$, $\Delta\eta_{jj}$,
$\phi^{*}$, $p_{Tt}^{\gamma\gamma}$, $\Delta R_{\gamma j}^{\text{min}}$,
$\eta^{*}$, $HT$, $\tilde{\eta}$, and the number of jets), and jet-related
information of the three leading jets (i.e., 4-vector momenta, jet-CNN scores,
and the girth, central/sided integrated jet shape of each jet without taking
summation or average). This BDT achieves the best AUC, 0.851, among all the
other BDTs and improve the baseline significantly. However, despite this
sizable improvement from the baseline, the event-CNN still outperforms “BDT:
all variables” with an even larger AUC.

##  V A Higgs-decay-agnostic VBF vs. GGF classifier

In the Introduction, we noted that event-level classifiers trained on low-
level inputs could potentially be agnostic to the Higgs decay mode, due to the
scalar nature of the Higgs and the $p_{T}$-balance of the whole event. Here we
will explore this idea further, by seeing to what extent $p_{T}$ balance
allows the Higgs momentum to be predicted from the hadronic activity in the
event, and then to what extent our classifiers suffer when the Higgs decay
products are removed from the event.

Shown in the left plot of Fig. 8 are histograms of the $p_{T}$ balance of the
whole event, $|\sum_{i\in{\rm reconstructed\,\,\,particles}}\vec{p}_{Ti}|$,
normalized by the $p_{T}$ of the Higgs. We see that the $p_{T}$ is well-
balanced amongst the low-level features, so the Higgs transverse momentum can
be well-reconstructed from the non-photon reconstructed particles. Meanwhile,
the right plot of Fig. 8 depicts the $p_{T}$ balance between the photons and
the leading three jets, again normalized by the $p_{T}$ of the Higgs. Here we
see that while the leading three jets can capture the $p_{T}$ information of
the photons to some extent, it is not as informative as the responses and,
therefore, the balance is not as complete.

Figure 8: The fractional $p_{T}$-balance of the leading di-photon and other
objects, non-photon responses on the left and up to leading three jets on the
right. To calculate the balance, we first vector-sum the momenta of the di-
photon and other objects, and then take its transverse momentum. Finally, the
balanced $p_{T}$ is divided by the $p_{T}$ of the di-photon.

Fig. 9 shows the impact of removing the Higgs decay products (in our case, the
two photons) from the event before training the VBF vs. GGF classifiers. We
see that removing photon information from the event-CNN only makes a very
small degradation in AUC, from 0.874 to 0.873. On the other hand, removing
photon information from the high-level features for the BDT reduces the AUC
from 0.851 to 0.840.333In more detail, in Fig. 9, “all variables with photons”
refers to the feature set used in “BDT: all variables” in Fig. 7, while “all
variables without photons” refers to the same feature set but with the photon-
related variables (i.e., $\phi^{*}$, $p_{Tt}^{\gamma\gamma}$, $\Delta
R_{\gamma j}^{\text{min}}$, $\eta^{*}$) all excluded. The degradation in
performance in the BDT is still not very large, but it is larger than that of
the event-CNN. This is completely in line with the histograms shown in Fig. 8.

All in all, we confirm here that due to the $p_{T}$ balance of the events, the
performance of VBF vs. GGF classification does not depend much on the Higgs
decay products, especially for the whole-event CNN that is based on low-level
inputs. This raises the intriguing possibility that one could train a single
VBF vs. GGF classifier that is agnostic to the Higgs decay mode, and could be
applied equally optimally to a variety of Higgs decay channels in a uniform
way. This could have benefits for data-driven calibration and reducing
systematic uncertainties associated with VBF tagging.

Figure 9: ROC curves of event-level classifiers with and without the photon
information.

##  VI Conclusions

In this paper, we have studied machine learning approaches to event-level
classification of Higgs production modes, focusing on the important problem of
VBF vs. GGF discrimination. Building on previous studies Chan  _et al._
(2017); Chung  _et al._ (2020), we have shown that full-event deep learning
classifiers that utilize low-level inputs (full-event images, sequences of
particle 4-vectors) significantly outperform classifiers based on high-level
physics features (kinematics, jet shapes). We have explored both CNNs trained
on full-event images, and permutation-invariant self-attention networks
trained on sequences of particle 4-vectors. Although the full-event CNN
achieved the best performance in our studies – improving beyond the baseline
shallow network by more than a factor of $2-3$ in background rejection across
a wide range of signal efficiencies – perhaps our work provides a useful
starting point for further optimization of the attention-based approach.

We also studied why the event-level CNNs perform so much better than the
shallow networks based on high-level features. Using saliency maps we saw how
additional jets in the event beyond the first two contribute to the CNN
classification, as well as unclustered hadronic activity (jets below the
$p_{T}$ threshold). By adding high-level features derived from these
additional jets, we have confirmed that the performance of the shallow
networks can indeed be improved and be brought somewhat closer to the event-
level CNN.

Finally, in this work we have gone beyond previous approaches and explored the
possibility of a VBF vs. GGF classifier that is agnostic to the Higgs decay
mode. A classifier trained on the low-level information of the full event
should be able to reconstruct the Higgs transverse momentum from $p_{T}$
balance and, since the Higgs is a scalar, its decay products (at least when it
decays to electroweak states) should be well-factorized from the rest of the
event. Therefore, a full-event, low-level classifier should be largely
independent of the Higgs decay channel. We have taken the first steps towards
verifying that in this work, by showing how the performance of the full-event
CNN is virtually unchanged when trained on the events with and without the
diphotons from the Higgs decay.

Some future directions of our work include: generalizing our work to more
Higgs production modes (e.g., $ZH$, $WH$ and $ttH$) using a multi-class
classifier; further fleshing out our idea of a decay-agnostic classifier by
studying other Higgs decay modes besides $H\to\gamma\gamma$; studying even
more recent deep learning classifiers such as graph networks; adding particle
interaction information into the event-level self-attention model (as was
inspired by Ref. Qu  _et al._ (2022)); and incorporating symmetries such as
Lorentz invariance into the architecture of the neural network to achieve even
better performance (as was done recently for top-tagging in Ref. Gong  _et
al._ (2022)).

###### Acknowledgements.

We are grateful to Tae Min Hong and Yi-Lun Chung for helpful discussions. We
also thank Kai-Feng Chen and Iftah Galon for their participation in this
project at the early stage. CC and SW were supported in part by the Ministry
of Science and Technology of Taiwan under Grant Nos.
MOST-108-2112-M-002-005-MY3 and 111-2112-M-002-018-MY3. The work of DS was
supported by DOE grant DOE-SC0010008.

##  Appendix A Architectures of various neural networks

Figure 10: The model structure of the jet-CNN. Figure 11: The model structure
of the event-CNN. Figure 12: The model structure of the self-attention model.
Figure 13: The model structure of the multi-stream CNN model.

##  Appendix B Multi-stream CNN

In this section, we examine an extension of CNN. In Ref. Chung  _et al._
(2020), an architecture called 2CNN extracts event-level and jet-level
features simultaneously with two streams. One stream applies filters on event
images, and the other deals with the leading non-Higgs jet images. Then the
two streams are connected together and combined to form a single model.
Inspired by this study, we want to investigate the possible improvement using
this multi-stream architecture.

We adopt a three-stream CNN where one stream applies filters on the event
images, and the other two process the images of the two leading jets
respectively. Each stream is a toy ResNet model used in Sec. III.3. We will
call this architecture “event + 2jet-CNN.” The model structure is shown in
Fig. 13. All of the event images and jet images are pixelated into $40\times
40$ pixels. The images are pre-processed as in Sec. III.2 and III.3. The
hyperparameters are the same as those in Table 4.

The ROC curves of our original event-CNN and this event + 2jet-CNN are plotted
in Fig. 14. The curves almost overlap with each other and the AUCs are very
similar, indicating that the event-CNN has already captured useful information
for classification. The additional high-resolution jet images do not provide
significant extra help to the performance.

Figure 14: ROC curves of the event-CNN and event+2jet-CNN.

## References

  * Aad  _et al._ (2012) G. Aad _et al._ (ATLAS), Phys. Lett. B 716, 1 (2012), arXiv:1207.7214 [hep-ex] .
  * Chatrchyan  _et al._ (2012) S. Chatrchyan _et al._ (CMS), Phys. Lett. B 716, 30 (2012), arXiv:1207.7235 [hep-ex] .
  * Cepeda  _et al._ (2019) M. Cepeda _et al._ , CERN Yellow Rep. Monogr. 7, 221 (2019), arXiv:1902.00134 [hep-ph] .
  * Grazzini (2019) M. Grazzini, “Sm higgs production cross sections at $\sqrt{S}$ = 13, 14 and 27 tev (update in cern hl-lhc yr 2019),”  (2019).
  * ATL (2020a) _A combination of measurements of Higgs boson production and decay using up to $139$ fb-1 of proton–proton collision data at $\sqrt{s}=$ 13 TeV collected with the ATLAS experiment_, Tech. Rep. (CERN, Geneva, 2020) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2020-027.
  * Aad  _et al._ (2020a) G. Aad _et al._ (ATLAS), Eur. Phys. J. C 80, 957 (2020a), [Erratum: Eur.Phys.J.C 81, 29 (2021), Erratum: Eur.Phys.J.C 81, 398 (2021)], arXiv:2004.03447 [hep-ex] .
  * ATL (2020b) _Measurement of the properties of Higgs boson production at $\sqrt{s}$=13 TeV in the $H\to\gamma\gamma$ channel using 139 fb-1 of $pp$ collision data with the ATLAS experiment_, Tech. Rep. (CERN, Geneva, 2020) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2020-026.
  * ATL (2021) _Measurements of gluon fusion and vector-boson-fusion production of the Higgs boson in $H\rightarrow WW^{*}\rightarrow e\nu\mu\nu$ decays using $pp$ collisions at $\sqrt{s}=13$ TeV with the ATLAS detector_, Tech. Rep. (CERN, Geneva, 2021) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/CONFNOTES/ATLAS-CONF-2021-014.
  * Sirunyan  _et al._ (2021a) A. M. Sirunyan _et al._ (CMS), JHEP 07, 027 (2021a), arXiv:2103.06956 [hep-ex] .
  * CMS (2022a) _Measurements of properties of the Higgs boson in the W boson pair decay channel in proton-proton collisions at $\sqrt{s}=13~{}\text{TeV}$_, Tech. Rep. (CERN, Geneva, 2022).
  * Sirunyan  _et al._ (2021b) A. M. Sirunyan _et al._ (CMS), Eur. Phys. J. C 81, 488 (2021b), arXiv:2103.04956 [hep-ex] .
  * Chan  _et al._ (2017) C.-H. Chan, K. Cheung, Y.-L. Chung, and P.-H. Hsu, Phys. Rev. D 96, 096009 (2017), arXiv:1706.02864 [hep-ph] .
  * Chung  _et al._ (2020) Y.-L. Chung, S.-C. Hsu, and B. Nachman,  (2020), 10.1088/1748-0221/16/07/P07002, arXiv:2009.05930 [hep-ph] .
  * Lin  _et al._ (2018) J. Lin, M. Freytsis, I. Moult, and B. Nachman, JHEP 10, 101 (2018), arXiv:1807.10768 [hep-ph] .
  * Aad  _et al._ (2021a) G. Aad _et al._ (ATLAS), (2021a), arXiv:2109.13808 [hep-ex] .
  * Aad  _et al._ (2021b) G. Aad _et al._ (ATLAS), Phys. Lett. B 812, 135980 (2021b), arXiv:2007.07830 [hep-ex] .
  * CMS (2022b) (2022b), arXiv:2204.12945 [hep-ex] .
  * Sirunyan  _et al._ (2020a) A. M. Sirunyan _et al._ (CMS), Phys. Lett. B 805, 135425 (2020a), arXiv:2002.06398 [hep-ex] .
  * Lin  _et al._ (2017) Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio, CoRR abs/1703.03130 (2017), 1703.03130 .
  * Vaswani  _et al._ (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”  (2017).
  * Pumplin (1991) J. Pumplin, Phys. Rev. D 44, 2025 (1991).
  * Cogan  _et al._ (2015) J. Cogan, M. Kagan, E. Strauss, and A. Schwarztman, JHEP 02, 118 (2015), arXiv:1407.5675 [hep-ph] .
  * Almeida  _et al._ (2015) L. G. Almeida, M. Backović, M. Cliche, S. J. Lee, and M. Perelstein, JHEP 07, 086 (2015), arXiv:1501.05968 [hep-ph] .
  * de Oliveira  _et al._ (2016) L. de Oliveira, M. Kagan, L. Mackey, B. Nachman, and A. Schwartzman, JHEP 07, 069 (2016), arXiv:1511.05190 [hep-ph] .
  * Baldi  _et al._ (2016) P. Baldi, K. Bauer, C. Eng, P. Sadowski, and D. Whiteson, Phys. Rev. D 93, 094034 (2016), arXiv:1603.09349 [hep-ex] .
  * Komiske  _et al._ (2017) P. T. Komiske, E. M. Metodiev, and M. D. Schwartz, JHEP 01, 110 (2017), arXiv:1612.01551 [hep-ph] .
  * Kagan  _et al._ (2016) M. Kagan, L. d. Oliveira, L. Mackey, B. Nachman, and A. Schwartzman, EPJ Web Conf. 127, 00009 (2016).
  * Guest  _et al._ (2016) D. Guest, J. Collado, P. Baldi, S.-C. Hsu, G. Urban, and D. Whiteson, Phys. Rev. D 94, 112002 (2016), arXiv:1607.08633 [hep-ex] .
  * Barnard  _et al._ (2017) J. Barnard, E. N. Dawe, M. J. Dolan, and N. Rajcic, Phys. Rev. D 95, 014018 (2017), arXiv:1609.00607 [hep-ph] .
  * Komiske  _et al._ (2018a) P. T. Komiske, E. M. Metodiev, and J. Thaler, JHEP 04, 013 (2018a), arXiv:1712.07124 [hep-ph] .
  * ATL (2017a) _Quark versus Gluon Jet Tagging Using Jet Images with the ATLAS Detector_, Tech. Rep. (CERN, Geneva, 2017) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-PHYS-PUB-2017-017.
  * Pearkes  _et al._ (2017) J. Pearkes, W. Fedorko, A. Lister, and C. Gay, (2017), arXiv:1704.02124 [hep-ex] .
  * Kasieczka  _et al._ (2017) G. Kasieczka, T. Plehn, M. Russell, and T. Schell, JHEP 05, 006 (2017), arXiv:1701.08784 [hep-ph] .
  * Datta and Larkoski (2017) K. Datta and A. Larkoski, JHEP 06, 073 (2017), arXiv:1704.08249 [hep-ph] .
  * Butter  _et al._ (2018) A. Butter, G. Kasieczka, T. Plehn, and M. Russell, SciPost Phys. 5, 028 (2018), arXiv:1707.08966 [hep-ph] .
  * Datta and Larkoski (2018) K. Datta and A. J. Larkoski, JHEP 03, 086 (2018), arXiv:1710.01305 [hep-ph] .
  * Egan  _et al._ (2017) S. Egan, W. Fedorko, A. Lister, J. Pearkes, and C. Gay, (2017), arXiv:1711.09059 [hep-ex] .
  * Schramm (2018) S. Schramm (ATLAS), EPJ Web Conf. 182, 02113 (2018).
  * Louppe  _et al._ (2019) G. Louppe, K. Cho, C. Becot, and K. Cranmer, JHEP 01, 057 (2019), arXiv:1702.00748 [hep-ph] .
  * Cheng (2018) T. Cheng, Comput. Softw. Big Sci. 2, 3 (2018), arXiv:1711.02633 [hep-ph] .
  * Sirunyan  _et al._ (2018) A. M. Sirunyan _et al._ (CMS), JINST 13, P05011 (2018), arXiv:1712.07158 [physics.ins-det] .
  * Komiske  _et al._ (2018b) P. T. Komiske, E. M. Metodiev, B. Nachman, and M. D. Schwartz, Phys. Rev. D 98, 011502 (2018b), arXiv:1801.10158 [hep-ph] .
  * Choi  _et al._ (2019) S. Choi, S. J. Lee, and M. Perelstein, JHEP 02, 132 (2019), arXiv:1806.01263 [hep-ph] .
  * Macaluso and Shih (2018) S. Macaluso and D. Shih, JHEP 10, 121 (2018), arXiv:1803.00107 [hep-ph] .
  * Komiske  _et al._ (2019) P. T. Komiske, E. M. Metodiev, and J. Thaler, JHEP 01, 121 (2019), arXiv:1810.05165 [hep-ph] .
  * Kasieczka  _et al._ (2019) G. Kasieczka, N. Kiefer, T. Plehn, and J. M. Thompson, SciPost Phys. 6, 069 (2019), arXiv:1812.09223 [hep-ph] .
  * Dreyer  _et al._ (2018) F. A. Dreyer, G. P. Salam, and G. Soyez, JHEP 12, 064 (2018), arXiv:1807.04758 [hep-ph] .
  * Fraser and Schwartz (2018) K. Fraser and M. D. Schwartz, JHEP 10, 093 (2018), arXiv:1803.08066 [hep-ph] .
  * Chen  _et al._ (2020) Y.-C. J. Chen, C.-W. Chiang, G. Cottin, and D. Shih, Phys. Rev. D 101, 053001 (2020), arXiv:1908.08256 [hep-ph] .
  * Datta  _et al._ (2019) K. Datta, A. Larkoski, and B. Nachman, Phys. Rev. D 100, 095016 (2019), arXiv:1902.07180 [hep-ph] .
  * Qu and Gouskos (2020) H. Qu and L. Gouskos, Phys. Rev. D 101, 056019 (2020), arXiv:1902.08570 [hep-ph] .
  * Chakraborty  _et al._ (2019) A. Chakraborty, S. H. Lim, and M. M. Nojiri, JHEP 07, 135 (2019), arXiv:1904.02092 [hep-ph] .
  * Lee  _et al._ (2019a) J. S. H. Lee, I. Park, I. J. Watson, and S. Yang, J. Korean Phys. Soc. 74, 219 (2019a), arXiv:2012.02531 [hep-ex] .
  * Lee  _et al._ (2019b) J. S. H. Lee, S. M. Lee, Y. Lee, I. Park, I. J. Watson, and S. Yang, J. Korean Phys. Soc. 75, 652 (2019b), arXiv:2012.02540 [hep-ph] .
  * Diefenbacher  _et al._ (2020) S. Diefenbacher, H. Frost, G. Kasieczka, T. Plehn, and J. M. Thompson, SciPost Phys. 8, 023 (2020), arXiv:1906.11265 [hep-ph] .
  * Moreno  _et al._ (2020a) E. A. Moreno, T. Q. Nguyen, J.-R. Vlimant, O. Cerri, H. B. Newman, A. Periwal, M. Spiropulu, J. M. Duarte, and M. Pierini, Phys. Rev. D 102, 012010 (2020a), arXiv:1909.12285 [hep-ex] .
  * Andreassen  _et al._ (2019) A. Andreassen, I. Feige, C. Frye, and M. D. Schwartz, Phys. Rev. Lett. 123, 182001 (2019), arXiv:1906.10137 [hep-ph] .
  * Moreno  _et al._ (2020b) E. A. Moreno, O. Cerri, J. M. Duarte, H. B. Newman, T. Q. Nguyen, A. Periwal, M. Pierini, A. Serikova, M. Spiropulu, and J.-R. Vlimant, Eur. Phys. J. C 80, 58 (2020b), arXiv:1908.05318 [hep-ex] .
  * Erdmann (2020) J. Erdmann, JINST 15, P01021 (2020), arXiv:1907.07505 [physics.ins-det] .
  * Li  _et al._ (2021) J. Li, T. Li, and F.-Z. Xu, JHEP 04, 156 (2021), arXiv:2008.13529 [hep-ph] .
  * Bols  _et al._ (2020) E. Bols, J. Kieseler, M. Verzetti, M. Stoye, and A. Stakia, JINST 15, P12012 (2020), arXiv:2008.10519 [hep-ex] .
  * Chakraborty  _et al._ (2020) A. Chakraborty, S. H. Lim, M. M. Nojiri, and M. Takeuchi, JHEP 07, 111 (2020), arXiv:2003.11787 [hep-ph] .
  * Bernreuther  _et al._ (2021) E. Bernreuther, T. Finke, F. Kahlhoefer, M. Krämer, and A. Mück, SciPost Phys. 10, 046 (2021), arXiv:2006.08639 [hep-ph] .
  * Lim and Nojiri (2022) S. H. Lim and M. M. Nojiri, Phys. Rev. D 105, 014004 (2022), arXiv:2010.13469 [hep-ph] .
  * Guo  _et al._ (2021) J. Guo, J. Li, T. Li, and R. Zhang, Phys. Rev. D 103, 116025 (2021), arXiv:2010.05464 [hep-ph] .
  * Dolan and Ore (2021) M. J. Dolan and A. Ore, Phys. Rev. D 103, 074022 (2021), arXiv:2012.00964 [hep-ph] .
  * Mikuni and Canelli (2020) V. Mikuni and F. Canelli, Eur. Phys. J. Plus 135, 463 (2020), arXiv:2001.05311 [physics.data-an] .
  * Li and Sun (2020) J. Li and H. Sun, (2020), arXiv:2009.00170 [hep-ph] .
  * Kagan (2020) M. Kagan, (2020), arXiv:2012.09719 [physics.data-an] .
  * Erdmann  _et al._ (2021) J. Erdmann, O. Nackenhorst, and S. V. Zeißner, JINST 16, P08039 (2021), arXiv:2011.10736 [hep-ex] .
  * Dreyer and Qu (2021) F. A. Dreyer and H. Qu, JHEP 03, 052 (2021), arXiv:2012.08526 [hep-ph] .
  * Nakai  _et al._ (2020) Y. Nakai, D. Shih, and S. Thomas, (2020), arXiv:2003.09517 [hep-ph] .
  * Bhattacharya  _et al._ (2022) S. Bhattacharya, M. Guchait, and A. H. Vijay, Phys. Rev. D 105, 042005 (2022), arXiv:2010.11778 [hep-ph] .
  * Sirunyan  _et al._ (2020b) A. M. Sirunyan _et al._ (CMS), JINST 15, P06005 (2020b), arXiv:2004.08262 [hep-ex] .
  * Andrews  _et al._ (2021) M. Andrews _et al._ , EPJ Web Conf. 251, 04030 (2021), arXiv:2104.14659 [physics.data-an] .
  * Filipek  _et al._ (2021) J. Filipek, S.-C. Hsu, J. Kruper, K. Mohan, and B. Nachman, (2021), arXiv:2105.04582 [hep-ph] .
  * Mikuni and Canelli (2021) V. Mikuni and F. Canelli, Mach. Learn. Sci. Tech. 2, 035027 (2021), arXiv:2102.05073 [physics.data-an] .
  * Konar  _et al._ (2022) P. Konar, V. S. Ngairangbam, and M. Spannowsky, JHEP 02, 060 (2022), arXiv:2109.14636 [hep-ph] .
  * Shimmin (2021) C. Shimmin (2021) arXiv:2107.02908 [hep-ph] .
  * Dreyer  _et al._ (2021) F. Dreyer, G. Soyez, and A. Takacs, (2021), arXiv:2112.09140 [hep-ph] .
  * Aguilar-Saavedra (2021) J. A. Aguilar-Saavedra, Eur. Phys. J. C 81, 734 (2021), arXiv:2102.01667 [hep-ph] .
  * Khosa and Marzani (2021) C. K. Khosa and S. Marzani, Phys. Rev. D 104, 055043 (2021), arXiv:2105.03989 [hep-ph] .
  * Gong  _et al._ (2022) S. Gong, Q. Meng, J. Zhang, H. Qu, C. Li, S. Qian, W. Du, Z.-M. Ma, and T.-Y. Liu, JHEP 07, 030 (2022), arXiv:2201.08187 [hep-ph] .
  * Kim and Martin (2021) T. Kim and A. Martin, (2021), arXiv:2102.05124 [hep-ph] .
  * Qu  _et al._ (2022) H. Qu, C. Li, and S. Qian, (2022), arXiv:2202.03772 [hep-ph] .
  * ATL (2017b) _Identification of Jets Containing $b$-Hadrons with Recurrent Neural Networks at the ATLAS Experiment_, Tech. Rep. (CERN, Geneva, 2017) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-PHYS-PUB-2017-003.
  * ATL (2020c) _Deep Sets based Neural Networks for Impact Parameter Flavour Tagging in ATLAS_, Tech. Rep. (CERN, Geneva, 2020) all figures including auxiliary figures are available at https://atlas.web.cern.ch/Atlas/GROUPS/PHYSICS/PUBNOTES/ATL-PHYS-PUB-2020-014.
  * Nguyen  _et al._ (2019) T. Q. Nguyen, D. Weitekamp, D. Anderson, R. Castello, O. Cerri, M. Pierini, M. Spiropulu, and J.-R. Vlimant, Comput. Softw. Big Sci. 3, 12 (2019), arXiv:1807.00083 [hep-ex] .
  * Andrews  _et al._ (2020) M. Andrews, M. Paulini, S. Gleyzer, and B. Poczos, Comput. Softw. Big Sci. 4, 6 (2020), arXiv:1807.11916 [physics.data-an] .
  * Du  _et al._ (2020) Y.-L. Du, K. Zhou, J. Steinheimer, L.-G. Pang, A. Motornenko, H.-S. Zong, X.-N. Wang, and H. Stöcker, Eur. Phys. J. C 80, 516 (2020), arXiv:1910.11530 [hep-ph] .
  * Tannenwald  _et al._ (2020) B. Tannenwald, C. Neu, A. Li, G. Buehlmann, A. Cuddeback, L. Hatfield, R. Parvatam, and C. Thompson, (2020), arXiv:2009.06754 [hep-ph] .
  * Aad  _et al._ (2020b) G. Aad _et al._ (ATLAS), Eur. Phys. J. C 80, 942 (2020b), arXiv:2004.03969 [hep-ex] .
  * Sirunyan  _et al._ (2021c) A. M. Sirunyan _et al._ (CMS), Phys. Rev. D 104, 052004 (2021c), arXiv:2104.12152 [hep-ex] .
  * Alwall  _et al._ (2014) J. Alwall, R. Frederix, S. Frixione, V. Hirschi, F. Maltoni, O. Mattelaer, H. S. Shao, T. Stelzer, P. Torrielli, and M. Zaro, JHEP 07, 079 (2014), arXiv:1405.0301 [hep-ph] .
  * Lai  _et al._ (2010) H.-L. Lai, M. Guzzi, J. Huston, Z. Li, P. M. Nadolsky, J. Pumplin, and C. P. Yuan, Phys. Rev. D 82, 074024 (2010), arXiv:1007.2241 [hep-ph] .
  * Alloul  _et al._ (2014) A. Alloul, N. D. Christensen, C. Degrande, C. Duhr, and B. Fuks, Comput. Phys. Commun. 185, 2250 (2014), arXiv:1310.1921 [hep-ph] .
  * Sjöstrand  _et al._ (2015) T. Sjöstrand, S. Ask, J. R. Christiansen, R. Corke, N. Desai, P. Ilten, S. Mrenna, S. Prestel, C. O. Rasmussen, and P. Z. Skands, Comput. Phys. Commun. 191, 159 (2015), arXiv:1410.3012 [hep-ph] .
  * Sjöstrand  _et al._ (2006) T. Sjöstrand, S. Mrenna, and P. Skands, Journal of High Energy Physics 2006, 026–026 (2006).
  * de Favereau  _et al._ (2014) J. de Favereau, C. Delaere, P. Demin, A. Giammanco, V. Lemaître, A. Mertens, and M. Selvaggi (DELPHES 3), JHEP 02, 057 (2014), arXiv:1307.6346 [hep-ex] .
  * Cacciari  _et al._ (2012) M. Cacciari, G. P. Salam, and G. Soyez, Eur. Phys. J. C 72, 1896 (2012), arXiv:1111.6097 [hep-ph] .
  * Cacciari  _et al._ (2008) M. Cacciari, G. P. Salam, and G. Soyez, JHEP 04, 063 (2008), arXiv:0802.1189 [hep-ph] .
  * Chen and Guestrin (2016) T. Chen and C. Guestrin, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2016), 10.1145/2939672.2939785.
  * Aaboud  _et al._ (2018) M. Aaboud _et al._ (ATLAS), Phys. Rev. D 98, 052005 (2018), arXiv:1802.04146 [hep-ex] .
  * Shelton (2013) J. Shelton, “Tasi lectures on jet substructure,”  (2013).
  * Ioffe and Szegedy (2015) S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,”  (2015).
  * Nair and Hinton (2010) V. Nair and G. Hinton (2010) pp. 807–814.
  * Abadi  _et al._ (2015) M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,”  (2015), software available from tensorflow.org.
  * Chollet  _et al._ (2015) F. Chollet _et al._ , “Keras,”  (2015).
  * Kingma and Ba (2014) D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”  (2014).
  * He  _et al._ (2015) K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” (2015), arXiv:1512.03385 [cs.CV] .
  * Simonyan  _et al._ (2014) K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” (2014), arXiv:1312.6034 [cs.CV] .
  * Kubota (2021) Y. Kubota, “tf-keras-vis,”  (2021).