MLG-23-001

MLG-23-001

# Portable acceleration of CMS computing workflows with coprocessors as a
service

###### Abstract

Computing demands for large scientific experiments, such as the CMS experiment
at the CERN LHC, will increase dramatically in the next decades. To complement
the future performance increases of software running on central processing
units (CPUs), explorations of coprocessor usage in data processing hold great
potential and interest. Coprocessors are a class of computer processors that
supplement CPUs, often improving the execution of certain functions due to
architectural design choices. We explore the approach of Services for
Optimized Network Inference on Coprocessors (SONIC) and study the deployment
of this as-a-service approach in large-scale data processing. In the studies,
we take a data processing workflow of the CMS experiment and run the main
workflow on CPUs, while offloading several machine learning (ML) inference
tasks onto either remote or local coprocessors, specifically graphics
processing units (GPUs). With experiments performed at Google Cloud, the
Purdue Tier-2 computing center, and combinations of the two, we demonstrate
the acceleration of these ML algorithms individually on coprocessors and the
corresponding throughput improvement for the entire workflow. This approach
can be easily generalized to different types of coprocessors and deployed on
local CPUs without decreasing the throughput performance. We emphasize that
the SONIC approach enables high coprocessor usage and enables the portability
to run workflows on different types of coprocessors.

##  0.1 Introduction

During the first two runs of the CERN LHC [1], the ATLAS [2] and CMS [3]
Collaborations have analyzed trillions of high-energy proton-proton or lead-
lead collisions and produced an extensive suite of physics results. Among
these are the discovery of the Higgs boson [4, 5, 6] in the standard model
(SM) and stringent constraints on various beyond the SM physics scenarios,
such as supersymmetry [7, 8, 9, 10, 11, 12, 13, 14] and exotic heavy-particle
or dark matter candidate production [15, 16, 17, 18, 19, 20, 21]. In order to
measure the SM with higher precision and search for new processes with lower
cross sections, the amount of data that is delivered by the LHC and processed
by the experiments is expected to increase dramatically in ongoing and future
physics runs [22, 23].

The high data-taking rates and increasing event complexity of the ATLAS and
CMS experiments present a significant computational challenge for data
processing [24, 25]. A two-level trigger system is employed to run fast
algorithms and reduce the data rate from 40TB/s to about 10GB/s [26, 27, 28].
While this is a significantly smaller rate, it is still very challenging for
subsequent processing steps. As discussed in Refs. [29, 30, 31], even with
optimistic expectations for computing research and development, the projected
computing needs for CMS will be only narrowly satisfied.

At present, data processing is mainly carried out using central processing
units (CPUs) but their expected performance increase is limited [32].
Nevertheless, data processing can be supported with a variety of modern
architectures, such as graphics processing units (GPUs), field-programmable
gate arrays (FPGAs), application-specific integrated circuits (ASICs), or
Graphcore intelligence processing units (IPUs) [33, 34], which can
collectively be referred to as coprocessors. These architectures are becoming
increasingly popular because of their large numbers of processing units,
inherent parallelization designs, and more energy-efficient and
environmentally friendly computing, especially suited for machine learning
(ML) algorithm computations.

Within high-energy physics (HEP), deep-learning (DL) algorithms are already
widely used for regression and classification tasks, and their popularity is
growing rapidly [35, 36, 37, 38, 39]. Because of this growth, inference
execution for these algorithms consumes increasingly large fractions of the
overall processing load. However, the processing load can be shared and
accelerated by using heterogeneous computing architectures. Therefore,
developing a framework to enable and optimize the deployment and portability
of coprocessors is of considerable interest to HEP experiments [40]. As a
proof of concept, this paper focuses on accelerating the inference for ML
algorithms in one of the CMS data processing stages, explained in Section
0.2.3. These algorithms collectively take about 10% of the total processing
time of that stage. For this approach to resolve the future computing
challenge, ML algorithms would need to be used for larger portions of CMS data
processing. Utilizing ML for tasks such as tracking [41, 42, 43, 44],
clustering [45, 46, 47], particle reconstruction [48, 49, 50, 51, 52], and
particle identification [53, 54, 55, 56, 57, 58, 59] is an active area of
research.

One of the approaches for the coprocessor deployment is to equip every CPU
machine with coprocessors, referred to as directly connected. In this
scenario, every CPU thread within the machine can communicate with the
coprocessor. However, since the coprocessor-to-CPU ratio must be determined
before deployment, the coprocessor resources are unlikely to be optimally
utilized, leading to either unused resources when undersaturating, or
decreased performance when oversaturating. In addition, it is difficult to
utilize additional or more advanced coprocessor resources after deployment.

An alternative approach is inference as a service (IaaS), where coprocessor
resources are separated from CPU machines. As illustrated in Fig. 1, in this
scheme CPU-based _clients_ can send the computing request with the necessary
information to coprocessor-based _servers_ via network calls. The servers
running on coprocessor resources can perform computing tasks upon request.
This removes the restriction of a coprocessor only being used by the CPUs
directly connected to it, allowing it to accept processing requests from any
CPUs (local or remote), as long as network communications are available.
Certain types of coprocessors can be allocated for specific tasks, and the
coprocessor-to-CPU ratio is flexible. Resource utilization can therefore be
optimized based on specific tasks, as the number of client-side jobs using a
single server can be varied depending on the computational demands of a given
task. Furthermore, at the software level, since the support for coprocessors
and CPU workflows is separated, it is easier to support different types of
coprocessors. This ensures algorithm portability with minimal maintenance
burden. The implementation of IaaS in experimental software frameworks can be
accomplished using the Services for Optimized Network Inference on
Coprocessors (SONIC) approach [60], as described in Section 0.3.

Figure 1:  An example inference as a service setup with multiple coprocessor
servers. Clients usually run on CPUs, shown on the left side; servers hosting
different models run on coprocessors, shown on the right side.

The SONIC approach has previously been demonstrated using FPGAs [60, 61] and
GPUs [62, 63] with a variety of ML algorithms. These studies demonstrated that
offloading ML algorithms with the SONIC approach adds little additional
computational overhead from the client-server communications and other
operations. Therefore, a corresponding increase in the throughput has been
observed when offloading algorithms to these faster coprocessors.

Heterogeneous computing frameworks using GPUs have appeared recently in
multiple non-IaaS contexts in HEP as well. One of the first significant real-
time applications was in the ALICE high-level trigger (HLT) system [64], where
GPUs were used to accelerate track reconstruction. Similarly, a fully GPU-
based first-level trigger has been implemented in LHCb [65], which runs on 200
GPUs [66]. In CMS, a GPU-specific version of the pixel tracking domain (non-
ML) algorithm called Patatrack was developed, along with several other
reconstruction algorithms, which are now employed in the HLT for Run 3 to
reduce the processing time per event [67]. Further review of GPU usage in
real-time applications for HEP can be found in Ref. [68].

In this paper, we take one data production workflow, called the Mini-AOD
production workflow [69], as an example, and study the performance gains of
applying the IaaS framework to this workflow. The abbreviation “AOD” comes
from a lower-level data format called Analysis Object Data (AOD) discussed in
Section 0.2. In the current Mini-AOD production workflow, which is a data
refinement and slimming step, about 10% of the computing time is consumed by
ML algorithm inference, which can be easily accelerated on GPUs. We first
summarize studies of the optimization and acceleration of the inference of
each ML algorithm on GPUs. Then we show that the IaaS scheme, which is
implemented in the CMS software framework [70, 71] via the SONIC approach, not
only decreases processing time but can also be applied in large-scale
production to optimize GPU utilization. Finally, we show that the SONIC
approach can be easily ported to different types of coprocessors such as IPUs,
and it can also run on local CPUs without decreasing the throughput. The study
of power, power savings, and sustainability is also important, but is beyond
the scope of this paper.

The paper is organized as follows. Section 0.2 provides a brief overview of
the CMS computing architecture and different data tiers for production.
Section 0.3 discusses the SONIC approach in detail, including the technical
implementation in , the current inference servers, and features of the
approach. Section 0.4 includes the data set used for the studies and the
algorithms that can currently use the SONIC approach for inference. Sections
0.5 and 0.6 provide detailed studies evaluating the computational performance
of this approach in the Mini-AOD production workflow. Finally, Section 0.7
summarizes the studies and discusses future plans.

##  0.2 The CMS detector, software, and computing

###  0.2.1 Introduction to the CMS experiment

The LHC provides countercirculating beams of high energy protons or heavy
ions, such that bunches of particles in these beams can interact with each
other in the center of the CMS detector [3] nearly every 25ns. When particles
from the countercirculating beams collide, a large variety of physical
processes can occur, which lead to the creation of either fundamental or
composite particles. These particles, or their decay products, can then
propagate into the CMS detector, which is designed to measure their energy and
momentum. In the context of this paper, we will refer to a readout cycle of
the detector as an event. The detector itself comprises multiple layers
including silicon pixels and strips, crystal electromagnetic calorimeters,
sampling hadron calorimeters, and muon spectrometers. Each component of the
detector has active elements that create electrical signals when particles
interact with the detector. Each discrete signal is called a hit.

Events of interest are selected using a two-tiered trigger system. The first
level (L1), composed of custom hardware processors, uses information from the
calorimeters and muon detectors to select events at a rate of around 100kHz
within a fixed latency of 4 [72]. The second level, known as the HLT, consists
of a farm of processors running a version of the full event reconstruction
software optimized for fast processing, and reduces the event rate to around
1kHz before data storage [28]. Processing in the trigger system is referred to
as online computing, while subsequent processing is known as offline
computing.

###  0.2.2 The framework

The framework is an open-source software framework that is used in triggering,
data formatting and processing, simulation, and offline analysis [70, 71].
Components of the framework are used in combination to extract high-level
physics information for each event from detector hits. The framework processes
each event with a sequence of algorithms, converting hits from electrical
signals to position and energy measurements, linking these measurements into
clusters [73] and trajectories [74], combining trajectories and clusters into
single-particle representations or jets corresponding to hadronic showers
[75]. Additional algorithms in , ML-based reconstruction algorithms, can be
run to determine quantities such as an overall imbalance of the momentum in
the direction perpendicular to the beam or to tag jets as containing or being
produced by certain particles.

The framework uses Intel Threading Building Blocks [76] to enable task-based
multithreading. As explained in Ref. [77], this multithreading implementation
allows for asynchronous nonblocking calls to external resources, such as a
GPU, via ExternalWork. This setup optimizes CPU resource utilization by
minimizing downtime; the CPU is allowed to continue executing algorithms that
do not require a coprocessor or depend on the results of the coprocessor-
dependent algorithm while waiting for the external call to return.

###  0.2.3 The CMS data tiers

The framework is used both in online and offline contexts within CMS. While
the L1 trigger is hardware-based, the HLT is composed of algorithms in . The
framework also contains the algorithms that are used to process the raw data
after they have been stored, deriving the higher-level information useful for
a wide range of data analyses based on reconstructed objects. The centralized
CMS offline data processing flow involves three steps, which are performed
both for raw data and simulated data sets. In the first step, an AOD format
for every event is derived. This contains high-level information, such as
reconstructed particles and jets, but the data size is large. In the second
step, a slimmed, higher-level Mini-AOD derivation is created [69]. Mini-AOD
files are designed to be relatively small and accessible, serving as an
intermediate step and the standard foundation for a variety of CMS physics
analyses. Finally, in the third step, a further slimmed Nano-AOD format is
created that contains only very high-level physics observables [78]. This
format is commonly used directly for physics analyses. Data sets are
reprocessed regularly to incorporate the latest Monte Carlo event generator
tunes [79], calibrations, and algorithm improvements. Table 1 summarizes the
average event size of these different data tiers with 2016–2018 (Run 2) data-
taking conditions [80].

Table 1: Average event size of different CMS data tiers with Run 2 data-taking conditions [81, 69, 78]. Data tier | Event size [kB/event]
---|---
Raw | 1000
AOD | 480
Mini-AOD | 35–60
Nano-AOD | 1–2

In the scope of the studies in this paper, we choose Mini-AOD production as
our test case. Mini-AOD files are derived from the AOD data format, reducing
the size per event by an order of magnitude. Mini-AOD processing involves a
wide variety of algorithms that propagate, filter, and reanalyze the AOD input
objects.

##  0.3 The SONIC approach

This section describes the implementation of the SONIC approach in and the
server technology currently used, which is the NVIDIA Triton Inference Server
(Triton) [82]. The benefits of running inference with the SONIC approach along
with additional complexity and other implications are also discussed in this
section.

###  0.3.1 Implementation in

The SONIC approach is implemented in through the ExternalWork framework
component [77] and accesses coprocessor resources on remote servers via gRPC
Remote Procedure Calls (gRPCs), which is a cross-platform open-source high-
performance remote procedure call framework originally developed by Google
[83]. An illustration of this procedure, where client jobs make asynchronous,
nonblocking gRPC calls to remote servers, is shown in Fig. 2. Multiple servers
can run on multiple coprocessors, with load balancers in between. An important
aspect of this scheme is that the client-side code does not need to be able to
run any particular inference packages or frameworks; it simply has to collect
the relevant input data for a trained model, communicate that information to
the server in the expected format, and handle the output from the server.

Figure 2:  Illustration of the SONIC implementation of IaaS in . The figure
also shows the possibility of an additional load-balancing layer in the SONIC
scheme. For example, if multiple coprocessor-enabled machines are used to host
servers, a Kubernetes engine can be set up to distribute inference calls
across the machines [84]. Image adapted from Ref. [63].

The client-side framework code for the SONIC approach is split into two
packages: a core package [85] containing class templates and other common
infrastructure for the IaaS approach, and a dedicated package [86] to interact
specifically with the Triton server. SONIC modules provide a similar interface
to standard modules. The partitioning into two packages reflects that the
SONIC approach can be implemented for multiple server backends. For example,
the first implementation [87, 60] used the Microsoft Brainwave service [88],
which provides FPGA resources. Another potential backend service is the
TensorFlow as a Service framework, which can provide access to TensorFlow-
based ML models via the HTTP protocol [89]. This framework was introduced
along with the Machine Learning as a Service pipeline for HEP (MLaaS4HEP),
which is a streamlined mechanism for training and deploying models from data
in ROOT data format files [90]. In practice, given the generality and openness
of the protocols used by Triton, which are themselves an extension of the
KServe standard [91], we expect that future development of the SONIC approach
will continue to use these protocols, even if the backends or coprocessors
change. Further discussion of different backends and coprocessors can be found
in Section 0.6.

A central TritonService is provided to keep track of all available servers and
which models each of them serves. By default, each client-side module only
needs to specify the model it needs, and the TritonService will automatically
find a server hosting that model. A client-side module can optionally specify
a preferred server, which the TritonService will use if the server is found
and confirmed to serve the required model.

Within , a mechanism has been implemented to account for the possibility that
a client job cannot access a specified server for whatever reason. In this
case, a “fallback” server is automatically created using either local GPU
resources if they are available or the CPU resources allocated to the client
job in question. The client then makes gRPC calls to that local fallback
server, which introduces negligible latency. Detailed studies related to these
fallback servers are discussed later in Section 0.6.1. In general, the server
overhead consumes very little of the CPU resources beyond what would be used
for conventional inference, such that the per-event processing time is not
strongly affected by the SONIC approach relative to running without it.
Fallback servers are automatically shut down when the job finishes.

###  0.3.2 The NVIDIA Triton Inference Server

As shown in Fig. 2, the server-side implementation of the SONIC approach in
currently uses Triton for inference on coprocessors [82, 92]. Triton is an
open-source solution whose protocols are public and extensible, as noted
above. It supports inference of ML algorithms, called models, in most modern
formats, including PyTorch [93], TensorRT (TRT) [94], ONNX Runtime (ONNX)
[95], TensorFlow [96], and XGBoost [97]. It also supports custom backends for
alternative tasks, such as classical rule-based domain algorithms and
inference on different types of coprocessors. Several features of Triton are
worth highlighting:

  * •

_Multiple model instances_ : A single server can host multiple models at the
same time or even multiple instances of the same model to allow concurrent
inference requests.

  * •

_Dynamic batching_ : Usually coprocessors have many processing units and the
number of operations from one inference call is not enough to fully utilize
the coprocessors. With dynamic batching, if multiple inference calls are made
within a window of time, the server can concatenate the inputs of all calls
into a single batch, which improves the GPU utilization and therefore
increases the overall throughput. In practice, the time window is typically
chosen to be about the client-side CPU processing time per event, such that
the inference can be processed in time without delay.

  * •

_Model analyzer_ : Parameters like the number of model instances on a single
server, the length of a batching window, and the optimal batch size are
tunable and can be optimized on a case-by-case basis. Triton provides a model
analyzer tool to aid in this optimization [98]. It can mimic the clients and
send randomized inputs or some pre-saved data to the server. The server
performance is measured, and the optimal deployment configuration can be
determined by scanning the parameter space.

  * •

_Ragged batching_ : Traditionally, inference can be performed on a batch of
multiple inputs as long as each input is of the same size. In HEP data, the
size of the input for inference can vary from one instance to another, making
it harder to batch them together. For example, if an algorithm uses
information from every particle in an event as input, it is difficult to batch
inference requests from multiple events because events can have a wide range
of numbers of particles. Ragged batching allows inference requests with
different sizes to be batched together, thereby improving the performance.
This feature is relatively new and not yet fully studied in this paper.

Triton servers can use one or multiple GPUs on the same machine with a built-
in load balancer. They can also run purely on CPU resources when there are no
GPU resources available. For other types of coprocessors, Triton servers can
also be used with the help of custom backends. A server requires a trained
model file and a configuration file specifying input and output variable
names, shapes, types, and model versions, along with the preferred batch size
and other details that can be acquired through the inference optimizations.
These model and configuration files are currently accessible through the
CernVM-File System (CVMFS) [99], and are tracked by the release management
system.

###  0.3.3 Advantages of the SONIC approach

Inference as a service, as implemented in the SONIC approach, provides several
advantages and benefits, which are summarized here. Many of these features
arise from the differences between the IaaS approach and the more traditional
approach of HEP software frameworks to use only local computing resources.
These features include:

  * •

_Containerization_ : The SONIC approach factorizes ML frameworks out of the
client software stack, i.e., , reducing the workload to support a wide variety
of ML models. With the SONIC approach, one can use any framework supported by
Triton, including custom backends, without needing to modify the software
stack to resolve library dependencies and ensure compatibility between all
external packages. This allows us to pick the best inference backend for one
algorithm, with less concern for the implementation details.

  * •

_Simplicity_ : Because of the containerization discussed above, SONIC client-
side code is simpler and more general than the corresponding direct inference
code. SONIC modules need only implement the conversion of input data into the
server’s desired format and the reverse operation for output data.

  * •

_Flexibility_ : In the SONIC paradigm, the connections between client CPUs and
server coprocessors are not fixed. The servers can be physically located
nearby or far from clients. Clients from many machines can access a single
server running on either one or multiple coprocessors. Similarly, a single
client can access multiple different servers running on multiple different
machines.

  * •

_Efficiency_ : The SONIC approach enables balanced utilization of coprocessor
resources. By optimizing the coprocessor-to-CPU ratio for different tasks, it
is easier to fully utilize coprocessor resources without oversaturating them.

  * •

_Portability_ : Through the use of the SONIC approach, client-side code does
not have to be modified to take advantage of different types of coprocessors.
Only a consistent protocol for communicating with the inference server is
required, regardless of the underlying hardware: CPU, GPU, FPGA, IPU, or any
other architecture.

  * •

_Accessibility_ : If GPUs or other coprocessors are not available locally, the
only way to accelerate workflows is to access those resources remotely, as a
service. The SONIC architecture implements this use case for CMS, allowing the
use of remote coprocessors.

###  0.3.4 Limitations and production requirements

However, with these advantages come additional complexity and changes in
resource usage. Production jobs using the SONIC approach rely on a separate
server running different software, compared to the existing scheme in which
jobs only execute on local hardware. This implies several additional
considerations:

  * •

_Server failures_ : Inference servers, remote or local, may experience
software or hardware failures that prevent them from running. These new
failure modes are mostly independent from existing known sources of failures,
potentially leading to an overall increase in the rate of job failures.
However, these failures can be mitigated with server-side technology, such as
the load balancer Kubernetes [84], and client-side protocols, like the local
fallback server.

  * •

_Network usage_ : The use of remote inference servers necessarily implies an
increase in network traffic, as input and output data must be communicated
over the network. Typically, input data are much larger than output data; the
total usage depends on the algorithm. For the Mini-AOD production workflow
tests presented here, the network usage is discussed in Section 0.5.3. High
network usage can be mitigated using compression, with some tradeoff in
throughput from the additional operations to compress and decompress the data.
In the studies done here, we have not observed significant issues with network
usage, and as a result, analysis of the tradeoff between compression and
network usage is not included.

  * •

_Memory usage_ : The use of remote inference servers reduces the local memory
usage of production jobs, compared to the direct inference approach. However,
the use of local fallback servers, whether to mitigate remote server failures
or take advantage of the containerization and portability of the SONIC
approach, implies increased memory usage. Running the server process locally
is generally expected to use more memory than the corresponding direct
inference libraries. Measurements of memory usage are presented in Section
0.6.1.

These potential drawbacks, especially uncorrelated failures and network usage,
are similar to those from other distributed services used in CMS production,
such as the conditions database or XRootD [100, 101]. These can potentially
impact the processing performance and should be studied more intensively.

Handling this additional complexity requires new components to be deployed in
the CMS workflow management system. We provide below several examples of such
operational concerns.

  * •

_Server discovery_ : The SONIC approach allows the use of remote coprocessor
resources, which requires the information of servers running on these
resources, , IP addresses, port numbers, served models, and number of GPUs, to
be available to the client jobs. Such information can be collected in the site
configurations and provided by the job submission system or a central service.
For the studies presented in this paper, servers are launched manually, and
their addresses are written into the configuration files.

  * •

_Load balancing_ : Different production jobs running on different data sets
have different coprocessor resource demands. A load balancer, such as
Kubernetes, can be set up to dynamically load and unload models on different
servers and distribute client inference requests to these servers. The load
balancing for large-scale inference requests is tested and discussed in
Section 0.5.3.

  * •

_Versioning_ : CMS production requires the versions of , servers, ML models,
and ML backends to be controlled and recorded for proper provenance tracking.
The ML model versions are already tracked by the release management system;
the server and backend versioning should be included in the same system for
consistency. For the studies presented in this paper, the prebuilt Triton
server provided by NVIDIA was used.

  * •

_Optimization procedure_ : While the SONIC approach enables the benefits
described in Section 0.3.3, it does not necessarily make them trivially
attainable. Some use-case-specific optimizations are required before large-
scale deployment, as explained in more detail in Section 0.5. For example,
each model should be analyzed individually to find a preferred batch size.
Similarly, while the flexible coprocessor-to-CPU ratio permits more efficient
utilization of resources, one must first determine an appropriate coprocessor-
to-CPU ratio. This is further complicated by the fact that the ratio can
depend on the hardware used, such as CPU or GPU type, and on the physics
content of the data set being processed.

##  0.4 Physics data sets, algorithms, and benchmark setup

This section describes the physics data set, the ML algorithms in that can
currently use the SONIC approach for inference, and the computing resources
used in the studies.

###  0.4.1 Data set and software versions

In the studies, we chose to process a simulated Run 2 data set of events with
one top quark and one anti-top quark (), as it includes many types of physics
objects, including leptons, heavy-flavor jets, and missing transverse
momentum. Here, a heavy-flavor jet is one that originates from a charm or
bottom quark, and missing transverse momentum, or , is defined as the negative
vector sum of the transverse momenta of all of the reconstructed particles in
an event, and its magnitude is denoted as [102]. The data set was copied to
local disk to factor out the effects of remote input-output (I/O) limitations
for the benchmarks. The version CMSSW_12_0_0_pre5 and Triton server version
21.06 were used for the studies.

###  0.4.2 Algorithms supported by the SONIC approach

Adapting an ML algorithm to work with the SONIC approach requires a small
effort to write code to prepare the network inputs and save the network
outputs within . A comparison of the producers used for direct inference and
for the SONIC approach can be found in Refs. [103, 104]. Most of the pre- and
post-processing steps are the same, while the operations that send inference
requests to the ML backends directly or to the servers are different.

In these studies, we tested three independent and computing-intensive ML-based
algorithms in the Mini-AOD workflow. These algorithms were chosen to
illustrate the performance of the SONIC approach for models with differing
input and output sizes, physics applications, and backends, as detailed below.

#### The ParticleNet algorithm

Graph neural networks (GNNs) have been demonstrated to achieve state-of-the-
art performance for identifying jets as arising from specific particles, a
task known as jet tagging [53, 54, 55]. ParticleNet [54] is a GNN-based
algorithm for jet tagging and regression that represents jets as “particle
clouds.” This algorithm was trained in PyTorch [93] and exported to the ONNX
format. With the SONIC approach, it is possible to perform inference with
ParticleNet in the following formats: ONNX; PyTorch; and PyTorch with TRT
[94], which optimizes model performance on NVIDIA devices.

There are four different trained versions of ParticleNet currently running in
the Mini-AOD workflow for different purposes:

  1. 1.

tagging anti-jets [105], clustered with the package [106], with a radius
parameter of 0.4 (AK4 jets), abbreviated PN-AK4 [107],

  2. 2.

tagging anti-jets with a radius parameter of 0.8 (AK8 jets) [108],

  3. 3.

mass-decorrelated tagging for AK8 jets [108], and

  4. 4.

mass regression for AK8 jets [109].

The three AK8 ParticleNet algorithms are abbreviated PN-AK8 in the following
sections. All ParticleNet variations can be hosted on Triton servers.

The inputs to ParticleNet are the kinematic and flavor properties of the
particle constituents of each jet and the secondary vertices associated with
the jet, including 20 features for one particle and 11 features for one
vertex. Up to 100 particles and 10 vertices are used in the inputs for AK8
jets; if there are more than 100 particles in a jet, the 100 particles with
the highest transverse momenta are used, and if there are more than 10
vertices, then the 10 with the highest displacement from the beamline are
used. For AK4 jets, the maximum numbers of particles and vertices are 50 and
5, respectively. For the three tagging versions of ParticleNet, the outputs of
each inference are category probabilities for a variety of predefined jet
categories, such as the presence of a Higgs boson or the presence of a top
quark. For the mass regression, the output is a single value: the predicted
jet mass.

In Mini-AOD processing, inference is performed separately for each jet in a
given event, so the number of ParticleNet inferences depends on the specific
physics processes and can vary substantially from event to event. When running
the standard version of ParticleNet, no inference batching is performed, so
each inference is truly performed separately. In this context, each inference
can have a variable number of inputs with no padding involved. When using the
SONIC approach for ParticleNet, it is simplest to batch all jets in an event
into a single inference request, such that input particle and secondary vertex
information for each jet in an event is sent to the server in a single
request. Because ragged batching is not yet fully implemented in the inference
server, the different inference requests must have the same number of inputs
per batch. Therefore, in subsequent performance studies, we adopt a maximally
padded approach, where zeros are added to the vectors of particles and
vertices for AK8 (AK4) jets such that they have consistent lengths of 100 (50)
and 10 (5), respectively.

An illustration of the jet content of the Run 2 simulated data set is given in
Fig. 3. The distributions of the number of jets per event and particles per
jet are provided for both AK4 and AK8 jets. As shown in the figure, the number
of jets can vary dramatically from event to event, indicating the importance
of dynamic batching.

Figure 3:  Illustration of the jet information in the Run 2 simulated data set
used in subsequent studies. Distributions of the number of jets per event
(left) and the number of particles per jet (right) are shown for AK4 jets
(upper) and AK8 jets (lower). For the distributions of the number of
particles, the rightmost bin is an overflow bin.

In the configuration where all AK4 jets are padded to 50 particles and 5
vertices and all AK8 jets are padded to 100 particles and 10 vertices, a
single four-threaded job processing a Run 2 event will generate about 140kB/s
of server input for AK4 jets and about 10kB/s of server input for each of the
AK8 jet versions of ParticleNet. At a processing rate of about 4 events per
second, with about 16 AK4 jets per event, this corresponds to about 2kB of
information per jet, which is consistent with the number of float inputs per
jet.

#### The DeepMET algorithm

DeepMET [110] is a TensorFlow-based deep neural network model that estimates
the in an event. The vector is associated with either the production of
neutrinos or potential beyond the SM particles that could propagate through
the detector interacting only weakly. The inputs to DeepMET are 11 features
from each particle [75] in an event, with zero-padding up to 4 500 particle
candidates for a given event. This zero padding is used for both the standard
version of DeepMET and its SONIC implementation. In both cases, only one
inference is made per event. DeepMET outputs two values for each event: the
components in the transverse plane. Because every event necessarily has the
same number of inputs, dynamic batching is automatically available via the
SONIC approach, aiding inference efficiency in cases where many client jobs
make concurrent requests within a certain time window (each client thread will
make a request about once per second). A single four-threaded job processing a
Run 2 event will generate about 1.3MB/s of server input for DeepMET.

#### The DeepTau algorithm

The DeepTau algorithm [111] is a TensorFlow-based deep neural network model to
identify hadronically decaying leptons from the jet collections. The inputs to
the network include low-level particle features of electrons/photons, muons,
and hadrons, and high-level features such as the lepton candidate kinematic
information. The algorithm splits nearby cones into grid cells, and loops over
the cells to collect the particle features. The low-level particle features
are then processed by a convolutional neural network (CNN) to extract the
particle-level information. The high-level features are processed by a fully
connected neural network (FCNN) to extract the lepton candidate-level
information. The outputs of the CNN and FCNN are combined and processed by a
final FCNN to produce a four-dimensional vector that represents the
probability that the candidate originates from a genuine lepton, a muon, an
electron, or a quark or gluon jet. In the implementation of direct inference
in , the network is split into three sub-models. The first two networks
process the lower-level input information separately, and the third network
combines the information from the first two models with the high-level
information and outputs the discriminator values. In the SONIC implementation,
we use a combined model with zero-padded inputs, so that dynamic batching can
be used and GPUs can be utilized more efficiently. A single four-threaded job
processing a Run 2 event will generate about 4.7MB/s of server input for
DeepTau, making it the most demanding algorithm explored in this study in
terms of server input.

###  0.4.3 Average processing time

The processing time is defined as the real-world time spent between starting
and finishing processing one event. The processing time breakdown for the
three algorithms highlighted above in the Run 2 events is presented in Table
2, measured with 1-thread jobs using one single CPU core. The per-event per-
thread average processing time is about 1s; PN-AK4 consumes about 4.3% of this
time and PN-AK8 about 1.1%, while DeepMET and DeepTau take about 1.3 and 2.1%,
respectively. Thus, for this collection of events, the algorithms supported by
the SONIC approach account for about 9% of the total processing time. This
fraction is dependent on the type of events. For example, if there are fewer
jets per event, ParticleNet will consume a smaller fraction of the total
processing time. While the fraction of the total workflow accelerated with the
SONIC approach in this paper is less than 10%, an increasing number of ML
algorithms are being integrated into CMS data processing. Because of this, it
will be possible to accelerate a larger fraction of the total processing time
with the SONIC approach in the future.

Table 2: The average time of the Mini-AOD processing (without the SONIC approach) with one thread on one single CPU core. The average processing times of the algorithms supported by the SONIC approach are listed in the column labeled “Time.” The column labeled “Fraction” refers to the fraction of the full workflow’s processing time that the algorithm in question consumes. Together, the algorithms currently supported by the SONIC approach consume about 9% of the total processing time. This table also contains the expected server input for each model type created per event in Run 2 events in the column labeled “Input.” Algorithm | Time [ms] | Fraction [%] | Input [MB]
---|---|---|---
PN-AK4 | 42 | 4.3 | 0.04
PN-AK8 | 11 | 1.2 | 0.01
DeepMET | 13 | 1.3 | 0.33
DeepTau | 21 | 2.1 | 1.18
PN-AK4+PN-AK8+DeepMET+DeepTau | 88 | 8.9 | 1.55
Full workflow | 990 | 100.0 |

###  0.4.4 Computing resources

Fermilab, via the LHC Physics Center, provides CPU-only batch resources and a
set of interactive machines with NVIDIA Tesla T4 GPUs [112]. Through Fermilab,
we were also able to steer the allocation of cloud resources (see below) using
the HEPCloud [113] framework. These resources are physically located in
Illinois.

The Google Cloud Platform (GCP) provides virtual machines (VMs) that are
either CPU-only or enabled with NVIDIA Tesla T4 GPUs. By default, the CPUs are
a mix of Skylake, Broadwell, Haswell, Sandy Bridge, and Ivy Bridge
architectures [114]. In GCP, we created customized machines with specified
numbers of CPU threads or different ratios of CPU threads to numbers of GPUs.
Similarly, a customized, dynamic SLURM [115] cluster was created that could
instantiate and deplete four-thread VMs on demand for medium-scale tests that
ran jobs across $\mathcal{O}(1000)$ CPUs. The cluster’s four-thread
configuration was chosen so that four-threaded jobs would saturate the node’s
resources, improving the reproducibility of timing tests. CPU-only VMs could
also be instantiated through HEPCloud. In GCP, we also maintained GPU-enabled
VMs running Triton servers that both the SLURM and HEPCloud client nodes could
access. These resources are physically located in Iowa.

At the Purdue CMS Tier-2 computing cluster, tests were performed with reserved
CPU-only and GPU-enabled machines. The CPU-only machines are 20-core Intel
E5-2660 v3 machines, and the GPU-enabled machines each have two AMD EPYC-7702
CPUs [116] with an NVIDIA Tesla T4 GPU. Reserving these nodes allowed for
controlled resource utilization, leading to more reproducible timing tests.
These resources are physically located in Indiana.

The diversity of resource locations used in these studies demonstrates one of
this approach’s key features, namely that it enables the use of nonlocal
resources. We were able to start a server in one location and have client jobs
running at another site. One such study is presented in Section 0.5.2.

Within a GCP project, VMs do not have ingress bandwidth limits other than
machine limits. These are above 10GB/s, which, based on the scans in Section
0.5.1 and Table 2, is far above the amount of information that can be sent
from client VMs to a server-hosting machine without saturating GPU resources.
For example, in GCP, if a single GPU is used to host all model types, then in
a typical running scenario, it can service about 33 simultaneous four-threaded
client jobs, or 132 client cores. A single client core generates about
1.55MB/s of traffic, so a total of 200MB/s of ingress is expected per GPU at
the saturation point. Similarly, the ingress bandwidth from the server into
the client VMs will be very small, as inference results are no more than
$\mathcal{O}(10)$ float values per algorithm. The GCP does impose VM egress
bandwidth limits, which is typically 0.25GB/s per CPU for a VM to an internal
IP address. However, given that a single client-side core will generate about
1.55MB/s of network traffic, this is also well within the allowed limit. There
are slightly more stringent restrictions for GCP bandwidth to external IP
addresses. However, when we performed such tests as in Section 0.5.2,
typically only one external machine was involved, with levels of network
traffic well within that allowed by the restrictions.

##  0.5 Performance

In this section, we discuss the performance of running the Mini-AOD production
workflow with the SONIC approach. We compare to direct inference, which refers
to the standard approach where the inference is performed using the ML
backends integrated into on CPUs.

As mentioned in Section 0.3.2, we first optimize the per-algorithm workflows
to find the optimal configurations for the full production workflow. Next we
check the impact of deploying servers on different sites. Finally, we mimic
the real production jobs by running scale-up tests and evaluating the
performance.

###  0.5.1 Per-algorithm inference optimization

To maximize the resource efficiency and throughput benefits of the SONIC
approach, we first perform single-model characterization studies independent
of . For example, to maximize GPU usage without oversaturation, we need to
find the optimal ratio of client-side CPUs to server-side GPUs, batch size for
inference in the Triton server, and model configuration that will provide the
highest throughput.

The latter two optimizations can be performed with the Triton Model Analyzer
tool [98]. This tool feeds inputs in the correct tensor format (either
randomized numbers or real data) to a loaded model hosted on a server,
allowing for robust characterization of processing time per inference or
exploration of the impact of batch size. As an example, we measure the
processing time and throughput of the ParticleNet algorithm for AK4 jet
tagging on one NVIDIA Tesla T4 GPU, with different inference backends
supported in Triton: ONNX, ONNX with TRT, and PyTorch [93] (labeled PT in the
figures). The results are shown in Fig. 4.

Figure 4:  Average processing time (left) and throughput (right) of the PN-AK4
algorithm served by a Triton server running on one NVIDIA Tesla T4 GPU,
presented as a function of the batch size. Values are shown for different
inference backends: ONNX (orange), ONNX with TRT (green), and PyTorch (red).
Performance values for these backends when running on a CPU-based Triton
server are given in dashed lines, with the same color-to-backend
correspondence.

For smaller batch sizes, the TRT version of the ParticleNet algorithm leads to
the highest throughput in total inferences per second, while at higher batch
sizes, the PyTorch version gives higher throughput. In the version of
ParticleNet supported by SONIC in , all the jets in a single event are batched
together, and there are on average 16 AK4 jets per event in our data set. To
achieve higher batch sizes in a production scenario, multiple clients would
need to make an inference request to the same server within a relatively
narrow time window. Triton allows us to specify a preferred batch size, such
that if many inference requests are queued within the time window, the server
will perform inference with batches of approximately the specified size. For
example, the peak throughput seems to plateau around a batch size of 100 for
the PyTorch version of ParticleNet.

Figure 5:  Average processing time (left) and throughput (right) of one of the
AK8 ParticleNet algorithms served by a Triton server running on one NVIDIA
Tesla T4 GPU, presented as a function of the batch size. Values are shown for
different inference backends: ONNX (orange), ONNX with TRT (green), and
PyTorch (red). Performance values for these backends when running on a CPU-
based Triton server are given in dashed lines, with the same color-to-backend
correspondence.

Similar studies can be performed on other models as well. Figures 5, 6, and 7
show the processing time and throughput scans of PN-AK8 jet tagging, DeepMET,
and DeepTau models, respectively. Some backend tests are skipped due to the
model availability.

Figure 6:  Average processing time (left) and throughput (right) of the
DeepMET algorithm served by a Triton server running on one NVIDIA Tesla T4
GPU, presented as a function of the batch size. Similar performance when
running on a CPU-based Triton server is given in dashed lines.  Figure 7:
Average processing time (left) and throughput (right) of the DeepTau algorithm
served by a Triton server running on one NVIDIA Tesla T4 GPU, presented as a
function of the batch size. Values are shown for different inference backends:
TensorFlow (TF) (orange), and TensorFlow with TRT (blue). Performance values
for these backends when running on a CPU-based Triton server are given in
dashed lines, with the same color-to-backend correspondence.

The model analyzer can also determine approximately how many inferences per
second a single GPU can perform before saturation. For example, for the
PyTorch version of ParticleNet for AK4 jets, a single Tesla T4 GPU can perform
about 5500 inferences per second without saturating. This corresponds to about
350 events per second, given the typical number of jets per event. Based on
this, we can estimate how many CPU clients one GPU can support in parallel. A
typical production configuration runs four-threaded Mini-AOD jobs, each of
which processes about 3.9 events per second. Therefore, a single GPU should be
able to handle about 90 four-threaded jobs in parallel running asynchronously,
assuming it is being used exclusively for a single server hosting the PyTorch
PN-AK4 model.

This expected saturation point can be tested directly by running the Mini-AOD
workflow in and scanning the throughput as a function of the number of CPU
clients pinging one GPU server. Figure 8 shows such tests for the PN-AK4, PN-
AK8 jet tagging, DeepMET, and DeepTau models, which were performed in GCP
using a custom SLURM cluster. For each model, a single server running on one
NVIDIA T4 GPU was started on one cloud VM, and client-side jobs were executed
in VMs that had 4 CPU threads. The tests for each model class were performed
separately, and as one model class was being tested, the direct-inference
versions of the other models were used.

The accelerated versions of the workflow can be compared with the dashed black
line, which represents the average throughput of the workflow when the direct-
inference versions of all the models were used. In this case, jobs were also
started in the VMs with 4 CPU threads, but there was no communication with an
external server. As there were no shared resources between jobs, there is no
expected dependence on the number of synchronized jobs. The average processing
time for this setup was determined very accurately by simply running a large
number of jobs, so no associated error is shown in Fig. 8.

When offloading PN-AK4 inference to the GPU, we expect an improvement in the
overall throughput of about 4% compared with direct inference, corresponding
to the fraction of time taken by the total PN-AK4 processing shown in Table 2.
Such an improvement is observed before saturation, where the throughput is
stable as a function of the number of simultaneous CPU clients. The throughput
decreases as the GPU starts to saturate because individual client-side jobs
have to wait longer for inference requests to complete and return. The point
where the throughput starts to decrease is the saturation point, where the GPU
is running at the maximum throughput and cannot handle any additional incoming
requests. The throughput becomes lower than CPU-only inference slightly above
160 four-threaded parallel jobs, , equivalent to 640 single-threaded jobs,
comparable with the expected saturation point from the model analyzer.

Figure 8:  The GPU saturation scan performed in GCP, where the per-event
throughput is shown as a function of the number of parallel CPU clients for
the PyTorch version of PN-AK4 (black), DeepMET (blue), DeepTau optimized with
TRT (red), and all PyTorch versions of PN-AK8 on a single GPU (green). Each of
the parallel jobs was run in a four-threaded configuration. The CPU tasks ran
in four-threaded GCP VMs, and the Triton servers were hosted on separate
single GPU VMs also in GCP. The line for direct-inference jobs represents the
baseline configuration measured by running all algorithms without the use of
the SONIC approach or any GPUs. Each solid line represents running one of the
specified models on GPU via the SONIC approach.

Similar analyses were performed for the other models. When all three variants
of ParticleNet for AK8 jets are hosted on a single GPU, that GPU can serve
about 190 simultaneous four-threaded client jobs. For DeepTau and DeepMET, a
single GPU hosting only one of the algorithms could serve about 64 and 520
client jobs, respectively. The differences between these saturation points are
primarily due to different model sizes and numbers of objects per event. From
these saturation values, it is possible to determine the number of GPUs needed
to serve a production job that uses many client-side CPUs, and thus to
determine the ratio of the number of GPUs hosting different models. For
ParticleNet and DeepTau, the saturation points will depend on the number of
jets and tau leptons in the processed events, such that the ratio of GPUs
hosting different models and the required ratio of GPUs to CPUs is dependent
on the type of events. If dynamic batching is enabled, the number of GPUs
required for a model scales approximately linearly with the number of objects
per event that require an ML inference. For example, if there are half as many
jets per event, it would require about half as many GPUs to serve ParticleNet,
while the number of GPUs required for DeepMET should not change, as that model
makes one inference per event.

A single GPU can host multiple models such as DeepTau and DeepMET. In
practice, it was found that loading only a single model on each GPU (split-
model) led to about a 3–5% increase in the overall performance relative to
loading every model on each GPU (all-on-one). If a 10% improvement in
throughput is observed in the split-model configuration, one would expect
better than a 9.5% improvement in the all-on-one configuration. Because of
this slight improvement, the split-model configuration was used in subsequent
large-scale tests. However, in scenarios where different physics processes are
combined into a single data set, using the all-on-one configuration may be the
most straightforward deployment option. Preparatory model profiling should
still be performed to ensure that enough GPUs and model instances will be
available, but the difference in performance between model partitioning
schemes is small.

###  0.5.2 Cross-site tests

A potential bottleneck in the SONIC paradigm is the increased inference
latency caused by the physical distance between client and server and other
network traffic. While a previous study observed that the average processing
time difference between remote and on-premises servers is negligible [62], we
tested this observation explicitly with the Mini-AOD workflow, with results
presented in Fig. 9. In this test, client-side jobs were executed at Purdue’s
Tier-2 computing cluster in Indiana. All the models are loaded into one server
running on a single GPU for simplicity. The blue points and lines show the
throughput improvement in the Mini-AOD workflow when the client jobs
communicate with Triton servers hosting all the models at the same time,
running on a single GPU also physically located at Purdue. The improvement is
shown as a function of the number of simultaneous four-threaded client-side
jobs running at once. The single GPU server begins to saturate when about 10
client-side jobs are sending requests at once and the Mini-AOD workflow
running with the SONIC approach becomes slower than the CPU-only workflow if
more than about 17 four-threaded client-side jobs are running at once. The
direct-inference line in Fig. 9 was made in the same way discussed in Section
0.5.1 for Fig. 8, though the CPU-only jobs were run at Purdue rather than GCP
in Iowa.

These cross-site tests were performed before model configuration optimization
as discussed in the preceding section. The exact model configuration is less
important for this test, as long as the near and far servers have the same
type of GPUs and host the same models. Here, the non-TRT version of DeepTau
was used, which saturated close to 40 synchronized four-threaded jobs. When
all the models are loaded on a single GPU server, the approximate saturation
point can be found with reciprocal addition. With the saturation points of 90
(AK4 jet ParticleNet), 190 (AK8 jet ParticleNet, 3 models), 40 (DeepTau), and
520 (DeepMET, 2 models), the expected saturation point is around 18
synchronized four-threaded jobs, when running all the models together on one
GPU. As noted in the previous section, this estimate is often about 5% high.
The saturation point seen in Fig. 9 is lower than expected from the above
analysis due to the different configurations of the CPU machines in each
scenario. Twenty-threaded, hyperthreading-disabled Intel Xeon Processor
E5-2660 v3 units were used at Purdue, which are more powerful processors than
those used in the GCP-based tests. The faster the client-side resources
process events, the lower the saturation point is for a given type of GPU.

The blue points and line in Fig. 9 show the throughput improvement when the
client-side jobs are once again run at Purdue, while the Triton server hosting
all the models on a single GPU is run on GCP resources, which were physically
located in Iowa for this test. Both the observed throughput increase and
observed GPU saturation point are about the same for both server locations, so
we conclude that the client-to-server distance has little impact on
performance up to a few hundred kilometers. In the future, it will be
important to monitor the impact of distance, especially beyond a few hundred
kilometers.

Figure 9:  Production tests across different sites. The CPU tasks always run
at Purdue, while the servers with GPU inference tasks for all the models run
at Purdue (blue) and at GCP in Iowa (red). The throughput values are higher
than those shown in Fig. 8 because the CPUs at Purdue are more powerful than
those comprising the GCP VMs.

###  0.5.3 Large-scale tests

Finally, we perform large-scale tests to emulate realistic Mini-AOD production
scenarios. These tests were performed exclusively using GCP resources. Here,
24 NVIDIA Tesla T4 GPUs were used to host the PyTorch version of ParticleNet
for AK4 jets, 20 GPUs were used to host the PyTorch version of all three
ParticleNet models for AK8 jets, 48 GPUs were used to host DeepTau, and 10
GPUs were used to host DeepMET. The ratios of the number of GPUs hosting each
model do not exactly match those expected from Section 0.5.1; this was done
for multiple reasons. First, a larger number of GPUs than strictly necessary
were used in the tests to ensure that the performance would meet the
expectations described in Section 0.5.1, and thereby avoid the cost of
repeating the test multiple times. For instance, based on a single-GPU
saturation point of 150 four-threaded jobs, one would expect that only about
17 GPUs would be needed to serve the AK4 jet ParticleNet model. It is worth
noting that while the demonstration illustrated here uses a CPU-to-GPU ratio
of 9820:102 (about 1 GPU per 96 CPU cores), achieving a higher ratio is likely
possible as we could have safely decreased the number of GPUs used. Second, it
was more practical to instantiate VMs with factors of 4 GPUs. This had the
added benefit of maximizing the allowed I/O bandwidth for VMs, which in GCP is
restricted for VMs with fewer CPU cores. A maximal CPU-to-GPU ratio for VMs in
GCP is achieved for machines with 4 GPUs. While I/O bandwidth was not expected
to create problems for this large test, a conservative approach was taken to
mitigate the risk of needing repeated trials. Thus, the GPU allocation
approach was to find the minimum number of GPUs expected based on single
saturation scans (17 for the AK4 jet ParticleNet model, 14 for the AK8 jet
ParticleNet models, 42 for DeepTau, and 5 for DeepMET), find the next largest
number divisible by 4, then add one extra server corresponding to 4 additional
GPUs. For DeepMET, two 4-GPU VMs were used along with one 2-GPU machine, as
DeepMET is a relatively lightweight algorithm, and the approach used for the
other algorithms would have more than doubled the number of GPUs allocated for
the algorithm.

A separate Kubernetes load-balancer was used for each model type to distribute
inference requests evenly among server-hosting, GPU-enabled VMs. Thus, client
VMs used separate IP addresses for each model type, and each inference for a
particular model type was passed through a single load-balancing machine,
allowing for network monitoring for each model separately.

Client-side jobs also ran on CPU-only GCP resources, using HEPCloud to
dynamically allocate preemptible resources and assign jobs to the client-side
VMs. Each client job was run in a four-threaded configuration, with input data
files stored locally, and each VM created in this HEPCloud setup had 32 cores
and 160GB of memory, meaning up to 8 simultaneous jobs could run in a single
VM.

The largest test had 2500 simultaneous client-side jobs, which amounts to 10
000 CPU cores. Because these jobs were run on preemptible resources, Google
reserves the right to reallocate any VM to higher-priority requests from other
GCP users. Of the 2500 jobs, 2455 jobs completed successfully without
preemption, so in total 9820 client-side CPUs were used.

The results of this large-scale test are shown in Fig. 10. The jobs running
with the SONIC approach achieved an average throughput of 4.0events/s, while
CPU-only benchmarking jobs had a throughput of 3.5events/s. This 13% increase
in throughput matches the expectation of completely removing the ParticleNet,
DeepTau, and DeepMET inference from the total Mini-AOD per-event processing
time. The throughput values for this test are slightly different from those
shown in Figs. 8 and 9. As noted previously, this is due to the difference in
the CPUs used in the tests.

As mentioned before, server-side VMs were optimized to allow maximal input and
output bandwidth. No bottlenecks due to bandwidth were observed in this scale-
out test. We noted that the maximum data input rate received by one of the
Kubernetes load-balancers was 11.5GB/s, which was for DeepTau. The next-
highest data input rate was 3.3GB/s for DeepMET, and less than 500MB/s of
input was needed for all ParticleNet models combined. These values are
consistent with expectations from Table 2, as there were roughly 10 000 CPU
cores running simultaneously, each processing about 1 event per second. The
output rate was significantly smaller for each model, as most return only one
or a few floating-point values as the inference result. In the future, more
algorithms will be able to be run on coprocessors. These could have larger I/O
sizes than the algorithms considered here, so continuing to monitor the
network usage will be important.

Figure 10:  Scale-out test results on Google Cloud. The average throughput of
the workflow with the SONIC approach is 4.0events/s (solid blue), while the
average throughput of the direct-inference workflow is 3.5events/s (dashed
red).

##  0.6 Portability

While the inference servers discussed so far have exclusively utilized GPU
resources, servers are easily portable and can run on other processing
platforms. Previous uses of the SONIC approach with FPGAs are reported in
detail in Refs. [60, 61], where the portability was demonstrated by changing
the coprocessor technology and server backend without modifying the client-
side software. In the new studies reported here, we have additionally run the
Mini-AOD workflow with servers using both CPUs and Graphcore IPUs [33, 34].

###  0.6.1 The CPU fallback server

When running with remote servers, one potentially common and important failure
mode is communication errors between clients and servers. To support automatic
local CPU inference as a backup option when communication failures occur with
remote servers, the SONIC implementation includes a service that can launch a
Triton server using local CPU resources for any SONIC approach-compatible
models, referred to as a fallback server. Fallback servers can also be used
for inference when third-party ML frameworks are not supported for direct
inference in .

Ideally, the use of fallback servers should have minimal impact on per-event
throughput relative to running direct-inference jobs without the SONIC
approach. This is contingent on two factors. First, the latency introduced by
sending data to or from local servers must be negligible. Second, servers
should introduce minimal overhead to maximize the CPU resources used to
perform inference. The first concern can be resolved using the shared-memory
option, which skips the gRPC communication and directly passes the data in
certain memory chunks between the server and client. The gRPC overhead in most
cases is found to be negligible. Regarding the second concern, local servers
are running on the same CPUs as the other modules in the workflow, so
scheduling efforts must be made to avoid CPU thread contention between the
two. This implies that the synchronous server mode is preferred for local CPU
fallback servers. Additionally, inference tasks should not create extra
threads to avoid contention. In our experiments, we found that using more
inference threads in the server than the number of threads allocated for the
job will slow processing down dramatically.

Having thus explored potential configurations, for the local CPU inference, we
run our servers in synchronous mode, with the number of model instances set to
the number of threads per job, and the number of inference threads always set
to one. This configuration mimics direct inference and avoids thread over-
subscription as much as possible. We compare the throughput between direct
inference and the SONIC approach with local CPU fallback servers using this
configuration. Tests were performed using resources at the Purdue Tier-2
cluster with the CPU-only nodes. There are $n_{\text{CPU}}=20$ Intel E5-2660
CPU cores on one node, and hyperthreading is disabled to ensure more stable
results. For all tests, we always saturate the CPU nodes by requiring the
product of the number of jobs ($n_{\text{j}}$) and number of threads per job
($n_{\text{T}}$) to be equal to the number of CPU cores:
$n_{\text{j}}n_{\text{T}}=n_{\text{CPU}}=20$. For example, to test a
configuration where a single job occupies four threads ($n_{\text{T}}=4$), we
would run five synchronized jobs ($n_{\text{j}}=5$), while a two-threaded
configuration would require 10 synchronized jobs. We scan the throughput as a
function of the number of threads, as shown in Fig. 11. The throughput of
running with local CPU fallback servers is similar to direct inference. The
higher throughput in some cases is a result of optimizations in more recent
versions of ONNX Runtime installed on the server, which can be controlled in
real production jobs.

Figure 11:  Throughput (upper) and throughput ratio (lower) between the SONIC
approach and direct inference in the local CPU tests at the Purdue Tier-2
cluster. To ensure the CPUs are always saturated, the number of threads per
job multiplied by the number of jobs is set to 20.

Memory usage was also monitored with the top command during the studies and
provided in Table 3. As the number of threads per job increases, the number of
model instances increases as well, leading to higher memory usage in the SONIC
approach compared with direct inference. On the other hand, if the number of
model instances is fixed to one, the server memory usage is always around
300MB, so the total memory usage is similar to direct inference, but the
throughput decreases by 5–10%, depending on the tasks. Further simultaneous
optimization of the throughput and memory usage will be explored in the
future.

Table 3:  Memory usage with direct inference and the SONIC approach in the local CPU tests at the Purdue Tier-2 cluster. The last column is calculated as the sum of client and server memory usage divided by the direct inference memory usage. To ensure the CPUs are always saturated, the number of threads $n_{\text{T}}$ per job multiplied by the number of jobs is set to 20.  $n_{\text{T}}$ |  with |  with SONIC | SONIC app. | SONIC/direct
---|---|---|---|---
per job | direct inference [MB] | app. (client) [MB] | server [MB]
1 | 1850 | 1700 | 200 | 103%
2 | 2000 | 1800 | 400 | 110%
5 | 2200 | 1950 | 800 | 125%
10 | 2500 | 2200 | 1200 | 136%
20 | 2900 | 2500 | 2000 | 155%

###  0.6.2 Studies with Graphcore IPUs

As discussed in Sections 0.3.2 and 0.3.3, NVIDIA Triton inference servers
support custom backends to run with different coprocessors and different (,
ML) backends. Since the SONIC approach’s client code only depends on the
Triton protocols, algorithms implemented in this way can easily be ported to
different types of coprocessors. One of the Mini-AOD production tests was run
together with the Graphcore IPU team, where a custom backend was prepared by
the developer team to support running ML inference with IPUs. The supported ML
frameworks on IPUs when the tests were performed included TensorFlow, ONNX,
and PyTorch. PyTorch-Geometric support was in development at that time and is
now available. With TensorFlow custom backend available for the server, we can
easily run DeepMET and DeepTau inference with the SONIC approach in on IPUs.

First, a per-algorithm throughput scan was carried out. Comparing running
inference on MK2 GC200 IPUs with NVIDIA Tesla V100s, a chip-to-chip factor of
3 throughput improvement was found, with larger gains expected for more
computing-intensive models. Running the entire Mini-AOD workflow was also
tested. We adapted the workflow configuration to point the clients running at
the Purdue Tier-2 cluster to IPU servers on the Graphcloud cloud. Without any
other modifications on the client side, the workflow ran successfully,
performing inference of the DeepMET and DeepTau models on the cloud and the
other parts of the workflow on the client local CPUs. Outputs were checked and
found to be consistent with direct inference, within $10^{-6}$ differences for
floats due to precision limits.

##  0.7 Summary

Within the next decade, the data-taking rate at the LHC will increase
dramatically, straining the expected computing resources of the LHC
experiments. At the same time, more algorithms that run on these resources
will be converted into either machine learning or domain algorithms that are
easily accelerated with the use of coprocessors, such as graphics processing
units (GPUs). By pursuing heterogeneous architectures, it is possible to
alleviate potential shortcomings of available central processing unit (CPU)
resources.

Inference as a service (IaaS) is a promising scheme to integrate coprocessors
into CMS computing workflows. In IaaS, client code simply assembles the input
data for an algorithm, sends that input to an inference server running either
locally or remotely, and retrieves output from the server. The implementation
of IaaS discussed throughout this paper is called the Services for Optimized
Network Inference on Coprocessors (SONIC) approach, which employs NVIDIA
Triton Inference Servers to host models on coprocessors, as demonstrated here
in studies on GPUs, CPUs, and Graphcore Intelligence Processing Units (IPUs).

In this paper, the SONIC approach in the CMS software framework () is
demonstrated in a sample Mini-AOD workflow, where algorithms for jet tagging,
tau lepton identification, and missing transverse momentum regression are
ported to run on inference servers. These algorithms account for nearly 10% of
the total processing time per event in a simulated data set of top quark-
antiquark events. After model profiling, which is used to optimize server
performance and determine the needed number of GPUs for a given number of
client jobs, the expected 10% decrease in per-event processing time was
achieved in a large-scale test of Mini-AOD production with the SONIC approach
that used about 10 000 CPU cores and 100 GPUs. The network bandwidth is large
enough to support high input-output model inference for the workflow tested,
and it will be monitored as the fraction of algorithms using remote GPUs
increases.

In addition to meeting performance expectations, we demonstrated that the
throughput results are not highly sensitive to the physical client-to-server
distance, at least up to distances of hundreds of kilometers. Running
inference through Triton servers on local CPU resources does not affect the
throughput compared with the standard approach of running inference directly
on CPUs in the job thread. We also performed a test using GraphCore IPUs to
demonstrate the flexibility of the SONIC approach.

The SONIC approach for IaaS represents a flexible method to accelerate
algorithms, which is increasingly valuable for LHC experiments. Using a
realistic workflow, we highlighted many of the benefits of the SONIC approach,
including the use of remote resources, workflow acceleration, and portability
to different processor technologies. To make it a viable and robust paradigm
for CMS computing in the future, additional studies are ongoing or planned for
monitoring and mitigating potential issues such as excessive network and
memory usage or server failures.

###### Acknowledgements.

We congratulate our colleagues in the CERN accelerator departments for the
excellent performance of the LHC and thank the technical and administrative
staffs at CERN and at other CMS institutes for their contributions to the
success of the CMS effort. In addition, we gratefully acknowledge the
computing centres and personnel of the Worldwide LHC Computing Grid and other
centres for delivering so effectively the computing infrastructure essential
to our analyses. Finally, we acknowledge the enduring support for the
construction and operation of the LHC, the CMS detector, and the supporting
computing infrastructure provided by the following funding agencies: SC
(Armenia), BMBWF and FWF (Austria); FNRS and FWO (Belgium); CNPq, CAPES,
FAPERJ, FAPERGS, and FAPESP (Brazil); MES and BNSF (Bulgaria); CERN; CAS,
MoST, and NSFC (China); MINCIENCIAS (Colombia); MSES and CSF (Croatia); RIF
(Cyprus); SENESCYT (Ecuador); ERC PRG, RVTT3 and TK202 (Estonia); Academy of
Finland, MEC, and HIP (Finland); CEA and CNRS/IN2P3 (France); SRNSF (Georgia);
BMBF, DFG, and HGF (Germany); GSRI (Greece); NKFIH (Hungary); DAE and DST
(India); IPM (Iran); SFI (Ireland); INFN (Italy); MSIP and NRF (Republic of
Korea); MES (Latvia); LAS (Lithuania); MOE and UM (Malaysia); BUAP, CINVESTAV,
CONACYT, LNS, SEP, and UASLP-FAI (Mexico); MOS (Montenegro); MBIE (New
Zealand); PAEC (Pakistan); MES and NSC (Poland); FCT (Portugal); MESTD
(Serbia); MCIN/AEI and PCTI (Spain); MOSTR (Sri Lanka); Swiss Funding Agencies
(Switzerland); MST (Taipei); MHESI and NSTDA (Thailand); TUBITAK and TENMAK
(Turkey); NASU (Ukraine); STFC (United Kingdom); DOE and NSF (USA).
Individuals have received support from the Marie-Curie programme and the
European Research Council and Horizon 2020 Grant, contract Nos. 675440,
724704, 752730, 758316, 765710, 824093, and COST Action CA16108 (European
Union); the Leventis Foundation; the Alfred P. Sloan Foundation; the Alexander
von Humboldt Foundation; the Science Committee, project no. 22rl-037
(Armenia); the Belgian Federal Science Policy Office; the Fonds pour la
Formation à la Recherche dans l’Industrie et dans l’Agriculture (FRIA-
Belgium); the Agentschap voor Innovatie door Wetenschap en Technologie (IWT-
Belgium); the F.R.S.-FNRS and FWO (Belgium) under the “Excellence of Science –
EOS” – be.h project n. 30820817; the Beijing Municipal Science & Technology
Commission, No. Z191100007219010 and Fundamental Research Funds for the
Central Universities (China); the Ministry of Education, Youth and Sports
(MEYS) of the Czech Republic; the Shota Rustaveli National Science Foundation,
grant FR-22-985 (Georgia); the Deutsche Forschungsgemeinschaft (DFG), under
Germany’s Excellence Strategy – EXC 2121 “Quantum Universe” – 390833306, and
under project number 400140256 - GRK2497; the Hellenic Foundation for Research
and Innovation (HFRI), Project Number 2288 (Greece); the Hungarian Academy of
Sciences, the New National Excellence Program - ÚNKP, the NKFIH research
grants K 124845, K 124850, K 128713, K 128786, K 129058, K 131991, K 133046, K
138136, K 143460, K 143477, 2020-2.2.1-ED-2021-00181, and TKP2021-NKTA-64
(Hungary); the Council of Science and Industrial Research, India; ICSC –
National Research Centre for High Performance Computing, Big Data and Quantum
Computing, funded by the EU NexGeneration program (Italy); the Latvian Council
of Science; the Ministry of Education and Science, project no. 2022/WK/14, and
the National Science Center, contracts Opus 2021/41/B/ST2/01369 and
2021/43/B/ST2/01552 (Poland); the Fundação para a Ciência e a Tecnologia,
grant CEECIND/01334/2018 (Portugal); the National Priorities Research Program
by Qatar National Research Fund; MCIN/AEI/10.13039/501100011033, ERDF “a way
of making Europe”, and the Programa Estatal de Fomento de la Investigación
Científica y Técnica de Excelencia María de Maeztu, grant MDM-2017-0765 and
Programa Severo Ochoa del Principado de Asturias (Spain); the Chulalongkorn
Academic into Its 2nd Century Project Advancement Project, and the National
Science, Research and Innovation Fund via the Program Management Unit for
Human Resources & Institutional Development, Research and Innovation, grant
B37G660013 (Thailand); the Kavli Foundation; the Nvidia Corporation; the
SuperMicro Corporation; the Welch Foundation, contract C-1845; and the Weston
Havens Foundation (USA).

## References

  * [1] L. Evans and P. Bryant, “LHC machine”,  JINST 3 (2008) S08001, 10.1088/1748-0221/3/08/S08001.
  * [2] ATLAS Collaboration, “The ATLAS experiment at the CERN Large Hadron Collider”,  JINST 3 (2008) S08003, 10.1088/1748-0221/3/08/S08003.
  * [3] CMS Collaboration, “The CMS experiment at the CERN LHC”,  JINST 3 (2008) S08004, 10.1088/1748-0221/3/08/S08004.
  * [4] ATLAS Collaboration, “Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC”,  Phys. Lett. B 716 (2012) 1, 10.1016/j.physletb.2012.08.020, arXiv:1207.7214.
  * [5] CMS Collaboration, “Observation of a new boson at a mass of 125 with the CMS experiment at the LHC”,  Phys. Lett. B 716 (2012) 30, 10.1016/j.physletb.2012.08.021, arXiv:1207.7235.
  * [6] CMS Collaboration, “Observation of a new boson with mass near 125 in $\Pp\Pp$ collisions at $\sqrt{s}=7$ and 8”,  JHEP 06 (2013) 081, 10.1007/JHEP06(2013)081, arXiv:1303.4571.
  * [7] CMS Collaboration, “Search for supersymmetry in proton-proton collisions at 13in final states with jets and missing transverse momentum”,  JHEP 10 (2019) 244, 10.1007/JHEP10(2019)244, arXiv:1908.04722.
  * [8] CMS Collaboration, “Combined searches for the production of supersymmetric top quark partners in proton-proton collisions at $\sqrt{s}=13\TeV$”,  Eur. Phys. J. C 81 (2021) 970, 10.1140/epjc/s10052-021-09721-5, arXiv:2107.10892.
  * [9] CMS Collaboration, “Search for higgsinos decaying to two Higgs bosons and missing transverse momentum in proton-proton collisions at $\sqrt{s}=13\TeV$”,  JHEP 05 (2022) 014, 10.1007/JHEP05(2022)014, arXiv:2201.04206.
  * [10] CMS Collaboration, “Search for supersymmetry in final states with two oppositely charged same-flavor leptons and missing transverse momentum in proton-proton collisions at $\sqrt{s}=13\TeV$”,  JHEP 04 (2021) 123, 10.1007/JHEP04(2021)123, arXiv:2012.08600.
  * [11] ATLAS Collaboration, “Search for squarks and gluinos in final states with hadronically decaying $\PGt$-leptons, jets, and missing transverse momentum using $\Pp\Pp$ collisions at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  Phys. Rev. D 99 (2019) 012009, 10.1103/PhysRevD.99.012009, arXiv:1808.06358.
  * [12] ATLAS Collaboration, “Search for top squarks in events with a Higgs or boson using 139of $\Pp\Pp$ collision data at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  Eur. Phys. J. C 80 (2020) 1080, 10.1140/epjc/s10052-020-08469-8, arXiv:2006.05880.
  * [13] ATLAS Collaboration, “Search for charginos and neutralinos in final states with two boosted hadronically decaying bosons and missing transverse momentum in $\Pp\Pp$ collisions at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  Phys. Rev. D 104 (2021) 112010, 10.1103/PhysRevD.104.112010, arXiv:2108.07586.
  * [14] ATLAS Collaboration, “Search for direct pair production of sleptons and charginos decaying to two leptons and neutralinos with mass splittings near the $\PW$-boson mass in $\sqrt{s}=13\TeV$ $\Pp\Pp$ collisions with the ATLAS detector”,  JHEP 06 (2023) 031, 10.1007/JHEP06(2023)031, arXiv:2209.13935.
  * [15] ATLAS Collaboration, “Search for new phenomena in events with an energetic jet and missing transverse momentum in $\Pp\Pp$ collisions at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  Phys. Rev. D 103 (2021) 112006, 10.1103/PhysRevD.103.112006, arXiv:2102.10874.
  * [16] CMS Collaboration, “Search for new particles in events with energetic jets and large missing transverse momentum in proton-proton collisions at $\sqrt{s}=13\TeV$”,  JHEP 11 (2021) 153, 10.1007/jhep11(2021)153, arXiv:2107.13021.
  * [17] ATLAS Collaboration, “Search for new resonances in mass distributions of jet pairs using 139of $\Pp\Pp$ collisions at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  JHEP 03 (2020) 145, 10.1007/jhep03(2020)145, arXiv:1910.08447.
  * [18] ATLAS Collaboration, “Search for high-mass dilepton resonances using 139of $\Pp\Pp$ collision data collected at $\sqrt{s}=13\TeV$ with the ATLAS detector”,  Phys. Lett. B 796 (2019) 68, 10.1016/j.physletb.2019.07.016, arXiv:1903.06248.
  * [19] CMS Collaboration, “Search for narrow and broad dijet resonances in proton-proton collisions at $\sqrt{s}=13\TeV$ and constraints on dark matter mediators and other new particles”,  JHEP 08 (2018) 130, 10.1007/jhep08(2018)130, arXiv:1806.00843.
  * [20] CMS Collaboration, “Search for high mass dijet resonances with a new background prediction method in proton-proton collisions at $\sqrt{s}=13\TeV$”,  JHEP 05 (2020) 033, 10.1007/jhep05(2020)033, arXiv:1911.03947.
  * [21] CMS Collaboration, “Search for resonant and nonresonant new phenomena in high-mass dilepton final states at $\sqrt{s}=13\TeV$”,  JHEP 07 (2021) 208, 10.1007/jhep07(2021)208, arXiv:2103.02708.
  * [22] O. Aberle et al., “High-Luminosity Large Hadron Collider (HL-LHC): Technical design report”,  CERN Yellow Rep. Monogr. 10 (2020) 10.23731/CYRM-2020-0010.
  * [23] O. Brüning and L. Rossi, eds., “The High Luminosity Large Hadron Collider: the new machine for illuminating the mysteries of universe”, volume 24.  World Scientific, 2015.  10.1142/9581, ISBN 978-981-4675-46-8, 978-981-4678-14-8.
  * [24] CMS Collaboration, “The Phase-2 upgrade of the CMS level-1 trigger”, CMS Technical Design Report CERN-LHCC-2020-004, CMS-TDR-021, 2020.
  * [25] ATLAS Collaboration, “Technical design report for the Phase-II upgrade of the ATLAS TDAQ system”, ATLAS Technical Design Report CERN-LHCC-2017-020, ATLAS-TDR-029, 2017.  10.17181/CERN.2LBB.4IAL.
  * [26] A. Ryd and L. Skinnari, “Tracking triggers for the HL-LHC”,  Ann. Rev. Nucl. Part. Sci. 70 (2020) 171, 10.1146/annurev-nucl-020420-093547, arXiv:2010.13557.
  * [27] ATLAS Collaboration, “Operation of the ATLAS trigger system in Run 2”,  JINST 15 (2020) P10004, 10.1088/1748-0221/15/10/P10004, arXiv:2007.12539.
  * [28] CMS Collaboration, “The CMS trigger system”,  JINST 12 (2017) P01020, 10.1088/1748-0221/12/01/P01020, arXiv:1609.02366.
  * [29] CMS Collaboration, “The Phase-2 upgrade of the CMS data acquisition and high level trigger”, CMS Technical Design Report CERN-LHCC-2021-007, CMS-TDR-022, 2021.
  * [30] CMS Offline Software and Computing Group, “CMS Phase-2 computing model: Update document”, CMS Note CMS-NOTE-2022-008, 2022.
  * [31] ATLAS Collaboration, “ATLAS software and computing HL-LHC roadmap”, LHCC Public Document CERN-LHCC-2022-005, LHCC-G-182, 2022.
  * [32] R. H. Dennard et al., “Design of ion-implanted MOSFET’s with very small physical dimensions”,  IEEE J. Solid-State Circuits 9 (1974) 256, 10.1109/JSSC.1974.1050511.
  * [33] Graphcore, “Intelligence processing unit”. https://www.graphcore.ai/products/ipu.  Accessed: 2023-11-08.
  * [34] Z. Jia, B. Tillman, M. Maggioni, and D. P. Scarpazza, “Dissecting the Graphcore IPU architecture via microbenchmarking”, 2019\. arXiv:1912.03413.
  * [35] D. Guest, K. Cranmer, and D. Whiteson, “Deep learning and its application to LHC physics”,  Ann. Rev. Nucl. Part. Sci. 68 (2018) 161, 10.1146/annurev-nucl-101917-021019, arXiv:1806.11484.
  * [36] K. Albertsson et al., “Machine learning in high energy physics community white paper”,  J. Phys. Conf. Ser. 1085 (2018) 022008, 10.1088/1742-6596/1085/2/022008, arXiv:1807.02876.
  * [37] D. Bourilkov, “Machine and deep learning applications in particle physics”,  Int. J. Mod. Phys. A 34 (2020) 1930019, 10.1142/S0217751X19300199, arXiv:1912.08245.
  * [38] A. J. Larkoski, I. Moult, and B. Nachman, “Jet substructure at the Large Hadron Collider: A review of recent advances in theory and machine learning”,  Phys. Rept. 841 (2020) 1, 10.1016/j.physrep.2019.11.001, arXiv:1709.04464.
  * [39] Feickert, Matthew and Nachman, Benjamin, “A living review of machine learning for particle physics”, 2021. arXiv:2102.02770.
  * [40] P. Harris et al., “Physics community needs, tools, and resources for machine learning”, in  Proc. 2021 US Community Study on the Future of Particle Physics.  2022\.  arXiv:2203.16255.
  * [41] S. Farrell et al., “Novel deep learning methods for track reconstruction”, in  4th International Workshop Connecting The Dots 2018.  2018\.  arXiv:1810.06111.
  * [42] S. Amrouche et al., “The Tracking Machine Learning challenge: Accuracy phase”, ch. 9, p. 231.  Springer Cham, 4, 2019.  arXiv:1904.06778.  10.1007/978-3-030-29135-8_9.
  * [43] X. Ju et al., “Performance of a geometric deep learning pipeline for HL-LHC particle tracking”,  Eur. Phys. J. C 81 (2021) 876, 10.1140/epjc/s10052-021-09675-8, arXiv:2103.06995.
  * [44] G. DeZoort et al., “Charged particle tracking via edge-classifying interaction networks”,  Comput. Softw. Big Sci. 5 (2021) 26, 10.1007/s41781-021-00073-z, arXiv:2103.16701.
  * [45] S. R. Qasim, J. Kieseler, Y. Iiyama, and M. Pierini, “Learning representations of irregular particle-detector geometry with distance-weighted graph networks”,  Eur. Phys. J. C 79 (2019) 608, 10.1140/epjc/s10052-019-7113-9, arXiv:1902.07987.
  * [46] J. Kieseler, “Object condensation: one-stage grid-free multi-object reconstruction in physics detectors, graph and image data”,  Eur. Phys. J. C 80 (2020) 886, 10.1140/epjc/s10052-020-08461-2, arXiv:2002.03605.
  * [47] CMS Collaboration, “GNN-based end-to-end reconstruction in the CMS Phase 2 high-granularity calorimeter”,  J. Phys. Conf. Ser. 2438 (2023) 012090, 10.1088/1742-6596/2438/1/012090, arXiv:2203.01189.
  * [48] J. Pata et al., “MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks”,  Eur. Phys. J. C 81 (2021) 381, 10.1140/epjc/s10052-021-09158-w, arXiv:2101.08578.
  * [49] CMS Collaboration, “Machine learning for particle flow reconstruction at CMS”,  J. Phys. Conf. Ser. 2438 (2023) 012100, 10.1088/1742-6596/2438/1/012100, arXiv:2203.00330.
  * [50] F. Mokhtar et al., “Progress towards an improved particle flow algorithm at CMS with machine learning”, in  Proc. 21st Intern. Workshop on Advanced Computing and Analysis Techniques in Physics Research: AI meets Reality.  2023\.  arXiv:2303.17657.
  * [51] F. A. Di Bello et al., “Reconstructing particles in jets using set transformer and hypergraph prediction networks”,  Eur. Phys. J. C 83 (2023) 596, 10.1140/epjc/s10052-023-11677-7, arXiv:2212.01328.
  * [52] J. Pata et al., “Improved particle-flow event reconstruction with scalable neural networks for current and future particle detectors”, 2023. arXiv:2309.06782.
  * [53] E. A. Moreno et al., “JEDI-net: a jet identification algorithm based on interaction networks”,  Eur. Phys. J. C 80 (2020) 58, 10.1140/epjc/s10052-020-7608-4, arXiv:1908.05318.
  * [54] H. Qu and L. Gouskos, “ParticleNet: Jet tagging via particle clouds”,  Phys. Rev. D 101 (2020) 056019, 10.1103/PhysRevD.101.056019, arXiv:1902.08570.
  * [55] E. A. Moreno et al., “Interaction networks for the identification of boosted $\PH\to\bbbar$ decays”,  Phys. Rev. D 102 (2020) 012010, 10.1103/PhysRevD.102.012010, arXiv:1909.12285.
  * [56] E. Bols et al., “Jet flavour classification using DeepJet”,  JINST 15 (2020) P12012, 10.1088/1748-0221/15/12/P12012, arXiv:2008.10519.
  * [57] CMS Collaboration, “Identification of heavy, energetic, hadronically decaying particles using machine-learning techniques”,  JINST 15 (2020) P06005, 10.1088/1748-0221/15/06/P06005, arXiv:2004.08262.
  * [58] H. Qu, C. Li, and S. Qian, “Particle transformer for jet tagging”, in  Proc. 39th Intern. Conf. on Machine Learning, K. Chaudhuri et al., eds., volume 162, p. 18281.  2022\.  arXiv:2202.03772.
  * [59] CMS Collaboration, “Muon identification using multivariate techniques in the CMS experiment in proton-proton collisions at $\sqrt{s}=13\TeV$”, 2023. arXiv:2310.03844. Submitted to _JINST_.
  * [60] J. Duarte et al., “FPGA-accelerated machine learning inference as a service for particle physics computing”,  Comput. Softw. Big Sci. 3 (2019) 13, 10.1007/s41781-019-0027-2, arXiv:1904.08986.
  * [61] D. Rankin et al., “FPGAs-as-a-service toolkit (FaaST)”, in  Proc. 2020 IEEE/ACM Intern. Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC).  IEEE, 2020.  10.1109/h2rc51942.2020.00010.
  * [62] J. Krupa et al., “GPU coprocessors as a service for deep learning inference in high energy physics”,  Mach. Learn. Sci. Tech. 2 (2021) 035005, 10.1088/2632-2153/abec21, arXiv:2007.10359.
  * [63] M. Wang et al., “GPU-accelerated machine learning inference as a service for computing in neutrino experiments”,  Front. Big Data 3 (2021) 604083, 10.3389/fdata.2020.604083, arXiv:2009.04509.
  * [64] ALICE Collaboration, “Real-time data processing in the ALICE high level trigger at the LHC”,  Comput. Phys. Commun. 242 (2019) 25, 10.1016/j.cpc.2019.04.011, arXiv:1812.08036.
  * [65] R. Aaij et al., “Allen: A high level trigger on GPUs for LHCb”,  Comput. Softw. Big Sci. 4 (2020) 7, 10.1007/s41781-020-00039-7, arXiv:1912.09161.
  * [66] LHCb Collaboration, “The LHCb upgrade I”, 2023. arXiv:2305.10515.
  * [67] A. Bocci et al., “Heterogeneous reconstruction of tracks and primary vertices with the CMS pixel tracker”,  Front. Big Data 3 (2020) 601728, 10.3389/fdata.2020.601728, arXiv:2008.13461.
  * [68] D. Vom Bruch, “Real-time data processing with GPUs in high energy physics”,  JINST 15 (2020) C06010, 10.1088/1748-0221/15/06/C06010, arXiv:2003.11491.
  * [69] CMS Collaboration, “Mini-AOD: A new analysis data format for CMS”,  J. Phys. Conf. Ser. 664 (2015) 7, 10.1088/1742-6596/664/7/072052, arXiv:1702.04685.
  * [70] CMS Collaboration, “CMS physics: Technical design report volume 1: Detector performance and software”, CMS Technical Design Report CERN-LHCC-2006-001, CMS-TDR-8-1, 2006.
  * [71] CMS Collaboration, “CMSSW on Github”. Accessed: 2023-11-08. http://cms-sw.github.io/.
  * [72] CMS Collaboration, “Performance of the CMS Level-1 trigger in proton-proton collisions at $\sqrt{s}=13\TeV$”,  JINST 15 (2020) P10017, 10.1088/1748-0221/15/10/P10017, arXiv:2006.10165.
  * [73] CMS Collaboration, “Electron and photon reconstruction and identification with the CMS experiment at the CERN LHC”,  JINST 16 (2021) P05014, 10.1088/1748-0221/16/05/P05014, arXiv:2012.06888.
  * [74] CMS Collaboration, “Description and performance of track and primary-vertex reconstruction with the CMS tracker”,  JINST 9 (2014) P10009, 10.1088/1748-0221/9/10/P10009, arXiv:1405.6569.
  * [75] CMS Collaboration, “Particle-flow reconstruction and global event description with the CMS detector”,  JINST 12 (2017) P10003, 10.1088/1748-0221/12/10/P10003, arXiv:1706.04965.
  * [76] oneTBB, “oneAPI Threading Building Blocks”. https://github.com/oneapi-src/oneTBB.  Accessed: 2023-11-08.
  * [77] A. Bocci et al., “Bringing heterogeneity to the CMS software framework”,  Eur. Phys. J. Web Conf. 245 (2020) 05009, 10.1051/epjconf/202024505009, arXiv:2004.04334.
  * [78] CMS Collaboration, “A further reduction in CMS event data for analysis: the NANOAOD format”,  Eur. Phys. J. Web Conf. 214 (2019) 06021, 10.1051/epjconf/201921406021.
  * [79] CMS Collaboration, “Extraction and validation of a new set of CMS 8 tunes from underlying-event measurements”,  Eur. Phys. J. C 80 (2020) 4, 10.1140/epjc/s10052-019-7499-4, arXiv:1903.12179.
  * [80] CMS Collaboration, “Pileup mitigation at CMS in 13data”,  JINST 15 (2020) P09018, 10.1088/1748-0221/15/09/P09018, arXiv:2003.00503.
  * [81] CMS Collaboration, “NANOAOD: A new compact event data format in CMS”,  Eur. Phys. J. Web Conf. 245 (2020) 06002, 10.1051/epjconf/202024506002.
  * [82] NVIDIA, “NVIDIA Triton Inference Server”. https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html.  Accessed: 2023-11-08.
  * [83] gRPC, “gRPC – A high performance, open source universal RPC framework”. https://grpc.io/.  Accessed: 2023-11-08.
  * [84] Kubernetes, “Kubernetes documentation”. Accessed: 2023-11-08. https://kubernetes.io/docs/home/.
  * [85] K. Pedro et al., “SonicCore”. https://github.com/cms-sw/cmssw/tree/master/HeterogeneousCore/SonicCore.  Accessed: 2023-11-08.
  * [86] K. Pedro et al., “SonicTriton”. https://github.com/cms-sw/cmssw/tree/master/HeterogeneousCore/SonicTriton.  Accessed: 2023-11-08.
  * [87] K. Pedro, “SonicCMS”. https://github.com/fastmachinelearning/SonicCMS.  Accessed: 2023-11-08.
  * [88] A. M. Caulfield et al., “A cloud-scale acceleration architecture”, in  Proc. 2016 49th Annual IEEE/ACM Intern. Symp. on Microarchitecture (MICRO), p. 1.  2016\.  10.1109/MICRO.2016.7783710.
  * [89] V. Kuznetsov, “vkuznet/TFaaS: First public version”, 2018.  10.5281/zenodo.1308049.
  * [90] V. Kuznetsov, L. Giommi, and D. Bonacorsi, “MLaaS4HEP: Machine learning as a service for HEP”,  Comput. Softw. Big Sci. 5 (2021) 17, 10.1007/s41781-021-00061-3, arXiv:2007.14781.
  * [91] KServe, “KServe documentation website”. https://kserve.github.io/website/.  Accessed: 2023-11-08.
  * [92] NVIDIA, “NVIDIA Triton Inference Server README (release 22.08)”. https://github.com/triton-inference-server/server/blob/r22.08/README.md#documentation.  Accessed: 2023-11-08.
  * [93] A. Paszke et al., “PyTorch: An imperative style, high-performance deep learning library”, in  Advances in Neural Information Processing Systems 32, H. Wallach et al., eds., p. 8024.  Curran Associates, Inc., 2019.  arXiv:1912.01703.
  * [94] NVIDIA, “NVIDIA TensorRT”. https://developer.nvidia.com/tensorrt.  Accessed: 2023-11-08.
  * [95] ONNX, “Open Neural Network Exchange (ONNX)”. https://github.com/onnx/onnx.  Accessed: 2023-11-08.
  * [96] M. Abadi et al., “TensorFlow: A system for large-scale machine learning”, 2016. arXiv:1605.08695.
  * [97] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system”, in  Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.  Association for Computing Machinery, 2016.  arXiv:1603.02754.  10.1145/2939672.2939785.
  * [98] NVIDIA, “NVIDIA Triton Inference Server Model Analyzer”. https://github.com/triton-inference-server/model_analyzer.  Accessed: 2023-11-08.
  * [99] P. Buncic et al., “CernVM – a virtual software appliance for LHC applications”,  J. Phys. Conf. Ser. 219 (2010) 042003, 10.1088/1742-6596/219/4/042003.
  * [100] S. D. Guida et al., “The CMS condition database system”,  J. Phys. Conf. Ser. 664 (2015) 042024, 10.1088/1742-6596/664/4/042024.
  * [101] L. Bauerdick et al., “Using Xrootd to federate regional storage”,  J. Phys. Conf. Ser. 396 (2012) 042009, 10.1088/1742-6596/396/4/042009.
  * [102] CMS Collaboration, “Performance of missing transverse momentum reconstruction in proton-proton collisions at $\sqrt{s}=13\TeV$ using the CMS detector”,  JINST 14 (2019) P07004, 10.1088/1748-0221/14/07/P07004, arXiv:1903.06078.
  * [103] CMSSW, “ParticleNet producer in CMSSW”. https://github.com/cms-sw/cmssw/blob/CMSSW_13_0_0/RecoBTag/ONNXRuntime/plugins/BoostedJetONNXJetTagsProducer.cc.  Accessed: 2023-11-08.
  * [104] CMSSW, “ParticleNet SONIC producer in CMSSW”. https://github.com/cms-sw/cmssw/blob/CMSSW_13_0_0/RecoBTag/ONNXRuntime/plugins/ParticleNetSonicJetTagsProducer.cc.  Accessed: 2023-11-08.
  * [105] M. Cacciari, G. P. Salam, and G. Soyez, “The anti-jet clustering algorithm”,  JHEP 04 (2008) 063, 10.1088/1126-6708/2008/04/063, arXiv:0802.1189.
  * [106] M. Cacciari, G. P. Salam, and G. Soyez, “FastJet user manual”,  Eur. Phys. J. C 72 (2012) 1896, 10.1140/epjc/s10052-012-1896-2, arXiv:1111.6097.
  * [107] CMS Collaboration, “Performance of the ParticleNet tagger on small and large-radius jets at high level trigger in Run 3”, CMS Detector Performance Note CMS-DP-2023-021, 2023.
  * [108] CMS Collaboration, “Identification of highly Lorentz-boosted heavy particles using graph neural networks and new mass decorrelation techniques”, CMS Detector Performance Note CMS-DP-2020-002, 2020.
  * [109] CMS Collaboration, “Mass regression of highly-boosted jets using graph neural networks”, CMS Detector Performance Note CMS-DP-2021-017, 2021.
  * [110] Y. Feng, “A new deep-neural-network-based missing transverse momentum estimator, and its application to recoil”.  PhD thesis, University of Maryland, College Park, 2020.  10.13016/e6ze-zycc.
  * [111] CMS Collaboration, “Identification of hadronic tau lepton decays using a deep neural network”,  JINST 17 (2022) P07023, 10.1088/1748-0221/17/07/P07023, arXiv:2201.08458.
  * [112] NVIDIA Corporation, “NVIDIA T4 70W low profile PCIe GPU accelerator”.  NVIDIA Corporation, Santa Clara, 2020.
  * [113] B. Holzman et al., “HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation”,  Comput. Softw. Big Sci. 1 (2017) 1, 10.1007/s41781-017-0001-9, arXiv:1710.00100.
  * [114] Intel Corporation, “Intel 64 and IA-32 architectures software developer’s manual”.  Intel Corporation, Santa Clara, 2023.
  * [115] SchedMD, “Slurm workload manager”. https://slurm.schedmd.com/documentation.html.  Accessed: 2023-11-08.
  * [116] Advanced Micro Devices, Inc., “AMD EPYC 7002 series processors power electronic health record solutions”.  Advanced Micro Devices, Inc., Santa Clara, 2020.

##  .8 The CMS Collaboration

Yerevan Physics Institute, Yerevan, Armenia A. Hayrapetyan, A.
Tumasyan10009-0000-0684-6742

Institut für Hochenergiephysik, Vienna, Austria W. Adam0000-0001-9099-4341,
J.W. Andrejkovic, T. Bergauer0000-0002-5786-0293, S.
Chatterjee0000-0003-2660-0349, K. Damanakis0000-0001-5389-2872, M.
Dragicevic0000-0003-1967-6783, P.S. Hussain0000-0002-4825-5278, M.
Jeitler20000-0002-5141-9560, N. Krammer0000-0002-0548-0985, A.
Li0000-0002-4547-116X, D. Liko0000-0002-3380-473X, I.
Mikulec0000-0003-0385-2746, J. Schieck20000-0002-1058-8093, R.
Schöfbeck0000-0002-2332-8784, D. Schwarz0000-0002-3821-7331, M.
Sonawane0000-0003-0510-7010, S. Templ0000-0003-3137-5692, W.
Waltenberger0000-0002-6215-7228, C.-E. Wulz20000-0001-9226-5812

Universiteit Antwerpen, Antwerpen, Belgium M.R. Darwish30000-0003-2894-2377,
T. Janssen0000-0002-3998-4081, P. Van Mechelen0000-0002-8731-9051

Vrije Universiteit Brussel, Brussel, Belgium E.S. Bols0000-0002-8564-8732, J.
D’Hondt0000-0002-9598-6241, S. Dansana0000-0002-7752-7471, A. De
Moor0000-0001-5964-1935, M. Delcourt0000-0001-8206-1787, H. El
Faham0000-0001-8894-2390, S. Lowette0000-0003-3984-9987, I.
Makarenko0000-0002-8553-4508, D. Müller0000-0002-1752-4527, A.R.
Sahasransu0000-0003-1505-1743, S. Tavernier0000-0002-6792-9522, M.
Tytgat40000-0002-3990-2074, G.P. Van Onsem0000-0002-1664-2337, S. Van
Putte0000-0003-1559-3606, D. Vannerom0000-0002-2747-5095

Université Libre de Bruxelles, Bruxelles, Belgium B.
Clerbaux0000-0001-8547-8211, A.K. Das, G. De Lentdecker0000-0001-5124-7693, L.
Favart0000-0003-1645-7454, P. Gianneios0009-0003-7233-0738, D.
Hohov0000-0002-4760-1597, J. Jaramillo0000-0003-3885-6608, A. Khalilzadeh,
F.A. Khan0009-0002-2039-277X, K. Lee0000-0003-0808-4184, M.
Mahdavikhorrami0000-0002-8265-3595, A. Malara0000-0001-8645-9282, S.
Paredes0000-0001-8487-9603, L. Thomas0000-0002-2756-3853, M. Vanden
Bemden0009-0000-7725-7945, C. Vander Velde0000-0003-3392-7294, P.
Vanlaer0000-0002-7931-4496

Ghent University, Ghent, Belgium M. De Coen0000-0002-5854-7442, D.
Dobur0000-0003-0012-4866, Y. Hong0000-0003-4752-2458, J.
Knolle0000-0002-4781-5704, L. Lambrecht0000-0001-9108-1560, G. Mestdach, K.
Mota Amarilo0000-0003-1707-3348, C. Rendón, A. Samalan, K.
Skovpen0000-0002-1160-0621, N. Van Den Bossche0000-0003-2973-4991, J. van der
Linden0000-0002-7174-781X, L. Wezenbeek0000-0001-6952-891X

Université Catholique de Louvain, Louvain-la-Neuve, Belgium A.
Benecke0000-0003-0252-3609, A. Bethani0000-0002-8150-7043, G.
Bruno0000-0001-8857-8197, C. Caputo0000-0001-7522-4808, C.
Delaere0000-0001-8707-6021, I.S. Donertas0000-0001-7485-412X, A.
Giammanco0000-0001-9640-8294, K. Jaffel0000-0001-7419-4248, Sa.
Jain0000-0001-5078-3689, V. Lemaitre, J. Lidrych0000-0003-1439-0196, P.
Mastrapasqua0000-0002-2043-2367, K. Mondal0000-0001-5967-1245, T.T.
Tran0000-0003-3060-350X, S. Wertz0000-0002-8645-3670

Centro Brasileiro de Pesquisas Fisicas, Rio de Janeiro, Brazil G.A.
Alves0000-0002-8369-1446, E. Coelho0000-0001-6114-9907, C.
Hensel0000-0001-8874-7624, T. Menezes De Oliveira, A.
Moraes0000-0002-5157-5686, P. Rebello Teles0000-0001-9029-8506, M. Soeiro

Universidade do Estado do Rio de Janeiro, Rio de Janeiro, Brazil W.L. Aldá
Júnior0000-0001-5855-9817, M. Alves Gallo Pereira0000-0003-4296-7028, M.
Barroso Ferreira Filho0000-0003-3904-0571, H. Brandao
Malbouisson0000-0002-1326-318X, W. Carvalho0000-0003-0738-6615, J.
Chinellato5, E.M. Da Costa0000-0002-5016-6434, G.G. Da
Silveira60000-0003-3514-7056, D. De Jesus Damiao0000-0002-3769-1680, S.
Fonseca De Souza0000-0001-7830-0837, R. Gomes De Souza, J.
Martins70000-0002-2120-2782, C. Mora Herrera0000-0003-3915-3170, L.
Mundim0000-0001-9964-7805, H. Nogima0000-0001-7705-1066, J.P.
Pinheiro0000-0002-3233-8247, A. Santoro0000-0002-0568-665X, A.
Sznajder0000-0001-6998-1108, M. Thiel0000-0001-7139-7963, A. Vilela
Pereira0000-0003-3177-4626

Universidade Estadual Paulista, Universidade Federal do ABC, São Paulo, Brazil
C.A. Bernardes60000-0001-5790-9563, L. Calligaris0000-0002-9951-9448, T.R.
Fernandez Perez Tomei0000-0002-1809-5226, E.M. Gregores0000-0003-0205-1672,
P.G. Mercadante0000-0001-8333-4302, S.F. Novaes0000-0003-0471-8549, B.
Orzari0000-0003-4232-4743, Sandra S. Padula0000-0003-3071-0559

Institute for Nuclear Research and Nuclear Energy, Bulgarian Academy of
Sciences, Sofia, Bulgaria A. Aleksandrov0000-0001-6934-2541, G.
Antchev0000-0003-3210-5037, R. Hadjiiska0000-0003-1824-1737, P.
Iaydjiev0000-0001-6330-0607, M. Misheva0000-0003-4854-5301, M.
Shopova0000-0001-6664-2493, G. Sultanov0000-0002-8030-3866

University of Sofia, Sofia, Bulgaria A. Dimitrov0000-0003-2899-701X, L.
Litov0000-0002-8511-6883, B. Pavlov0000-0003-3635-0646, P.
Petkov0000-0002-0420-9480, A. Petrov0009-0003-8899-1514, E.
Shumka0000-0002-0104-2574

Instituto De Alta Investigación, Universidad de Tarapacá, Casilla 7 D, Arica,
Chile S. Keshri0000-0003-3280-2350, S. Thakur0000-0002-1647-0360

Beihang University, Beijing, China T. Cheng0000-0003-2954-9315, T.
Javaid0009-0007-2757-4054, L. Yuan0000-0002-6719-5397

Department of Physics, Tsinghua University, Beijing, China Z.
Hu0000-0001-8209-4343, J. Liu, K. Yi8,90000-0002-2459-1824

Institute of High Energy Physics, Beijing, China G.M.
Chen100000-0002-2629-5420, H.S. Chen100000-0001-8672-8227, M.
Chen100000-0003-0489-9669, F. Iemmi0000-0001-5911-4051, C.H. Jiang, A.
Kapoor110000-0002-1844-1504, H. Liao0000-0002-0124-6999, Z.-A.
Liu120000-0002-2896-1386, R. Sharma130000-0003-1181-1426, J.N. Song12, J.
Tao0000-0003-2006-3490, C. Wang10, J. Wang0000-0002-3103-1083, Z. Wang10, H.
Zhang0000-0001-8843-5209

State Key Laboratory of Nuclear Physics and Technology, Peking University,
Beijing, China A. Agapitos0000-0002-8953-1232, Y. Ban0000-0002-1912-0374, A.
Levin0000-0001-9565-4186, C. Li0000-0002-6339-8154, Q. Li0000-0002-8290-0517,
Y. Mao, S.J. Qian0000-0002-0630-481X, X. Sun0000-0003-4409-4574, D.
Wang0000-0002-9013-1199, H. Yang, L. Zhang0000-0001-7947-9007, C.
Zhou0000-0001-5904-7258

Sun Yat-Sen University, Guangzhou, China Z. You0000-0001-8324-3291

University of Science and Technology of China, Hefei, China N.
Lu0000-0002-2631-6770

Nanjing Normal University, Nanjing, China G. Bauer14

Institute of Modern Physics and Key Laboratory of Nuclear Physics and Ion-beam
Application (MOE) - Fudan University, Shanghai, China X.
Gao150000-0001-7205-2318, D. Leggat, H. Okawa0000-0002-2548-6567

Zhejiang University, Hangzhou, Zhejiang, China Z. Lin0000-0003-1812-3474, C.
Lu0000-0002-7421-0313, M. Xiao0000-0001-9628-9336

Universidad de Los Andes, Bogota, Colombia C. Avila0000-0002-5610-2693, D.A.
Barbosa Trujillo, A. Cabrera0000-0002-0486-6296, C. Florez0000-0002-3222-0249,
J. Fraga0000-0002-5137-8543, J.A. Reyes Vega

Universidad de Antioquia, Medellin, Colombia J. Mejia
Guisao0000-0002-1153-816X, F. Ramirez0000-0002-7178-0484, M.
Rodriguez0000-0002-9480-213X, J.D. Ruiz Alvarez0000-0002-3306-0363

University of Split, Faculty of Electrical Engineering, Mechanical Engineering
and Naval Architecture, Split, Croatia D. Giljanovic0009-0005-6792-6881, N.
Godinovic0000-0002-4674-9450, D. Lelas0000-0002-8269-5760, A.
Sculac0000-0001-7938-7559

University of Split, Faculty of Science, Split, Croatia M.
Kovac0000-0002-2391-4599, T. Sculac0000-0002-9578-4105

Institute Rudjer Boskovic, Zagreb, Croatia P. Bargassa0000-0001-8612-3332, V.
Brigljevic0000-0001-5847-0062, B.K. Chitroda0000-0002-0220-8441, D.
Ferencek0000-0001-9116-1202, S. Mishra0000-0002-3510-4833, A.
Starodumov160000-0001-9570-9255, T. Susa0000-0001-7430-2552

University of Cyprus, Nicosia, Cyprus A. Attikis0000-0002-4443-3794, K.
Christoforou0000-0003-2205-1100, S. Konstantinou0000-0003-0408-7636, J.
Mousa0000-0002-2978-2718, C. Nicolaou, F. Ptochos0000-0002-3432-3452, P.A.
Razis0000-0002-4855-0162, H. Rykaczewski, H. Saka0000-0001-7616-2573, A.
Stepennov0000-0001-7747-6582

Charles University, Prague, Czech Republic M. Finger0000-0002-7828-9970, M.
Finger Jr.0000-0003-3155-2484, A. Kveton0000-0001-8197-1914

Escuela Politecnica Nacional, Quito, Ecuador E. Ayala0000-0002-0363-9198

Universidad San Francisco de Quito, Quito, Ecuador E. Carrera
Jarrin0000-0002-0857-8507

Academy of Scientific Research and Technology of the Arab Republic of Egypt,
Egyptian Network of High Energy Physics, Cairo, Egypt A.A.
Abdelalim17,180000-0002-2056-7894, E. Salama19,200000-0002-9282-9806

Center for High Energy Physics (CHEP-FU), Fayoum University, El-Fayoum, Egypt
M.A. Mahmoud0000-0001-8692-5458, Y. Mohammed0000-0001-8399-3017

National Institute of Chemical Physics and Biophysics, Tallinn, Estonia K.
Ehataht0000-0002-2387-4777, M. Kadastik, T. Lange0000-0001-6242-7331, S.
Nandan0000-0002-9380-8919, C. Nielsen0000-0002-3532-8132, J.
Pata0000-0002-5191-5759, M. Raidal0000-0001-7040-9491, L.
Tani0000-0002-6552-7255, C. Veelken0000-0002-3364-916X

Department of Physics, University of Helsinki, Helsinki, Finland H.
Kirschenmann0000-0001-7369-2536, K. Osterberg0000-0003-4807-0414, M.
Voutilainen0000-0002-5200-6477

Helsinki Institute of Physics, Helsinki, Finland S.
Bharthuar0000-0001-5871-9622, E. Brücken0000-0001-6066-8756, F.
Garcia0000-0002-4023-7964, K.T.S. Kallonen0000-0001-9769-7163, R. Kinnunen, T.
Lampén0000-0002-8398-4249, K. Lassila-Perini0000-0002-5502-1795, S.
Lehti0000-0003-1370-5598, T. Lindén0009-0002-4847-8882, L.
Martikainen0000-0003-1609-3515, M. Myllymäki0000-0003-0510-3810, M.m.
Rantanen0000-0002-6764-0016, H. Siikonen0000-0003-2039-5874, E.
Tuominen0000-0002-7073-7767, J. Tuominiemi0000-0003-0386-8633

Lappeenranta-Lahti University of Technology, Lappeenranta, Finland P.
Luukka0000-0003-2340-4641, H. Petrow0000-0002-1133-5485

IRFU, CEA, Université Paris-Saclay, Gif-sur-Yvette, France M.
Besancon0000-0003-3278-3671, F. Couderc0000-0003-2040-4099, M.
Dejardin0009-0008-2784-615X, D. Denegri, J.L. Faure, F.
Ferri0000-0002-9860-101X, S. Ganjour0000-0003-3090-9744, P.
Gras0000-0002-3932-5967, G. Hamel de Monchenault0000-0002-3872-3592, V.
Lohezic0009-0008-7976-851X, J. Malcles0000-0002-5388-5565, J. Rander, A.
Rosowsky0000-0001-7803-6650, M.Ö. Sahin0000-0001-6402-4050, A. Savoy-
Navarro210000-0002-9481-5168, P. Simkina0000-0002-9813-372X, M.
Titov0000-0002-1119-6614, M. Tornago0000-0001-6768-1056

Laboratoire Leprince-Ringuet, CNRS/IN2P3, Ecole Polytechnique, Institut
Polytechnique de Paris, Palaiseau, France C. Baldenegro
Barrera0000-0002-6033-8885, F. Beaudette0000-0002-1194-8556, A. Buchot
Perraguin0000-0002-8597-647X, P. Busson0000-0001-6027-4511, A.
Cappati0000-0003-4386-0564, C. Charlot0000-0002-4087-8155, M.
Chiusi0000-0002-1097-7304, F. Damas0000-0001-6793-4359, O.
Davignon0000-0001-8710-992X, A. De Wit0000-0002-5291-1661, B.A. Fontana Santos
Alves0000-0001-9752-0624, S. Ghosh0009-0006-5692-5688, A.
Gilbert0000-0001-7560-5790, R. Granier de Cassagnac0000-0002-1275-7292, A.
Hakimi0009-0008-2093-8131, B. Harikrishnan0000-0003-0174-4020, L.
Kalipoliti0000-0002-5705-5059, G. Liu0000-0001-7002-0937, J.
Motta0000-0003-0985-913X, M. Nguyen0000-0001-7305-7102, C.
Ochando0000-0002-3836-1173, L. Portales0000-0002-9860-9185, R.
Salerno0000-0003-3735-2707, J.B. Sauvan0000-0001-5187-3571, Y.
Sirois0000-0001-5381-4807, A. Tarabini0000-0001-7098-5317, E.
Vernazza0000-0003-4957-2782, A. Zabi0000-0002-7214-0673, A.
Zghiche0000-0002-1178-1450

Université de Strasbourg, CNRS, IPHC UMR 7178, Strasbourg, France J.-L.
Agram220000-0001-7476-0158, J. Andrea0000-0002-8298-7560, D.
Apparu0009-0004-1837-0496, D. Bloch0000-0002-4535-5273, J.-M.
Brom0000-0003-0249-3622, E.C. Chabert0000-0003-2797-7690, C.
Collard0000-0002-5230-8387, S. Falke0000-0002-0264-1632, U.
Goerlach0000-0001-8955-1666, C. Grimault, R. Haeberle0009-0007-5007-6723,
A.-C. Le Bihan0000-0002-8545-0187, M. Meena0000-0003-4536-3967, G.
Saha0000-0002-6125-1941, M.A. Sessini0000-0003-2097-7065, P. Van
Hove0000-0002-2431-3381

Institut de Physique des 2 Infinis de Lyon (IP2I ), Villeurbanne, France S.
Beauceron0000-0002-8036-9267, B. Blancon0000-0001-9022-1509, G.
Boudoul0009-0002-9897-8439, N. Chanon0000-0002-2939-5646, J.
Choi0000-0002-6024-0992, D. Contardo0000-0001-6768-7466, P.
Depasse0000-0001-7556-2743, C. Dozen230000-0002-4301-634X, H. El Mamouni, J.
Fay0000-0001-5790-1780, S. Gascon0000-0002-7204-1624, M.
Gouzevitch0000-0002-5524-880X, C. Greenberg, G. Grenier0000-0002-1976-5877, B.
Ille0000-0002-8679-3878, I.B. Laktineh, M. Lethuillier0000-0001-6185-2045, L.
Mirabito, S. Perries, A. Purohit0000-0003-0881-612X, M. Vander
Donckt0000-0002-9253-8611, P. Verdier0000-0003-3090-2948, J.
Xiao0000-0002-7860-3958

Georgian Technical University, Tbilisi, Georgia I.
Bagaturia240000-0001-8646-4372, I. Lomidze0009-0002-3901-2765, Z.
Tsamalaidze160000-0001-5377-3558

RWTH Aachen University, I. Physikalisches Institut, Aachen, Germany V.
Botta0000-0003-1661-9513, L. Feld0000-0001-9813-8646, K.
Klein0000-0002-1546-7880, M. Lipinski0000-0002-6839-0063, D.
Meuser0000-0002-2722-7526, A. Pauls0000-0002-8117-5376, N.
Röwert0000-0002-4745-5470, M. Teroerde0000-0002-5892-1377

RWTH Aachen University, III. Physikalisches Institut A, Aachen, Germany S.
Diekmann0009-0004-8867-0881, A. Dodonova0000-0002-5115-8487, N.
Eich0000-0001-9494-4317, D. Eliseev0000-0001-5844-8156, F.
Engelke0000-0002-9288-8144, J. Erdmann, M. Erdmann0000-0002-1653-1303, P.
Fackeldey0000-0003-4932-7162, B. Fischer0000-0002-3900-3482, T.
Hebbeker0000-0002-9736-266X, K. Hoepfner0000-0002-2008-8148, F.
Ivone0000-0002-2388-5548, A. Jung0000-0002-2511-1490, M.y.
Lee0000-0002-4430-1695, F. Mausolf0000-0003-2479-8419, M.
Merschmeyer0000-0003-2081-7141, A. Meyer0000-0001-9598-6623, S.
Mukherjee0000-0001-6341-9982, D. Noll0000-0002-0176-2360, F. Nowotny, A.
Pozdnyakov0000-0003-3478-9081, Y. Rath, W. Redjeb0000-0001-9794-8292, F. Rehm,
H. Reithler0000-0003-4409-702X, U. Sarkar0000-0002-9892-4601, V.
Sarkisovi0000-0001-9430-5419, A. Schmidt0000-0003-2711-8984, A.
Sharma0000-0002-5295-1460, J.L. Spah0000-0002-5215-3258, A.
Stein0000-0003-0713-811X, F. Torres Da Silva De Araujo250000-0002-4785-3057,
L. Vigilante, S. Wiedenbeck0000-0002-4692-9304, S. Zaleski

RWTH Aachen University, III. Physikalisches Institut B, Aachen, Germany C.
Dziwok0000-0001-9806-0244, G. Flügge0000-0003-3681-9272, W. Haj
Ahmad260000-0003-1491-0446, T. Kress0000-0002-2702-8201, A.
Nowack0000-0002-3522-5926, O. Pooth0000-0001-6445-6160, A.
Stahl0000-0002-8369-7506, T. Ziemons0000-0003-1697-2130, A.
Zotz0000-0002-1320-1712

Deutsches Elektronen-Synchrotron, Hamburg, Germany H. Aarup
Petersen0009-0005-6482-7466, M. Aldaya Martin0000-0003-1533-0945, J.
Alimena0000-0001-6030-3191, S. Amoroso, Y. An0000-0003-1299-1879, S.
Baxter0009-0008-4191-6716, M. Bayatmakou0009-0002-9905-0667, H. Becerril
Gonzalez0000-0001-5387-712X, O. Behnke0000-0002-4238-0991, A.
Belvedere0000-0002-2802-8203, S. Bhattacharya0000-0002-3197-0048, F.
Blekman270000-0002-7366-7098, K. Borras280000-0003-1111-249X, A.
Campbell0000-0003-4439-5748, A. Cardini0000-0003-1803-0999, C. Cheng, F.
Colombina0009-0008-7130-100X, S. Consuegra Rodríguez0000-0002-1383-1837, G.
Correia Silva0000-0001-6232-3591, M. De Silva0000-0002-5804-6226, G. Eckerlin,
D. Eckstein0000-0002-7366-6562, L.I. Estevez Banos0000-0001-6195-3102, O.
Filatov0000-0001-9850-6170, E. Gallo270000-0001-7200-5175, A.
Geiser0000-0003-0355-102X, A. Giraldi0000-0003-4423-2631, V.
Guglielmi0000-0003-3240-7393, M. Guthoff0000-0002-3974-589X, A.
Hinzmann0000-0002-2633-4696, A. Jafari290000-0001-7327-1870, L.
Jeppe0000-0002-1029-0318, N.Z. Jomhari0000-0001-9127-7408, B.
Kaech0000-0002-1194-2306, M. Kasemann0000-0002-0429-2448, C.
Kleinwort0000-0002-9017-9504, R. Kogler0000-0002-5336-4399, M.
Komm0000-0002-7669-4294, D. Krücker0000-0003-1610-8844, W. Lange, D. Leyva
Pernia0009-0009-8755-3698, K. Lipka300000-0002-8427-3748, W.
Lohmann310000-0002-8705-0857, R. Mankel0000-0003-2375-1563, I.-A. Melzer-
Pellmann0000-0001-7707-919X, M. Mendizabal Morentin0000-0002-6506-5177, A.B.
Meyer0000-0001-8532-2356, G. Milella0000-0002-2047-951X, A.
Mussgiller0000-0002-8331-8166, L.P. Nair0000-0002-2351-9265, A.
Nürnberg0000-0002-7876-3134, Y. Otarid, J. Park0000-0002-4683-6669, D. Pérez
Adán0000-0003-3416-0726, E. Ranken0000-0001-7472-5029, A.
Raspereza0000-0003-2167-498X, B. Ribeiro Lopes0000-0003-0823-447X, J.
Rübenach, A. Saggio0000-0002-7385-3317, M. Scham32,280000-0001-9494-2151, S.
Schnake280000-0003-3409-6584, P. Schütze0000-0003-4802-6990, C.
Schwanenberger270000-0001-6699-6662, D. Selivanova0000-0002-7031-9434, K.
Sharko0000-0002-7614-5236, M. Shchedrolosiev0000-0003-3510-2093, R.E. Sosa
Ricardo0000-0002-2240-6699, D. Stafford, F. Vazzoler0000-0001-8111-9318, A.
Ventura Barroso0000-0003-3233-6636, R. Walsh0000-0002-3872-4114, Q.
Wang0000-0003-1014-8677, Y. Wen0000-0002-8724-9604, K. Wichmann, L.
Wiens280000-0002-4423-4461, C. Wissing0000-0002-5090-8004, Y.
Yang0009-0009-3430-0558, A. Zimermmane Castro Santos0000-0001-9302-3102

University of Hamburg, Hamburg, Germany A. Albrecht0000-0001-6004-6180, S.
Albrecht0000-0002-5960-6803, M. Antonello0000-0001-9094-482X, S.
Bein0000-0001-9387-7407, L. Benato0000-0001-5135-7489, S. Bollweg, M.
Bonanomi0000-0003-3629-6264, P. Connor0000-0003-2500-1061, M. Eich, K. El
Morabit0000-0001-5886-220X, Y. Fischer0000-0002-3184-1457, C.
Garbers0000-0001-5094-2256, E. Garutti0000-0003-0634-5539, A.
Grohsjean0000-0003-0748-8494, J. Haller0000-0001-9347-7657, H.R.
Jabusch0000-0003-2444-1014, G. Kasieczka0000-0003-3457-2755, P. Keicher, R.
Klanner0000-0002-7004-9227, W. Korcari0000-0001-8017-5502, T.
Kramer0000-0002-7004-0214, V. Kutzner0000-0003-1985-3807, F.
Labe0000-0002-1870-9443, J. Lange0000-0001-7513-6330, A.
Lobanov0000-0002-5376-0877, C. Matthies0000-0001-7379-4540, A.
Mehta0000-0002-0433-4484, L. Moureaux0000-0002-2310-9266, M. Mrowietz, A.
Nigamova0000-0002-8522-8500, Y. Nissan, A. Paasch0000-0002-2208-5178, K.J.
Pena Rodriguez0000-0002-2877-9744, T. Quadfasel0000-0003-2360-351X, B.
Raciti0009-0005-5995-6685, M. Rieger0000-0003-0797-2606, D.
Savoiu0000-0001-6794-7475, J. Schindler0009-0006-6551-0660, P.
Schleper0000-0001-5628-6827, M. Schröder0000-0001-8058-9828, J.
Schwandt0000-0002-0052-597X, M. Sommerhalder0000-0001-5746-7371, H.
Stadie0000-0002-0513-8119, G. Steinbrück0000-0002-8355-2761, A. Tews, M.
Wolf0000-0003-3002-2430

Karlsruher Institut fuer Technologie, Karlsruhe, Germany S.
Brommer0000-0001-8988-2035, M. Burkart, E. Butz0000-0002-2403-5801, T.
Chwalek0000-0002-8009-3723, A. Dierlamm0000-0001-7804-9902, A. Droll, N.
Faltermann0000-0001-6506-3107, M. Giffels0000-0003-0193-3032, A.
Gottmann0000-0001-6696-349X, F. Hartmann330000-0001-8989-8387, R.
Hofsaess0009-0008-4575-5729, M. Horzela0000-0002-3190-7962, U.
Husemann0000-0002-6198-8388, J. Kieseler0000-0003-1644-7678, M.
Klute0000-0002-0869-5631, R. Koppenhöfer0000-0002-6256-5715, J.M.
Lawhorn0000-0002-8597-9259, M. Link, A. Lintuluoto0000-0002-0726-1452, S.
Maier0000-0001-9828-9778, S. Mitra0000-0002-3060-2278, M.
Mormile0000-0003-0456-7250, Th. Müller0000-0003-4337-0098, M. Neukum, M.
Oh0000-0003-2618-9203, M. Presilla0000-0003-2808-7315, G.
Quast0000-0002-4021-4260, K. Rabbertz0000-0001-7040-9846, B.
Regnery0000-0003-1539-923X, N. Shadskiy0000-0001-9894-2095, I.
Shvetsov0000-0002-7069-9019, H.J. Simonis0000-0002-7467-2980, M.
Toms0000-0002-7703-3973, N. Trevisani0000-0002-5223-9342, R.F. Von
Cube0000-0002-6237-5209, M. Wassmer0000-0002-0408-2811, S.
Wieland0000-0003-3887-5358, F. Wittig, R. Wolf0000-0001-9456-383X, X.
Zuo0000-0002-0029-493X

Institute of Nuclear and Particle Physics (INPP), NCSR Demokritos, Aghia
Paraskevi, Greece G. Anagnostou, G. Daskalakis0000-0001-6070-7698, A.
Kyriakis, A. Papadopoulos33, A. Stakia0000-0001-6277-7171

National and Kapodistrian University of Athens, Athens, Greece P.
Kontaxakis0000-0002-4860-5979, G. Melachroinos, A. Panagiotou, I.
Papavergou0000-0002-7992-2686, I. Paraskevas0000-0002-2375-5401, N.
Saoulidou0000-0001-6958-4196, K. Theofilatos0000-0001-8448-883X, E.
Tziaferi0000-0003-4958-0408, K. Vellidis0000-0001-5680-8357, I.
Zisopoulos0000-0001-5212-4353

National Technical University of Athens, Athens, Greece G.
Bakas0000-0003-0287-1937, T. Chatzistavrou, G. Karapostoli0000-0002-4280-2541,
K. Kousouris0000-0002-6360-0869, I. Papakrivopoulos0000-0002-8440-0487, E.
Siamarkou, G. Tsipolitis, A. Zacharopoulou

University of Ioánnina, Ioánnina, Greece K. Adamidis, I. Bestintzanos, I.
Evangelou0000-0002-5903-5481, C. Foudas, C. Kamtsikis, P. Katsoulis, P.
Kokkas0009-0009-3752-6253, P.G. Kosmoglou Kioseoglou0000-0002-7440-4396, N.
Manthos0000-0003-3247-8909, I. Papadopoulos0000-0002-9937-3063, J.
Strologas0000-0002-2225-7160

HUN-REN Wigner Research Centre for Physics, Budapest, Hungary M.
Bartók340000-0002-4440-2701, C. Hajdu0000-0002-7193-800X, D.
Horvath35,360000-0003-0091-477X, K. Márton, F. Sikler0000-0001-9608-3901, V.
Veszpremi0000-0001-9783-0315

MTA-ELTE Lendület CMS Particle and Nuclear Physics Group, Eötvös Loránd
University, Budapest, Hungary M. Csanád0000-0002-3154-6925, K.
Farkas0000-0003-1740-6974, M.M.A. Gadallah370000-0002-8305-6661, Á.
Kadlecsik0000-0001-5559-0106, P. Major0000-0002-5476-0414, K.
Mandal0000-0002-3966-7182, G. Pásztor0000-0003-0707-9762, A.J.
Rádl380000-0001-8810-0388, G.I. Veres0000-0002-5440-4356

Faculty of Informatics, University of Debrecen, Debrecen, Hungary P. Raics, B.
Ujvari0000-0003-0498-4265, G. Zilizi0000-0002-0480-0000

Institute of Nuclear Research ATOMKI, Debrecen, Hungary G. Bencze, S. Czellar,
J. Molnar, Z. Szillasi

Karoly Robert Campus, MATE Institute of Technology, Gyongyos, Hungary T.
Csorgo380000-0002-9110-9663, F. Nemes380000-0002-1451-6484, T.
Novak0000-0001-6253-4356

Panjab University, Chandigarh, India J. Babbar0000-0002-4080-4156, S.
Bansal0000-0003-1992-0336, S.B. Beri, V. Bhatnagar0000-0002-8392-9610, G.
Chaudhary0000-0003-0168-3336, S. Chauhan0000-0001-6974-4129, N.
Dhingra390000-0002-7200-6204, A. Kaur0000-0002-1640-9180, A.
Kaur0000-0003-3609-4777, H. Kaur0000-0002-8659-7092, M.
Kaur0000-0002-3440-2767, S. Kumar0000-0001-9212-9108, K.
Sandeep0000-0002-3220-3668, T. Sheokand, J.B. Singh0000-0001-9029-2462, A.
Singla0000-0003-2550-139X

University of Delhi, Delhi, India A. Ahmed0000-0002-4500-8853, A.
Bhardwaj0000-0002-7544-3258, A. Chhetri0000-0001-7495-1923, B.C.
Choudhary0000-0001-5029-1887, A. Kumar0000-0003-3407-4094, A.
Kumar0000-0002-5180-6595, M. Naimuddin0000-0003-4542-386X, K.
Ranjan0000-0002-5540-3750, S. Saumya0000-0001-7842-9518

Saha Institute of Nuclear Physics, HBNI, Kolkata, India S.
Baradia0000-0001-9860-7262, S. Barman400000-0001-8891-1674, S.
Bhattacharya0000-0002-8110-4957, S. Dutta0000-0001-9650-8121, S. Dutta, S.
Sarkar

Indian Institute of Technology Madras, Madras, India M.M.
Ameen0000-0002-1909-9843, P.K. Behera0000-0002-1527-2266, S.C.
Behera0000-0002-0798-2727, S. Chatterjee0000-0003-0185-9872, P.
Jana0000-0001-5310-5170, P. Kalbhor0000-0002-5892-3743, J.R.
Komaragiri410000-0002-9344-6655, D. Kumar410000-0002-6636-5331, P.R.
Pujahari0000-0002-0994-7212, N.R. Saha0000-0002-7954-7898, A.
Sharma0000-0002-0688-923X, A.K. Sikdar0000-0002-5437-5217, S.
Verma0000-0003-1163-6955

Tata Institute of Fundamental Research-A, Mumbai, India S. Dugad, M.
Kumar0000-0003-0312-057X, G.B. Mohanty0000-0001-6850-7666, P. Suryadevara

Tata Institute of Fundamental Research-B, Mumbai, India A.
Bala0000-0003-2565-1718, S. Banerjee0000-0002-7953-4683, R.M. Chatterjee, R.K.
Dewanjee420000-0001-6645-6244, M. Guchait0009-0004-0928-7922, Sh.
Jain0000-0003-1770-5309, A. Jaiswal, S. Karmakar0000-0001-9715-5663, S.
Kumar0000-0002-2405-915X, G. Majumder0000-0002-3815-5222, K.
Mazumdar0000-0003-3136-1653, S. Parolia0000-0002-9566-2490, A.
Thachayath0000-0001-6545-0350

National Institute of Science Education and Research, An OCC of Homi Bhabha
National Institute, Bhubaneswar, Odisha, India S.
Bahinipati430000-0002-3744-5332, C. Kar0000-0002-6407-6974, D.
Maity440000-0002-1989-6703, P. Mal0000-0002-0870-8420, T.
Mishra0000-0002-2121-3932, V.K. Muraleedharan Nair
Bindhu440000-0003-4671-815X, K. Naskar440000-0003-0638-4378, A.
Nayak440000-0002-7716-4981, P. Sadangi, P. Saha0000-0002-7013-8094, S.K.
Swain0000-0001-6871-3937, S. Varghese440009-0000-1318-8266, D.
Vats440009-0007-8224-4664

Indian Institute of Science Education and Research (IISER), Pune, India S.
Acharya450009-0001-2997-7523, A. Alpana0000-0003-3294-2345, S.
Dube0000-0002-5145-3777, B. Gomber450000-0002-4446-0258, B.
Kansal0000-0002-6604-1011, A. Laha0000-0001-9440-7028, B.
Sahu450000-0002-8073-5140, S. Sharma0000-0001-6886-0726, K.Y. Vaish

Isfahan University of Technology, Isfahan, Iran H.
Bakhshiansohi460000-0001-5741-3357, E. Khazaie470000-0001-9810-7743, M.
Zeinali480000-0001-8367-6257

Institute for Research in Fundamental Sciences (IPM), Tehran, Iran S.
Chenarani490000-0002-1425-076X, S.M. Etesami0000-0001-6501-4137, M.
Khakzad0000-0002-2212-5715, M. Mohammadi Najafabadi0000-0001-6131-5987

University College Dublin, Dublin, Ireland M. Grunewald0000-0002-5754-0388

INFN Sezione di Baria, Università di Barib, Politecnico di Baric, Bari, Italy
M. Abbresciaa,b0000-0001-8727-7544, R. Alya,c,170000-0001-6808-1335, A.
Colaleoa,b0000-0002-0711-6319, D. Creanzaa,c0000-0001-6153-3044, B.
D’Anzia,b0000-0002-9361-3142, N. De Filippisa,c0000-0002-0625-6811, M. De
Palmaa,b0000-0001-8240-1913, A. Di Florioa,c0000-0003-3719-8041, W.
Elmetenaweea,b,170000-0001-7069-0252, L. Fiorea0000-0002-9470-1320, G.
Iasellia,c0000-0003-2546-5341, M. Loukaa,b, G. Maggia,c0000-0001-5391-7689, M.
Maggia0000-0002-8431-3922, I. Margjekaa,b0000-0002-3198-3025, V.
Mastrapasquaa,b0000-0002-9082-5924, S. Mya,b0000-0002-9938-2680, S.
Nuzzoa,b0000-0003-1089-6317, A. Pellecchiaa,b0000-0003-3279-6114, A.
Pompilia,b0000-0003-1291-4005, G. Pugliesea,c0000-0001-5460-2638, R.
Radognaa0000-0002-1094-5038, G. Ramirez-Sancheza,c0000-0001-7804-5514, D.
Ramosa0000-0002-7165-1017, A. Ranieria0000-0001-7912-4062, L.
Silvestrisa0000-0002-8985-4891, F.M. Simonea,b0000-0002-1924-983X, Ü.
Sözbilira0000-0001-6833-3758, A. Stamerraa0000-0003-1434-1968, R.
Vendittia0000-0001-6925-8649, P. Verwilligena0000-0002-9285-8631, A.
Zazaa,b0000-0002-0969-7284

INFN Sezione di Bolognaa, Università di Bolognab, Bologna, Italy G.
Abbiendia0000-0003-4499-7562, C. Battilanaa,b0000-0002-3753-3068, D.
Bonacorsia,b0000-0002-0835-9574, L. Borgonovia0000-0001-8679-4443, R.
Campaninia,b0000-0002-2744-0597, P. Capiluppia,b0000-0003-4485-1897, A.
Castroa,b0000-0003-2527-0456, F.R. Cavalloa0000-0002-0326-7515, M.
Cuffiania,b0000-0003-2510-5039, G.M. Dallavallea0000-0002-8614-0420, T.
Diotalevia,b0000-0003-0780-8785, A. Fanfania,b0000-0003-2256-4117, D.
Fasanellaa,b0000-0002-2926-2691, P. Giacomellia0000-0002-6368-7220, L.
Giommia,b0000-0003-3539-4313, C. Grandia0000-0001-5998-3070, L.
Guiduccia,b0000-0002-6013-8293, S. Lo Meoa,500000-0003-3249-9208, L.
Lunertia,b0000-0002-8932-0283, S. Marcellinia0000-0002-1233-8100, G.
Masettia0000-0002-6377-800X, F.L. Navarriaa,b0000-0001-7961-4889, A.
Perrottaa0000-0002-7996-7139, F. Primaveraa,b0000-0001-6253-8656, A.M.
Rossia,b0000-0002-5973-1305, T. Rovellia,b0000-0002-9746-4842, G.P.
Sirolia,b0000-0002-3528-4125

INFN Sezione di Cataniaa, Università di Cataniab, Catania, Italy S.
Costaa,b,510000-0001-9919-0569, A. Di Mattiaa0000-0002-9964-015X, R.
Potenzaa,b, A. Tricomia,b,510000-0002-5071-5501, C. Tuvea,b0000-0003-0739-3153

INFN Sezione di Firenzea, Università di Firenzeb, Firenze, Italy P.
Assiourasa0000-0002-5152-9006, G. Barbaglia0000-0002-1738-8676, G.
Bardellia,b0000-0002-4662-3305, B. Camaiania,b0000-0002-6396-622X, A.
Cassesea0000-0003-3010-4516, R. Ceccarellia0000-0003-3232-9380, V.
Ciullia,b0000-0003-1947-3396, C. Civininia0000-0002-4952-3799, R.
D’Alessandroa,b0000-0001-7997-0306, E. Focardia,b0000-0002-3763-5267, T.
Kelloa, G. Latinoa,b0000-0002-4098-3502, P. Lenzia,b0000-0002-6927-8807, M.
Lizzoa0000-0001-7297-2624, M. Meschinia0000-0002-9161-3990, S.
Paolettia0000-0003-3592-9509, A. Papanastassioua,b, G.
Sguazzonia0000-0002-0791-3350, L. Viliania0000-0002-1909-6343

INFN Laboratori Nazionali di Frascati, Frascati, Italy L.
Benussi0000-0002-2363-8889, S. Bianco0000-0002-8300-4124, S.
Meola520000-0002-8233-7277, D. Piccolo0000-0001-5404-543X

INFN Sezione di Genovaa, Università di Genovab, Genova, Italy P.
Chatagnona0000-0002-4705-9582, F. Ferroa0000-0002-7663-0805, E.
Robuttia0000-0001-9038-4500, S. Tosia,b0000-0002-7275-9193

INFN Sezione di Milano-Bicoccaa, Università di Milano-Bicoccab, Milano, Italy
A. Benagliaa0000-0003-1124-8450, G. Boldrinia,b0000-0001-5490-605X, F.
Brivioa0000-0001-9523-6451, F. Cetorellia0000-0002-3061-1553, F. De
Guioa,b0000-0001-5927-8865, M.E. Dinardoa,b0000-0002-8575-7250, P.
Dinia0000-0001-7375-4899, S. Gennaia0000-0001-5269-8517, R.
Gerosaa,b0000-0001-8359-3734, A. Ghezzia,b0000-0002-8184-7953, P.
Govonia,b0000-0002-0227-1301, L. Guzzia0000-0002-3086-8260, M.T.
Lucchinia,b0000-0002-7497-7450, M. Malbertia0000-0001-6794-8419, S.
Malvezzia0000-0002-0218-4910, A. Massironia0000-0002-0782-0883, D.