# A Roadmap for
HEP Software and Computing R&D
for the 2020s

HEP Software Foundation111Authors are listed at the end of this report.

###### Abstract

Particle physics has an ambitious and broad experimental programme for the
coming decades. This programme requires large investments in detector
hardware, either to build new facilities and experiments, or to upgrade
existing ones. Similarly, it requires commensurate investment in the R&D of
software to acquire, manage, process, and analyse the shear amounts of data to
be recorded. In planning for the HL-LHC in particular, it is critical that all
of the collaborating stakeholders agree on the software goals and priorities,
and that the efforts complement each other. In this spirit, this white paper
describes the R&D activities required to prepare for this software upgrade.

|  | HSF-CWP-2017-01
---|---|---
|  | December 15, 2017

##  1 Introduction

Particle physics has an ambitious experimental programme for the coming
decades. The programme supports the strategic goals of the particle physics
community that have been laid out by the European Strategy for Particle
Physics ESPP2013 (1) and by the Particle Physics Project Prioritization Panel
(P5) DOE-P5 (2) in the United States 2014pwa (3). Broadly speaking, the
scientific goals are:

  * •

Exploit the discovery of the Higgs boson as a precision tool for investigating
Standard Model (SM) and Beyond the Standard Model (BSM) physics.

  * •

Study the decays of $b$\- and $c$-hadrons, and tau leptons, in the search for
manifestations of BSM physics, and investigate matter-antimatter differences.

  * •

Search for signatures of dark matter.

  * •

Probe neutrino oscillations and masses.

  * •

Study the Quark Gluon Plasma state of matter in heavy-ion collisions.

  * •

Explore the unknown.

The High-Luminosity Large Hadron Collider (HL-LHC) HL-LHC (4, 5, 6) will be a
major upgrade of the current LHC LHC (7) supporting the aim of an in-depth
investigation of the properties of the Higgs boson and its couplings to other
particles (Figure 1). The ATLAS ATLAS (8) and CMS CMS (9) collaborations will
continue to make measurements in the Higgs sector, while searching for new
physics Beyond the Standard Model (BSM). Should a BSM discovery be made, a
full exploration of that physics will be pursued. Such BSM physics may help
shed light on the nature of dark matter, which we know makes up the majority
of gravitational matter in the universe, but which does not interact via the
electromagnetic or strong nuclear forces Mangano2016 (10).

The LHCb experiment at the LHC LHCb (11) and the Belle II experiment at KEK
BelleII (12) study various aspects of heavy flavour physics (b- and c-quark,
and tau-lepton physics), where quantum influences of very high mass particles
manifest themselves in lower energy phenomena. Their primary goal is to look
for BSM physics, either by studying CP violation (that is, asymmetries in the
behaviour of particles and their corresponding antiparticles) or modifications
in rate or angular distributions in rare heavy-flavour decays. Current
manifestations of such asymmetries do not explain why our universe is so
matter dominated. These flavour physics programmes are related to BSM searches
through effective field theory, and powerful constraints on new physics keep
coming from such studies.

Figure 1: The current schedule for the LHC and HL-LHC upgrade and run HL-LHC
(4). Currently, the start of the HL-LHC run is foreseen for mid 2026\. The
long shutdowns, LS2 and LS3, will be used to upgrade both the accelerator and
the detector hardware.

The study of neutrinos, their mass and oscillations, can also shed light on
matter-antimatter asymmetry. The DUNE experiment will provide a huge
improvement in our ability to probe neutrino physics, detecting neutrinos from
the Long Baseline Neutrino Facility at Fermilab, as well as linking to astro-
particle physics programmes, in particular through the potential detection of
supernovas and relic neutrinos. An overview of the experimental programme
scheduled at the Fermilab facility is given in Figure 2.

Figure 2: Run schedule for the Fermilab facility until 2026 fnal-schedule
(13).

In the study of the early universe immediately after the Big Bang, it is
critical to understand the phase transition between the highly compressed
quark-gluon plasma and the nuclear matter in the universe today. The ALICE
experiment at the LHC ALICE (14) and the CBM CBM (15) and PANDA PANDA (16)
experiments at the Facility for Antiproton and Ion Research (FAIR) are
specifically designed to probe this aspect of nuclear and particle physics. In
addition ATLAS, CMS and LHCb all contribute to the LHC heavy-ion programme.

These experimental programmes require large investments in detector hardware,
either to build new facilities and experiments (e.g., FAIR and DUNE) or to
upgrade existing ones (HL-LHC, Belle II). Similarly, they require commensurate
investment in the research and development necessary to deploy software to
acquire, manage, process, and analyse the data recorded.

For the HL-LHC, which is scheduled to begin taking data in 2026 (Figure 1) and
to run into the 2030s, some 30 times more data than the LHC has currently
produced will be collected by ATLAS and CMS. As the total amount of LHC data
already collected is close to an exabyte, it is clear that the problems to be
solved require approaches beyond simply scaling current solutions, assuming
Moore’s Law and more or less constant operational budgets. The nature of
computing hardware (processors, storage, networks) is evolving with radically
new paradigms, the quantity of data to be processed is increasing
dramatically, its complexity is increasing, and more sophisticated analyses
will be required to maximise physics yield. Developing and deploying
sustainable software for future and upgraded experiments, given these
constraints, is both a technical and a social challenge, as detailed in this
paper. An important message of this report is that a “software upgrade” is
needed to run in parallel with the hardware upgrades planned for the HL-LHC in
order to take full advantage of these hardware upgrades and to complete the
HL-LHC physics programme.

In planning for the HL-LHC in particular, it is critical that all of the
collaborating stakeholders agree on the software goals and priorities, and
that the efforts complement each other. In this spirit, the HEP Software
Foundation (HSF) began a planning exercise in late 2016 to prepare a Community
White Paper (CWP) HSF2017 (17) at the behest of the Worldwide LHC Computing
Grid (WLCG) project WLCG2016 (18). The role of the HSF is to facilitate
coordination and common efforts in HEP software and computing internationally
and to provide a structure for the community to set goals and priorities for
future work. The objective of the CWP is to provide a roadmap for software R&D
in preparation for the HL-LHC and for other HEP experiments on a similar
timescale, which would identify and prioritise the software research and
development investments required:

  * •

to achieve improvements in software efficiency, scalability and performance,
and to make use of advances in CPU, storage and network technologies in order
to cope with the challenges ahead;

  * •

to enable new approaches to computing and software that can radically extend
the physics reach of the detectors;

  * •

to ensure the long-term sustainability of the software through the lifetime of
the HL- LHC;

  * •

to ensure data and knowledge preservation beyond the lifetime of individual
experiments;

  * •

to attract the required new expertise by offering appropriate career
recognition to physicists specialising in software development and by an
effective training effort to target all software contributors in the
community.

The CWP process, organised by the HSF with the participation of the LHC
experiments and the wider HEP software and computing community, began with a
kick-off workshop at the San Diego Supercomputer Centre (SDSC), USA, in
January 2017 and concluded after a final workshop in June 2017 at the
Laboratoire d’Annecy de Physique des Particules (LAPP), France, with a large
number of intermediate topical workshops and meetings (Appendix A). The entire
CWP process involved an estimated 250 participants.

To reach more widely than the LHC experiments, specific contact was made with
individuals with software and computing responsibilities in the Fermilab muon
and neutrino experiments, Belle II, the Linear Collider community, as well as
various national computing organisations. The CWP process was able to build on
all the links established since the inception of the HSF in 2014.

Working groups were established on various topics which were expected to be
important parts of the HL-LHC roadmap: _Careers, Staffing and Training;
Conditions Database; Data Organisation, Management and Access; Data Analysis
and Interpretation; Data and Software Preservation; Detector Simulation; Data-
Flow Processing Frameworks; Facilities and Distributed Computing; Machine
Learning; Physics Generators; Security; Software Development, Deployment and
Validation/Verification; Software Trigger and Event Reconstruction;_ and
_Visualisation._ The work of each working group is summarised in this
document.

This document is the result of the CWP process. Investing in the roadmap
outlined here will be fruitful for the whole of the HEP programme and may also
benefit other projects with similar technical challenges, particularly in
astrophysics, e.g., the Square Kilometre Array (SKA) SKA (19), the Cherenkov
Telescope Array (CTA) CTA (20) and the Large Synoptic Survey Telescope (LSST)
LSST (21).

##  2 Software and Computing Challenges

Run 2 for the LHC started in 2015 and delivered a proton-proton collision
energy of 13 TeV. By the end of LHC Run 2 in 2018, it is expected that about
150 fb-1 of physics data will have been collected by both ATLAS and CMS.
Together with ALICE and LHCb, the total size of LHC data storage pledged by
sites for the year 2017 is around 1 exabyte, as shown in Table 1 from the
LHC’s Computing Resource Scrutiny Group (CRSG) Lucchesi:2284575 (22). The CPU
allocation from the CRSG for 2017 to each experiment is also shown.

Experiment |  2017 Disk Pledges (PB) |  2017 Tape Pledges (PB) |  Total Disk and Tape Pledges (PB) |  2017 CPU Pledges (kHS06)
---|---|---|---|---
ALICE |  67 |  68 |  138 |  807
ATLAS |  172 |  251 |  423 |  2194
CMS |  123 |  204 |  327 |  1729
LHCb |  35 |  67 |  102 |  413
Total |  400 |  591 |  990 |  5143
Table 1: Resources pledged by WLCG sites to the 4 LHC experiments for the year
2017 as described at the September 2017 session of the Computing Resources
Scrutiny Group (CRSG).

Using an approximate conversion from HS06 HS06 (23) to CPU cores of 10 means
that LHC computing in 2017 is supported by about 500k CPU cores. These
resources are deployed ubiquitously, from close to the experiments themselves
at CERN to a worldwide distributed computing infrastructure, the WLCG WLCG
(24). Each experiment has developed its own workflow management and data
management software to manage its share of WLCG resources.

In order to process the data, the 4 largest LHC experiments have written tens
of millions of lines of program code over the last 15 years
1742-6596-898-7-072013 (25, 26, 27, 28). This has involved contributions from
thousands of physicists and many computing professionals, encompassing a wide
range of skills and abilities. The majority of this code was written for a
single architecture (x86_64) and with a serial processing model in mind. There
is considerable anxiety in the experiments that much of this software is not
sustainable, with the original authors no longer in the field and much of the
code itself in a poorly maintained state, ill-documented, and lacking tests.
This code, which is largely experiment-specific, manages the entire experiment
data flow, including data acquisition, high-level triggering, calibration and
alignment, simulation, reconstruction (of both real and simulated data),
visualisation, and final data analysis.

HEP experiments are typically served with a large set of integrated and
configured common software components, which have been developed either in-
house or externally. Well-known examples include ROOT Brun1996 (29), which is
a data analysis toolkit that also plays a critical role in the implementation
of experiments’ data storage systems, and Geant4 Agostinelli2003 (30), a
simulation framework through which most detector simulation is achieved. Other
packages provide tools for supporting the development process; they include
compilers and scripting languages, as well as tools for integrating, building,
testing, and generating documentation. Physics simulation is supported by a
wide range of event generators provided by the theory community (PYTHIA PYTHIA
(31), SHERPA Gleisberg:2008ta (32), ALPGEN Mangano:565290 (33), MADGRAPH
MADGRAPH (34), HERWIG HERWIG (35), amongst many others). There is also code
developed to support the computing infrastructure itself, such as the CVMFS
distributed caching filesystem 1742-6596-331-4-042003 (36), the Frontier
database caching mechanism Frontier (37), the XRootD file access software
XRootD (38) and a number of storage systems (dCache, DPM, EOS). This list of
packages is by no means exhaustive, but illustrates the range of software
employed and its critical role in almost every aspect of the programme.

Already in Run 3 LHCb will process more than 40 times the number of collisions
that it does today, and ALICE will read out Pb-Pb collisions continuously at
50 kHz. The upgrade to the HL-LHC for Run 4 then produces a step change for
ATLAS and CMS. The beam intensity will rise substantially, giving bunch
crossings where the number of discrete proton-proton interactions (pileup)
will rise to about 200, from about 60 today. This has important consequences
for the operation of the detectors and for the performance of the
reconstruction software. The two experiments will upgrade their trigger
systems to record 5-10 times as many events as they do today. It is
anticipated that HL-LHC will deliver about 300 fb-1 of data each year.

The steep rise in resources that are then required to manage this data can be
estimated from an extrapolation of the Run 2 computing model and is shown in
Figures 3 and 4.

(a)

(b)

Figure 3: CMS estimated CPU (3(a)) and disk space (3(b)) resources required
into the HL-LHC era, using the current computing model with parameters
projected out for the next 12 years 1742-6596-1085-2-022006 (39).

(a) Estimated CPU resources (in kHS06) needed for the years 2018 to 2028 for
both data and simulation processing. The blue points are estimates based on
the current software performance estimates and using the ATLAS computing model
parameters from 2017. The solid line shows the amount of resources expected to
be available if a flat funding scenario is assumed, which implies an increase
of 20% per year, based on the current technology trends.

(b) Estimated total disk resources (in PB) needed for the years 2018 to 2028
for both data and simulation processing. The blue points are estimates based
on the current event sizes estimates and using the ATLAS computing model
parameters from 2017. The solid line shows the amount of resources expected to
be available if a flat funding scenario is assumed, which implies an increase
of 15% per year, based on the current technology trends.

Figure 4: ATLAS resources required into the HL-LHC era, using the current
computing model and software performance.ATLAS-HL-LHC-Plots (40)

In general, it can be said that the amount of data that experiments can
collect and process in the future will be limited by affordable software and
computing, and therefore the physics reach during HL-LHC will be limited by
how efficiently these resources can be used.

The ATLAS numbers, in Figure 4, are particularly interesting as they estimate
the resources that will be available to the experiment if a flat funding
profile is maintained, taking into account the expected technology
improvements given current trends Panzer2017 (41). As can be seen, the
shortfall between needs and bare technology gains is considerable: a factor 4
in CPU and a factor 7 in disk in 2027.

While the density of transistors on silicon continues to increase following
Moore’s Law (albeit more slowly than in the past), power density constraints
have limited the clock speed of processors for more than a decade NAP12980
(42). This has effectively stalled any progress in the processing capacity of
a single CPU core. Instead, increases in potential processing capacity come
from increases in the core count of CPUs and wide CPU registers. Alternative
processing architectures have become more commonplace. These range from the
many-core architecture based on standard x86_64 cores to numerous alternatives
such as GPUs. For GPUs, the processing model is very different
Cook:2012:CPD:2430671 (43), allowing a much greater fraction of the die to be
dedicated to arithmetic calculations, but at a price in programming difficulty
and memory handling for the developer that tends to be specific to each
processor generation. Further developments may even see the use of FPGAs for
more general-purpose tasks. Fully exploiting these evolutions requires a shift
in programming model to one based on _concurrency_.

Even with the throttling of clock speed to limit power consumption, power
remains a major issue. Low power architectures are in huge demand. At one
level this might challenge the dominance of x86_64 by simply replacing it
with, for example, AArch64 devices that may achieve lower power costs for the
scale of HEP computing needs than Intel has achieved with its Xeon
architecture 1742-6596-608-1-012001 (44, 45, 46). More extreme is an
architecture that would see specialised processing units dedicated to
particular tasks, but with possibly large parts of the device switched off
most of the time, so-called dark silicon martin2014 (47, 48).

Limitations in affordable storage also pose a major challenge, as does the I/O
rates of higher capacity hard disks. Network bandwidth will probably continue
to increase at the required level, but the ability to use it efficiently will
need a closer integration with applications. This will require software
developments to support distributed computing (data and workload management,
software distribution and data access) and an increasing awareness of the
extremely hierarchical view of data, from long latency tape access and medium-
latency network access through to the CPU memory hierarchy.

Taking advantage of these new architectures and programming paradigms will be
critical for HEP to increase the ability of our code to deliver physics
results efficiently, and to meet the processing challenges of the future. Some
of this work will be focused on re-optimised implementations of existing
algorithms. This will be complicated by the fact that much of our code is
written for the much simpler model of serial processing, and without the
software engineering needed for sustainability. Proper support for taking
advantage of concurrent programming techniques, such as vectorisation and
thread-based programming, through frameworks and libraries, will be essential,
as the majority of the code will still be written by physicists. Other
approaches should examine new algorithms and techniques, including highly
parallelised code that can run on GPUs or the use of machine learning
techniques to replace computationally expensive pieces of simulation or
pattern recognition. The ensemble of computing work that is needed by the
experiments must remain sufficiently flexible to take advantage of different
architectures that will provide computing to HEP in the future. The use of
high performance computing sites and commercial cloud providers will very
likely be a requirement for the community and will bring particular
constraints and demand flexibility.

These technical challenges are accompanied by significant human challenges.
Software is written by many people in the collaborations, with varying levels
of expertise, from a few experts with precious skills to novice coders. This
implies organising training in effective coding techniques and providing
excellent documentation, examples and support. Although it is inevitable that
some developments will remain within the scope of a single experiment,
tackling software problems coherently as a community will be critical to
achieving success in the future. This will range from sharing knowledge of
techniques and best practice to establishing common libraries and projects
that will provide generic solutions to the community. Writing code that
supports a wider subset of the community than just a single experiment will
almost certainly be mandated upon HEP and presents a greater challenge, but
the potential benefits are huge. Attracting, and retaining, people with the
required skills who can provide leadership is another significant challenge,
since it impacts on the need to give adequate recognition to physicists who
specialise in software development. This is an important issue that is treated
in more detail later in the report.

Particle physics is no longer alone in facing these massive data challenges.
Experiments in other fields, from astronomy 10.1371/journal.pbio.1002195 (49)
to genomics ijms18020412 (50), will produce huge amounts of data in the
future, and will need to overcome the same challenges that we face, i.e.,
massive data handling and efficient scientific programming. Establishing links
with these fields has already started. Additionally, interest from the
computing science community in solving these data challenges exists, and
mutually beneficial relationships would be possible where there are genuine
research problems that are of academic interest to that community and provide
practical solutions to ours. The efficient processing of massive data volumes
is also a challenge faced by industry, in particular the internet economy,
which developed novel and major new technologies under the banner of _Big
Data_ that may be applicable to our use cases 47224 (51, 52, 53).

Establishing a programme of investment in software for the HEP community, with
a view to ensuring effective and sustainable software for the coming decades,
will be essential to allow us to reap the physics benefits of the multi-
exabyte data to come. It was in recognition of this fact that the HSF itself
was set up and already works to promote these common projects and community
developments HSF2015 (54).

##  3 Programme of Work

In the following we describe the programme of work being proposed for the
range of topics covered by the CWP working groups. We summarise the main
specific challenges each topic will face, describe current practices, and
propose a number of R&D tasks that should be undertaken in order to meet the
challenges. R&D tasks are grouped in two different timescales: short term (by
2020, in time for the HL-LHC Computing Technical Design Reports of ATLAS and
CMS) and longer-term actions (by 2022, to be ready for testing or deployment
during LHC Run 3).

###  3.1 Physics Generators

#### Scope and Challenges

Monte-Carlo event generators are a vital part of modern particle physics,
providing a key component of the understanding and interpretation of
experiment data. Collider experiments have a need for theoretical QCD
predictions at very high precision. Already in LHC Run 2, experimental
uncertainties for many analyses are at the same level as, or lower than, those
from theory. Many analyses have irreducible QCD-induced backgrounds, where
statistical extrapolation into the signal region can only come from theory
calculations. With future experiment and machine upgrades, as well as
reanalysis of current data, measured uncertainties will shrink even further,
and this will increase the need to reduce the corresponding errors from
theory.

Increasing accuracy will compel the use of higher-order perturbation theory
generators with challenging computational demands. Generating Monte Carlo
events using leading order (LO) generators is only a small part of the overall
computing requirements for HEP experiments. Next-to-leading order (NLO) event
generation, used more during LHC Run 2, is already using significant
resources. Higher accuracy theoretical cross sections calculated at next-to-
next-to-leading (NNLO), already important in some Run 2 analyses, are not
widely used because of computational cost. By HL-LHC the use of NNLO event
generation will be more widely required, so these obstacles to their adoption
must be overcome. Increasing the order of the generators increases greatly the
complexity of the phase space integration required to calculate the
appropriate QCD matrix elements. The difficulty of this integration arises
from the need to have sufficient coverage in a high-dimensional space (10-15
dimensions, with numerous local maxima); the appearance of negative event
weights; and the fact that many terms in the integration cancel, so that a
very high degree of accuracy of each term is required. Memory demands for
generators have generally been low and initialisation times have been fast,
but an increase in order means that memory consumption becomes important and
initialisation times can become a significant fraction of the job’s run time.

For HEP experiments, in many cases, meaningful predictions can only be
obtained by combining higher-order perturbative calculations with parton
showers. This procedure is also needed as high-multiplicity final states
become more interesting at higher luminosities and event rates. Matching
(N)NLO fixed-order calculations to parton shower algorithms can have a very
low efficiency, and increases further the computational load needed to
generate the necessary number of particle-level events. In addition, many of
the current models for the combination of parton-level event generators and
parton shower codes are incompatible with requirements for concurrency on
modern architectures. It is a major challenge to ensure that this software can
run efficiently on next generation hardware and software systems.

Developments in generator software are mainly done by the HEP theory
community. Theorists typically derive career recognition and advancement from
making contributions to theory itself, rather than by making improvements to
the computational efficiency of generators _per se_. So, improving the
computational efficiency of event generators, and allowing them to run
effectively on resources such as high performance computing facilities (HPCs),
will mean engaging with experts in computational optimisation who can work
with the theorists who develop generators.

The challenge in the next decade is to advance the theory and practical
implementation of event generators to support the needs of future experiments,
reaching a new level of theory precision and recognising the demands for
computation and computational efficiency that this will bring.

#### Current Practice

Extensive use of LO generators and parton shower algorithms are still made by
most HEP experiments. Each experiment has its own simulation needs, but for
the LHC experiments tens of billions of generated events are now used each
year for Monte Carlo simulations. During LHC Run 2 more and more NLO
generators were used, because of their increased theoretical precision and
stability. The raw computational complexity of NLO amplitudes, combined with
many-body phase-space evaluations and the inefficiencies of the matching
process, leads to a potentially much-increased CPU budget for physics event
simulation for ATLAS and CMS.

The use of NLO generators by the experiments today is also limited because of
the way the generators are implemented, producing significant numbers of
negative event weights. This means that the total number of events the
experiments need to generate, simulate, and reconstruct can be many times
larger for NLO than for LO samples. At the same time, the experiments budget
only a similar number of Monte Carlo simulation events as from the real data.
Having large NLO samples is thus not consistent with existing computing
budgets until a different scheme is developed that does not depend on negative
event weights or produces them only at a significantly reduced rate.

While most event generation is run on “standard” grid resources, effort is
ongoing to run more demanding tasks on HPC resources, e.g., W-boson + 5-jet
events at the Argonne Mira HPC). However, scaling for efficient running on
some of the existing HPC resources is not trivial and requires effort.

Standard HEP libraries such as LHAPDF LHAPDF (55), HepMCHepMC (56), and Rivet
Rivet (57) are used by the generators for integration into the experiments’
event generation workflows. These require extensions and sustained maintenance
that should be considered a shared responsibility of the theoretical and
experimental communities in the context of large-scale experiments. In
practice, however, it has been difficult to achieve the level of support that
is really needed as there has been a lack of recognition for this work. To
help improve the capabilities and performance of generators as used by the
experimental HEP programme, and to foster interaction between the communities,
the MCnet MCnet (58) short-term studentship programme has been very useful.
Interested experimental PhD students can join a generator group for several
months to work on improving a physics aspect of the simulation that is
relevant to their work, or to improve the integration of the generator into an
experimental framework.

#### Research and Development Programme

As the Monte Carlo projects are funded mainly to develop theoretical
improvements, and not mainly as “suppliers” to the experimental HEP programme,
any strong requests towards efficiency improvements from the experimental
community would need to be backed up by plausible avenues of support that can
fund contributions from software engineers with the correct technical skills
in software optimisation to work within the generator author teams.

In a similar way to the MCnet studentships, a matchmaking scheme could focus
on the software engineering side, and transfer some of the expertise available
in the experiments and facilities teams to the generator projects. Sustainable
improvements are unlikely to be delivered by graduate students “learning on
the job” and then leaving after a few months, so meeting the requirement of
transferring technical expertise and effort will likely require placements for
experienced optimisation specialists and a medium- to long-term connection to
the generator project.

HEP experiments, which are now managed by very large collaborations including
many technical experts, can also play a key role in sustaining a healthy
relationship between theory and experiment software. Effort to work on common
tools that benefit both the experiment itself and the wider community would
provide shared value that justifies direct investment from the stakeholders.
This model would also be beneficial for core HEP tools like LHAPDF, HepMC and
Rivet, where future improvements have no theoretical physics interest anymore,
putting them in a similar situation to generator performance improvements. One
structural issue blocking such a mode of operation is that some experiments do
not currently recognise contributions to external projects as experiment
service work — a situation deserving of review in areas where external
software tools are critical to experiment success.

In the following we describe specific areas of R&D for event generation up to
2022 and beyond.

  * •

The development of new and improved theoretical algorithms provides the
largest potential for improving event generators. While it is not guaranteed
that simply increasing the effort dedicated to this task will bring about the
desired result, the long-term support of event generator development, and the
creation of career opportunities in this research area, are critical given the
commitment to experiments on multi-decade scales.

  * •

Expand development in reweighting event samples, where new physics signatures
can be explored by updating the partonic weights according to new matrix
elements. It is necessary that the phase space for the updated model be a
subset of the original one, which is an important limitation. The procedure is
more complex at NLO and can require additional information to be stored in the
event files to properly reweight in different cases. Overcoming the technical
issues from utilising negative event weights is crucial. Nevertheless, the
method can be powerful in many cases, and would hugely reduce the time needed
for the generation of BSM samples.

  * •

At a more technical level, concurrency is an avenue that has yet to be
explored in depth for event generation. As the calculation of matrix elements
requires VEGAS-style integration, this work would be helped by the development
of a new Monte-Carlo integrator. For multi-particle interactions, factorising
the full phase space integration into lower dimensional integrals would be a
powerful method of parallelising, while the interference between different
Feynman graphs can be handled with known techniques.

  * •

For many widely used generators, basic problems of concurrency and thread
hostility need to be tackled, to make these packages suitable for efficient
large scale use on modern processors and within modern HEP software
frameworks. Providing appropriate common tools for interfacing, benchmarking
and optimising multithreaded code would allow expertise to be shared
effectively HSF-CWP-2017-13 (59).

  * •

In most generators, parallelism was added post-facto, which leads to scaling
problems when the level of parallelism becomes very large, e.g., on HPC
machines. These HPC machines will be part of the computing resource pool used
by HEP, so solving scaling issues on these resources for event generation is
important, particularly as the smaller generator code bases can make porting
to non-x86_64 architectures more tractable. The problem of long and
inefficient initialisation when a job utilises hundreds or thousands of cores
on an HPC needs to be tackled. While the memory consumption of event
generators is generally modest, the generation of tree-level contributions to
high multiplicity final states can use significant memory, and gains would be
expected from optimising here.

  * •

An underexplored avenue is the efficiency of event generation as used by the
experiments. An increasingly common usage is to generate very large inclusive
event samples, which are filtered on event final-state criteria to decide
which events are to be retained and passed onto detector simulation and
reconstruction. This naturally introduces a large waste of very CPU-expensive
event generation, which could be reduced by developing filtering tools within
the generators themselves, designed for compatibility with the experiments’
requirements. A particularly wasteful example is where events are separated
into orthogonal subsamples by filtering, in which case the same large
inclusive sample is generated many times, with each stream filtering the
events into a different group: allowing a single inclusive event generation to
be filtered into several orthogonal output streams would improve efficiency.

###  3.2 Detector Simulation

#### Scope and Challenges

For all its success so far, the challenges faced by the HEP field in the
simulation domain are daunting. During the first two runs, the LHC experiments
produced, reconstructed, stored, transferred, and analysed tens of billions of
simulated events. This effort required more than half of the total computing
resources allocated to the experiments. As part of the HL-LHC physics
programme, the upgraded experiments expect to collect 150 times more data than
in Run 1; demand for larger simulation samples to satisfy analysis needs will
grow accordingly. In addition, simulation tools have to serve diverse
communities, including accelerator-based particle physics research utilising
proton-proton colliders, neutrino, dark matter, and muon experiments, as well
as the cosmic frontier. The complex detectors of the future, with different
module- or cell-level shapes, finer segmentation, and novel materials and
detection techniques, require additional features in geometry tools and bring
new demands on physics coverage and accuracy within the constraints of the
available computing budget. The diversification of the physics programmes also
requires new and improved physics models. More extensive use of Fast
Simulation is a potential solution, under the assumption that it is possible
to improve time performance without an unacceptable loss of physics accuracy.

The gains that can be made by speeding up critical elements of the Geant4
simulation toolkit can be leveraged for all applications that use it, and it
is therefore well worth the investment in effort needed to achieve it. The
main challenges to be addressed if the required physics and software
performance goals are to be achieved are:

  * •

Review the implementations of physics models, including the assumptions,
approximations, and limitations. In the best cases this can achieve higher
precision and improve runtime performance through code modernisation Novak2018
(60). The extension of the validity of models up to energies of 100 TeV is
foreseen for the Future Circular Collider (FCC) project FCC (61) and provides
a good opportunity for this modernisation.

  * •

Redesigning, developing, and commissioning detector simulation toolkits to be
more efficient when executed on current vector CPUs and emerging new
architectures, including GPUs, where use of SIMD vectorisation is vital; this
includes porting and optimising the experiments’ simulation applications to
allow exploitation of large HPC facilities.

  * •

Exploring different Fast Simulation options, where the full detector
simulation is replaced, in whole or in part, by computationally efficient
techniques. An area of investigation is common frameworks for fast tuning and
validation.

  * •

Developing, improving and optimising geometry tools that can be shared among
experiments to make the modeling of complex detectors computationally more
efficient, modular, and transparent.

  * •

Developing techniques for background modeling, including contributions of
multiple hard interactions overlapping the event of interest in collider
experiments (pileup).

  * •

Revisiting digitisation algorithms to improve performance and exploring
opportunities for code sharing among experiments.

It is obviously of critical importance that the whole community of scientists
working in the simulation domain continue to work together in as efficient a
way as possible in order to deliver the required improvements. Very specific
expertise is required across all simulation domains, such as physics modeling,
tracking through complex geometries and magnetic fields, and building
realistic applications that accurately simulate highly complex detectors.
Continuous support is needed to recruit, train, and retain people with a
unique set of skills needed to guarantee the development, maintenance, and
support of simulation codes over the long timeframes foreseen in the HEP
experimental programme.

#### Current Practices

The Geant4 detector simulation toolkit is at the core of simulation in almost
every HEP experiment. Its continuous development, maintenance, and support for
the experiments is of vital importance. New or refined functionality in
physics coverage and accuracy continues to be delivered in the ongoing
development programme and software performance improvements are introduced
whenever possible.

Physics models are a critical part of the detector simulation, and are
continuously being reviewed, and in some cases reimplemented, in order to
improve accuracy and software performance. Electromagnetic (EM) transport
simulation is challenging as it occupies a large part of the computing
resources used in full detector simulation. Significant efforts have been made
in the recent past to better describe the simulation of electromagnetic shower
shapes, in particular to model the $H\to\gamma\gamma$ signal and background
accurately at the LHC. This effort is being continued with an emphasis on
reviewing the models’ assumptions, approximations, and limitations, especially
at very high energy, with a view to improving their respective software
implementations. In addition, a new “theory-based” model (Goudsmit-
Saunderson), for describing the _multiple scattering_ of electrons and
positrons, has been developed that has been demonstrated to outperform, in
terms of physics accuracy and speed, the current models in Geant4. The models
used to describe the _bremsstrahlung_ process have also been reviewed, and
recently an improved theoretical description of the Landau-Pomeranchuk-Migdal
effect was introduced that plays a significant role at high energies.
Theoretical review of all electromagnetic models, including those of hadrons
and ions, is therefore of high priority both for HL-LHC and for FCC studies.

Hadronic physics simulation covers purely hadronic interactions. It is not
possible for a single model to describe all the physics encountered in a
simulation due to the large energy range that needs to be covered and the
simplified approximations that are used to overcome the difficulty of solving
the full theory (QCD). Currently the most-used reference physics list for high
energy and space applications is FTFP_BERT. It uses the Geant4 Bertini cascade
for hadron–nucleus interactions from 0 to 12 GeV incident hadron energy and
the FTF parton string model for hadron–nucleus interactions from 3 GeV
upwards. QGSP_BERT is a popular alternative which replaces the FTF model with
the QGS model over the high energy range. The existence of more than one model
(for each energy range) is very valuable in order to be able to determine the
systematics effects related to the approximations used. The use of highly
granular calorimeters, such as the ones being designed by the CALICE
collaboration for future linear colliders, allows a detailed validation of the
development of hadronic showers with test-beam data. Preliminary results
suggest that the lateral profiles of Geant4 hadronic showers are too narrow.
Comparisons with LHC test-beam data have shown that a fundamental ingredient
for improving the description of the lateral development of showers is the use
of intermediate and low energy models that can describe the cascading of
hadrons in nuclear matter. Additional work is currently being invested in the
further improvement of the QGS model, which is a more theory-based approach
than the phenomenological FTF model, and therefore offers better confidence at
high energies, up to a few TeV. This again is a large endeavour and requires
continuous effort over a long time.

The Geant4 collaboration is working closely with user communities to enrich
the physics models’ validation system with data acquired during physics runs
and test beam campaigns. In producing new models of physics interactions and
improving the fidelity of the models that exist, it is absolutely imperative
that high-quality data are available. Simulation model tuning often relies on
test beam data, and a program to improve the library of available data could
be invaluable to the community. Such data would ideally include both thin-
target test beams for improving interaction models and calorimeter targets for
improving shower models. This data could potentially be used for directly
tuning Fast Simulation models as well.

There are specific challenges associated with the Intensity Frontier
experimental programme, in particular simulation of the beamline and the
neutrino flux. Neutrino experiments rely heavily on detector simulations to
reconstruct neutrino energy, which requires accurate modelling of energy
deposition by a variety of particles across a range of energies. Muon
experiments such as Muon g-2 and Mu2e also face large simulation challenges;
since they are searching for extremely rare effects, they must grapple with
very low signal to background ratios and the modeling of low cross-section
background processes. Additionally, the size of the computational problem is a
serious challenge, as large simulation runs are required to adequately sample
all relevant areas of experimental phase space, even when techniques to
minimise the required computations are used. There is also a need to simulate
the effects of low energy neutrons, which requires large computational
resources. Geant4 is the primary simulation toolkit for all of these
experiments.

Simulation toolkits do not include effects like charge drift in an electric
field or models of the readout electronics of the experiments. Instead, these
effects are normally taken into account in a separate step called
digitisation. Digitisation is inherently local to a given sub-detector and
often even to a given readout element, so that there are many opportunities
for parallelism in terms of vectorisation and multiprocessing or
multithreading, if the code and the data objects are designed optimally.
Recently, both hardware and software projects have benefitted from an
increased level of sharing among experiments. The LArSoft Collaboration
develops and supports a shared base of physics software across Liquid Argon
(LAr) Time Projection Chamber (TPC) experiments, which includes providing
common digitisation code. Similarly, an effort exists among the LHC
experiments to share code for modeling radiation damage effects in silicon. As
ATLAS and CMS expect to use similar readout chips in their future trackers,
further code sharing might be possible.

The Geant4 simulation toolkit will also evolve over the next decade to include
contributions from various R&D projects, as described in the following
section. This is required to ensure the support of experiments through
continuous maintenance and improvement of the Geant4 simulation toolkit. This
is necessary until production versions of potentially alternative engines,
such as those resulting from ongoing R&D work, become available, integrated,
and validated by experiments. The agreed ongoing strategy to make this
adoption possible is to ensure that new developments resulting from the R&D
programme can be tested with realistic prototypes and then be integrated,
validated, and deployed in a timely fashion in Geant4.

#### Research and Development Programme

To meet the challenge of improving the performance by a large factor, an
ambitious R&D programme is underway to investigate each component of the
simulation software for the long term. In the following we describe in detail
some of the studies to be performed in the next 3-5 years.

  * •

Particle Transport and Vectorisation: the study of an efficient transport of
particles (tracks) in groups so as to maximise the benefit of using SIMD
operations.

  * •

Modularisation: improvement of Geant4 design to allow for a tighter and easier
integration of single sub-packages of the code into experimental frameworks.

  * •

Physics Models: extensions and refinements of the physics algorithms to
provide new and more performant physics capabilities.

  * •

Other activities: integration of multi-threading capabilities in experiment
applications; experiment-agnostic software products to cope with increased
pileup, fast simulation, digitisation, and efficient production of high-
quality random numbers.

##### Particle Transport and Vectorisation

One of the most ambitious elements of the simulation R&D programme is a new
approach to managing particle transport, which has been introduced by the
GeantV project. The aim is to deliver a multithreaded vectorised transport
engine that has the potential to deliver large performance benefits. Its main
feature is track-level parallelisation, bundling particles with similar
properties from different events to process them in a single thread. This
approach, combined with SIMD vectorisation coding techniques and improved data
locality, is expected to yield significant speed-ups, which are to be measured
in a realistic prototype currently under development. For the GeantV transport
engine to display its best computing performance, it is necessary to vectorise
and optimise the accompanying modules, including geometry, navigation, and the
physics models. These are developed as independent libraries so that they can
also be used together with the current Geant4 transport engine. Of course,
when used with the current Geant4 they will not expose their full performance
potential, since transport in Geant4 is currently sequential, but this allows
for a preliminary validation and comparison with the existing implementations.
The benefit of this approach is that new developments can be delivered as soon
as they are available. The new vectorised geometry package (VecGeom),
developed as part of GeantV R&D and successfully integrated into Geant4, is an
example that demonstrated the benefit of this approach. By the end of 2018 it
is intended to have a proof-of-concept for the new particle transport engine
that includes vectorised EM physics, vectorised magnetic field propagation and
that uses the new vectorised geometry package. This will form a sound basis
for making performance comparisons for simulating EM showers in a realistic
detector.

  * •

2019: the _beta_ release of the GeantV transport engine will contain enough
functionality to build the first real applications. This will allow
performance to be measured and give sufficient time to prepare for HL-LHC
running. It should include the use of vectorisation in most of the components,
including physics modelling for electrons, gammas and positrons, whilst still
maintaining simulation reproducibility, and I/O in a concurrent environment
and multi-event user data management.

##### Modularisation

Starting from the next release, a modularisation of Geant4 is being pursued
that will allow an easier integration in experimental frameworks, with the
possibility to include only the Geant4 modules that are actually used. A
further use case is the possibility to use one of the Geant4 components in
isolation, e.g., to use hadronic interaction modeling without kernel
components from a fast simulation framework. As a first step a preliminary
review of libraries’ granularity is being pursued, which will be followed by a
review of intra-library dependencies with the final goal of reducing their
dependencies.

  * •

2019: Redesign of some Geant4 kernel components to improve the efficiency of
the simulation on HPC systems, starting from improved handling of Geant4
_database_ s on large core-count systems. A review will be made of the
multithreading design to be closer to task-based frameworks, such as Intel’s
Threaded Building Blocks (TBB) TBB (62).

##### Physics Models

It is intended to develop new and extended physics models to cover extended
energy and physics processing of present and future colliders, Intensity
Frontier experiments, and direct dark matter search experiments. The goal is
to extend the missing models (e.g., neutrino interactions), improve models’
physics accuracy and, at the same time, improve CPU and memory efficiency. The
deliverables of these R&D efforts include physics modules that produce
equivalent quality physics, and will therefore require extensive validation in
realistic applications.

  * •

2020: Improved implementation of hadronic cascade models for LHC and, in
particular, Liquid Argon detectors. Improved accuracy models of EM
interactions of photons and electrons. To address the needs of cosmic frontier
experiments, optical photon transport must be improved and made faster.

  * •

2022: Implementation of EPOS string model for multi-GeV to multi-TeV
interactions, for FCC detector simulation and systematic studies of HL-LHC
detectors.

##### Experiment Applications

The experiment applications are essential for validating the software and
physics performance of new versions of the simulation toolkit. ATLAS and CMS
have already started to integrate Geant4 multithreading capability in their
simulation applications; in the case of CMS the first Full Simulation
production in multithreaded mode was delivered in the autumn of 2017. Specific
milestones are as follows:

  * •

2020: LHC, Neutrino, Dark Matter, and Muon experiments to demonstrate the
ability to run their detector simulation in multithreaded mode, using the
improved navigation and electromagnetic physics packages. This should bring
experiments more accurate physics and improved performance.

  * •

2020: Early integration of the beta release of the GeantV transport engine in
the experiments’ simulation, including the implementation of the new user
interfaces, which will allow the first performance measurements and physics
validation to be made.

  * •

2022: The availability of a production version of the new track-level
parallelisation and fully vectorised geometry, navigation, and physics
libraries will offer the experiments the option to finalise integration into
their frameworks; intensive work will be needed in physics validation and
computing performance tests. If successful, the new engine could be in
production on the timescale of the start of the HL-LHC run in 2026.

##### Pileup

Backgrounds to hard-scatter events have many components including in-time
pileup, out-of-time pileup, cavern background and beam-gas collisions. All of
these components can be simulated, but they present storage and I/O challenges
related to the handling of the large simulated minimum bias samples used to
model the extra interactions. An R&D programme is needed to study different
approaches to managing these backgrounds within the next 3 years:

  * •

Real zero-bias events can be collected, bypassing any zero suppression, and
overlaid on the fully simulated hard scatters. This approach faces challenges
related to the collection of non-zero-suppressed samples or the use of
suppressed events, non-linear effects when adding electronic signals from
different samples, and sub-detector misalignment consistency between the
simulation and the real experiment. Collecting calibration and alignment data
at the start of a new Run would necessarily incur delays such that this
approach is mainly of use in the final analyses. The experiments are expected
to invest in the development of the zero-bias overlay approach by 2020.

  * •

The baseline option is to “pre-mix” together the minimum bias collisions into
individual events that have the full background expected for a single
collision of interest. Experiments will invest effort on improving their pre-
mixing techniques, which allow the mixing to be performed at the digitisation
level, reducing the disk and network usage for a single event.

##### Fast Simulation

The work on Fast Simulation is also accelerating with the objective of
producing a flexible framework that permits Full and Fast simulation to be
combined for different particles in the same event. Various approaches to Fast
Simulation are being tried all with the same goal of saving computing time,
under the assumption that it is possible to improve time performance without
an unacceptable loss of physics accuracy. There has recently been a great deal
of interest in the use of Machine Learning in Fast Simulation, most of which
has focused on the use of multi-objective regression and generative
adversarial networks (GANs) NIPS2014_5423 (63). Since use of GANs allows for
non-parametric learning in cases such as calorimetric shower fluctuations, it
is a promising avenue for generating non-Gaussian and highly correlated
physical effects. This is an obvious area for future expansion and
development, as it is currently in its infancy.

  * •

2018: Assessment of the benefit of machine learning approach for Fast
Simulation.

  * •

2019: ML-based Fast Simulation for some physics observables.

  * •

2022: Demonstrate the potential of a common Fast Simulation infrastructure
applicable to the variety of detector configurations.

##### Digitisation

It is expected that, within the next 3 years, common digitisation efforts are
well-established among experiments, and advanced high-performance generic
digitisation examples, which experiments could use as a basis to develop their
own code, become available. For example, the development of next generation
silicon detectors requires realistic simulation of the charge collection and
digitisation processes. Owing to the large variety of technologies, common
software frameworks need to be flexible and modular to cater for the different
needs.

  * •

2020: Deliver advanced high-performance, SIMD-friendly generic digitisation
examples that experiments can use as a basis to develop their own code.

  * •

2022: Fully tested and validated optimised digitisation code that can be used
by the HL-LHC and DUNE experiments.

##### Pseudorandom Number Generation

The selection of pseudorandom number generators (PRNGs) presents challenges
when running on infrastructures with a large degree of parallelism, as
reproducibility is a key requirement. HEP will collaborate with researchers in
the development of PRNGs, seeking to obtain generators that address better our
challenging requirements. Specific milestones are:

  * •

2020: Develop a single library containing sequential and vectorised
implementations of the set of state-of-the-art PRNGs, to replace the existing
ROOT and CLHEP implementations. Potential use of C++11 PRNG interfaces and
implementations, and their extension for our further requirements (output of
multiple values, vectorisation) will be investigated.

  * •

2022: Promote a transition to the use of this library to replace existing
implementations in ROOT and Geant4.

###  3.3 Software Trigger and Event Reconstruction

#### Scope and Challenges

The reconstruction of raw detector data and simulated data, and its processing
in real time, represent a major component of today’s computing requirements in
HEP. Advances in the capabilities of facilities and future experiments bring
the potential for a dramatic increase in physics reach, at the price of
increased event complexities and rates. It is therefore essential that event
reconstruction algorithms and software triggers continue to evolve so that
they are able to efficiently exploit future computing architectures, and deal
with the increase in data rates without loss of physics. Projections into
future, e.g., at HL-LHC conditions, show that without significant changes in
approach or algorithms the increase in resources needed would be incompatible
with the expected budget.

At the HL-LHC, the central challenge for object reconstruction is to maintain
excellent efficiency and resolution in the face of high pileup values,
especially at low transverse momentum ($p_{T}$). Detector upgrades, such as
increases in channel density, high-precision timing, and improved detector
geometric layouts, are essential to overcome these problems. In many cases
these new technologies bring novel requirements to software trigger and/or
event reconstruction algorithms, or require new algorithms to be developed.
Ones of particular importance at the HL-LHC include high-granularity
calorimetry, precision timing detectors, and hardware triggers based on
tracking information, which may seed later software trigger and reconstruction
algorithms.

At the same time, trigger systems for next-generation experiments are evolving
to be more capable, both in their ability to select a wider range of events of
interest for the physics programme, and their ability to stream a larger rate
of events for further processing. ATLAS and CMS both target systems where the
output of the hardware trigger system is increased by an order of magnitude
over the current capability, up to 1 MHz CERN-LHCC-2015-020 (64, 65). In LHCb
CERN-LHCC-2014-016 (66) and ALICE Buncic:2011297 (67), the full collision rate
(between 30 to 40 MHz for typical LHC proton-proton operations) will be
streamed to real-time or near-real-time software trigger systems. The increase
in event complexity also brings a “problem” of an overabundance of signals to
the experiments, and specifically to the software trigger algorithms. The
evolution towards a genuine real-time analysis of data has been driven by the
need to analyse more signal than can be written out for traditional
processing, and technological developments that enable this without reducing
the analysis sensitivity or introducing biases.

Evolutions in computing technologies are an opportunity to move beyond
commodity x86_64 technologies, which HEP has used very effectively over the
past 20 years, but also represent a significant challenge if we are to derive
sufficient event processing throughput per cost to reasonably enable our
physics programmes Bird:1695401 (68). Among these challenges, important items
identified include the increase of SIMD capabilities, the evolution towards
multi- or many-core architectures, the slow increase in memory bandwidth
relative to CPU capabilities, the rise of heterogeneous hardware, and the
possible evolution in facilities available to HEP production systems.

The move towards open source software development and continuous integration
systems brings opportunities to assist developers of software trigger and
event reconstruction algorithms. Continuous integration systems based on
standard open-source tools have already allowed automated code quality and
performance checks, both for algorithm developers and code integration teams.
Scaling these up to allow for sufficiently high-statistics checks is still an
outstanding challenge. Also, code quality demands increase as traditional
offline analysis components migrate into trigger systems, where algorithms can
only be run once, and any problem means losing data permanently.

#### Current Practices

Substantial computing facilities are in use for both online and offline event
processing across all experiments surveyed. In most experiments, online
facilities are dedicated to the operation of the software trigger, but a
recent trend has been to use them opportunistically for offline processing
too, when the software trigger does not make them 100% busy. On the other
hand, offline facilities are shared with event reconstruction, simulation, and
analysis. CPU in use by experiments is typically measured at the scale of tens
or hundreds of thousands of x86_64 processing cores.

The CPU needed for event reconstruction tends to be dominated by charged
particle reconstruction (tracking), especially when the number of collisions
per bunch crossing is high and an efficient reconstruction low pT particles is
required. Calorimetric reconstruction, particle flow reconstruction, and
particle identification algorithms also make up significant parts of the CPU
budget in some experiments. Disk storage is typically 10s to 100s of PBs per
experiment. It is dominantly used to make the output of the event
reconstruction, both for real data and simulation, available for analysis.

Current experiments have moved towards smaller, but still flexible, tiered
data formats. These tiers are typically based on the ROOT file format and
constructed to facilitate both skimming of interesting events and the
selection of interesting pieces of events by individual analysis groups or
through centralised analysis processing systems. Initial implementations of
real-time analysis systems are in use within several experiments. These
approaches remove the detector data that typically makes up the raw data tier
kept for offline reconstruction, and keep only final analysis objects Aaij2016
(69, 70, 71).

Systems critical for reconstruction, calibration, and alignment generally
implement a high level of automation in all experiments. They are an integral
part of the data taking and data reconstruction processing chain, both in the
online systems as well as the offline processing setup.

#### Research and Development Programme

Seven key areas, itemised below, have been identified where research and
development is necessary to enable the community to exploit the full power of
the enormous datasets that we will be collecting. Three of these areas concern
the increasingly parallel and heterogeneous computing architectures that we
will have to write our code for. In addition to a general effort to vectorise
our codebases, we must understand what kinds of algorithms are best suited to
what kinds of hardware architectures. It is an area where collaboration with
the computer science community is required. We also need to develop benchmarks
that allow us to compare the physics-per-dollar-per-watt performance of
different algorithms across a range of potential architectures, and find ways
to optimally utilise heterogeneous processing centres. The consequent increase
in the complexity and diversity of our codebase will necessitate both a
determined push to educate physicists in modern algorithmic approaches and
coding practices, and a development of more sophisticated and automated
quality assurance and control. The increasing granularity of our detectors,
and the addition of timing information, which seems mandatory to cope with the
extreme pileup conditions at the HL-LHC, will require new kinds of
reconstruction algorithms that are sufficiently fast for use in real-time.
Finally, the increased signal rates will mandate a push towards real-time
analysis in many areas of HEP, in particular those with low-pT signatures.

  * •

HEP developed toolkits and algorithms typically make poor use of vector units
on commodity computing systems. Improving this will bring speedups to
applications running on both current computing systems and most future
architectures. The goal for work in this area is to evolve current toolkit and
algorithm implementations, and best programming techniques, to better use SIMD
capabilities of current and future CPU architectures.

  * •

Computing platforms are generally evolving towards having more cores in order
to increase processing capability. This evolution has resulted in
multithreaded frameworks in use, or in development, across HEP. Algorithm
developers can improve throughput by being thread-safe and enabling the use of
fine-grained parallelism. The goal is to evolve current event models, toolkits
and algorithm implementations, and best programming techniques, to improve the
throughput of multithreaded software trigger and event reconstruction
applications.

  * •

Computing architectures using technologies beyond CPUs offer an interesting
alternative for increasing throughput of the most time-consuming trigger or
reconstruction algorithms. Examples such as GPUs and FPGAs could be integrated
into dedicated trigger or specialised reconstruction processing facilities, in
particular online computing farms. The goal is to demonstrate how the
throughput of toolkits or algorithms can be improved in a production
environment and to understand how much these new architectures require
rethinking the algorithms used today. In addition, it is necessary to assess
and minimise possible additional costs coming from the maintenance of multiple
implementations of the same algorithm on different architectures.

  * •

HEP experiments have extensive continuous integration systems, including
varying code regression checks that have enhanced the quality assurance (QA)
and quality control (QC) procedures for software development in recent years.
These are typically maintained by individual experiments and have not yet
reached the point where statistical regression, technical, and physics
performance checks can be performed for each proposed software change. The
goal is to enable the development, automation, and deployment of extended QA
and QC tools and facilities for software trigger and event reconstruction
algorithms.

  * •

Real-time analysis techniques are being adopted to enable a wider range of
physics signals to be saved by the trigger for final analysis. As rates
increase, these techniques can become more important and widespread by
enabling only the parts of an event associated with the signal candidates to
be saved, reducing the disk space requirement. The goal is to evaluate and
demonstrate the tools needed to facilitate real-time analysis techniques.
Research topics include the study of compression and custom data formats,
toolkits for real-time detector calibration and validation that enable full
offline analysis chains to be ported into real-time, and frameworks that allow
non-expert offline analysts to design and deploy real-time analyses without
compromising data taking quality.

  * •

The central challenge for object reconstruction at the HL-LHC is to maintain
excellent efficiency and resolution in the face of high pileup, especially at
low object pT. Trigger systems and reconstruction software need to exploit new
techniques and higher granularity detectors to maintain or even improve
physics measurements in the future. It is also becoming increasingly clear
that reconstruction in very high pileup environments, such as the HL-LHC or
FCC-hh, will not be possible without adding some timing information to our
detectors, in order to exploit the finite time during which the beams cross
and the interactions are produced. The goal is to develop and demonstrate
efficient techniques for physics object reconstruction and identification in
complex environments.

  * •

Future experimental facilities will bring a large increase in event
complexity. The performance scaling of current-generation algorithms with this
complexity must be improved to avoid a large increase in resource needs. In
addition, it may become necessary to deploy new algorithms in order to solve
these problems, including advanced machine learning techniques. The goal is to
evolve or rewrite existing toolkits and algorithms focused on their physics
and technical performance at high event complexity, e.g., high pileup at HL-
LHC. Most important targets are those which limit expected throughput
performance at future facilities, most significantly charged-particle
tracking. A number of efforts in this area are already in progress recocwp
(72).

###  3.4 Data Analysis and Interpretation

#### Scope and Challenges

Scientific questions are answered by analysing the data obtained from suitably
designed experiments and comparing measurements with predictions from models
and theories. Such comparisons are typically performed long after data taking,
but can sometimes also be executed in near-real time on selected samples of
reduced size.

The final stages of analysis are undertaken by small groups or even individual
researchers. The baseline analysis model utilises successive stages of data
reduction, finally reaching a compact dataset for quick real-time iterations.
This approach aims at exploiting the maximum possible scientific potential of
the data, whilst minimising the “time to insight” for a large number of
different analyses performed in parallel. It is a complicated combination of
diverse criteria, ranging from the need to make efficient use of computing
resources to the management styles of the experiment collaborations. Any
analysis system has to be flexible enough to cope with deadlines imposed by
conference schedules. Future analysis models must adapt to the massive
increases in data taken by the experiments, while retaining this essential
“time to insight” optimisation.

Over the past 20 years the HEP community has developed and gravitated around a
single analysis ecosystem based on ROOT.

ROOT is a general-purpose object oriented framework that addresses the
selection, integration, development, and support of a number of foundation and
utility class libraries that can be used as a basis for developing HEP
application codes. The added value to the HEP community is that it provides an
integrated and validated toolkit, and its use encompasses the full event
processing chain; it has a major impact on the way HEP analysis is performed.
This lowers the hurdle to start an analysis, enabling the community to
communicate using a common analysis language, as well as making common
improvements as additions to the toolkit quickly become available. The ongoing
ROOT programme of work addresses important new requirements, in both
functionality and performance, and this is given a high priority by the HEP
community.

An important new development in the analysis domain has been the emergence of
new analysis tools coming from industry and open-source projects (e.g. Jupyter
notebooks Jupyter (73), the scikit-learn package
Pedregosa:2011:SML:1953048.2078195 (74)), and this presents new opportunities
for improving the HEP analysis software ecosystem. The HEP community is very
interested in using these software tools, together with established
components, in an interchangeable way. The main challenge will be to enable
new open-source tools to be plugged in dynamically to the existing ecosystem
and to provide mechanisms that allow the existing and new components to
interact and exchange data efficiently. To improve our ability to analyse much
larger datasets, R&D will be needed to investigate file formats, compression
algorithms, and new ways of storing and accessing data for analysis and to
adapt workflows to run on future computing infrastructures.

Reproducibility is the cornerstone of scientific results. It is currently
difficult to repeat most HEP analyses in exactly the manner they were
originally performed. This difficulty mainly arises due to the number of
scientists involved, the large number of steps in a typical HEP analysis
workflow, and the complexity of the analyses themselves. A challenge specific
to data analysis and interpretation is tracking the evolution of relationships
between all the different components of an analysis, i.e. the provenance of
each step.

Reproducibility of scientific results goes in hand with the need to preserve
both the data and the software. Section 3.12 develops this latter topic where
the FAIR principles of data management are embraced.

Robust methods for data reinterpretation are also critical. Collaborations
typically interpret results in the context of specific models for new physics
searches and sometimes reinterpret those same searches in the context of
alternative theories. However, understanding the full implications of these
searches requires the interpretation of the experimental results in the
context of many more theoretical models than are currently explored at the
time of publication. Analysis reproducibility and reinterpretation strategies
need to be considered in all new approaches under investigation, so that they
become a fundamental component of the system as a whole.

Adapting to the rapidly evolving landscape of software tools, as well as to
methodological approaches to data analysis, requires effort in continuous
training, both for novices as well as for experienced researchers, as detailed
in the Section 4. The maintenance and sustainability of the current analysis
ecosystem also present a major challenge, as currently this effort is provided
by just a few institutions. Legacy and less-used parts of the ecosystem need
to be managed appropriately. New policies are needed to retire little used or
obsolete components and free up effort for the development of new components.
These new tools should be made attractive and useful to a significant part of
the community to attract new contributors.

#### Current Practices

Methods for analysing HEP data have been developed over many years and
successfully applied to produce physics results, including more than 2000
publications, during LHC Runs 1 and 2. Analysis at the LHC experiments
typically starts with users running code over centrally managed data that is
of O(100kB/event) and contains all of the information required to perform a
typical analysis leading to publication. The most common approach is through a
campaign of _data reduction_ and _refinement_ , ultimately producing
simplified data structures of arrays of simple data types (“flat ntuples”) and
histograms used to make plots and tables, from which physics results can be
derived.

The current centrally-managed data typically used by a Run 2 data analysis at
the LHC (hundreds of TB) is far too large to be delivered locally to the user.
An often-stated requirement of the data reduction steps is to arrive at a
dataset that “can fit on a laptop”, in order to facilitate low-latency, high-
rate access to a manageable amount of data during the final stages of an
analysis. Creating and retaining intermediate datasets produced by data
reduction campaigns, bringing and keeping them “close” to the analysers, is
designed to minimise latency and the risks related to resource contention. At
the same time, disk space requirements are usually a key constraint of the
experiment computing models. The LHC experiments have made a continuous effort
to produce optimised analysis-oriented data formats with enough information to
avoid the need to use intermediate formats. Another effective strategy has
been to combine analyses from different users and execute them within the same
batch jobs (so-called “analysis trains”), thereby reducing the number of times
data must be read from the storage systems. This has improved performance and
usability, and simplified the task of the bookkeeping.

There has been a huge investment in using C++ for performance-critical code,
in particular in event reconstruction and simulation, and this will continue
in the future. However, for analysis applications, Python has emerged as the
language of choice in the data science community, and its use continues to
grow within HEP. Python is highly appreciated for its ability to support fast
development cycles, for its ease-of-use, and it offers an abundance of well-
maintained and advanced open source software packages. Experience shows that
the simpler interfaces and code constructs of Python could reduce the
complexity of analysis code, and therefore contribute to decreasing the “time
to insight” for HEP analyses, as well as increasing their sustainability.
Increased HEP investment is needed to allow Python to become a first class
supported language.

One new model of data analysis, developed outside of HEP, maintains the
concept of sequential reduction, but mixes interactivity with batch
processing. These exploit new cluster management systems, most notably Apache
Spark Spark (75, 53), which uses open-source tools contributed both by
industry and the data-science community. Other products implementing the same
analysis concepts and workflows are emerging, such as TensorFlow
tensorflow2015-whitepaper (76), Dask rocklin2015dask (77, 78), Pachyderm
thepachydermteam (79), Blaze wiebe2014blaze (80), Parsl
babuji_yadu_2017_853492 (81), and Thrill bingmann2016thrill (82). This
approach can complement the present and widely adopted Grid processing of
datasets. It may

potentially simplify the access to data and the expression of parallelism,
thereby improving the exploitation of cluster resources.

An alternative approach, which was pioneered in astronomy but has become more
widespread throughout the Big Data world, is to perform _fast querying_ of
centrally managed data and compute remotely on the queried data to produce the
analysis products of interest. The analysis workflow is accomplished without
focus on persistence of data traditionally associated with data reduction,
although transient data may be generated in order to efficiently accomplish
this workflow and optionally can be retained to facilitate an analysis
“checkpoint” for subsequent execution. In this approach, the focus is on
obtaining the analysis end-products in a way that does not necessitate a data
reduction campaign. It is of interest to understand the role that such an
approach could have in the global analysis infrastructure, and if it can bring
an optimisation of the global storage and computing resources required for the
processing of raw data to analysis.

Another active area regarding analysis in the world outside HEP is the switch
to a functional or declarative programming model, as for example provided by
Scala Scala (83) in the Spark environment. This allows scientists to express
the intended data transformation as a query on data. Instead of having to
define and control the “how”, the analyst declares the “what” of their
analysis, essentially removing the need to define the event loop in an
analysis, and leave it to underlying services and systems to optimally iterate
over events. It appears that these high-level approaches will allow
abstraction from the underlying implementations, allowing the computing
systems more freedom in optimising the utilisation of diverse forms of
computing resources. R&D is already under way, e.g., TDataFrame TDataFrame
(84) in ROOT, and this needs to be continued with the ultimate goal of
establishing a prototype functional or declarative programming paradigm.

#### Research and Development Programme

Towards HL-LHC, we envisage dedicated data analysis facilities for
experimenters, offering an extendable environment that can provide fully
functional analysis capabilities, integrating all these technologies relevant
for HEP. Initial prototypes of such analysis facilities are currently under
development. On the time scale of HL-LHC, such dedicated analysis facilities
would provide a complete system engineered for latency optimisation and
stability.

The following R&D programme lists the tasks that need to be accomplished. By
2020:

  * •

Enable new open-source software tools to be plugged in dynamically to the
existing ecosystem, and provide mechanisms to dynamically exchange parts of
the ecosystem with new components.

  * •

Prototype a comprehensive set of mechanisms for interacting and exchanging
data between new open-source tools and the existing analysis ecosystem.

  * •

Complete an advanced prototype of a low-latency response, high-capacity
analysis facility, incorporating fast caching technologies to explore a query-
based analysis approach and open-source cluster-management tools. It should in
particular include an evaluation of additional storage layers, such as SSD
storage and NVRAM-like storage, and cloud and Big Data orchestration systems.

  * •

Expand support of Python in our ecosystem with a strategy for ensuring long-
term maintenance and sustainability. In particular in ROOT, the current Python
bindings should evolve to reach the ease of use of native Python modules.

  * •

Develop a prototype based on a functional or declarative programming model for
data analysis.

  * •

Conceptualise and prototype an analysis “Interpretation Gateway”, including
data repositories, e.g., HEPData HEPData (85, 86), and analysis preservation
and reinterpretation tools.

By 2022:

  * •

Evaluate chosen architectures for analysis facilities, verify their design and
provide input for corrective actions to test them on a larger scale during Run
3.

  * •

Develop a blueprint for remaining analysis facility developments, system
design and support model.

###  3.5 Machine Learning

Machine Learning (ML) is a rapidly evolving approach to characterising and
describing data with the potential to radically change how data is reduced and
analysed. Some applications will qualitatively improve the physics reach of
datasets. Others will allow much more efficient use of processing and storage
resources, effectively extending the physics reach of experiments. Many of the
activities in this area will explicitly overlap with those in the other focus
areas, whereas others will be more generic. As a first approximation, the HEP
community will build domain-specific applications on top of existing toolkits
and ML algorithms developed by computer scientists, data scientists, and
scientific software developers from outside the HEP world. Work will also be
done to understand where problems do not map well onto existing paradigms and
how these problems can be recast into abstract formulations of more general
interest.

#### Scope and Challenges

The Machine Learning, Statistics, and Data Science communities have developed
a variety of powerful ML approaches for classification (using pre-defined
categories), clustering (where categories are discovered), regression (to
produce continuous outputs), density estimation, dimensionality reduction,
etc. Some of these have been used productively in HEP for more than 20 years,
others have been introduced relatively recently. The portfolio of ML
techniques and tools is in constant evolution, and a benefit is that many have
well-documented open source software implementations. ML has already become
ubiquitous in some HEP applications, most notably in classifiers used to
discriminate between signals and backgrounds in final offline analyses. It is
also increasingly used in both online and offline reconstruction and particle
identification algorithms, as well as the classification of reconstruction-
level objects, such as jets.

The abundance of, and advancements in, ML algorithms and implementations
present both opportunities and challenges for HEP. The community needs to
understand which are most appropriate for our use, tradeoffs for using one
tool compared to another, and the tradeoffs of using ML algorithms compared to
using more traditional software. These issues are not necessarily
“factorisable”, and a key goal will be to ensure that, as HEP research teams
investigate the numerous approaches at hand, the expertise acquired and
lessons learned, get adequately disseminated to the wider community. In
general, each _team_ , typically a small group of scientists from a
collaboration, will serve as a source of expertise, helping others develop and
deploy experiment-specific ML-based algorithms in their software stacks. It
should provide training to those developing new ML-based algorithms, as well
as those planning to use established ML tools.

With the advent of more powerful hardware, particularly GPUs and ML dedicated
processors, as well as more performant ML algorithms, the ML toolset will be
used to develop application software that could potentially, amongst other
things:

  * •

Replace the most computationally expensive parts of pattern recognition
algorithms and parameter extraction algorithms for characterising
reconstructed objects. For example, investigating how ML algorithms could
improve the physics performance or execution speed of charged track and vertex
reconstruction, one of the most CPU intensive elements of our current
software.

  * •

Extend the use of ML algorithms for real-time event classification and
analysis, as discussed in more detail in Section 3.3.

  * •

Extend the physics reach of experiments by extending the role of ML at the
analysis stage: handling data/MC or control/signal region differences,
interpolating between mass points, training in a systematics-aware way, etc.

  * •

Compress data significantly with negligible loss of fidelity in terms of
physics utility.

As already discussed, many particle physics detectors produce much more data
than can be moved to permanent storage. The process of reducing the size of
the datasets is managed by the trigger system. ML algorithms have already been
used very successfully for triggering, to rapidly characterise which events
should be selected for additional consideration and eventually saved to long-
term storage. In the era of the HL-LHC, the challenges will increase both
quantitatively and qualitatively as the number of proton-proton collisions per
bunch crossing increases. The scope of ML applications in the trigger will
need to expand in order to tackle the challenges to come.

#### Current Practices

The use of ML in HEP analyses has become commonplace over the past two
decades, and the most common use case has been in signal/background
classification. The vast majority of HEP analyses published in recent years
have used the HEP-specific software package TMVA TMVA (87) included in ROOT.
Recently, however, many HEP analysts have begun migrating to non-HEP ML
packages such as scikit-learn Pedregosa:2011:SML:1953048.2078195 (74) and
Keras Keras (88), although these efforts have yet to result in physics
publications from major collaborations. Data scientists at Yandex created a
Python package that provides a consistent API to most ML packages used in HEP
REP (89). Packages like Spearmint Spearmint (90) and scikit-optimize scikit-
optimize (91) perform Bayesian optimisation and can improve HEP Monte Carlo
work.

This shift in the set of ML techniques and packages utilised is especially
strong in the neutrino physics community, where new experiments such as DUNE
place ML at the very heart of their reconstruction algorithms and event
selection. The shift is also occurring among LHC collaborations, where ML is
becoming more and more commonplace in reconstruction and real-time
applications. Examples where ML has already been deployed in a limited way
include charged and neutral particle reconstruction and identification, jet
reconstruction and identification, and determining a particle’s production
properties (flavour tagging), based on information from the rest of the event.
In addition, ML algorithms have been developed that are insensitive to
changing detector performance, for use in real-time applications, and
algorithms that are minimally biased with respect to the physical observables
of interest.

At present, much of this development has happened in specific collaborations.
While each experiment has, or is likely to have, different specific use cases,
we expect that many of these will be sufficiently similar to each other that
R&D can be done in common. Even when this is not possible, experience with one
type of problem will provide insights into how to approach other types of
problem. This is why the Inter-experiment Machine Learning forum (IML IML
(92)) was created at CERN in 2016, as a compliment to experiment specific ML
R&D groups. It has already fostered closer collaboration between LHC and non-
LHC collaborations in the ML field.

#### Research and Development Roadmap and Goals

The R&D roadmap presented here is based on the preliminary work done in recent
years, coordinated by the IML, which will remain the main forum to coordinate
work in ML in HEP and ensure the proper links with the data science
communities. The following programme of work is foreseen.

By 2020:

  * •

_Particle identification and particle properties:_ in calorimeters or time
projection chambers (TPCs), where the data can be represented as a 2D or 3D
image (or even in 4D, including timing information), the problems can be cast
as a computer vision task. Deep Learning (DL), one class of ML algorithm, in
which neural networks are used to reconstruct images from pixel intensities,
is a good candidate to identify particles and extract many parameters.
Promising DL architectures for these tasks include convolutional, recurrent,
and adversarial neural networks. A particularly important application is to
Liquid Argon TPCs (LArTPCs), which is the chosen detection technology for
DUNE, the new flagship experiment in the neutrino programme. A proof of
concept and comparison of DL architectures should be finalised by 2020\.
Particle identification can also be explored to tag the flavour of jets in
collider experiments (e.g., so-called b-tagging). The investigation of these
concepts, which connect to Natural Language Processing
Collobert:2011:NLP:1953048.2078186 (93), has started at the LHC and is to be
pursued on the same timescale.

  * •

_ML middleware and data formats for offline usage:_ HEP relies on the ROOT
format for its data, wheras the ML community has developed several other
formats, often associated with specific ML tools. A desirable data format for
ML applications should have the following attributes: high read-write speed
for efficient training, sparse readability without loading the entire dataset
into RAM, compressibility, and widespread adoption by the ML community. The
thorough evaluation of the different data formats and their impact on ML
performance in the HEP context must be continued, and it is necessary to
define a strategy for bridging or migrating HEP formats to the chosen ML
format(s), or vice-versa.

  * •

_Computing resource optimisations:_ managing large volume data transfers is
one of the challenges facing current computing facilities. Networks play a
crucial role in data exchange and so a network-aware application layer may
significantly improve experiment operations. ML is a promising technology to
identify anomalies in network traffic, to predict and prevent network
congestion, to detect bugs via analysis of self-learning networks, and for WAN
path optimisation based on user access patterns.

  * •

_ML as a Service (MLaaS):_ current cloud providers rely on a MLaaS model
exploiting interactive machine learning tools in order to make efficient use
of resources, however, this is not yet widely used in HEP. HEP services for
interactive analysis, such as CERN’s Service for Web-based Analysis, SWAN SWAN
(94), may play an important role in adoption of machine learning tools in HEP
workflows. In order to use these tools more efficiently, sufficient and
appropriately tailored hardware and instances other than SWAN will be
identified.

By 2022:

  * •

_Detector anomaly detection:_ data taking is continuously monitored by
physicists taking shifts to monitor and assess the quality of the incoming
data, largely using reference histograms produced by experts. A whole class of
ML algorithms called anomaly detection can be useful for automating this
important task. Such unsupervised algorithms are able to learn from data and
produce an alert when deviations are observed. By monitoring many variables at
the same time, such algorithms are sensitive to subtle signs forewarning of
imminent failure, so that pre-emptive maintenance can be scheduled. These
techniques are already used in industry.

  * •

_Simulation:_ recent progress in high fidelity fast generative models, such as
Generative Adversarial Networks (GANs) NIPS2014_5423 (63) and Variational
Autoencoders (VAEs) 2013arXiv1312.6114K (95), which are able to sample high
dimensional feature distributions by learning from existing data samples,
offer a promising alternative for Fast Simulation. A simplified first attempt
at using such techniques in simulation saw orders of magnitude increase in
speed over existing Fast Simulation techniques, but has not yet reached the
required accuracy CaloGAN (96).

  * •

_Triggering and real-time analysis:_ one of the challenges is the trade-off in
algorithm complexity and performance under strict inference time constraints.
To deal with the increasing event complexity at HL-LHC, the use of
sophisticated ML algorithms will be explored at all trigger levels, building
on the pioneering work of the LHC collaborations. A critical part of this work
will be to understand which ML techniques allow us to maximally exploit future
computing architectures.

  * •

_Sustainable Matrix Element Method (MEM):_ MEM is a powerful technique that
can be utilised for making measurements of physical model parameters and
direct searches for new phenomena. As it is very computationally intensive its
use in HEP is limited. Although the use of neural networks for numerical
integration is not new, it is a technical challenge to design a network
sufficiently rich to encode the complexity of the ME calculation for a given
process over the phase space relevant to the signal process. Deep Neural
Networks (DNNs) are good candidates Bendavid2017 (97, 98).

  * •

_Tracking:_ pattern recognition is always a computationally challenging step.
It becomes a huge challenge in the HL-LHC environment. Adequate ML techniques
may provide a solution that scales linearly with LHC intensity. Several
efforts in the HEP community have started to investigate ML algorithms for
track pattern recognition on many-core processors.

###  3.6 Data Organisation, Management and Access

The scientific reach of data-intensive experiments is limited by how fast data
can be accessed and digested by computational resources. Changes in computing
technology and large increases in data volume require new computational models
Butler2013 (99), compatible with budget constraints. The integration of newly
emerging data analysis paradigms into our computational model has the
potential to enable new analysis methods and increase scientific output. The
field, as a whole, has a window in which to adapt our data access and data
management schemes to ones that are more suited and optimally matched to
advanced computing models and a wide range of analysis applications.

#### Scope and Challenges

The LHC experiments currently provision and manage about an exabyte of
storage, approximately half of which is archival, and half is traditional disk
storage. Other experiments that will soon start data taking have similar
needs, e.g., Belle II has the same data volumes as ATLAS. The HL-LHC storage
requirements per year are expected to jump by a factor close to 10, which is a
growth rate faster than can be accommodated by projected technology gains.
Storage will remain one of the major cost drivers for HEP computing, at a
level roughly equal to the cost of the computational resources. The
combination of storage and analysis computing costs may restrict scientific
output and the potential physics reach of the experiments, so new techniques
and algorithms are likely to be required.

In devising experiment computing models for this era many factors have to be
taken into account. In particular, the increasing availability of very high-
speed networks may reduce the need for CPU and data co-location. Such networks
may allow for more extensive use of data access over the wide-area network
(WAN), which may provide failover capabilities, global and federated data
namespaces, and will have an impact on data caching. Shifts in data
presentation and analysis models, such as the use of event-based data
streaming along with more traditional dataset-based or file-based data access,
will be particularly important for optimising the utilisation of opportunistic
computing cycles on HPC facilities, commercial cloud resources, and campus
clusters. This can potentially resolve currently limiting factors such as job
eviction.

The three main challenges for data management in the HL-LHC follow:

  * •

The experiments will significantly increase both the data rate and the data
volume. The computing systems will need to handle this with as small a cost
increase as possible and within evolving storage technology limitations.

  * •

The significantly increased computational requirements for the HL-LHC era will
also place new requirements on data access. Specifically, the use of new types
of computing resources (cloud, HPC) that have different dynamic availability
and characteristics will require more dynamic data management and access
systems.

  * •

Applications employing new techniques, such as training for machine learning
or high rate data query systems, will likely be employed to meet the
computational constraints and to extend physics reach. These new applications
will place new requirements on how and where data is accessed and produced.
Specific applications, such as training for machine learning, may require use
of specialised processor resources, such as GPUs, placing further requirements
on data.

The projected event complexity of data from future HL-LHC runs with high
pileup and from high resolution Liquid Argon detectors at DUNE will require
advanced reconstruction algorithms and analysis tools to interpret the data.
The precursors of these tools, in the form of new pattern recognition and
tracking algorithms, are already proving to be drivers for the compute needs
of the HEP community. The storage systems that are developed, and the data
management techniques that are employed, will need to be matched to these
changes in computational work, so as not to hamper potential improvements.

As with computing resources, the landscape of storage solutions is trending
towards heterogeneity. The ability to leverage new storage technologies as
they become available into existing data delivery models is a challenge that
we must be prepared for. This also implies the need to leverage “tactical
storage”, i.e., storage that becomes more cost-effective as it becomes
available (e.g., from a cloud provider), and have a data management and
provisioning system that can exploit such resources at short notice. Volatile
data sources would impact many aspects of the system: catalogues, job
brokering, monitoring and alerting, accounting, the applications themselves.

On the hardware side, R&D is needed in alternative approaches to data
archiving to determine the possible cost/performance tradeoffs. Currently,
tape is extensively used to hold data that cannot be economically made
available online. While the data is still accessible, it comes with a high
latency penalty, limiting effective data access. We suggest investigating
either separate direct access-based archives (e.g., disk or optical) or new
models that hierarchically overlay online direct access volumes with archive
space. This is especially relevant when access latency is proportional to
storage density. Either approach would need to also evaluate reliability risks
and the effort needed to provide data stability. For this work, we should
exchange experiences with communities that rely on large tape archives for
their primary storage.

Cost reductions in the maintenance and operation of storage infrastructure can
be realised through convergence of the major experiments and resource
providers on shared solutions. This does not necessarily mean promoting a
monoculture, as different solutions will be adapted to certain major classes
of use cases, type of site, or funding environment. There will always be a
judgement to make on the desirability of using a variety of specialised
systems, or of abstracting the commonalities through a more limited, but
common, interface. Reduced costs and improved sustainability will be further
promoted by extending these concepts of convergence beyond HEP and into the
other large-scale scientific endeavours that will share the infrastructure in
the coming decade (e.g., the SKA and CTA experiments). Efforts must be made as
early as possible, during the formative design phases of such projects, to
create the necessary links.

Finally, all changes undertaken must not make the ease of access to data any
worse than it is under current computing models. We must also be prepared to
accept the fact that the best possible solution may require significant
changes in the way data is handled and analysed. What is clear is that current
practices will not scale to the needs of HL-LHC and other major HEP
experiments of the coming era.

#### Current Practices

The original LHC computing models were based on simpler models used before
distributed computing was a central part of HEP computing. This allowed for a
reasonably clean separation between four different aspects of interacting with
data, namely data organisation, data management, data access, and data
granularity. The meaning of these terms may be summarised in what follows.

  * •

_Data organisation_ is essentially how data is structured as it is written.
Most data is written in files, in ROOT format, typically with a column-wise
organisation of the data. The records corresponding to these columns are
compressed. The internal details of this organisation are visible only to
individual software applications.

  * •

In the past, the key challenge for _data management_ was the transition to use
distributed computing in the form of the grid. The experiments developed
dedicated data transfer and placement systems, along with catalogues, to move
data between computing centres. Originally, computing models were rather
static: data was placed at sites, and the relevant compute jobs were sent to
the right locations. Since LHC startup, this model has been made more flexible
to limit non-optimal pre-placement and to take into account data popularity.
In addition, applications might interact with catalogues or, at times, the
workflow management system does this on behalf of the applications.

  * •

_Data access_ : historically, various protocols have been used for direct
reads (rfio, dcap, xrootd, etc.) where jobs are reading data explicitly
staged-in or cached by the compute resource used or the site it belongs to. A
recent move has been the convergence towards xrootd as the main protocol for
direct access. With direct access, applications may use alternative protocols
to those used by data transfers between sites. In addition, LHC experiments
have been increasingly using remote access to the data, without any stage-in
operations, using the possibilities offered by protocols such as xrootd or
http.

  * •

_Data granularity_ : the data is split into datasets, as defined by physics
selections and use cases, consisting of a set of individual files. While
individual files in datasets can be processed in parallel, the files
themselves are usually processed as a whole.

Before LHC turn-on, and in the first years of the LHC, these four areas were
to first order optimised independently. As LHC computing matured, interest has
turned to optimisations spanning multiple areas. For example, the recent use
of “Data Federations” mixes up Data Management and Access. As we will see
below, some of the foreseen opportunities towards HL-LHC may require global
optimisations.

Thus, in this section we take a broader view than traditional data management
and consider the combination of “Data Organisation, Management and Access”
(DOMA) together. We believe that this fuller picture will provide important
opportunities for improving efficiency and scaleability, as we enter the many-
exabyte era.

#### Research and Development Programme

In the following, we describe tasks that will need to be carried out in order
to demonstrate that the increased volume and complexity of data expected over
the coming decade can be stored, accessed, and analysed at an affordable cost.

  * •

Sub-file granularity, e.g., event-based, will be studied to see whether it can
be implemented efficiently, and in a scalable, cost-effective manner, for all
applications making use of event selection, to see whether it offers an
advantage over current file-based granularity. The following tasks should be
completed by 2020:

    * –

Quantify the impact on performance and resource utilisation of the storage and
network for the main access patterns, i.e., simulation, reconstruction,
analysis.

    * –

Assess the impact on catalogues and data distribution.

    * –

Assess whether event-granularity makes sense in object stores that tend to
require large chunks of data for efficiency.

    * –

Test for improvement in recoverability from preemption, in particular when
using cloud spot resources and/or dynamic HPC resources.

  * •

We will seek to derive benefits from data organisation and analysis
technologies adopted by other big data users. A proof-of-concept that involves
the following tasks needs to be established by 2020 to allow full
implementations to be made in the years that follow.

    * –

Study the impact of column-wise, versus row-wise, organisation of data on the
performance of each kind of access.

    * –

Investigate efficient data storage and access solutions that support the use
of map-reduce or Spark-like analysis services.

    * –

Evaluate just-in-time decompression schemes and mappings onto hardware
architectures considering the flow of data, from spinning disk to memory and
application.

  * •

Investigate the role data placement optimisations can play, such as caching,
in order to use computing resources effectively, and the technologies that can
be used for this. The following tasks should be completed by 2020:

    * –

Quantify the benefit of placement optimisation for reconstruction, analysis,
and simulation.

    * –

Assess the benefit of caching for Machine Learning-based applications, in
particular for the learning phase, and follow-up the evolution of technology
outside HEP.

In the longer term the benefits that can be derived from using different
approaches to the way HEP is currently managing its data delivery systems
should be studied. Two different content delivery methods will be looked at,
namely Content Delivery Networks (CDN) and Named Data Networking (NDN).

  * •

Study how to minimise HEP infrastructure costs by exploiting varied quality of
service from different storage technologies. In particular, study the role
that opportunistic/tactical storage can play, as well as different archival
storage solutions. A proof-of-concept should be made by 2020, with a full
implementation to follow in the following years.

  * •

Establish how to globally optimise data access latency, with respect to the
efficiency of using CPU, at a sustainable cost. This involves studying the
impact of concentrating data in fewer, larger locations (the “data-lake”
approach), and making increased use of opportunistic compute resources located
further from the data. Again, a proof-of-concept should be made by 2020, with
a full implementation in the following years, if successful. This R&D will be
done in common with the related actions planned as part of Facilities and
Distributed Computing.

###  3.7 Facilities and Distributed Computing

#### Scope and Challenges

As outlined in Section 2, huge resource requirements are anticipated for HL-
LHC running. These need to be deployed and managed across the WLCG
infrastructure, which has evolved from the original ideas on deployment before
LHC data-taking started Aderholz:510694 (100), to be a mature and effective
infrastructure that is now exploited by LHC experiments. Currently, hardware
costs are dominated by disk storage, closely followed by CPU, followed by tape
and networking. Naive estimates of scaling to meet HL-LHC needs indicate that
the current system would need almost an order of magnitude more resources than
will be available from technology evolution alone. In addition, other
initiatives such as Belle II and DUNE in particle physics, but also other
science projects such as SKA, will require a comparable amount of resources on
the same infrastructure. Even anticipating substantial software improvements,
the major challenge in this area is to find the best configuration for
facilities and computing sites that make HL-LHC computing feasible. This
challenge is further complicated by substantial regional differences in
funding models, meaning that any solution must be sensitive to these local
considerations to be effective.

There are a number of changes that can be anticipated on the timescale of the
next decade that must be taken into account. There is an increasing need to
use highly heterogeneous resources, including the use of HPC infrastructures
(which can often have very particular setups and policies that make their
exploitation challenging); volunteer computing (which is restricted in scope
and unreliable, but can be a significant resource); and cloud computing, both
commercial and research. All of these offer different resource provisioning
interfaces and can be significantly more dynamic than directly funded HEP
computing sites. In addition, diversity of computing architectures is expected
to become the norm, with different CPU architectures, as well as more
specialised GPUs and FPGAs.

This increasingly dynamic environment for resources, particularly CPU, must be
coupled with a highly reliable system for data storage and a suitable network
infrastructure for delivering this data to where it will be processed. While
CPU and disk capacity is expected to increase by respectively 15% and 25% per
year for the same cost CERN-COST-EST (101), the trends of research network
capacity increases show a much steeper growth, such as two orders of magnitude
from now to HL-LHC times Roberts:17 (102). Therefore, the evolution of the
computing models would need to be more network centric.

In the network domain, there are new technology developments, such as Software
Defined Networks (SDNs), which enable user-defined high capacity network paths
to be controlled via experiment software, and which could help manage these
data flows Blikra:2221659 (103). Some projects already started to explore the
potential of these technologies OSiRIS (104) but a considerable R&D is
required to prove their utility and practicality. In addition, the networks
used by HEP are likely to see large increases in traffic from other science
domains.

Underlying storage system technology will continue to evolve, for example
towards object stores, and, as proposed in Data Organisation, Management and
Access (Section 3.6), R&D is also necessary to understand their usability and
their role in the HEP infrastructures. There is also the continual challenge
of assembling inhomogeneous systems and sites into an effective widely
distributed worldwide data management infrastructure that is usable by
experiments. This is particularly compounded by the scale increases for HL-LHC
where multiple replicas of data (for redundancy and availability) will become
extremely expensive.

Evolutionary change towards HL-LHC is required, as the experiments will
continue to use the current system. Mapping out a path for migration then
requires a fuller understanding of the costs and benefits of the proposed
changes. A model is needed in which the benefits of such changes can be
evaluated, taking into account hardware and human costs, as well as the impact
on software and workload performance that in turn leads to physics impact.
Even if HL-LHC is the use case used to build this cost and performance model,
because the ten years of experience running large-scale experiments helped to
define the needs, it is believed that this work, and the resulting model, will
be valuable for other upcoming data intensive scientific initiatives. This
includes future HEP projects, such as Belle II, DUNE and possibly ILC
experiments, but also non-HEP projects, such as SKA.

#### Current Practices

While there are many particular exceptions, most resources incorporated into
the current WLCG are done so in independently managed sites, usually with some
regional organisation structure, and mostly offering both CPU and storage. The
sites are usually funded directly to provide computing to WLCG, and are in
some sense then “owned” by HEP, albeit often shared with others. Frequently
substantial cost contributions are made indirectly, for example through
funding of energy costs or additional staff effort, particularly at smaller
centres. Tape is found only at CERN and at large national facilities, such as
the WLCG Tier-1s Bird:1695401 (68).

Interfaces to these computing resources are defined by technical operations in
WLCG. Frequently there are choices that sites can make among some limited set
of approved options for interfaces. These can overlap in functionality. Some
are very HEP specific and recognised as over-complex: work is in progress to
get rid of them. The acceptable architectures and operating systems are also
defined at the WLCG level (currently x86_64, running Scientific Linux 6 and
compatible), and sites can deploy these either directly onto “bare metal” or
can use an abstraction layer, such as virtual machines or containers.

There are different logical networks being used to connect sites: LHCOPN
connects CERN with the Tier-1 centres and a mixture of LHCONE and generic
academic networks connect other sites.

Almost every experiment layers its own customised workload and data management
system on top of the base WLCG provision, with several concepts, and a few
lower-level components, in common. The pilot job model for workloads is
ubiquitous, where a real workload is dispatched only once a job slot is
secured. Data management layers aggregate files in the storage systems into
datasets and manage experiment-specific metadata. In contrast to the MONARC
model, sites are generally used more flexibly and homogeneously by
experiments, both in workloads and in data stored.

In total, WLCG currently provides experiments with resources distributed at
about 170 sites, in 42 countries, which pledge every year the amount of CPU
and disk resources they are committed to delivering. The pledge process is
overseen by the Computing Resource Scrutiny Group (CRSG), mandated by the
funding agencies to validate the experiment requests, and to identify
mismatches with site pledges. These sites are connected by 10-100 Gb links,
and deliver approximately 500k CPU cores and 1 EB of storage, of which 400 PB
is disk. More than 200M jobs are executed each day Bird2017 (105).

#### Research and Development programme

The following areas of study are ongoing, and will involve technology
evaluations, prototyping, and scale tests. Several of the items below require
some coordination with other topical areas discussed in this document, and
some work is still needed to finalise the detailed action plan. These actions
will need to be structured to meet the common milestones of informing the HL-
LHC Computing Technical Design Reports (TDRs), and deploying advanced
prototypes during LHC Run 3.

  * •

Understand better the relationship between the performance and costs of the
WLCG system, and how it delivers the necessary functionality to support LHC
physics. This will be an ongoing process, started by the recently formed
System Performance and Cost Modeling Working Group SPCMWG (106), and aims to
provide a quantitative assessment for any proposed changes.

  * •

Define the functionality needed to implement a federated data centre concept
(“data lake”) that aims to reduce the operational cost of storage for HL-LHC,
and at the same time better manage network capacity, whilst maintaining the
overall CPU efficiency. This would include the necessary qualities of service,
and options for regionally distributed implementations, including the ability
to flexibly respond to model changes in the balance between disk and tape.
This work should be done in conjunction with the existing Data Organisation,
Management and Access Working Group WLCG-DOMA (107) to evaluate the impact of
the different access patterns and data organisations envisaged.

  * •

Building upon the experience of projects currently exploring SDN potential,
define the role for this technology in managing data transfers and access and
the integration strategy into experiment frameworks.

  * •

Establish an agreement on the common data management functionality that is
required by experiments, targeting a consolidation and a lower maintenance
burden. The intimate relationship between the management of elements in
storage systems and metadata must be recognised. This work requires
coordination with the Data Processing Frameworks Working Group. It needs to
address at least the following use cases:

    * –

processing sites that may have some small disk cache, but do not manage
primary data;

    * –

fine grained processing strategies that may enable processing of small chunks
of data, with appropriate bookkeeping support;

    * –

integration of heterogeneous processing resources, such as HPCs and clouds.

  * •

Explore scalable and uniform means of workload scheduling, which incorporate
dynamic heterogenous resources, and the capabilities of finer grained
processing that increases overall efficiency. The optimal scheduling of
special workloads that require particular resources is clearly required.

  * •

Contribute to the prototyping and evaluation of a quasi-interactive analysis
facility that would offer a different model for physics analysis, but would
also need to be integrated into the data and workload management of the
experiments. This is work to be done in collaboration with groups working on
new data analysis models.

###  3.8 Data-Flow Processing Framework

#### Scope and Challenges

Frameworks in HEP are used for the collaboration-wide data processing tasks of
triggering, reconstruction, and simulation, as well as other tasks that
subgroups of the collaboration are responsible for, such as detector alignment
and calibration. Providing framework services and libraries that will satisfy
the computing and data needs for future HEP experiments in the next decade,
while maintaining our efficient exploitation of increasingly heterogeneous
resources, is a huge challenge.

To fully exploit the potential of modern processors, HEP data processing
frameworks need to allow for the parallel execution of reconstruction or
simulation algorithms on multiple events simultaneously. Frameworks face the
challenge of handling the massive parallelism and heterogeneity that will be
present in future computing facilities, including multi-core and many-core
systems, GPUs, Tensor Processing Units (TPUs), and tiered memory systems, each
integrated with storage and high-speed network interconnections. Efficient
running on heterogeneous resources will require a tighter integration with the
computing models’ higher-level systems of workflow and data management.
Experiment frameworks must also successfully integrate and marshall other HEP
software that may have its own parallelisation model, such as physics
generators and detector simulation.

Common developments across experiments are desirable in this area, but are
hampered by many decades of legacy work. Evolving our frameworks also has to
be done recognising the needs of the different stakeholders in the system.
This includes physicists, who are writing processing algorithms for
triggering, reconstruction or analysis; production managers, who need to
define processing workflows over massive datasets; and facility managers, who
require their infrastructures to be used effectively. These frameworks are
also constrained by security requirements, mandated by the groups and agencies
in charge of it.

#### Current Practices

Although most frameworks used in HEP share common concepts, there are, for
mainly historical reasons, a number of different implementations; some of
these are shared between experiments. The Gaudi framework Barrand:2001ny (108)
was originally developed by LHCb, but is also used by ATLAS and various non-
LHC experiments. CMS uses its own CMSSW framework Bayatian:2006nff (109),
which was forked to provide the art framework for the Fermilab Intensity
Frontier experiments Green:2012gv (110). Belle II uses basf2
1742-6596-331-3-032024 (111). The linear collider community developed and uses
Marlin Gaede:2006pj (112). The FAIR experiments use FairROOT, closely related
to ALICE’s AliROOT. The FAIR experiments and ALICE are now developing a new
framework, which is called O2 O2 (113). At the time of writing, most major
frameworks support basic parallelisation, both within and across events, based
on a task-based model Jones:2015soc (114)Clemencic:2015paa (115).

Each framework has a processing model, which provides the means to execute and
apportion work. Mechanisms for this are threads, tasks, processes, and inter-
process communication. The different strategies used reflect different trade-
offs between constraints in the programming model, efficiency of execution,
and ease of adapting to inhomogeneous resources. These concerns also reflect
two different behaviours: firstly, maximising throughput, where it is most
important to maximise the number of events that are processed by a given
resource; secondly, minimising latency, where the primary constraint is on how
long it takes to calculate an answer for a particular datum.

Current practice for throughput maximising system architectures have
constrained the scope of framework designs. Framework applications have
largely been viewed by the system as a batch job with complex configuration,
consuming resources according to rules dictated by the computing model: one
process using one core on one node, operating independently with a fixed size
memory space on a fixed set of files (streamed or read directly). Only
recently has CMS broken this tradition starting at the beginning of Run 2, by
utilising all available cores in one process space using threading. ATLAS is
currently using a multi-process fork-and-copy-on-write solution to remove the
constraint of one core/process. Both experiments were driven to solve this
problem by the ever-growing need for more memory per process brought on by the
increasing complexity of LHC events. Current practice manages systemwide (or
facility-wide) scaling by dividing up datasets, generating a framework
application configuration, and scheduling jobs on nodes/cores to consume all
available resources. Given anticipated changes in hardware (heterogeneity,
connectivity, memory, storage) available at computing facilities, the
interplay between workflow and workload management systems and framework
applications need to be carefully examined. It may be advantageous to permit
framework applications (or systems) to span multi-node resources, allowing
them to be first-class participants in the business of scaling within a
facility. In our community some aspects of this approach, which maps features
with microservices or function as a service, is being pioneered by the O2
framework.

#### Research and Development programme

By the end of 2018: review the existing technologies that are the important
building blocks for data processing frameworks and reach agreement on the main
architectural concepts for the next generation of frameworks. Community
meetings and workshops, along the lines of the original Concurrency Forum, are
envisaged in order to foster collaboration in this work ConcurrencyForum
(116). This includes the following:

  * •

Libraries used for concurrency, their likely evolution and the issues in
integrating the models used by detector simulation and physics generators into
the frameworks.

  * •

Functional programming, as well as domain specific languages, as a way to
describe the physics data processing that has to be undertaken rather than how
it has to be implemented. This approach is based on the same concepts as the
idea for functional approaches for (statistical) analysis as described in
Section 3.4.

  * •

Analysis of the functional differences between the existing frameworks and the
different experiment use cases.

By 2020: prototype and demonstrator projects for the agreed architectural
concepts and baseline to inform the HL-LHC Computing TDRs and to demonstrate
advances over what is currently deployed. The following specific items will
have to be taken into account:

  * •

These prototypes should be as common as possible between existing frameworks,
or at least several of them, as a proof-of-concept of effort and component
sharing between frameworks for their future evolution. Possible migration
paths to more common implementations will be part of this activity.

  * •

In addition to covering the items mentioned for the review phase, they should
particularly demonstrate possible approaches for scheduling the work across
heterogeneous resources and using them efficiently, with a particular focus on
the efficient use of co-processors, such as GPUs.

  * •