cases, tests are run on a bare-metal machine with two AMD
$\mathrm{EPYC}^{\mathrm{TM}}$ 7302 16-core processors configured to be in the
_Performance_ mode with Simultaneous Multi-Threading (SMT) disabled. To
disentangle its effects, clock frequency boosting is disabled and the clock
frequency is capped at the base frequency of 3.0 GHz. The machine has 252 GB
of available memory. In all results, memory is measured by the proportional
set size, and time by the wall clock processing time unless stated otherwise.

Insofar as it is possible, the job configurations mirror what is used in the
production system by ATLAS. The number of events to be processed concurrently
is set to the number of available threads. The computing performance results
for each step of the MC simulation processing workflow are provided in their
respective subsections below. The MC simulation workloads use
$t\bar{t}$-production events. In all cases, a constant number of events per
thread or worker is processed (i.e. these are tests of weak scaling).

####  5.3.2 Monte Carlo processing chain

##### MC simulation

As described in Section 4.2, during LS2 the Geant4 Optimisation Task Force was
launched with the aim of reducing the Geant4 simulation CPU time in Run 3 by
more than 30% without making any compromise on the physics accuracy. In this
section, the simulation is benchmarked for the three latest ATLAS MC
simulation campaigns: MC20 (the last ATLAS MC campaign for Run 2), MC21 (the
first ATLAS MC Campaign for Run 3) and MC23 (the latest ATLAS MC Campaign for
Run 3).

Figure 20(a) shows the memory usage, which scales well with the number of
threads and similarly across the three campaigns. In these jobs, Athena
release 22.0.92 is used, with 100 events per thread. The increase in the
memory footprint is expected. It is explained partly by the different centre-
of-mass energy of the input events in Run 2 (13 TeV) and Run 3 (13.6 TeV),
some changes in the geometry and beam conditions and, most significantly, by
the introduction of a technical optimisation that concerns the way Athena code
is linked with Geant4 [153] (see Section 4.2.1, this optimisation reduces CPU
in turn).

Figure 20(b) shows the event throughput scaling of these simulation
configurations in the same benchmark jobs. The scaling is almost ideal; an
extrapolation of the single-thread performance is provided for reference. In
fact, ideal performance is not expected to be linear on modern processors
because, for example, the core frequency is automatically changed to optimise
instruction throughput against power (see, for example, Ref. [43]). The CPU
hyper-threading is also expected to result in somewhat lower throughput.
Overall, the three workflows scale in a similar way. The throughput increased
from MC20 to MC21 by 27%–32%, and from MC21 to MC23 by 45%–47%. These
improvements are the results from the CPU optimisations in MC21 and MC23 with
the Geant4 optimisations that are detailed in Section 4.2; a CPU speed-up of
33%–38% is observed in MC21 relative to the MC20 setup and the speed-up
increases to 50% in MC23.

((a))

((b))

Figure 20: (a) Memory usage as a function of number of threads and (b) event
throughput as a function of number of threads for three recent MC simulation
campaigns. In the memory figure linear fits are superimposed, and in the
throughput figure extrapolations of the single-thread results are shown to
guide the eye.

##### Overlay, trigger simulation and reconstruction for MC

Figure 21(a) shows the memory usage of several workloads normally run together
in MC simulation production. The MC23 production configuration is used with
Athena release 23.0.53. The specific MC campaign of this performance study is
MC23c, which corresponds to an average pile-up of 56 interactions. The first
step is MC Overlay, where the effects of pile-up are simulated by overlaying
hard-scattering events with pre-mixed RDOs to model the desired
$\langle\mu\rangle$ profile and other effects (see Section 4.3.2). The result
of this step is fed into the second step, offline trigger simulation, labelled
RDOtoRDOTrigger. The resulting file is then passed through the reconstruction,
labelled RAWtoALL, where physics objects are created (see Section 4.4) and
stored in an AOD. The memory usage increases by 200–400 MB per additional
thread, and each 8-thread configuration stays well below the standard 2 GB per
core available on the Grid.

Figure 21(b) shows the event throughput scaling of these MC simulation
production workloads in the same benchmark job. Standard production
configurations use eight threads, and thus optimisation efforts were focused
on this regime. Indeed, up to eight threads the scaling is relatively linear
in all of the three steps. At this point, MC Overlay reaches its Amdahl’s law
[242] plateau and the improvement due to additional threads is significantly
reduced. This is a subject for future improvement. The same happens for
offline trigger simulation above 24 threads, whereas reconstruction shows
almost ideal improvement throughout (within the caveats discussed above).
Nevertheless, the production setup is well-optimised for the most-used
hardware configurations on standard Grid production queues that offer eight
cores per job.

((a))

((b))

Figure 21: (a) Memory usage as a function of number of threads and (b) event
throughput as a function of number of threads for several MC simulation
workloads in MC23. In the memory figure linear fits are superimposed, and in
the throughput figure extrapolations of the single-thread results are shown to
guide the eye.

####  5.3.3 Reconstruction of collision data

Figure 22(a) shows the memory usage as a function of the number of threads,
using input proton–proton collision data collected in 2023. Athena release
23.0.53 is used for these tests, with 150 events per thread. The corresponding
average numbers of interactions per bunch crossing, $\langle\mu\rangle=65$. As
expected, the memory usage grows linearly with the number of worker threads.
However, it is always well below the standard 2 GB per core available on the
Grid for 8-core configurations, meaning that data reconstruction jobs can
successfully be executed on any Grid node. The memory is significantly larger
than that for the reconstruction of MC (see Figure 21(a)) because data
reconstruction includes additional data quality monitoring tools that involves
the creation of many histograms.

Figure 22(b) shows the event throughput as a function of the number of threads
in the same jobs. For the production configuration with eight threads, the
data throughput is measured to be 0.5 events per second. The scaling is close
to ideal, with small deviations in the region with many threads, but the
throughput continuously improves until all available cores on the node are
utilised. Although there is room for improvement, the reconstruction job is no
longer memory-bound in production configurations, a notorious problem in
previous versions of the Athena software.

((a))

((b))

Figure 22: (a) Memory usage as a function of number of threads and (b) event
throughput as a function of number of threads for 2023 detector data
reconstruction. The average pile-up $\langle\mu\rangle$ was 65. In the memory
figure a linear fit is superimposed.

####  5.3.4 Derivation production

Figure 23(a) shows the memory usage as a function of the number of worker
processes in three different multi-process derivation production
configurations. These performance tests are all done with Athena release
24.0.12, mirroring production configurations as closely as possible. The data
run was mostly in the range of $62<\langle\mu\rangle<67$. Derivation
production based on simulated $t\bar{t}$-production events is shown with
serial compression (see Section 4.5.2) and the Run 3-standard parallel
compression. As expected from the additional compression buffers, the memory
requirement for parallel compression is higher. The memory usage for
derivation production using 2023 detector data as input is also shown. The
memory usage is slightly lower for low numbers of workers because fewer
algorithms run on data (e.g. those operating on generator records run only on
MC simulation); the scaling with the number of workers is slightly worse owing
to subtle differences in the configuration of the forking (i.e. which memory
pages are fully shared between processes).

Figure 23(b) shows the event throughput as a function of the number of worker
processes in the same jobs. 1000 events per worker are included for MC
simulation, and 700 events per worker are included for detector data. Here the
advantage of parallel compression is clearly visible: there is dramatically
better scaling for high numbers of workers when parallel compression is
enabled. The derivation production for detector data runs slightly faster than
that of MC simulation owing again to the lack of algorithms and tools running
on generator records. With parallel compression in MC simulation, the scaling
is very close to ideal beyond 16 workers.

((a))

((b))

Figure 23: (a) Memory usage as a function of number of workers and (b) event
throughput as a function of number of workers for derivation production using
2023 data and MC simulation. The MC simulation workloads use
$t\bar{t}$-production events. Also shown is derivation production for MC
simulation with serial compression (‘Ser. Comp.’). In the memory figure linear
fits are superimposed, and in the throughput figure extrapolations of the
single-worker results are shown to guide the eye.

####  5.3.5 Disk sizes of formats and containers

The average disk sizes of the standard formats for MC simulation datasets
under several representative sets of pile-up conditions are listed in Table 5.
From MC20 to MC21, the ‘standard’ $t\bar{t}$-production sample was changed;
however, this has only a minor impact on the file size (at the 5%-level). The
dominant part of the change in HIT file size between MC20 and MC21 is due to
the full simulation optimisations described in Section 4.2. Changes like the
range cut application to all EM processes and the neutron and photon Russian
Roulette reduce the number of steps taken in the simulation, and particularly
the number of steps taken at large times. This has a significant impact on the
calorimeter output, reducing it by almost a factor of two. From MC21 to MC23
the compression algorithm was changed from ZLIB to LZMA, resulting in a
10%–15% reduction in file size after compression. Both these changes affected
both the Full and Fast Simulation outputs. The RDO and AOD sizes increased
over time due to the larger amount of pile-up. The composition of a background
RDO (which stores pile-up information only, to be later overlayed on hard-
scatter events) is displayed in Figure 24, and the compositions of a
$t\bar{t}$-production FullSim HITS file and AOD file are shown in Figure 25.
The RDO file is dominated by calorimeter and inner detector information. The
largest fraction in the HIT file is that of Silicon HITs (recorded in the
pixel and SCT detectors). The AOD is later reduced to a DAOD (derived AOD, see
Section 4.5), which is the format used for most physics analyses. The AOD does
not contain any reconstructed jet collections, because jets are built only at
the derivation step using the low-level inputs stored in the AOD. This
approach helps to reduce the file size of the AOD.

Table 5: Average size of an MC simulation $t\bar{t}$ event, for various MC production campaigns. The data taking year represented by each campaign and the average number of proton–proton interactions, $\langle\mu\rangle$, are reported. All the MC20 sub-campaigns use the same simulation configuration at $\sqrt{s}=13$ TeV; the MC23 sub-campaign simulation configurations differ only by the beamspot, which has a negligible effect on the output file size. Both MC21 and MC23a represent the 2022 data conditions at $\sqrt{s}=13.6$ TeV, but the former refers to production performed before and during data taking, while the latter represents production done in 2023 with an updated software release and conditions matching those of the data taking (rather than a prediction). The HITS files are the output of the Geant4-based or AtlFast3 simulation of the hard-scatter process; the RDO files are the output of digitisation and MC Overlay, and store the contribution of pile-up; and the AOD files are the output of reconstruction of objects needed for physics analysis. ‘N.A.’ indicates combinations that were not validated for physics analysis use.  Campaign | MC20a | MC20d | MC20e | MC21 | MC23a | MC23c
---|---|---|---|---|---|---
Representing data taken in | 2015–16 | 2017 | 2018 | 2022 | 2022 | 2023
$\langle\mu\rangle$ | 23.9 | 37.8 | 36.1 | 44.3 | 42.8 | 55.6
Full Simulation HITS (kB/event) | 906 | 711 | 620
Fast Simulation HITS (kB/event) | 801 | N.A. | 591
RDO (kB/event) | 1648 | 1897 | 1878 | 2073 | 2052 | 2240
AOD (kB/event) | 271 | 364 | 354 | 375 | 348 | 430
Figure 24: The composition of a simulated Run 3 background RDO file that
stores pile-up information, which is later overlayed on hard-scatter events.
The pile-up corresponds to $\langle\mu\rangle=43$, which approximates the run
conditions of data taking in 2022. Here ‘ID Si’ includes both the silicon
pixels and strips.

((a))  ((b))

Figure 25: The composition of a Run 3 (a) HITS file and (b) AOD file. The
process simulated in both cases is $t\bar{t}$ with a single lepton from a
$W$-boson decay. The AOD includes hard-scatter information and also simulated
pile-up. The pile-up corresponds to $\langle\mu\rangle=43$, which approximates
the run conditions of data taking in 2022.

###  5.4 Preparation for new campaigns

Major campaigns for improvements to reconstruction (of either data or MC
simulation) or detector simulation require significant resources and are only
undertaken after careful preparation and validation. The data and MC
simulation must be kept consistent to minimise analysis corrections, and
therefore often the reprocessing of one mandates the reprocessing of the
other. Typically, two major data (re)processing and MC simulation campaigns
accompany each LHC Run: one that is undertaken continuously as the data taking
proceeds, and one at the end of the Run, gathering the best-known geometry and
conditions and final optimisation of the reconstruction. The time required for
these campaigns is dominated by preparation time; the time for the processing
itself is typically dominated by fixing rare crashes and similar issues that
affect a few events, to ensure that every data event is reprocessed and
provided to analysis.

####  5.4.1 Monte Carlo simulation campaign preparations

The preparation for a new MC simulation campaign involves many inter-dependent
steps that must be performed and checked before large-scale production is
undertaken. The full sequency of changes described here is required for
updates of the detector simulation and its reconstruction; some new MC
simulation campaigns involve only changes to the reconstruction of the
simulated events, and the first of these steps can be skipped.

First among these is the preparation of the simulation software, including the
choice of Geant4 version. Because changes to the Geant4 version often result
in changes to physics observables (even with the same configuration), the
version must be fixed for the duration of an MC simulation campaign. Normally,
some large (e.g. 10 million event) sample of $t\bar{t}$ events is simulated to
detect any remaining, rare potential bugs, and several other samples are run
with modest numbers of events to check for computing performance issues
(speed, memory consumption, and so on). Small patches are normally allowed
after this stage, built as part of an ATLAS-specific Geant4 release, but the
official Geant4 patch version is fixed.

In parallel with the fixing of the Geant4 version, the initial geometry and
detector conditions must be prepared. Only coarse (large-scale) detector
alignments, the beam spot position and distribution, and the gas mixtures used
in detectors (e.g. in the TRT the gas mixture may change during the run and is
therefore recorded as a part of the conditions rather than as a part of the
geometry) enter the simulation; other conditions like detailed maps of
disabled channels only enter in the reconstruction and therefore can be
finalized later on. Small improvements to the geometry might arrive close to
the time of production, but most physics validation (see Section 5.2) is done
with a close-to-final geometry.

With the geometry and Geant4 versions in place, the calorimeter sampling
fractions can be checked, which calibrate the electromagnetic-scale response
of the calorimeters. These are normally calculated using bespoke single-
particle samples, where the particles are simulated starting from within the
uniform sampling region of the calorimeter. The sampling fractions are
dependent on the Geant4 version, owing primarily to the details of the
treatment of range cuts in various processes (see Section 4.2). Although these
are in principle purely geometric quantities, they have also been observed to
have dependence on particle incident angle in some detector geometries.

The next step towards production is the preparation of OFCs for the
calorimeter (see Section 4.3). The calorimeter OFCs are based on noise
pedestals that are pile-up dependent, and therefore a $\langle\mu\rangle$
value must be selected for their calculation. This $\langle\mu\rangle$ value
need not match the anticipated pile-up conditions as discrepancies can be re-
calibrated and high values result in the suppression of some physics signal,
but generally a close, round value is used (e.g. $\langle\mu\rangle=40$). The
OFCs (and sampling fractions) are calculated assuming perfect calorimeter
detector conditions (e.g. without disabled channels and with nominal high-
voltage conditions).

With these steps complete, frozen showers (see Section 4.2.1) can be prepared
for the MC simulation campaign. Calibrations also begin at this stage with the
preparation of some TopoCluster calibrations, calculated based on single
particle simulation. These each depend only weakly on final detector
conditions and non-calorimeter geometry.

At this stage, the geometry and those detector conditions important to the
simulation can be finalized, and physics validation of the simulation itself
can begin. Often the first validations are done using the reconstruction from
the previous MC simulation campaign, since it is well-understood. This
procedure allows the isolation of simulation changes, although changes that
require new calibrations (e.g. changes in the hadronic energy scale in the
calorimeter) must be inspected with care by experts.

With the simulation validated, the digitisation is the next workload to
finalize. This includes updates to digitisation algorithms (e.g. for improved
noise models or response treatments) and conditions that must be applied
during digitisation (e.g. abnormal high-voltage in the calorimeter). These
changes are normally validated with and without pile-up, and both the direct
digitisation of hard-scatter and pile-up as well as the MC Overlay workflow
must be validated. Normally these validations proceed with a preliminary
version of the reconstruction to be used in the MC simulation campaign,
because often in production MC Overlay and reconstruction are run in a single
job on the Grid.

Finally, the geometry, conditions and configuration to be used for
reconstruction must be finalised and validated. As a part of the Sample-A
production run during physics validation (see Section 5.2), computing
performance issues can also be checked on the Grid. The final validation and
production normally begins once the last updates to the detector and
accelerator conditions are included; this can include, for example, a final
update to the $\langle\mu\rangle$ distribution used in the sample, which then
requires an updated RDO production for MC Overlay before the reconstruction
configuration is finalised. Often, only once the full chain is validated and
the sample production begins can samples be produced for the training of the
various types of fast simulation, which then requires its own update and
validation process.

The production itself often begins from samples required for the derivation of
calibrations and systematic uncertainties. These must be available before data
analysis can begin. Sometimes, _transfer_ uncertainties are used, wherein the
uncertainty prescription from a previous MC simulation campaign is applied,
along with extra systematic uncertainties related to the known or expected
changes between the two campaigns. These temporary uncertainties allow faster
data analysis uptake in new software releases.

Depending on the number and scale of changes expected, this entire procedure
from start to end may take more than a year. This was the case in the lead-up
to Run 3, where the thread-safety of the software was being regularly checked
on increasingly large-scale samples, new detector elements from the Phase-I
upgrades had to be introduced and the corresponding reconstruction software
developed and commissioned, and significant uncertainty around the accelerator
conditions remained until significant data were recorded. For this reason,
often the initial MC simulation campaign in a Run is used as a prototype for
preparatory work, validation, early data tests, and very early analysis. A
subsequent campaign includes a more realistic picture of the data taking and
is used for most data analyses. This was the case, for example, for MC15 and
MC16 (the latter has run for seven years), and more recently for MC21 and
MC23. For the same reason, a small sub-campaign is often undertaken based on
estimates of the coming year’s data-taking conditions, and a longer campaign
follows the data-taking with conditions that are known to be representative of
the year. This was the case, for example, for MC16c (before 2017 data-taking
and meant to be representative of the 2017 data) and MC16d (after 2017).

####  5.4.2 Data reprocessing preparations

To begin a campaign to reprocess data, it is necessary first to identify what
improvements will be made and then to validate each of the improvements via
the process discussed in Sections 5.1 and 5.2. The improvements broadly fall
into three categories: updates to the detector conditions data, updates to the
detector geometry, and updates to reconstruction algorithms.

Conditions data updates [243] incorporate improvements in the knowledge of the
time-dependent status of the detector during periods of operation. Typical
examples include improved knowledge of detector alignment, detector noise and
instantaneous luminosity delivered by the LHC. This information is stored in
the ATLAS conditions database [60], an Oracle® database hosting a COOL
technology schema [50] (see Section 6.2.2). Each detector system may provide
updated conditions in advance of a reprocessing. Only coarse (large-scale)
alignment is applied in the detector simulation; small alignment corrections,
common in the inner detector and muon spectrometer, can be updated without
modifying the simulated geometry. Similarly, updates to disabled or noisy
channel maps do not require an update of the MC simulation.

Detector geometry updates incorporate improvements in knowledge of the
detector geometry over a long period of stability. Typical examples are
updates to maps of _dead material_ , the inactive parts of the detector that
are not read-out, such as supporting material of subdetectors or the cryostat
walls. The updates may also include corrections to the positioning or internal
geometry of sub-detectors assumed in event reconstruction. Significant updates
to the geometry may require a new simulation campaign to maintain good
agreement with the data.

Algorithm updates incorporate improvements to the algorithms used to
reconstruct the events. New tracking and vertexing algorithms, as well as
updates to lepton identification and algorithms are typical examples. The
updates may also include fixes to deal with crashes or floating point
exceptions observed in events that were previously problematic to reconstruct.
Many of these updates require corresponding changes to the reconstruction of
the MC simulation; some, however, might involve improvements to the rejection
of backgrounds that only occur in real detector data, in which case only the
detector data must be updated.

The validation of these changes uses the Data Quality tools described in
Section 5.1. Most often, a small sub-set of data are first processed to test
for technical issues. Later, a larger sub-set of the data are processed and
passed through a Data Quality validation process to check for issues with
various updates and ensure that the changes are as expected. Particularly
because conditions are period-specific, this larger sub-set includes runs from
several different conditions periods. Only after all of these checks are
completed is the reprocessing of the remainder of the data undertaken.
Afterwards there may be updates to the Good Runs List to include additional
data that could be declared ‘good’ thanks to improvements in the conditions or
reconstruction.

##  6 Infrastructure and databases

This section describes the significant central infrastructure required to
support the services, systems, and developers of the collaboration. Section
6.1 introduces the infrastructure maintained to support the software
development cycle. The database systems used throughout the experiment are
described in Section 6.2. Finally, the development and use of metadata, much
of which relies on these systems, is described in Section 6.3.

###  6.1 Software engineering process and infrastructure

The software developed for event simulation, reconstruction and analysis, as
well as for detector calibration and alignment, has to run on the computing
facilities that are available to the collaboration members; these facilities
are not necessarily uniform in their hardware architecture, nor their
operating systems and environments. To ensure that ATLAS software is robust
and portable, it is necessary to provide a common process and infrastructure
to integrate all new code with the existing code base, and then build and test
changes. A central ticket and workflow system is provided using Jira [241],
which provides functionality for bug reports, improvements, and tasks; for the
preparation of MC simulation requests before they enter the production system;
for tracking physics validation (see Section 5.2); and for the tracking of
long-term development. GitLab integration is used to ensure code updates and
tickets are cross-linked.

####  6.1.1 Software structure and compilation system

The ATLAS software, shown in Figure 26, is organized into a project for
general use (Athena) and several projects for specific purposes like event
generation (AthGeneration), detector simulation (AthSimulation), and data
analysis (the AthAnalysis and AnalysisBase projects). These rely on several
projects like ‘TDAQ Common’ (common trigger and data acquisition software) and
several external packages (e.g. event generators, ROOT, or Geant4). Each
project is further sub-divided into packages with much more specific purposes
(e.g. pixel detector digitisation, or muon momentum calibration); there are
about 2000 packages in the ATLAS software.

Figure 26: Schematic overview of the ATLAS software projects and external
dependencies. The overlaps between project areas indicate overlaps in code
included in the projects. The widths of AnalysisBaseExternals and
AthenaExternals are wider to indicate their support for the entirety of the
AnalysisBase and AthAnalysis projects, respectively.

ATLAS uses the CMake build system [38] for building all of its own code, in
conjunction with most of the HEP software projects that the experiment uses.
CMake is a very flexible system supporting Linux, MacOS and Windows hosts, x86
and ARM CPU (aarch64) architectures, GNU Make, and Ninja and additional low-
level build systems, with a rich high-level language allowing support for the
complex software building steps required for the ATLAS reconstruction,
simulation, trigger, and analysis code.

Thanks to the flexibility of CMake and to harmonise how different parts of the
code are built, ATLAS-specific functions and macros are used throughout the
build configuration of the offline and trigger software for setting up the
building of libraries and executables, and for dealing with the installation
of scripts and data files. For the offline software, that basic CMake
infrastructure is maintained in the |AtlasCMake| package [244].

One fundamental requirement for the build system is to allow the users to
build small parts of the software against a pre-built full software release (a
_base project_), as described in Section 6.1.5. This is achieved with imported
targets in CMake. Using |AtlasCMake|’s facilities the user is able to build
packages and libraries as part of a project that are already part of the base
project. This is achieved by only importing the targets of the base project
into the build of the current project that are not present in the current
project already.

The HEP_OSlibs meta package [245] provides the software package dependencies
for the Athena build and runtime environment for the two currently-used
operating systems (CentOS 7 and Alma Linux 9; see Section 6.1.2). The LCG
software stack [246], maintained by the CERN EP-SFT group in collaboration
with several large LHC experiments like ATLAS, provides several hundred
external packages that include recent versions of commonly used packages as
well as HEP-specific tools and MC event generators (see Section 4.1). There
are usually two or three major LCG software releases per year and development
builds are available every night. The major LCG versions are normally released
with a new major stable release of ROOT [40]. Minor LCG version updates are
done every few weeks and usually contain updates of MC event generators or
bug-fixes of other external packages. The LCG software stack is built for
x86_64 and aarch64 architectures and several recent versions of the GCC and
Clang compilers and C++ standards. As of autumn 2023, LCG version 104b with
ROOT 6.28/08 is used in Athena, compiled using GCC 13 with the C++20 language
standard.

_External software_ , software used by the ATLAS software that is maintained
separately from the experimental software and is not provided by an LCG
package, is built using CMake as part of an ATLAS external project. As shown
in Figure 26, every ATLAS offline software project (Athena, AthSimulation,
etc.) has a corresponding external project (AthenaExternals,
AthSimulationExternals, etc.), upon which it is built. This includes packages
like Gaudi [41], and allows a more rapid update of external packages that
receive rather frequent version updates or are not sufficiently common to be
included in the LCG stack directly. This separation is not present in the TDAQ
projects: TDAQ Common builds a combination of external and ATLAS packages as
part of the same project. Some parts of this TDAQ software are also required
in the ATLAS offline data processing, for example to read the ATLAS RAW data.

Any software build that uses CMake is separated into configuration, build,
installation and packaging steps. Configuring the build of small projects like
analysis projects, or the WorkDir project used for developing code in the
Athena repository, takes just a few seconds, while configuring the build of
the largest project, Athena, takes a few minutes.

Each package is typically built into one or more small dynamic libraries that
are loaded on-demand at runtime. This helps ensure a minimal memory footprint
from library loading despite the wide variety of jobs that can be run within
Athena. It also allows the user to override specific libraries provided in the
release or by the system, for example by pre-loading a different memory
allocation library or math library. There is a small CPU overhead associated
with dynamic library loading, discussed further in Section 4.2.1.

The packaging and distribution of the ATLAS software is achieved using RPM
packages built using the CPack tool within CMake [247]. The RPMs are
configured to depend on each other such that the installation of a full Athena
release can be done by requesting the installation of just the top-most Athena
RPM package. This top-most package in turn ensures that all dependent RPMs,
including those provided by LCG and packages providing various data files
(e.g. for Geant4) are installed in the correct place automatically.

####  6.1.2 Build system, CI, nightly and stable releases

The ATLAS Nightly and Continuous Integration (CI) Systems provide a modern
software development workflow for developers, feature fast development cycles
and assure confidence in new software deployments. Both the systems are
Jenkins-based [248] and have a long evolution history [249]. Interconnected
with the ATLAS GitLab code repository [25], the CI system performs up to 100
multi-project software builds daily, probing code changes proposed in GitLab
merge requests. A comprehensive test suite of unit and short integration tests
runs for each CI job. The Nightly System (separate from CI) performs daily
builds of all ATLAS software projects from the code repository (called
‘nightlies’ because the largest projects are built overnight), often on
several platforms. It maintains a multi-stream, parallel development
environment with many simultaneous branches. The Nightly System probes how the
changes from accepted merge requests work together. In addition, it helps to
support migrations to new platforms and compilers and verify patches to
external tools.

The CI and Nightly jobs run on the 1400-core build farm, including both real
hardware and virtual nodes. Various _operational intelligence_ techniques such
as incremental compilations, selective testing, operations parallelization,
and dimensionality reduction result in efficient resource use and faster
delivery of results. The CI and Nightly monitoring system provides dynamic
information about build and test results and installation status. It is based
on the Nightlies Database residing in the ATLAS database production cluster
dedicated to offline analysis (ATLR, see Section 6.2.1) [250]. The ATLAS
Nightlies Database is the source of dynamic content for the nightlies
dashboards hosted on the BigPanDA web application [251] (see Section 7.4.2).

The systems build software releases from the GitLab main branch and from
dedicated branches for the online high-level trigger, reconstruction and
simulation of Run 3 data, and legacy branches for Run 2 reconstruction,
simulation and analysis. Regular ‘sweeps’ are used to copy changes made in
various branches to the main branch. Special development nightlies are
available that use development versions of ROOT, LCG builds, or other
externals. These ensure that when preparing for significant changes (e.g. a
new operating system, GCC version, or C++ language standard), the changes can
be thoroughly vetted without putting at risk the main development stream.

When the compilation and test steps of the nightly build are successfully
completed, CPack [247] is used to produce RPMs, which are then uploaded to EOS
[252]. This acts as a staging area for the subsequent installation step that
fetches them and, using ayum [253], an ATLAS-specific wrapper around yum,
installs the nightly releases in the atlas-nightlies repository on the CernVM
FileSystem (CVMFS) [254], and thereby makes them globally accessible to
developers. CVMFS is the service by which ATLAS distributes all of its
development and production software, and auxiliary datasets.

The installation software [255] is executed locally on a publishing node
(officially, a Release Manager) that communicates with a backend Gateway node.
Having multiple publisher nodes allows concurrent transactions, increasing the
overall publication rate of the repository and improving the speed at which
nightly builds become available.

Currently the total size of the atlas-nightlies repository is 10 TB, with
approximately 4 TB in use. In 2020, the repository data was migrated to Ceph-
based S3 object storage, resulting in a performance enhancement over the
previous aufs system (advanced multi-layered unification system). All nightly
releases (currently about 35) built in ATLAS and using different architectures
(binaries, operating systems and compilers) are deployed on CVMFS in the form
of RPMs. The installation time ranges from 3 to 60 minutes, depending on the
number of external packages needed and the format of the build (e.g. whether
debugging symbols are included) and type of release (e.g.
DetCommon/AnalysisBase compared with full Athena). The installations are kept
for 30 days by default, with the possibility of extensions for developers to
test newer code against older nightlies or if a physics validation is
performed with a nightly release (see Section 5.2).

The nightly releases are rigorously tested in the ATLAS Release Tester (ART)
Grid-based framework [256] described in Section 6.1.3. When the set
development goals are achieved, a successful nightly release is transformed
into a stable release by the team of ATLAS offline release shifters. Stable
releases have unique numeric identifiers and indefinite lifetime. The
numbering of stable releases follows the pattern A.B.C(.D), where A is the
major release version that changes at most once a year, B generally indicates
the purpose of the release (e.g. 2 for analysis, or 6 for generation; these
numbers are often historical), C indicates the minor release version and is
typically increased about once a week for a new stable release, and D is an
optional patch version in case a stable release must be patched. Patching is
most often used so that a stable release, for example one to be used for
trigger simulation, which must therefore match the release used during data
taking, must be patched to fix a rare bug, add a feature for MC simulation, or
read a new file version.

All production releases until fall 2023 were built on the CentOS 7 Linux
operating system [257] on the x86_64 and aarch64 architectures using the CMake
build system described above. A migration to Alma Linux 9 [258] of all
infrastructure was completed by the end of 2023, though several legacy
releases continue to be based on CentOS 7.

The default compiler for all Run 3 releases is at present GCC version 11.2.0
[259] with the binutils 2.37 [260] and the C++17 standard. In parallel there
is a nightly release build from the GitLab main branch using the Clang 16.0.3
compiler [261] on the x86_64 architecture. Maintaining builds with at least
two compilers has helped identify a variety of minor issues in the software
and led to increased robustness. For the releases built on Alma Linux 9, the
recent stable version of GCC, version 13.1, with the C++20 language standard
and binutils 2.40 are used. Nightly builds on the aarch64 architecture exist
both on the CentOS 7 and Alma Linux 9 operating systems with the GCC 11.2 and
13.1 compiler versions respectively. Since summer 2023, the main branch
releases are built for the microarchitecture x86-64-v2 and newer [262], which
brings support among other things for vector instructions up to Streaming SIMD
Extensions 4.2 (SSE4.2). Newer microarchitecture levels are not yet supported
by a sufficiently large fraction of machines available on the WLCG Grid to
make adoption beneficial.

####  6.1.3 Release testing

The ART system [256] is designed to run test jobs on the Grid after an ATLAS
nightly release is built. The choice was made to exploit the Grid as a back-
end as it offers a huge resource pool, suitable for a deep set of integration
tests, and running the tests could be delegated to the highly scalable ATLAS
production system (PanDA, see Section 7.3). The challenge of enabling the Grid
as a test environment is met with the CVMFS file system for the software and
input data files. Test jobs are submitted to the Grid by the GitLab-CI system,
which itself is triggered at the end of a release build. Jobs can be adorned
with special headers that direct how to run the specific test, allowing many
options to be customised. These options might include which releases are to be
tested, input- and output-file specifications, and the number of cores to be
used for multi-process or multithreaded running, for example. The GitLab-CI
waits for the exit status, and output files are copied back from the Grid to
an EOS area accessible by the users. All GitLab-CI jobs run in ART virtual
machines using Docker images [263] for their ATLAS setup. ART jobs can be
tracked by using the PanDA monitoring system.

ART can also be used to run short test jobs locally. They use the same ART
command-line interface, where the back-end is replaced to access a local
machine for job submission rather than the Grid. This feature allows
developers to ensure their tests work correctly before adding them to the
system. In both the Grid and local machine options, running and result copying
are completely parallelised.

####  6.1.4 Software quality

To assist in the development of high-quality code, ATLAS has maintained a
Coding Standard [264]. The goal of the standard is to ensure some consistency
in code style (e.g. naming conventions for variables and files) so that
developers can quickly understand a piece of ATLAS code with which they are
not familiar, while acknowledging that with thousands of developers working
over tens of years, not all style issues must be rigorously enforced (e.g.
spacing and bracket placement is not enforced). The standard is divided into
requirements, which are checked using a static code checker implemented as a
GCC plugin [54], and recommendations that are taught but not enforced.

A wide variety of static code-checking tools are used, including Coverity
[265], CppCheck [266], and lizard [267] for C++, and flake8 [268] for Python.
Both flake8 and CppCheck (when available) are integrated into the build system
so that compilation warnings are raised if code problems are identified.

The primary mechanism used to enforce code quality and consistency is via
multi-level reviews of merge requests. Because of the size of the code base,
‘tidying’ the entirety of ATLAS code would be impractical. Instead, when
touching a particular piece of code, developers are encouraged to gently
improve the code around their changes, help identify and improve missing
documentation or code weakness, and only introduce new code that is of high
quality. For most merge requests, a first-level reviewer looks over the
changes to ensure that they are sensible and well explained in the merge
request, and that all of the automated tests have succeeded. If the merge
request is particularly complex, a second-level (expert) reviewer might be
asked to examine the code further. Some domain-specific review is often
performed, as the CI system automatically alerts some experts to changes to
code within their domain. Finally, a release coordinator is responsible for a
last check before the changes are merged into the central repository.

####  6.1.5 Development and run environment

As described in Section 6.1.2, the current default runtime environment is the
CentOS 7 Linux operating system. The AFS [269] and EOS [252] file systems are
used during the software nightly builds to access larger calibration files
that are then integrated into the respective software releases.

Although primarily designed for clusters of Linux processors with open
external connectivity that allows the software to be imported through CVMFS
and the conditions data accessed through FroNTier (see Section 6.2.2), ATLAS
software must be able to run on different hardware architectures and different
operating system versions. A notable example is High-Performance Computers
(HPCs), which may have large numbers of processors but may lack external
connectivity. In that case the software and conditions data need to be
packaged into a container that is then uploaded through a gateway to the HPC;
input and output data files also go through the same gateway.

The ATLASLocalRootBase (ALRB) [270] suite of shell scripts, whether accessed
from CVMFS or locally installed on a laptop or HPC site, provides a uniform
look and feel to configure many tools with their dependencies in the correct
order by encapsulating the details. It is also a wrapper to start containers
with the necessary environments and mount points for runtime Docker [263],
Singularity [271], Apptainer [272] and Shifter [273]. It provides mechanisms
to test tools and validate containers before deployment. Diagnostics are also
available in ALRB for user support and for tutorials. It works for bash/zsh
shells, x86_64 and aarch64 architectures, and RHEL-derived operating systems
with versions 5–9; it is Python 3 ready and supports containers on other Linux
flavours and MacOS.

The AtlasSetup (or simply asetup) script [270] provides users a convenient and
fast way (a few seconds) to locate the required release and configure its
environment for stable and nightly releases. For the latest nightly releases,
asetup checks the release completeness because a nightly release is built in a
few steps with a time gap of about an hour. It only requires users to specify
a few short tags, like the project (e.g. Athena or AthGeneration) and stable
version number or day of the month for a nightly release. There are a few
layers of configuration for asetup, and asetup can print out the origin of a
given configuration parameter. The original shell environment is saved before
the first asetup run, so that asetup can configure another release environment
in the same session, even in subshells. The details of the required release
are saved into the working directory, such that the same release under that
working directory can be restored easily and quickly the next time.

The asetup script works for bash/zsh shells, with Python 2 (including old
Python 2 in SLC5) and Python 3, and in both Singularity/Apptainer and Docker
containers. It supports releases in architectures of i686, x86_64 and aarch64
for RHEL-derived operating systems from SLC 5 to Alma Linux 9. In the case of
a conflict between the setup release environment and system commands, asetup
helps provide a solution via wrappers and aliases. A suite of regression tests
comes with asetup to validate new versions, as asetup can run in different
RHEL-derived Linux operating systems through Singularity/Apptainer and Docker
containers from CVMFS. The regression tests include more than 100 tests for
the CentOS 7 platform.

Most contributors to the ATLAS software only need to work on a subset of
packages (usually one or two packages) and do not require checking out the
whole Athena GitLab repository. A wrapper command, git-atlas [274], handles
the consistent checking out of the packages the developer needs using the
sparse checkout functionality, as well as an interface to the developer’s
fork, which is necessary for submitting merge requests. A user needs only to
specify the package name (e.g. |PixelDigitization|), and the code from the
relevant package area (in this case,
|InnerDetector/InDetDigitization/PixelDigitization|) is added to the existing
sparse checkout. This configuration enables local partial builds against the
nightlies, significantly speeding up development cycles. Since it aimed to
minimise the disruption to (non-core) feature developers, the git-atlas
command was very useful to many developers during the transition from SVN to
GIT, as it did not expose the complex commands required for sparse checkouts
to the end users.

####  6.1.6 Central services

The Central Services Operations (CSops) team is central to all ATLAS
activities. Its role is to support a wide-ranging set of projects given the
great variety of services needed by ATLAS computing. The team is responsible
for the configuration, deployment, and operation of different services on the
CERN IT infrastructure. They also act as the interface between CERN IT and the
service managers of different projects and ensure that security and good
computing practices are maintained.

CSops is responsible for the management of over 630 machines with more than
4,700 CPU cores, spread across 60 projects including PanDA [112] (see Section
7.3.1), Rucio [275] (see Section 7.2.1), the services described Section 6.1.2,
and others. These machines are hosted in the CERN-IT Openstack [276]
infrastructure and the configurations are managed by Puppet [277], which
allows a significant reduction of complicated central operation tasks, while
ensuring the high availability and scalability of all ATLAS central services.

These machines can be split up into three main groups: those for ADC, those
for ATLAS nightly software building, and the miscellaneous nodes. The last
group contains machines from many different detector and operations groups in
ATLAS and include critical web services, logbooks, monitoring and other
services that are used by both the offline and online teams in ATLAS.

More than half of the machines deployed in Openstack are maintained with
Puppet manifests. This helps to ensure that security and configurations are
reproducible on a large scale. All machines that are controlled with Puppet
can generally be destroyed and recreated within 45 minutes. There are 139
different configuration manifests in place that are used to define the major
configurations. Specific changes to each configuration can be made with YAML
[278] files that are used by the Puppet Database to fine tune individual
hosts. All of these manifests and configurations are stored in CERN GitLab
repositories to aid with rapid deployment and keep track of changes to
configurations over time.

Together with ADC and the ATLAS TDAQ system administrators, CSops also helps
to maintain the Simulation at Point 1 (Sim@P1, see Section 7.3.2) project,
which is an opportunistic cloud that makes use of the ATLAS TDAQ HLT computing
farm for offline grid workflows. The Sim@P1 project is able to spawn over
2,000 virtual machines using in excess of 111,000 CPU cores that can be
deployed in under 15 minutes; however, this is usually done over an hour for
maintainability and to avoid unnecessary pressure on critical systems.

###  6.2 Databases

####  6.2.1 Database infrastructure

ATLAS relies on several services for the management of non–event data; for
example data derived from the run configuration, detector calibration and
alignment, slow control system of the experiment, metadata associated with
events, raw data files or data and workload management systems. Most of these
services profit from a database infrastructure that is maintained by the CERN
IT department and is based on Oracle® technology. Other applications include
analytics clusters utilising the Hadoop [279] file system and Elasticsearch
[280]. The same infrastructure supports other database applications like those
for ATLAS publications and membership information [281].

The original (Run 2) relational database infrastructure, as shown in Figure
27, consisted of three production Oracle® clusters deployed and maintained by
CERN IT: ATONR for Point 1 (P1) usage, ATLR as a general purpose cluster and
ADCR for data and workload management systems like Rucio and PanDA (see
Section 7). In this configuration, for data-processing purposes a subset of
the ATLR data (which is limited to conditions, geometry and some trigger
information) is regularly replicated to two Tier-1 sites, TRIUMF and CC-IN2P3.
ATONR was and still is within the firewall at P1, protecting it from
accidental external use.

Figure 27: The database infrastructure employed by ATLAS before Run 3.

During the long shutdown between Run 2 and Run 3, the relational database
infrastructure was consolidated in view of changes to the Oracle® licensing
model at CERN. ATLAS therefore progressively migrated to a new infrastructure,
now in place and illustrated in Figure 28, completing the migration in early
2024. The cost and the complexity of the system was reduced by eliminating
Oracle® instances at the Tier-1 sites, and to consolidate the CERN resources
of the P1 cluster ATONR and its standby Active Data Guard (ADG) instance that
are accessible from outside of the P1 network in a read-only mode. An
important step towards consolidation was the migration of the ATLAS Metadata
Interface (AMI) database [282] (see Section 6.3), which was historically based
at CC-IN2P3, to CERN by the end of 2021. Although the independent
infrastructure had offered some advantages in terms of resilience, the
advantages of the tighter integration, central support, and centralised
authentication mechanisms available at CERN made the migration beneficial. The
migration of the conditions data in the offline schemas into ATONR is now
complete as well; to preserve isolation of Point 1 during data taking a
special proxy server was developed to permit updates of conditions in the
offline COOL schemas. Additional nodes were added into the ATONR cluster to
accommodate additional workflows originating from the General Purpose Network,
which were previously utilizing ATLR. This expansion ensures the preservation
of the existing resources for data-taking workflows at Point 1. On the ADCR
cluster, new more powerful hardware was deployed to support the heavy
workflows coming from the data management system. The most delicate parts of
this migration had to be performed during pauses in data taking (either short
technical stops or end of year shutdowns), but the migration could be done
during Run 3.

Figure 28: The database infrastructure employed by ATLAS for Run 3.

####  6.2.2 Conditions data distribution

The conditions data are distributed using several technologies. The main
service in place for access to conditions is FroNTier [283], a web application
that redirects SQL requests to Oracle® databases and provides a caching layer
implemented using Squid proxies. The system is in use since Run 1, and the
main difference in Run 3 is its deployment architecture as mentioned in the
previous section; the FroNTier launchpads are deployed only at CERN and
provide access via the ATONR_ADG cluster. The network access to conditions via
FroNTier may not be possible, especially for certain HPCs lacking outbound
network connectivity (see Section 7.3.2). In such cases, an alternative
approach is available; data can be extracted from Oracle® and distributed
through simple SQLite files. In this case there is the possibility to extract
data from Oracle® and distribute them via simple SQLite files. The tools for
the easy generation of an SQLite file with a selection of a subset of
conditions data are gathered in a GitLab repository [284]. The SQLite files
can then be distributed in a file system or via a Docker container provided
their volume are reasonable.

###  6.3 Metadata handling

The ATLAS experiment has developed three systems that build upon information
from other ATLAS systems to provide unique metadata services to the experiment
at the three fundamental levels of granularity of ATLAS data: services at the
event, run, and dataset levels are provided by the EventIndex, COMA, and AMI,
respectively. Each of these systems has evolved considerably since Run 1 in
mutual cooperation with many systems and services throughout ATLAS (having
connections to aid data collection and service optimisation). Furthermore,
they each have evolved into related areas to provide additional services as
described in this section.

The EventIndex [285] is the metadata catalogue of all ATLAS events. For each
real or simulated event in the primary processing formats (e.g. RAW and AOD),
it stores the main event identification parameters (run and event number,
trigger stream, luminosity block), the trigger record and the provenance
information, i.e. the Globally Unique Identifiers (GUIDs) of the files that
contain it. This information is used to check the completeness and correctness
of the processing campaigns, as the number of output events must match the
number of input events and each event must be found once and only once in the
output streams, and to monitor the overlaps between trigger chains or between
offline selection streams. The provenance information is used to extract one
or several events that are selected during any analysis and produce event
displays (see Section 8.2), to study details of the calibration, alignment and
reconstruction algorithms, or for special analysis workflows. In this last
instance, physics analyses that need specific reconstruction algorithms to be
applied on specific event samples (up to one million events) can use
EventIndex provenance information and the Event Picking Service [286] to
extract those events from the bulk of the RAW data and process them
separately. The EventIndex metadata are collected by Grid jobs that scan all
newly produced data files. These metadata are then sent to a central storage
system at CERN, implemented using Hadoop [279], HBase [287], and Phoenix
[288], an interface layer on top of HBase that allows SQL access. For Run 3,
the system was extensively revised and re-implemented [289]. The EventIndex
stores several hundred billion real and simulated event records and grows
linearly with data production rates.

The COMA system (Conditions/Configuration Metadata for ATLAS) [290] collects
run-level conditions and configuration metadata mainly from the Conditions
database, as well as essential run-related information from other systems: the
HLT farm, Tier-0 site, Trigger, AMI, and EventIndex data repositories and file
systems such as good runs list (GRL) XML files (see Section 5.1). Early in Run
1, COMA became the main repository to store the ATLAS Data Periods, which are
official sets of run numbers during physics data-taking with common detector
or machine conditions. These ATLAS Periods form the basis to group sets of
runs for many purposes for ATLAS data processing and reprocessing as well as
data analysis. COMA provides both web-based and command line interfaces
featuring collected data and unique related derived quantities, as well as
aggregated information across many runs (e.g. runs in specific projects,
periods and GRLs) such as event counts (by stream and trigger), data volumes,
luminosity and LHC beam-related quantities. To ease the management of the
ATLAS Conditions database, the COMA repository was expanded [291] to collect
metadata about conditions data structure, payloads and related metrics,
providing reports to help experts understand conditions database organisation,
content, and usage. COMA systems have undergone continuous refinements in data
collection and client utilities into Run 3.

The ATLAS Metadata Interface (AMI) [282] is a software ecosystem dedicated to
scientific metadata that provides two critical services; management of
metadata for all datasets produced by the experiment and bookkeeping of all
the parameters defining the data processing workflows. The AMI task server
aggregates metadata information from several sources (e.g. Rucio and PanDA)
via specific tasks and stores them in the AMI database. Metadata sets are
recorded for each processing step a dataset undergoes and the ‘filiation
links’ (provenance information) between processing steps of the same datasets
are saved.

In parallel with ongoing operations, the AMI environment is constantly
evolving: an internal rewrite of both the core server and Web interfaces were
done for Run 3 to provide improved stability, scalability and flexibility when
addressing the metadata needs of the users. A new administration interface for
the AMI task server was written to easily isolate, monitor and manage tasks.
In addition, the interface to make super containers (a set of datasets for a
given data taking period or year, or any other logically-grouped set of
datasets) was rewritten. A prototype was developed and tested for the revision
of the AMI-tags that define all the parameters of a processing workflow; it
will be finalised in coordination with the Tier-0 and database teams.

Users most often access AMI via Web applications to manage and display
metadata. These are also available via command-line interfaces for easy use in
scripts or data analyses. AMI also recently developed a whiteboard system that
allows users to set arbitrary tags onto a dataset to be able to retrieve them
in a more efficient manner. These tags are used to identify samples that are
standard samples to be used in data analysis; they can also be used to
identify all datasets in use by a single analysis, for example.

In addition to these external metadata systems, ATLAS employs _in-file_
metadata to transfer information from one processing step to the next. These
data include information about the configuration of the job that produced the
data (e.g. the parton distribution function used during event generation, or
the geometry used during simulation). They can be used for automatic
configuration of subsequent workflow steps. For example, the reconstruction
automatically initializes the correct geometry based on information in the
input file. This automatic configuration can be overridden, or the information
can be explicitly provided if it is not present. The in-file metadata can also
be used for consistency checks, for example by checking that files to be
combined in MC Overlay (see Section 4.3.2) were created with a consistent
geometry. For real data, the in-file metadata includes information about which
luminosity blocks were processed, to help ensure that the entire set of events
from each luminosity block is correctly processed in an analysis. During
derivation production (see Section 4.5.1), some of the in-file metadata are
converted into a simple ROOT-readable format. These can then be used in
analyses, for example for the configuration of calibration and uncertainty
tools. Including this in-file metadata helps reduce the necessity of database
connections in analysis jobs and enhances the ability to run analysis with
only simple ROOT or similar software.

##  7 Distributed computing

ATLAS Distributed Computing (ADC) comprises the hardware, software and
operations needed to support distributed processing, simulation and analysis
of ATLAS data and to support the evolving needs of the experiment. This can be
broken down into two closely related areas, which operate side by side:
Distributed Data Management (DDM), covering all aspects of storage, transfer,
and access to the (collision and simulated) data, and the Workflow Management
System (WFMS), which handles request, task and job definition, and manages the
various workloads performed on the ATLAS data, for both large scale production
tasks and user analysis. The first site through which all data move, the
Tier-0 site, uses special configurations and software due to the stringent
operational requirements of the site. This site is described in Section 7.1.
The remaining Grid sites use common software for both the data and workflow
management. The DDM model employed by ATLAS is presented in Section 7.2,
followed by a description of the many aspects of the WFMS in Section 7.3. The
data monitoring and analytics systems described in Section 7.4 naturally cover
both of these areas.

###  7.1 The Tier 0 site

The ATLAS Tier-0 site comprises the hardware, software and operations needed
to support the _prompt_ processing of the data produced by the ATLAS detector.
This prompt processing mainly consists of the first-pass reconstruction
workflow and many calibration workflows (see Section 2.4). The Tier-0 site
also provides support for ad-hoc reprocessing, and is frequently used for
various commissioning tasks. Reprocessing and commissioning tasks may include
the recall of data from tape storage if necessary.

The Tier-0 site uses the same conceptual models of _datasets_ (i.e.
collections of files) being processed by _tasks_ (i.e. collections of jobs
executed on some external batch service) as DDM (see Section 7.2) and WFMS
(see Section 7.3). However, it uses different workflow management software
from most Grid sites for the prompt processing of data.

####  7.1.1 Tier-0 resources

The Tier-0 site consists of about 40,000 cores in the CERN HTCondor-managed
batch system. The cluster is shared with Grid processing, which takes over the
resources if they are not needed for Tier-0 operations (see also Section
7.3.2). The Tier-0 site has about 2 PB of EOS disk space available to
temporarily store intermediate datasets, and about 100 GB of shared AFS disk
space to store all other (non-dataset) files needed for operation.

In addition, a single-core virtual machine is used to run the software that
drives Tier-0 site operations. Another single-core virtual machine is used to
host the web service allowing both the monitoring and operation of the system.

####  7.1.2 Tier-0 software

The Tier-0 software is designed to depend on as few external services as
possible. This not only minimises the chance of service interruptions caused
by them, but also the effort required to follow their evolution over time. The
services the Tier-0 software does depend on tend to be stable and highly
reliable. For the software, there is a similar strategy to depend on as little
third-party software as possible, for the same reasons.

The Tier-0 software system is a modular, self-contained system comprising a
workload management system (comparable to ProdSys2–PanDA, as described in
Section 7.3), a data management system (comparable to Rucio, as described in
Section 7.2), a monitoring system (probes, data collectors, alarms, etc.), and
a comprehensive web interface for monitoring and operations.

The Tier-0 workload management system consists of two entities: one operating
on the dataset/task level (the Tier-0 Manager, or TOM), and one operating on
the file/job level (the Supervisor). Both follow a component-oriented software
paradigm; each entity is composed of approximately ten components that
collaborate to provide the desired functionality. This approach not only
allows different implementations of the same component (e.g. interfacing to
different batch system flavours, or to different database backend flavours),
but also allows the sharing of components between the two entities (e.g. the
logging component). In addition, some components provide a plug-in mechanism
that allows loading of alternative implementations for particular key
algorithms.

This combined component-oriented and plug-in approach has not only facilitated
a smooth evolution of the system over the last 15 years, but also allowed
another CERN experiment (NA62) to use the ATLAS Tier-0 software system with
minimal effort [292].

##### Tier-0 manager

Each Tier-0 Manager (TOM) instance/process runs a set of configured plug-in
tasks, called _tomprocesses_ , at regular intervals in an infinite loop. Each
TOM is single-threaded, ensuring strict serialisation. The main task of the
TOMs is to implement the Tier-0 reconstruction and calibration workflows. The
paradigm followed here is that of a dataset blackboard: files created on EOS
and published in a handshake catalogue by the online system are organised by a
tomprocess into RAW datasets and put on a virtual blackboard. Other
tomprocesses regularly scan the blackboard for new datasets fulfilling certain
criteria and, if triggered, automatically define tasks that process these
datasets into one or more new output datasets. For Run 3, about 100 such
dataset-processing tomprocesses are defined, each configured with about 20
parameters.

Taking advantage of the generic architecture of the TOM system, most other
Tier-0 activities are also run as tomprocesses. Examples include the
registration of permanent output datasets and their files with Rucio, the
management of the temporary EOS disk space containing the transient datasets,
and the management of the temporary AFS disk space.

##### Supervisor

As the name suggests, the task of a Supervisor process is to supervise the
execution of the jobs defined by the T0 Manager(s). Each supervisor process is
single-threaded, and multiple processes can be configured and deployed
simultaneously. The standard configuration for Run 3, however, only needs to
deploy a single instance, which manages the about 10,000 jobs running
simultaneously on the HTCondor batch system.

##### Web control interface

The Tier-0 web interface, called _conTZole_ , is a classical
Javascript/Django/Apache single-page web application. It allows comprehensive
monitoring of the state of the Tier-0 system, with both live and historical
information. Subject to authentication and authorisation, it also allows the
Tier-0 operators to dynamically change the Tier-0 configuration, and third-
party users to request ad-hoc processing of selected datasets.

###  7.2 Distributed data management

ATLAS data are distributed over a worldwide network of data centres, also
called sites, under the umbrella of the WLCG [293]. These sites are
categorised into Tiers in a semi-rigid hierarchy with various capacities,
duties, and responsibilities. CERN is the origin of detector data and thus the
single Tier-0 centre. There are 11 ATLAS Tier-1 sites that are connected via
dedicated national research and education networks (NRENs) with typically 10
to 100 GBit optical private networks [294]. These Tier-1 sites provide disk
and tape storage and are charged with the perpetual archival of detector data.
Around 70 ATLAS Tier-2 sites, typically hosted by national universities and
laboratories, provide disk storage that is used for data processing and user
analysis. Several additional computing centres complement the storage and
computing available to ATLAS, including opportunistic sites (see Section
7.3.2) and sites dedicated to data analysis (see Section 8.1.7). As shown in
Figure 29, as of 2023 ATLAS has more than 800 PB of resident data. The total
volume is split roughly equally between tape and disk. All these data are
organised, managed, transferred, accessed, and accounted for via the Rucio
distributed data management system (see Section 7.2.1).

Figure 29: The evolution of the ATLAS total data volume over the last 15
years, since before Run 1.

####  7.2.1 Rucio

Rucio [275] is a flexible and modular software framework to build data
management federations. It allows seamless integration of scientific and
commercial storage, and their network systems. The data are stored in a single
global namespace and can contain any payload. Facilities hosting the storage
can be distributed at multiple locations, and belong to different
administrative domains. Rucio was designed with more than a decade of
operational experience in very large-scale data management, building from the
work of its predecessor DQ2 [295].

As a technology, Rucio is location-aware and manages data in heterogeneous
storage solutions and environments. It allows the creation, location,
transfer, deletion, annotation, and access of and to the data. The major,
standout feature is the orchestration of dataflows. These include both high-
level dataflows, such as institutional or experiment policies, and low-level
dataflows, such as specific user requests. Its main users are the PanDA and
ProdSys2 workflow management systems, including the PanDA Pilot job processing
framework, as described in Section 7.3. Individual users have command-line
interface and web-based access through dedicated clients, both for access to
data and for the manipulation of rules for data movement and storage.

At ATLAS mid-2022 scale, the interaction rate of operations with the Rucio
system is typically beyond 200 Hz and often reaches 500 Hz, from distributed
clients across the world interacting with the single Rucio instance at CERN.
This includes diverse operations such as registering new files, searching for
data, downloading files for processing jobs, scheduling files for
deletion,212121See Section 7.2.3 for the reason behind scheduling files for
deletion, rather than deleting them immediately. or modifying metadata. Rucio
handles on average 1.5 million file transfers per day, with a peak rate of
three million files per day. On average around 8 PB of data is moved per day
from job and user upload/download, peaking at 12 PB per day. Transfers between
data centres accounts for 2 PB per day on average, peaking at 4 PB per day.
The high-availability deployment of Rucio is done via Kubernetes [296] in the
CERN data centre and is built atop free, open-source technologies.
Additionally, extensive monitoring solutions were built and are the
cornerstone of daily operations.

Rucio was principally developed by and for the needs of the ATLAS experiment,
with a view to eventually making it community open source software. It has
since matured into the de-facto general solution for scientific data
management. Now, it is developed, used, and supported by a multitude of
different communities from diverse scientific fields, from neutrino
experiments and dark matter searches, to astronomical observatories. As
software, it is both free and open source, provided under the Apache 2.0
License [26].

####  7.2.2 Dataset nomenclature

ATLAS datasets follow strict nomenclature rules to ease findability and ensure
that dataset contents can be quickly identified. The names can be generated
algorithmically, are unique, and are case-insensitive, although the original
case is preserved throughout the system.

The dataset name is a series of fields separated with a ‘.’. For MC
simulation, the fields are:

  1. 1.

The project, a short indicator of the MC simulation campaign and, when
relevant, centre-of-mass energy ($<15$ characters);

  2. 2.

A numerical dataset identifier ($<8$ characters);

  3. 3.

A short description of the physics described by the dataset ($<50$
characters);

  4. 4.

The production step that generated the dataset (e.g. ‘simul’ for simulation or
‘evgen’ for event generation; $<15$ characters);

  5. 5.

The data type (e.g. ‘EVNT’, ‘HITS’, or ‘AOD’; $<15$ characters);

  6. 6.

A series of processing tags, called AMI tags (see Sections 6.3 and 3.2),
indicating the configuration of the software that was used for each production
step in the creation of the dataset, separated by underscores. Each step is
represented by a letter, followed by the numerical index of the configuration,
stored in the AMI database (e.g. ‘e1234’ for an event generation
configuration; $<32$ characters in total); and

  7. 7.

Optionally, a production task index and sub-task index.

An example of an MC simulation dataset name is
‘mc15_13TeV.300402.Pythia8B_A14_CTEQ6L1_Bs_mu3p5mu3p5.recon.AOD.e4397_s2608_r6869’.
In this case, ‘mc15_13TeV’ is the project; 300402 is the dataset identifier;
the production step is ‘recon’ (reconstruction); the data type is ‘AOD’; and
the configuration tags are ‘e4397’, ‘s2608’, and ‘r6869’ for event generation
(e), simulation (s), and reconstruction (r). The description means that the
sample was generated with the Pythia8B event generator (see Section 4.1) using
the A14 tune of underlying event and hadronisation parameters in Pythia and
the CTEQ6L1 parton distribution function set. The sample contains
$B_{s}^{0}\rightarrow\mu\mu$ decays, where both the muons are required to have
$p_{\text{T}}>3.5$ GeV. Although not trivial to decode, with some experience
these labels are enough to understand the contents of the sample.

For detector data, the fields are:

  1. 1.

The project, indicating the year and, when relevant, the centre-of-mass energy
for collision data;

  2. 2.

The unique run number for the dataset, established by the data acquisition
software;

  3. 3.

The name of the data stream (e.g. ‘physics_Main’ for the most commonly
analysed data stream [18]);

  4. 4.

The production step that generated the data, similar to MC simulation;

  5. 5.

The data type, similar to MC simulation;

  6. 6.

A series of processing tags, similar to MC simulation; and

  7. 7.

Optionally, a production task index and sub-task index, similar to MC
simulation.

One example is ‘data15_13TeV.00284484.physics_Main.merge.AOD.f644_m1518’. In
this case, ‘data15_13TeV’ is the project; 284484 is the run number; the data
are from the physics main stream; the data are merged (following the
reconstruction, in this case); the data format is ‘AOD’; and the production
tags are ‘f644’ and ‘m1518’ for prompt reconstruction at the Tier-0 site (f)
and merging (m).

User- or group-defined datasets are required to follow a less-strict set of
rules; they must begin with ‘user.’ followed by the user name, or ‘group.’
followed by the group name.

Rucio, used to organize the data, requires each dataset to be defined within a
scope. For MC simulation and detector data, the scope is identical to the
project in the dataset name. For group and user data, the scope is ‘group.’
followed by the group name, or ‘user.’ followed by the user name. Datasets can
also be grouped into containers (groups of like datasets) within Rucio;
containers follow the same naming conventions where possible.222222For
example, a container of multiple data runs follows the same general scheme,
but does not specify a unique run number if it contains many runs. It may
instead specify a Period (see Section 6.3) based on which the runs are
grouped.

These conventions were established before the start of data collection and
were only carefully updated since. This ensures that any ATLAS dataset in the
entire storage system can be quickly and easily identified by any users
familiar with the naming conventions.

####  7.2.3 Data policies and life cycle

One key feature of ATLAS data is immutability. Datasets are locked once
production is complete, and no additional files can be added, nor removed
except in rare cases of data loss. Changes are not allowed; if a fix must be
applied, an additional processing step is undertaken and the pre- and post-fix
data can be retained in the system independently if necessary. This ensures
that all data produced centrally (whether MC simulation datasets or real
detector data, at any stage of processing) can be reproduced at any time, and
the full chain of provenance is known.

Data on disk and tape are managed in slightly different ways. Disk-resident
data are divided into three categories:

  * •

_Persistent_ data are pinned on disk for an indefinite period of time,

  * •

_Temporary_ data are pinned for a limited period of time, and

  * •

_Cached_ data are not pinned.

When non-user-analysis data are produced by Grid jobs they are copied to their
final location and pinned there as persistent data. These data cannot be
deleted except through one of the mechanisms described later in this section.
Data that are moved around by the production system to be used as input for
jobs (including data that are replicated from tape) are pinned on disk for a
limited time and are thus labelled temporary. Once this time runs out, the
data are considered cached and are eligible for deletion. The total volume of
data stored on disk by ATLAS and the distribution within these different
categories is shown in Figure 30.

Figure 30: The distribution among persistent, temporary, and cached of ATLAS
data stored on disk in 2022–2023.

Each site hosting ATLAS data defines a quota of space available for use by
ATLAS, and publishes frequently the capacity, used and free space. Rucio sets
a watermark just below the defined capacity and aims to keep the used space at
this watermark. If the site reports used space over the watermark, Rucio
deletes cache data in order of least recent access until the watermark is
reached. In this way, ATLAS uses as much disk space as is possible, unpopular
data are automatically expunged from disk and popular data are likely to stay
and be reused.

A site can be assigned as a _nucleus_ when it can reliably aggregate output by
_satellite_ sites. The nucleus designation is made manually by operators in
the Computer Resource Information Catalogue, CRIC [297], and can change over
time. Job output data are transferred from the satellite sites and aggregated
at the nucleus for the full task. The output is kept at the nucleus for
archival or further processing. Brokering ensures that jobs are spread across
satellites, while taking the available free storage space of the nuclei into
account.

Data on tape are managed in a much less dynamic way, as the medium is not
designed for rapid turnover. Obsolete data on tape are marked as cached, but
they are only deleted in organised deletion campaigns coordinated with the
sites, which take place at most once per year. This is because space freed
from deleted data is usually only reclaimed when tape cassettes are _repacked_
(i.e. the remaining data on each cassette are packed into a new one).

The policies for data replication and lifetime are designed to optimise the
availability and redundancy of data for processing and analysis, whilst
ensuring that the limited storage capacity is effectively used. The data
replication factor and lifetime of the data are heavily dependent on how
actively the data are used and their reproducibility in the case of loss. The
policies defining whether to store data on disk or tape also depend on the
access patterns and expected lifetime of the data. Raw data from the detector,
for example, are always copied both to tape storage at CERN and one Tier-1
centre, because they are read infrequently and to ensure a negligible risk of
data loss (only one raw data file was lost since the start of ATLAS
operations). On the other hand, DAOD datasets used for analyses have at least
two copies on disk so those data are always available, but are not replicated
to tape because new versions are frequently produced and hence each version
has a limited lifetime.

The replication of new data to match the policies is handled by Rucio
subscriptions, which automatically create copies of data in the required
places. Archiving to tape of data produced on the Grid (HITS and AOD, and
reprocessed data AOD) is done through a separate asynchronous process that
delays the archival until one month after the data are produced, to avoid bad
data being written to tape that are then immediately deleted.

An important shift in the ATLAS data model in recent years is from viewing
tape as a pure archive or backup to be used in limited cases to a more active
use of tape as the main data store, and disk as a cache to which data are
staged from tape temporarily and then deleted. This change required detailed
investigation into how sites’ tape storage was organised and developments in
ATLAS services for optimally reading data back from tape. This work was
carried out under the auspices of the Data Carousel project [298] (see Section
7.3.3), which was successfully implemented during LS2. The amount of data
staged from tape increased from 20 PB in 2018 to 130 PB in 2021, without
disrupting writing to tape (which increased from 47 PB in 2018 to 75 PB in
2021).

Another important handle in controlling what data are cached on disk is the
automatic deletion of unused data from disk when there is an archival copy on
tape. This mechanism can be tuned to be more or less aggressive according to
how much cache space is available globally, which processing campaigns are
planned and so on.

The life cycle of data is controlled by the _lifetime model_ , through which
data are deleted from all centrally-managed storage a certain period of time
after they are produced. The lifetime depends on the type of data and their
expected use; following the examples above, raw data have an infinite lifetime
and DAOD data have a lifetime of six months. The lifetime may be extended in
two ways. When data are accessed (i.e. used as input to a PanDA job or
manually downloaded using Rucio), their lifetime is extended (usually by six
or twelve months). Alternatively, an analysis team may ask for an extension at
the point when data become eligible for deletion, for example if the team is
midway through a publication procedure and the data may be needed to recreate
results. The lifetime model has worked well to control the disk and tape space
used by ATLAS, allowing obsolete data to be easily removed to free up space
for more copies of actively used data. However, with the length of individual
ATLAS analysis efforts often stretching to many years, the amount of data that
is not active but was requested to be kept ‘just in case’ is steadily growing
and is now roughly 10 PB. Recent studies have shown that only a fraction of
these data are accessed after a lifetime extension request. Therefore, work is
ongoing and will continue throughout Run 3 to identify alternatives to keeping
such data pinned on disk, such as moving to ‘cold’ storage or streamlining
procedures to allow recreation of data on demand.

This lifecycle is applied to data on general ATLAS resources; some resources
exist outside of these rules. Analysis groups (e.g. the Standard Model or
Higgs group) have dedicated _group disk_ pools where data supporting ongoing
analyses can be stored without regard to the lifetime model. Similarly,
institutes have local group disk areas where data supporting the local data
analysis can be stored. Users or group space managers are responsible for
creating rules to transfer data to these disk areas and for deleting data from
them.

####  7.2.4 WLCG data challenges

In anticipation of HL-LHC data rates, the WLCG has decided to run increasingly
challenging stress tests of the data infrastructure, including disk, network,
and the tape systems. The plan is to run these tests every two years until
100% of the expected HL-LHC data rate is reached by 2027. Two target rates are
defined: the _minimal_ target, which is the rate expected with a rigid
hierarchical computing model, and the _flexible_ target, which is the rate
expected with flexible computing models allowing transfers across and between
all Tiers dynamically. The latter model is similar to the current computing
model.

In October 2021, the first disk and network challenge took place with the aim
of achieving a sustained overall transfer throughput of 10% of the expected
HL-LHC data rate, with the minimal target corresponding to the expected Run 3
rate [299]. Consequently, this data challenge also served as a commissioning
test for Run 3. The results are shown in Figure 31, showing that the minimal
target was easily met throughout the duration of the challenge and at peak
rate the ‘flexible’ target was achieved.

Figure 31: In purple, additional ‘Data Challenge’ traffic injected to reach
target rates during the October 2021 Data Challenge, along with the standard
operational traffic from the four LHC experiments.

Dedicated tape challenges focused on the capability of the Tier-0 and Tier-1
tape systems to handle the expected rate of Run 3 RAW data export, which is up
to 8 GB/s from all physics streams combined. These tape challenges were
successful, and these rates were met and even exceeded without any operational
incidents during Run 3 data taking. Different streams are distributed across
different sites, but all data within each stream are sent to a single site for
a single run. The largest stream is produced at a typical rate of 3.5 GB/s,
and the final tape challenge in March 2022 showed that all Tier-1 sites were
capable of handling this data rate.

In mid-2022, the planning for the follow-up data challenge has started, with a
projected date in early 2024. The major features to be tested are the new
token authentication mechanisms (see Section 7.2.5), software-defined
networks, and much-improved monitoring. Potential new dataflows introduced by
HPC centres (see Section 7.3.2) and dedicated analysis facilities (see Section
8.1.7) will be folded into the plan once their usage is more clearly
articulated within the community.

####  7.2.5 Further WLCG infrastructure changes during Run 3

A virtual organisation-based (VO-based) security architecture, using X.509
certificates, has been a reliable solution for authentication and
authorisation in ATLAS, but has also showed usability issues and required ad-
hoc services and libraries in the Grid middleware. The need to move beyond the
VO-based scheme was recognised as an important objective in WLCG
Authentication and Authorisation Infrastructure (AAI) [300], to overcome the
usability issues of the current AAI and embrace recent advancements in web
technologies. A token-based AAI was implemented in ATLAS using the Indigo
Identity and Access Management service [301], fully compliant with
OIDC/OAuth2.0 and capable of identity federations among scientific and
academic identity providers. The deployment of this new AAI infrastructure
across services is ongoing.

Until 2023, all WLCG experiments utilised the HEPSpec06 [302] benchmark for
accounting and pledges. However, this 32-bit benchmark is no longer a reliable
indicator of HEP workloads, whilst the underlying SPEC-CPU 2006 benchmark
[303] is no longer supported since 2018. Moreover, there is a coming need for
a benchmark that caters to non-x86 architectures such as ARM processors and
GPUs. To address these issues, a task force was established by the WLCG in
November 2020, to identify a new benchmark based on the current workloads of
the LHC experiments and a transition plan. A new benchmark, HEPScore, which
includes both x86 and ARM processors, was developed [304] and deployed in
April 2023. The experiments now use this new benchmark for resource requests,
and sites are expected to score new hardware purchases using this benchmark
instead of HEPSpec06. The benchmark is made up of a variety of workloads [305]
from all four LHC experiments and Belle II; the initial version is HEPScore23.
The transition to HEPScore23 will enable more accurate accounting and pledges,
providing a more realistic representation of HEP workloads. Furthermore,
HEPScore23 was normalised to the old HEPSpec06 benchmark, so that sites do not
have to re-benchmark existing systems, and only use it to measure newly
purchased hardware.

####  7.2.6 DDM R&D projects

In addition to the ongoing development work and data challenges, which come
with their own research and development communities, several distributed data
management-specific R&D projects were started: most importantly, integration
with commercial clouds, dynamic data handling, and distributed caching.

The first of these R&D projects, underway in Run 3, is the integration of
Rucio with commercial clouds, specifically Google Cloud [306], Amazon Web
Services [307], and Seal Storage Technologies [308]. The integration of such
cloud services is reaching a stage where these types of storage can be
included in the distributed data management system beyond simple
demonstrators. The main benefit is that funding agencies and institutes can
achieve more flexible storage installations (including, potentially, the
purchase of cloud-based resources), while Rucio takes care of abstracting the
peculiarities from the scientists.

A second R&D project is revamping the policies that drive ATLAS dataflows,
mostly given by experiment data agreements, processing requirements, and MoUs,
as well as operational and infrastructural needs. To address this, a working
group was formed to address dynamic data handling in a coherent and consistent
way. The working group will study and tune ATLAS dataflows, most importantly
placement of new data, rebalancing between data centres, deletion of obsolete
and unused data, and data replication both for production and analysis. The
eventual goal is to reduce workload execution time, reduce data access time
and make better use of available storage through improved placement, movement,
addition, and deletion of replicas of ATLAS data, under the hard constraint of
limited storage space.

Finally, a third R&D project deals with distributed caching to reduce wide-
area-network (WAN, site-to-site) traffic, reduce latency to the analysis
software, and eventually to increase CPU efficiency. This is an integration of
Rucio with the Xcache system [309] to assign processing jobs not only with the
common ‘job to data’ paradigm, but to allow a more flexible approach that
includes significant staging of data to sites with extra computing capacity.
During LS2 a prototype deployment was put in place. First observations have
already provided insights, including increased cache use and reduced WAN
traffic. During Run 3 this mechanism will be exercised with real analysis use
cases and, if proven successful, will be integrated into Rucio.

###  7.3 Workflow management

A large variety of workflows are required by the ATLAS experiment, including
data processing (and reprocessing), MC event generation, simulation and
reconstruction, and derivation data production for physics groups (see Section
4 for details). This is in addition to data analysis conducted by physics
groups and individual users. Figure 32 shows the number of running ATLAS jobs
since the beginning of 2022, for each of the different activities.

Figure 32: Running ATLAS jobs since the beginning of 2022, grouped by
activity.

To fulfil this workload demand, the ATLAS experiment uses more than 250
computer centres worldwide, primarily made up of the WLCG Grid sites described
in the previous section, but also including HPC centres and national,
academic, and commercial cloud computing resources. Figure 33 again shows the
number of running ATLAS jobs since the beginning of 2022, but this time
grouped by resource type. It can be seen that although the contribution from
the Grid dominates, as expected, there are many running jobs on opportunistic
resources such as HPCs and cloud, the latter contribution mainly coming from
Sim@P1232323Because originally Sim@P1 was managed using OpenStack, it was
accounted for as part of the cloud resources. Although this is no longer the
case, for historical consistency it remains in this accounting category. (see
Section 7.3.2). Further contributions are present from assigning dedicated
production tasks to both HPCs (‘hpc_special’) and clouds (‘cloud_special’),
where the latter is mainly made up of BOINC [21, 22] job submissions and jobs
running on the Tier-0 site. These additional resources are described in
Section 7.3.2.

Figure 33: Running ATLAS jobs since the beginning of 2022 grouped by resource
type.

####  7.3.1 A universal workflow management system for ATLAS

The processing, simulation and analysis of data from modern HEP experiments
requires the orchestration of multiple diverse computing facilities. This
section describes the methods developed and the approaches taken to arrive at
a universal workflow management system for ATLAS, ProdSys2–PanDA [310, 112],
which has been used since the beginning of 2014. New elements such as iDDS
[311] and Harvester [312] have since been developed and brought into
production before the beginning of LHC Run 3 to address the heterogeneity of
resources and the introduction of new data processing scenarios such as the
Data Carousel [298] (Section 7.3.3).

##### Design concept

When designing a suitable workflow management system for a distributed and
heterogeneous computing infrastructure, there are many aspects to consider.
The primary role of such a system is the management of the various production
and analysis workloads, as well as defining how the resources are to be used
and the application of appropriate fair–share policies. The system should also
work regardless of the type of infrastructure and level of heterogeneity and
be dynamic enough for possible changes in the available resources. The various
resources used and the associated access protocols, are described using CRIC
[297]. User identification and security considerations are handled via the use
of X.509 certificates, which are to be replaced by tokens during Run 3 as
described in Section 7.2.5.

After analysing the various classes of workflows, which are described in
Sections 4 and 5, the essential components of the production system were
identified and a logical data model built. The following entities are defined:

  * •

A _request_ is the upper level of abstraction, and is made of tasks of one
type. A typical example would be the (re)processing of all data for a certain
period or a collection of MC simulations with common parameters, which are to
be used together to compare to a given period of data. A request may comprise
multiple steps, where event generation is one step, for example. Each step on
each input dataset is represented as a separate task.

  * •

A _task_ is an entity for passing parameters to the payload submission system,
and is composed of jobs. The result of the task is a set (or several sets) of
files, organised in datasets, typically with one dataset per task per output
file type.

  * •

A _job_ is a single executable workload, where each task consists of one or
many jobs (up to ten thousand). A job is executed on a single Grid computing
element, opportunistic CPU or worker node. A job may have input data and
writes the result of the work to an output file or files.

The system architecture is designed to ensure continuous and optimal access of
the scientific community to computing resources, which is achieved with an
extensible layered architecture. Figure 34 shows the different levels of the
workflow management system, where the relationship between the relevant
entities is schematically described on the right [313]. The implementation of
these levels as part of the actual system, shown in the left of Figure 34, is
described in the following.

Figure 34: The several levels of the ATLAS workflow management system. Figure
from Ref. [313].

##### ProdSys2 web user interface

The ProdSys2 [310] Web User Interface (Web-UI) is used by the ATLAS production
managers to interact with the task management system. A production manager
represents a group activity in the ATLAS experiment and defines workflows for
their group: Data (re)processing, MC production (event generation, simulation
and reconstruction), and Derivation (DAOD) production. Each activity has its
own workflow and specific task requirements.

The heart of a work process with the Web-UI is the request. The production
manager creates a request using the Web-UI, and defines its parameters and
input datasets. For each input dataset, a sequential set of steps to be
executed is defined, which is translated by the system into a sequence of
tasks to be executed. A single step might correspond to event generation
alone, with detector simulation as a second step following it. The data model
features the logical division of the request into horizontal _slices_ defining
all the steps beginning from a single input dataset. Each slice may comprise
several sequential steps in one production workflow, each executed as a task,
which itself is divided into jobs.

The Web-UI is also used to monitor the progress of running tasks and adjust
them as required. Active bookkeeping is provided, metadata is stored,
including user-defined tags, and the aggregation of task statistics is done,
which may be used for the fine-tuning of running and historical task analysis.
Advanced error handling and reporting is included to help quickly understand
the root of any problem and to fix it by redefining and resubmitting the task.
The chaining together of slices in separate requests is also possible and
successfully used, for example when connecting MC event generation, simulation
and reconstruction together with derivation production.

##### DEfT

The Database Engine for Tasks (DEfT) is a top-level sub-system and is the
engine beneath the Web-UI. DEfT accepts task requests via a dedicated user
interface or from prepared lists (e.g. a text file, or Google or Excel
document). DEfT processes requests and is responsible for the formation of
processing steps, tasks, and input data and parameters.

ProdSys2 sets default parameters via DEfT, for example memory and CPU limits,
which are often not defined by the user. ATLAS has many default parameters for
complex workflows, which may be viewed and changed via a special ProdSys2
interface. After parameters are set, the system checks for their consistency
and compatibility. For example, it is verified that the input dataset has
enough events or that the parameters defined in the task are valid for the
ATLAS software release to be used. A further check is performed to see if
similar tasks were defined and successfully executed, to avoid event
duplication. During task definition, if ProdSys2 detects that some of the
events in an input dataset were already used for a given configuration, DEfT
sets an offset (in files when input datasets are used, or in event numbers and
random number seeds for steps without input data like event generation) and
the new task uses only unique events.

##### JEDI

The Job Execution and Definition Interface, or JEDI, is a middle-level sub-
system that receives formalised job descriptions from DEfT. JEDI dynamically
determines the number of jobs for each task and is responsible for launching
and executing individual jobs. JEDI verifies information about the data via
Rucio and about the job queues via CRIC. A common database is used by JEDI and
PanDA (see Section 7.3.1) to store information about the status of jobs and
tasks.

#####  PanDA

PanDA [112] is the engine of the underlying system and the most complex layer.
It determines which resources are used when for each of the jobs, receives
information from Pilot jobs (see below) and the CRIC information system, and
manages the progress of jobs. The various PanDA components are described
below.

A database of jobs and tasks is used system-wide, storing comprehensive static
and dynamic information and meta information about all jobs and tasks defined,
running, and completed in the system, including the history of their execution
and errors that have occurred in the process.

_Pilot_ jobs are used to collect information about the state of computing
resources and requirements of a task. Jobs are submitted to successfully
activated and verified Pilots by a PanDA server based on resource selection
criteria. Late binding of jobs to the execution location prevents delays and
failures and maximises the flexibility of resource allocation for a job, based
on the dynamic state of processing resources and job priorities. A Pilot is
the main ‘isolating layer’ for WFMS, encapsulating complex heterogeneous
environments and the Grid interfaces and tools with which the WFMS interacts.
Pilot jobs also serve to identify the resource requirements of a task and
adjust subsequent job definitions, for example to request additional memory.
In the case of failures, pilots ensure that only a small fraction of the task
runs, and the task is marked as failed before wasting significant resources.

The _Intelligent Data Delivery Service_ (iDDS) [311] is an additional layer
developed to orchestrate workflow management and distributed data management
systems to optimise resource usage in various workflows. The input data to a
task are dynamically transformed, so that the data pre-processing, delivery,
and main processing in each workflow are decoupled, which allows them to run
asynchronously. iDDS was introduced in 2020 to address inefficiencies in the
Data Carousel (see Section 7.3.3), which previously required many input events
due to constraints in the workflow, creating delays in bulk reprocessing
campaigns. iDDS propagates the detailed information about the input data
status from Rucio to JEDI, allowing the incremental release of tasks so that
processing can begin even if input data are only partially staged-in. iDDS is
now also used in new workflows such as some AI/ML workflows [314, 315], and
supports several workflow definition languages.

_Harvester_ [312] mediates the control and information flow between PanDA and
the resources to enable more intelligent workload management and dynamic
resource provisioning based on detailed knowledge of resource capabilities and
their real-time state. Harvester was designed around a modular structure to
separate core functions and resource-specific plugins, simplifying the
operation with heterogeneous resources and providing a uniform monitoring
view.

PanDA also includes functionality to automatically retry or re-broker jobs.
Sometimes, site failures can be identified and jobs can be re-directed to an
alternative site that has a copy of the input data. Additionally, a catalogue
of known errors is kept, defining rules for the number of retry attempts that
should be made when a specific error is encountered. With this functionality,
jobs that are suffering from system or site failures can be retried and moved
around the Grid, while jobs with clear failures that will not converge can be
stopped before significant resources are consumed.

The concept of _global shares_ is used to set the amount of resources
available instantaneously for a certain activity, for example MC simulation,
as a fraction of the total amount of resources available to ATLAS. Global
shares have recently been updated to be measured in the new benchmark,
HEPScore23 (see Section 7.2.5); previously, they were measured in HEPSpec06,
the previous standard unit for core-power benchmarking used in HEP community.
A nested structure of global shares is used, where siblings have the
preference to occupy unused shares, before the unused share is taken by higher
levels. For example, 75% of the resources might be assigned to Production
(Level 1 share); of those, 25% might be assigned to Simulation (Level 2
share); and of those, 20% might be assigned to the MC16 simulation campaign
(Level 3 share). If MC16 does not fully occupy its fraction of the resources,
idle resources would be preferentially reallocated first to run other
simulation tasks, then to run other production tasks, and then to run any
available tasks. Within a share, jobs and tasks are assigned priorities,
allowing urgent tasks to run quickly, ahead of large, low-priority tasks.
Tasks and jobs are assigned a global share at creation time, and jobs are
sorted by priority and creation time within a particular global share.

Figure 35: Data flows in the BigPanDA monitoring system.

The _BigPanDA_ monitoring system [251] is a Django-based [316] web application
consisting of a set of standard monitoring pages and separate modules that can
be plugged in or removed on demand. Data are collected from a variety of
different sources including the PanDA database, the CRIC information system,
payload logs of jobs from Rucio, PanDA/JEDI logs from the ElasticSearch
cluster, and the Monit Grafana instance where accounting data are available
(see Section 7.4.1). The data flow of these sources is illustrated in Figure
35. More information about monitoring is provided in Section 7.4.

####  7.3.2 Opportunistic resources

One of the challenges of the ATLAS production system from the beginning has
been the integration of a wide variety of computing resources whilst at the
same time hiding the heterogeneity from the users. The initial computing model
was based on the Grid [5], developed for the LHC experiments and realised via
homogeneous resources situated in the computing centres distributed worldwide.
This homogenisation was based on a list of software and hardware requirements
to be met by the centres, where any remaining differences were absorbed by the
associated Grid middleware.

More recently, new and often more exotic types of resources have become
available to ATLAS, requiring adaptation and evolution of the production
system to integrate them. Among these resources are the Tier-0 site when not
in use for operations (see Section 7.1), HPCs, various cloud resource
providers and other _opportunistic resources_ ,242424The name ‘opportunistic’
is meant to distinguish these resources from those upon which the experiment
relies to deliver its primary physics programme. The WLCG Grid sites pledge
resources annually to the experiment to satisfy that physics programme.
Recently, because of the scale of these opportunistic resources and their
consistent availability, some amount of these resources were accounted for in
the core resource needs, to more fairly request the necessary resources from
the Grid sites. Another term for these resources that is in use, therefore, is
_unpledged_ , to be distinguished from _pledged_ resources at WLCG Grid sites
and _beyond-pledge_ resources that Grid sites deliver in addition to their
pledged resources. such as those running volunteer computing initiatives like
‘ATLAS@Home’ [317, 318].

Such resources may not be dedicated specifically to ATLAS and therefore miss
critical component infrastructure, are often constrained in their use in that
they are optimal only for some workflows, and may not be permanently
available, depending on the allocated hours or budget. A measure of the
success of the integration of these special resources is the production
system’s ability to effectively broker jobs to them, dispatching the
appropriate job type, and to detect changes in resource availability in real
time. These various additional resources are described in the following.

##### Use of high performance computing resources

ATLAS has a long history of exploiting the potential of High Performance
Computing (HPC) centres to provide additional computing cores for ATLAS
workloads. Employing HPCs presents significant challenges, not least that
access to such systems is usually strictly limited, so that connections to the
nodes themselves are heavily restricted and installation of local software is
tightly controlled. Furthermore, the CPU architecture may not be suitable for
ATLAS software, the expected job structure and number may be atypical of an
ATLAS workflow, and network issues may arise due to geographic location.

Nevertheless, ATLAS has successfully used a series of HPCs for more than a
decade, many of which appeared or continue to appear near the top of the
Top500 list of supercomputers [319]. These included Cori [320] at the National
Energy Research Scientific Computing Center (NERSC), Titan [321] at Oak Ridge
National Laboratory, both in the USA, SuperMUC-NG [322] at the Leibniz
Supercomputing Centre in Germany, and Toubkal [323] at the African
Supercomputing Centre in Morocco. Typically, these machines ran dedicated
ATLAS Full Simulation tasks, delivered to the site with all required inputs
and database information, allowing them to run in an isolated way via the use
of edge services at each machine. In the case of Titan, this was mainly used
in back-filling mode, where ATLAS exploited spare cores not used by existing
tasks to run ATLAS Fast Simulation jobs.

More recently, Perlmutter [324] at NERSC, Karolina [325] at the IT4Innovations
National Supercomputing Center in Czechia and in particular Vega [326] at the
Institute of Information Science in Slovenia have provided significant
additional computing resources to ATLAS. Karolina and Vega are part of the
EuroHPC project [327]. The more widespread use and acceptance of native CVMFS
and the adoption of containerised workflows by ATLAS has enabled a more
generic use model of HPCs for ATLAS. All production workflows now run on most
HPCs, allowing such machines to be fully integrated as additional resources,
essentially no different to a standard Grid site. This has meant reduced
operational load, more flexibility and periods where the number of available
CPU cores available to ATLAS has more doubled relative to the WLCG pledged
resources. Some HPC centres are sufficiently well integrated that they can run
as part of standard WLCG sites, like MareNostrum [328] at the Barcelona
Supercomputing Center in Spain. The large-scale use of these resources is
expected to continue throughout Run 3 and beyond.

##### Opportunistic use of the HLT farm: Sim@P1

The ATLAS Trigger and Data Acquisition (TDAQ) high-level trigger (HLT)
computing farm [329] is a critical part of online data taking for the ATLAS
experiment. It facilitates the final selection of events to be stored for
further physics analysis based on full event reconstruction [18]. It consists
of 145,000 computing cores spread amongst three different flavours of
hypervisors with RAM between 0.9 and 1.1 GB/core. The nodes are located within
the private network of the ATLAS experiment, which is separated from the
internet and are accessible to ATLAS Distributed Computing via a VLAN on a
data link layer through two 100 Gbps connections. Networking and RAM per core
are the two limiting factors that define the type of workflows that can be run
efficiently on this resource.

Since 2013, ATLAS has been running MC full simulation in longer periods of
inactivity of the LHC within the _Simulation at Point One_ (Sim@P1) project on
these resources [330]. Recently it was shown that they can also successfully
be used for other workflows, such as MC reconstruction, whenever necessary,
albeit with lower efficiency.

Current studies [331] show that thanks to improvements in the new ATLAS fast
simulation, AtlFast3 (see Section 4.2.2), and optimisation of the job
submission configuration, ATLAS is able to process many simulated events
within one hour. This speed allows Sim@P1 to process simulation events between
LHC Run 3 fills when the inter-fill break lasts longer than one hour.
Furthermore, with the move of the project to Kubernetes, it may be possible to
run Sim@P1 in parallel to the trigger whenever part of the compute resources
are not needed by the ATLAS online system.

##### Volunteer computing: ATLAS@Home

ATLAS@Home [317, 318] is a volunteer computing project based on BOINC [21, 22]
for utilising the free CPU cycles of volunteer computers around the world for
MC simulation of the ATLAS experiment. ATLAS@Home was one of the first
volunteer computing initiatives in HEP.

This resource is fully integrated into the Grid computing infrastructure of
the ATLAS experiment through PanDA. Detector simulation tasks assigned to this
resource are assigned to a single PanDA queue that covers all the volunteer
resources. The jobs from the tasks are pulled by an ARC Control Tower [332],
and then the control tower submits the jobs to an ARC compute element, which
forwards the jobs to the BOINC server. The ARC Control Tower and Compute
Element handle all interactions with PanDA and Grid services and hence no
credentials are required on the volunteer hosts. BOINC clients request jobs
from the BOINC server and process them whenever they have idle CPU cycles.

In addition to members of the public, ATLAS@Home was used in a back-filling
mode at several Grid sites to make full use of CPU not normally used due to
job or scheduling inefficiencies [333]. BOINC has become a very reliable
computing resource for the ATLAS experiment; major simulation tasks are run,
and it contributes about 1% of the CPU available to ATLAS computing daily.

####  7.3.3 The data carousel

The evolution of the computing facilities and the way storage is organised and
consolidated will play a key role in how the LHC experiments will address the
possible shortage of resources in the HL-LHC era. In particular, to reduce
storage costs to the experiments at a time when the data volume is expected to
significantly increase, it is anticipated that the use of tape may be
expanded. To address this data handling challenge, the Data Carousel project
[298] was established to study the feasibility of directly receiving input
data from tape for various ATLAS workflows.

The Data Carousel is the result of a successful orchestration between the
workflow management system ProdSys2–PanDA, the distributed data management
system Rucio, and the tape services at the Tier-1 sites. It enables a bulk
production campaign, with input data resident on tape, to be executed by
staging and promptly processing a sliding window of a fraction of the input
onto buffer disk such that only a percentage of the data is pinned on disk at
any one time. The production system follows site-specific preferred staging
profiles, provided by the Tier-0 and each Tier-1 site, which define the upper
and lower limits of concurrent staging requests that can be handled. The
typical staging pattern over a two-week period can be seen in Figure 36, where
the peak data staging performance reaches over 15 GB/s and the colours
represent different Tier-1 sites.

Staging inputs from tape rather than having them resident on (more expensive)
disk allows the dedication of significant disk space for more popular data
such as DAODs. Without Data Carousel, a large fraction of HITS or AODs would
need to be kept permanently on disk to run regular processing campaigns. In
the case of data reprocessing campaigns, this would require complete (and
large) RAW datasets to be pre-staged onto disk before they could begin. The
Data Carousel model therefore brings significant disk space savings for ATLAS,
where, for example, less than half of the AODs are on disk at any one time.

To promptly process the staged data and to improve turnaround time, iDDS (see
Section 7.3.1), was developed and integrated with the existing system. The
collaboration between the Data Carousel and iDDS R&D projects is an excellent
example of early HL-LHC R&D delivery and commissioning for LHC Run 3.

Figure 36: The data-staging pattern over a two-week period corresponding to
the Data Challenge (see Section 7.2.4) across many Tier-1 sites by the Data
Carousel.

####  7.3.4 WFMS R&D projects

In addition to the distributed data management research and development
projects described in Section 7.2.6, several areas related to ATLAS workflow
management are being investigated in further R&D initiatives.

The integration of additional HPC resources is a priority for ATLAS, so that
the experiment can benefit from new machines in the US, in Europe via the
EuroHPC project, or anywhere they become available. Whilst an easier
integration than a decade ago is possible thanks to the evolution described in
Section 7.3.2, each new machine still presents a different challenge.
Discussions are continuously underway to identify possible new HPCs of
interest, whether in production or in planning, to understand if they might
provide significant resources to the experiment.

An evaluation of commercial cloud resources complements the HPC effort, and
includes the commissioning of an ATLAS site at Google, together with a Total
Cost of Ownership analysis. Both HPC and commercial cloud resources provide an
opportunity to investigate and integrate non-x86 resources before such
technologies are provided by WLCG pledged resources. This includes both GPUs
and ARM architectures, which are expected to be more prevalent in the future
(see Sections 7.2.5 and 9).

Related to the dynamic data handling R&D project described in Section 7.2.6,
the future creation, storage and lifetime of DAODs is under investigation. The
benefits and consequences of the adoption of a model where DAODs are recreated
on demand will be evaluated.

###  7.4 Data monitoring and analytics

The distributed computing systems produce a wealth of data that can be used to
monitor the health of the many servers and services, and at the same time
investigate the ways collaboration members use these services and find ways to
optimise the overall resource usage.

####  7.4.1 ADC monitoring

The monitoring infrastructure collects information from the workflow
management system PanDA [112] and the data management system Rucio [275],
complements it with static information about site configurations, aggregates
it into time bins and stores it in appropriate data storage systems.

Figure 37 shows a schema of the data flow through the main monitoring system
based on Kafka [334], using the infrastructure provided by the CERN IT Monit
team [335, 336]. Several data sources are queried or send periodically
information to the central monitoring system; here the information is
processed and aggregated using Apache Spark [337] jobs, and finally the
records are stored in ElasticSearch [280], InfluxDB [338] or HDFS [339]
systems (depending on their type). These data stores can be used as sources of
data for visualisation in Grafana [340] or Kibana [341] dashboards, or treated
interactively using the CERN Swan suite [342].

Figure 37: The information flow through the monitoring infrastructure provided
by the CERN IT Monit team. Figure from Ref. [336].

####  7.4.2 Dashboards

The same data may be used to feed dashboards that have several purposes. For
example, information about successful and failed jobs grouped in time bins of
10 minutes and displayed over a few hours can be useful to detect faults in
software or site operations, or grouped in bins of one week and displayed over
several years can be used to prepare accounting reports. Several dashboards
[343] were developed to cover the main needs, from short-term operation
monitoring to long-term accounting, for the main ATLAS systems, PanDA and
Rucio, as well as for the operation of WLCG sites that support ATLAS and data
transfers between them. Auxiliary dashboards cover specific needs and other
smaller systems. All dashboards can be customised in real time, changing the
time range, the time binning, the quantities to be displayed and the data
grouping. Figure 38 shows, as an example, a screenshot of the top part of the
job accounting dashboard for a period of one month, in 1-hour bins, grouped by
ATLAS activity; the pull-down menus at the top of the page allow filtering the
data and customising the displays, and many other charts are displayed in the
lower parts of the page.

Figure 38: An example of a customisable dashboard showing the number of jobs
and CPU resources used during a set time period.

The BigPanDA monitoring suite [251] is a web application that provides various
processing capabilities and representations of the PanDA system state.
Analysing hundreds of millions of computation entities, such as tasks or jobs,
BigPanDA monitoring builds reports with several scales and levels of
abstraction in real time. The reports allow users to understand specific
failures or observe the broad picture by tracking the computation performance
or the progress of a whole production campaign. BigPanDA is a core component
of the PanDA system, commissioned in the middle of 2014, and is now the
primary source of information for ATLAS users about the state of their
computations and a key source of information for shifters, site operators and
production managers.

####  7.4.3 ADC analytics

An ADC Analytics coordination activity was started to better link the diverse
efforts in ADC, raise awareness of the existing projects, and share experience
with tools and available data. In addition, the effort aims to improve the
link between users (e.g. physicists doing data analysis) and developers of
analytics solutions. Current activities cover a wide range of topics,
including optimising the use of FroNTier, anomaly detection in networks, data
popularity, studies of access patterns at the file and tree level, prediction
of the time to completion of tasks, development toward alerts for managers of
operations, anomaly detection in data management, and the composition of
dashboards that allow effective analysis of operational incidents and issues.

####  7.4.4 ElasticSearch clusters for analytics

The ElasticSearch infrastructures at CERN and at the University of Chicago
(UC) form the backbone of many monitoring and analytics activities in ADC. At
CERN, the cluster is a 20-core two-node Openstack Kubernetes cluster that
collects WFMS data and ships it to the ElasticSearch infrastructure at UC. The
29-node cluster at UC serves as the infrastructure for the ‘ATLAS Alarm and
Alert System’ and provides resources to the new ADC analytics platform [344].
This platform will offer a common front-end to different ML solutions in
analytics and monitoring and serve as a registry for analytics projects.

New data sources were added to ElasticSearch over the years. For example, xAOD
data-access monitoring data enables the study of access patterns in great
detail, with the potential to speed up data access by tuning the configuration
of |TTreeCache| in real time, based on the profile of similar jobs. To keep
Squid monitoring data (see Section 6.2.2) with high precision for a longer
time period and allow analytics tasks to combine the data with other datasets,
the monitoring data is now sent via Logstash to ElasticSearch. To discover
unused files in disk storage, access information was exported from dCache
[345] metadata databases to the ElasticSearch infrastructure. Two sites are
currently part of the first implementation of this project.

##  8 Analysis tools, event displays, and tutorials

Performing physics analysis is the final step of the data and MC processing
chain, with the aim to publish groundbreaking results using stringent
scientific methods. A variety of tools are available that assist physicists
with their analyses. Among these, event displays are special tools for
visualizing recorded or simulated events and the detector for analysis
purposes, and they can also be useful for visual inspection and
troubleshooting. This section describes these tools and also the efforts to
train physicists to use them.

###  8.1 Analysis tools

Analysis tools enable the data processing steps from centrally provided
derivation formats for analysis to final results for publication and public
dissemination. Many analyses are carried out in ATLAS, with several hundred
ongoing at any one time, which cover a broad range of topics and focus on a
wide range of signatures. As a result, analysis tools and workflows are
necessarily diverse, flexible, and focused on analysis-specific needs. This
section covers commonly used tools and methods and highlights examples of how
some analysis-specific needs are accommodated.

Analysis workflows in ATLAS often entail the following steps, described in
more detail below:

  1. 1.

Processing from central derivations, which may include applying corrections to
analysis objects (calibrations), steps towards data reduction, and systematic
variations.

  2. 2.

Analysis design and development, which may include the creation of additional
intermediate data formats and processing steps.

  3. 3.

Statistical analysis to quantify the correspondence between experimental
observations and theoretical predictions.

  4. 4.

Creation of data products (e.g. figures, tables, likelihood functions) for
publications and other publicly available material, analysis preservation, and
reinterpretation.

Carrying out an analysis typically relies on a combination of centrally
provided code, community supported tools, and analysis-specific software. In
addition, several computing and storage resources are available for processing
of analysis data, ranging from the Grid to local machines and personal
computers.

####  8.1.1 Analysis data formats and workflows

The starting point for an analysis workflow is the central derivation formats
for analysis, described in Section 4.5. The model for Run 3 is characterised
by the introduction of a new common central derivation format for analysis,
`DAOD_PHYS`, whose goal is to support most analyses in Run 3 [346]. This
format contains information for all physics objects (muons, electrons,
photons, hadronically-decaying $\tau$-leptons, jets including identification
and flavour tagging, and missing transverse momentum), which makes it possible