Adopt hardware abstraction layer: The current R&D version of the LST
algorithms were written using CUDA; it cannot be run on non-NVIDIA resources
such as those available on DOE’s (AMD-based) Frontier, currently the fastest
supercomputer in the world. A hardware abstraction library will be required
for an algorithm to survive through the HL-LHC era.

  * •

Increasing parallelism: The current LST algorithms are memory-hungry compared
to what’s available on modern GPUs (and what might be affordable during the
HL-LHC era). The amount of work assigned per GPU device - and hence the total
parallelism and speedup of the algorithm - is limited by the available memory,
leaving GPU cores idle. An initial goal of the next phase is to reduce the
memory footprint to fully leverage the available silicon.

Machine Learning approaches to tracking

Funding Scenarios: High

Description: Unlike traditional tracking approaches, a GNN pipeline scales
linearly, not quadratically, with data density – making it an interesting
option for the HL-LHC environment. Various research efforts have established
pipelines for researching GNN architectures and training approaches applied to
different tracking algorithms. While these have not caught up to the technical
(speed) or physics (accuracy) performance relative to the highly-tuned
existing algorithms, they are rapidly advancing and potential remains to help
close the raw resource gap for HL-LHC.

Current and Potential Future Activities:

  * •

More Compact GNNs: Evaluating novel GNN techniques and have focused partially
on the inner detector (pixel-based) tracking. Small and thus quick to evaluate
GNN architectures are promising but need further research to reach adequate
physics performance. One important use case is the online system, where
computational speed is more important than precision.

Defining tracking pipelines for ATLAS and CMS

Funding Scenarios: Medium, High

Description: Individual algorithms are insufficient; they must be combined
into full tracking pipelines. The performance challenges for this process are
in the data exchange at algorithm boundaries as well as balancing overall
physics performance with computational costs.

Current and Potential Future Activities:

  * •

Experiments are in the process of defining tracking pipelines over the next
five or so years. For example, a likely vision for CMS tracking in HL-LHC is
one that begins with pixel seeds created by Patatrack, then unpacking /
clustering of the outer tracker information and matching outer tracker hits
using LST. The combination of these two could be done on an accelerator which
avoids the costly moving data between GPU memory and main memory. After these
steps, a more traditional Kalman filter approach via mkFit would then be used
to complete the track finding and do the final track fitting. This setup would
be appropriate for both online and offline environments and they could meet
the differing goals of the two applications through different configurations.
Piecing together such a pipeline requires active involvement of algorithm
developers to properly configure and optimize it.

#### Impact and Success Criteria

Investment into the research of techniques for pattern recognition algorithms
– specifically charged-particle tracking algorithms – will enable a tracking
approach that meets the goals of the HL-LHC scientific program; perform within
the resource budget used for Run 3 tracking; and be flexible enough to exploit
evolving computing hardware through the HL-LHC era. Primary success metrics
are related to achieving superior technical performance and delivering
superior science performance relative to state-of-the-art algorithms and other
R&D approaches:

  * •

Performance: Both online and offline, the critical metric is event throughput
per unit (compute) cost. Given current performance estimates, a factor of two
reduction in the full tracking pipeline would translate into a 25% reduction
in compute for CMS at the HL-LHC. Large factors have already been demonstrated
in individual tracking algorithms through ongoing research. Now developments
must realize these gains in practice. In the next phase of IRIS-HEP, our
research aims to achieve factors of two to three in speedup of critical
tracking algorithms in both the offline and trigger environments.

  * •

Correctness: Physics performance – computing the “correct” result for the
underlying inputs – is critical. The HL-LHC program sets a minimum threshold
of algorithm’s physics performance to have the desired scientific reach. This
is particularly critical as tracking underlies many other reconstruction
components, meaning that deficiencies will propagate throughout the physics
data formats for analysis. Given the excellent physics performance of current
algorithms, the primary goal for this metric is to maintain current levels
while focusing on improving potential gaps. Gaps foreseen include tracking
efficiency in particularly dense environments (eg, in high-$p_{t}$ jets), for
highly displaced tracks (eg, from long-lived particles), and to further reduce
the level of fake tracks (tracks “found” by the algorithm that were not part
of the actual physics).

Tracking code written but not adopted by the experiment has minimal impact.
Unlike the analysis environment, where a variety of approaches are taken by
different groups, each HL-LHC experiment can only afford to run one set of the
most resource-intensive tracking algorithms online and offline. The next five
years will include a long process of evaluation and decision making to define
the basis for HL-LHC tracking within each experiment. This process must be
tracked in order to facilitate adoption of promising research directions and
to cut short those that will not be adopted; the structure and long-term view
of an institute is critical for the desired impact.

Research in algorithms for event reconstruction and trigger applications build
upon numerous previous and still on-going NSF investments and serve as an
excellent training ground for early-career HEP researchers. For example, three
postdocs in this area who were associated to IRIS-HEP have since taken faculty
positions: Philip Chang is now faculty at the University of Florida (worked on
LST tracking at UCSD); Xiaocong Ai is a professor at Zhengzhou University
(worked on ACTS at UC, Berkeley) and Dylan Rankin is a professor at
Pennsylvania University (worked on hadron calorimetry reconstruction
algorithms on accelerators at MIT).

HL-LHC Computing Gap Impact: Research in Reconstruction and Trigger Algorithms
impacts gaps G1 (raw resource usage) and G4 (Sustainability). As a major
consumer of resources - estimated to be a majority consumer in the case of CMS
- investment in this area has the largest impact in closing the resource gap.
To be sustainable, implementations must be both flexible and adaptable across
hardware platforms and generations. High-energy physics depends on being able
to make effective use of both specialized high-performance (online) and
commodity (offline) hardware as well as high-performance computing. Today this
means having codes engineered well enough to be adapted to new hardware
architectures either through an interoperability layer or by other means. As
the LHC community has no control over the hardware market, and only indirect
control over the hardware architectures at the distributed resources, tracking
algorithms must design in mechanisms to support hardware architectures most
prevalent during the HL-LHC era. HEP detector designs will continue to evolve
beyond the initial HL-LHC phase. In the energy-frontier, there is R&D for both
future colliders and so-called phase-3 LHC upgrades; the software should be
sufficiently adaptable to be used for these or other future experiments. While
achieving the ultimate performance from tracking algorithms in a specific
detector geometry typically requires detailed tuning, algorithms which seek
applicability beyond a single experiments must demonstrate how easily they can
be used by another and whether there is any trade off in performance.

Success Criteria - Milestones & Deliverables

D2.1 ATLAS demonstrators (2024,Q2). The ATLAS experiment is currently defining
a list of demonstrator projects. Selected demonstrators need to be released by
Q2 2024 after which time they will be evaluated by the experiment for physics
performance, technical performance and overall readiness.

D2.2 CDR/TDR documents of experiments (2024). Both experiments have a timeline
to prepare documents describing their software and computing model for HL-LHC
and to submit documents to support the LHCC review of the WLCG. Performance
and readiness of potential tracking algorithms is an important component of
these documents.

D2.3 Downselect and validation by experiments (2026 or later). Transitions
from research to evaluation and validation by experiments must be complete for
final adoption by experiments. Timelines will be established closer to the
start of HL-LHC operations.

Success Criteria - Metrics:

M2.1. Number of deployed tracking algorithm components that meet or exceed the
performance requirements of the HL-LHC science program.

M2.2. Number of deployed tracking algorithm components that achieve factors of
two or more reduction in resource needs for HL-LHC in either the offline and
trigger environments.

M2.3. Number of deployed tracking algorithm components that are able to run
efficiency with abstraction libraries as adopted by experimental frameworks.

###  5.3 Translational AI

Cutting-edge research in artificial intelligence and machine learning – “AI”
for short – directly benefits the physics program of the HL-LHC. In many ways,
AI is transforming the way field does physics. A dedicated effort in
Translational AI will increase adoption of promising new techniques and reduce
the time and effort needed to deploy these solutions in the experiments. This
would significantly impact all aspects of HL-LHC physics, including trigger,
reconstruction, simulation, and analysis.

HEP has a long history of using AI and has been very active in embracing and
contributing to modern AI research. The field has developed experts in AI and
is beginning to see wider adoption of the techniques by non-experts.

New AI techniques are able to work with the complex, low-level data, which
generally enhances performance or sensitivity compared to traditional
approaches. Similarly, new AI techniques targeting simulation, multivaraite
unfolding, and anomaly detection are providing qualitatively new capabilities
that can change the HL-LHC physics program in more fundamental ways.

Exploratory AI R&D is often done outside of experiments. This work occurs in
the context of other NSF projects like IAIFI [49] and A3D3 [50], or in small
research teams, or is explicitly funded as part of experimentalist’s base
physics research grants. There are a number of barriers to the adoption of
this research in the experiments. For example, the papers written by AI
researchers are often not easily translated into the HEP context without the
aid of expertise in both fields - making them accessible only to the ML
experts within the field. Even if the papers are accessible, the
implementation of the new techniques require further development to be adapted
to the experiment’s context. Finally, the integration of ML techniques into an
experiment’s workflow is often accomplished with bespoke solutions, and often
different groups within one experiment will approach this differently.

The faster the community can absorb new AI research and translate it into use
by experiments, the more impact it will have on the physics program.

Ad-hoc collaborations and communication between AI research groups and
experiments have formed around collaborations between AI and particle physics
researchers. These links have grown a small, but active, group of AI experts
in particle physics. The communication does run in both directions: as
particle physics provides opportunities for use-inspired research in AI, these
collaborations do influence AI research resulting in work that is well-matched
to the field’s problems. This healthy communication must be maintained and
strengthened as the field must find ways to bring advanced techniques to a
broader audience. Graph Neural Networks (GNNs) are an illustrative example.
This type of AI model is particularly well suited to our problem space. A
multidisciplinary team including physicists and AI researchers at Deep Mind
have prepared a review article and position piece outlining this development
[51]. GNNs are now used for bottom-quark identification [52], replacing an
RNN-based solution, and are being explored for tracking, jet reconstruction,
and tagging. However, there are many advances remaining before GNNs could be a
solution for tracking (see Section 5.2).

Coordination is needed to guide the evolution from this period of rapid R&D,
prototyping, and bespoke solutions for deployment to a more mature and
established set of practices for ML in various contexts (e.g. trigger,
reconstruction, simulation, analysis). US funding agencies have invested
resources in making foundational progress in AI. This new strategic area will
improve the research deployment pipeline, help to further normalize the use of
ML, and provide a set of best practices that can be used by all researchers,
not just experts.

#### Specific Challenges and Opportunities

The challenges and opportunities that will help transform how ML is used at
the HL-LHC can be split into three broad categories.

##### Bringing research from the AI Community to the HL-LHC:

Most AI research happens outside the experiments. For example, a team of
Computer Science or Statistics researchers will develop a new technique.
Another common pattern is that of a small team, including some people from the
HL-LHC community and some people from CS, will use greatly simplified detector
data to publish a paper directly addressing problems in the field. The new
techniques are not easily incorporated into the experiment. Currently, bespoke
solutions are employed for development, training, and deployment. Worse, the
target environments - trigger, reconstruction, and analysis - all require
different solutions for inference. There is an opportunity to work with the AI
research community and the particle physics community to build tools that span
the domains and make the inclusion of new AI techniques more straightforward.

The NSF and DOE fund a large amount of foundational AI research. Two NSF
institutes, IAIFI and A3D3, are good examples of this work. IRIS-HEP’s broad
reach via its intellectual hub role and work on Analysis Facilities and
training infrastructure provide opportunities for bringing this research into
the field.

##### Improving HL-LHC’s interaction with the AI Research Community:

Our problem space is interesting to AI researchers. Delivering realistic data
to the researchers in a way that a non-domain expert can understand is not
easy, however. This is further complicated by some experiments not wanting to
release their simulation data openly. Finally, it is desirable to use
experiment-agnostic data when collaborating with AI researchers. A toolchain
exists to generate experiment-agnostic data [53], but it does not fully
represent the complexity of an HL-LHC detector, and it writes data in a field-
specific format (ROOT). Improvements to these tools and data formats will make
collaboration simpler. With this work and a better understanding of the
challenge and benchmarking tool set, the community can participate in public
challenges more effectively and collaborate more easily with non-HEP AI
research teams.

##### Sustainable AI at the HL-LHC:

The AI landscape is evolving rapidly. New techniques replace old ones. People
move from one task to another or leave an experiment entirely. As AI is used
more and more in reconstruction and the trigger, it becomes important to have
a process in place to maintain it. The data streaming from the detector will
evolve over the course of the HL-LHC (expected to run more than a decade) -
due to changes in the position of the detector components or due to radiation
damage, for example. The retraining of such a mission-critical ML algorithm
cannot be dependent on a (former) student’s laptop configuration. The field
needs to develop infrastructure and best practices around the long-term
sustainability of ML algorithms in reconstruction and the trigger.

#### Current Approaches and Development Roadmap

Translational AI will be a new area for IRIS-HEP, and there is already a
tremendous amount of work on AI in the field. IRIS-HEP has at least two places
it can offer a unique contribution that play to its already existing
strengths: helping to bridge the foundational AI research community and HL-LHC
physicists, and helping to build sustainable AI within the experiments. The
work to accomplish this first relies on building consensus in the field and
then helping to execute on that consensus. It is a mix of blueprint meetings
and concrete development projects. It would be a new area for the next phase
of IRIS-HEP, having appeared in the initial $S^{2}I^{2}$ strategic plan but
being largely de-scoped due to available resources. Some of the tasks are
important for the long-term sustainability of the experiments and will have to
be solved by the field; IRIS-HEP can make important cross-experiment
contributions and help set the direction for the community. The current work
and roadmap is much more exploratory than in the other areas of this strategic
plan as this is a new area for IRIS-HEP and a covers a lot of ground the field
has not yet given careful thought to.

Training and Community

Building Community Consensus
Funding Scenarios: Medium
Description: Holding Blueprint meetings (as part of the Intellectual Hub
activity in Section 4.2) will help build community consensus and guide the
exploratory activities in this area. These meetings will work to improve the
understanding between experiments and groups in the experiments and also
explore best practices from outside the field of particle physics and improve
collaboration with the AI research community. The activities listed below are
all meetings planned to help bootstrap the activities in this new area.
Current and Potential Future Activities:

  * •

Streamlining ML Data Delivery: Different cyberinfrastructure is used to
process data during prototyping, training, and deployment. Frequently, the
methods and code to extract the input data to build training/prototyping
datasets are different from the way the same task is performed for a deployed
solution. A common set of tools to address this will reduce deployment time
and may allow uniform access to national or international resources for
complex training. This work may also allow caching of training datasets
reducing the DOMA burden.

  * •

Deploying Trained Models in different environments: There are experiment
hardware and software experts for the trigger, reconstruction, and analysis.
Each is a radically different environment in which we wish to run ML
algorithms. The way ML is prototyped and deployed in each is quite different,
resulting in drastically different infrastructure. This meeting will
understand the different environments and explore how to make the training
environment as common as possible.

  * •

Using the AI Benchmarking and Challenge infrastructure in HEP: With the
overall goal of using more common tools in training, deployment, benchmarking,
and challenges, this blueprint meeting will bring together the various
communities to understand the current state-of-the-art and how it could evolve
to help quickly move new ideas and solutions between communities. There is a
lot of community expertise already existing. Challenges and benchmarking
efforts are initiated, organized, and reported in workshops like ML4Jets,
ML4PS, NeurIPS, Hammer & Nails, MODE, Aspen, IPAM, Dagstuhl, and MIAPbP. A
recent DOE award, “A Fair Universe: Unbiased Data Benchmark Ecosystem for
Physics”, is explicitly addressing the benchmark side of this story.

  * •

Disseminating Advanced ML Techniques within the field: An advanced library of
ML techniques - translations of ML papers - is hard to build. For most
physicists reading an ML paper is difficult due to the different problem
domains and vocabulary (sometimes even for the same concept!). Nor is it
possible to have one or two people translate all papers into some sort of
common example format as the approach won’t scale. This meeting will pull
together people from similar institutes, such as the eScience Institute at the
University of Washington (which actively deals with a similar problem), and
experts within the field of particle physics to discuss scalable ways of
pushing advanced AI techniques into the experiments.

  * •

Foundational Models: Many areas of AI are adopting a pattern referred to as
foundational models, where a large, multi-purpose model is used as a common
backbone for many tasks. While the foundation model may be expensive to train
over large amounts of data, those costs are amortized over relatively
inexpensive fine tuning or task-specific modules. Thus far, HEP has not
adopted this pattern, but it offers potential advantages. However, it is also
not clear that this pattern holds well for HEP use cases. A blueprint to
explore this pattern would be valuable.

  * •

Prompt-based assistants based on language models: Recently released models
such as GitHub Copilot and ChatGPT, allow one to quickly translate
instructions formulated as plain English text prompts into code or other
actions. This blueprint meeting would gather experts on such models and
experts from HEP and HEP/ML to develop a program of research that can be
followed in the next years (in or outside of IRIS-HEP) to use the power of
these new models.

Training and Dissemination
Funding Scenarios: High
Description: Getting advanced AI techniques into the hands of physicists
requires a training program addressing multiple levels - especially when this
must be scaled out to reach the full field of HL-LHC physicists. Many users of
AI at the HL-LHC are still learning the basics - so their needs must be
addressed in a way that they can later then use the more advanced techniques.
Current and Potential Future Activities:

  * •

Basic Machine Learning Skills: The HEP Software Foundation and IRIS-HEP and
many others collaborate on building a library of training resources for basic
software skills using the Software Carpentry’s model. The library already
contains two ML-related trainings - one for basic PyTorch training and one
using GPU’s to accelerate training; advanced training materials will be added
to cover LHC-specific topics instead of generic basic topics.

  * •

Advanced Techniques: A library of example uses of advanced ML techniques will
be built. This library will provide physicists who have absorbed the more
basic training the ability to apply the techniques with best practices being
integrated from the blueprint meetings.

Making Fundamental AI Research Sustanibly Work for the HL-LHC Experiments

Sustainable AI at the HL-LHC
Funding Scenarios: High
Description: Currently, AI solutions deployed in the experiments are rapidly
replaced by new and improved solutions that are developed through a relatively
ad-hoc process. As the AI solutions stabilize, systems will be needed to
streamline the retraining or fine-tuning of these AI models in response to
changing running conditions. This is akin to the routine recalibration and
data reprocessing that the experiments manage, but brings in different
computing challenges (access to GPUs, multiple passes over batches of training
events, etc.) as the algorithmic patterns encountered in training are
different than those found with recalibration. This also adds new requirements
to the metadata that tracks run conditions and software versioning. There is
an opportunity to design systems that can leverage computing resources that
are well suited for AI, and reduce inefficiencies in the data transfer and
redundancies that will be encountered without a more streamlined solution.
Current and Potential Future Activities:

  * •

Retraining Challenge: Working with reconstruction authors within an experiment
to use community tools to automate the retraining of a reconstruction
component. Once implemented, track its use and updates required to keep it
working as run conditions and the detector evolve.

Benchmarks and AI Challenges
Funding Scenarios: High
Description: Coordination is needed to guide the evolution from this period of
rapid R&D, prototyping, and bespoke solutions for deployment to a more mature
or established set of practices for ML in various contexts (trigger,
reconstruction, simulation, analysis). There is an opportunity to establish
the cyberinfrastructure components (both human and technical) that can serve
as a bridge between the vibrant AI R&D activities taking place outside of the
context of the individual experiments and the experiment-specific software and
computing frameworks.
Current and Potential Future Activities:

  * •

Community Challenge and Benchmark Infrastructure Efforts: Use the roadmap from
the blueprint meeting “Using the AI Benchmarking and Challenge infrastructure
in HEP” to form a program of work to better integrate challenge and benchmark
tools in to the HL-LHC’s ML infrastructure.

  * •

Using Common Infrastructure in the Experiments: Close the gap between the
backend infrastructure used for benchmarks and challenges and what is used in
the experiments. The experiments have multiple environments that would require
separate studies (trigger, reconstruction, analysis). Especially important
will be supporting prototyping efforts as new techniques are explored inside
the experiments.

Hardware Accelerators
Funding Scenarios: High
Description: R&D on AI with accelerators (e.g. FPGAs) and in highly
constrained computing environments (e.g. trigger) requires both specialized
hardware and specialized expertise. This poses a high barrier to entry and
hinders participation from a more distributed and diverse group of teams.
There is an opportunity to facilitate research and lower the barrier to
participation by investing in a centralized resource with dedicated expertise
and hardware to support distributed research on AI in these highly constrained
computing environments.
Current and Potential Future Activities:

  * •

Making ML on FPGA’s Accessible: Work with the A3D3 institute [50] to best
understand how to move their R&D work into a place where others in the
experiment can make use of it. In particular, the software pipeline and
testbeds that can be hosted at a community resource (e.g. an Analysis
Facility). The goal of this work is to make the barrier of entry to developing
ML trigger solutions lower.

Improving AI Research on HEP Data

Expanding connections between AI and HEP Communities
Funding Scenarios: High
Description: Data from HL-LHC detectors is ideal for many types of AI
research. The problems that need to be solved (e.g. jet tagging, track
reconstruction, signal identification) require learning complex processes,
physics concepts, and even geometric relationships. Further, the problems
solved can have large impacts on the physics ability of the HL-LHC and
resource requirements. The common tools in the field to generate data suitable
for ML are more tuned for internal use.
Current and Potential Future Activities:

  * •

Simulating a Generic HL-LHC Detector: Tools like DELPHES implement a
simplified simulation of an LHC detector. This tool set needs to be updated to
reflect some of the complexities of an HL-LHC detector: adding more modern
tracking architectures and an active calorimeter much like CMS’s next-
generation forward calorimeter. This work would also include building a more
automated pipeline for producing datasets that could be used in challenges.

  * •

Data Formats For Research: The data formats used by the HL-LHC community are
mostly ROOT based, which is not easy for AI researchers and others outside the
field to use. Tools like NumPy and Pandas DataFrames or Arrow Tables are more
commonly used. This task will work on building these common formats as first-
class citizens.

  * •

AI Challenges: As the work above comes together this will enable the field to
build and run AI challenges. Running a successful challenge is far from
simple, however. Working with others within the field that have run
challenges, a set of best practices will be built by example.

#### Impact and Success Criteria

Success Criteria – Milestones & Deliverables:

The milestones and deliverables outlined below focus on the first three years
of an Institute. These high-level items would be expanded and later years
filled in as part of the execution of any project.

  1. D4.1.

Each Blueprint meeting will have one of its products a report. A milestone
will be associated with the circulation of the report to the community. The
expected dates will be tied to the schedule (about 4 months after the
occurrence of the blueprint meeting). Various

  2. D4.2.

Retraining Challenge completed: a ML algorithm, used in the reconstruction or
trigger, has its retraining automated. December 2025.

  3. D4.3.

Launching an FPGA testbed (like SSL). December 2026.

  4. D4.4.

Run a Challenge with entrants from the AI Research Community. December 2027

  5. D4.5.

Access to a FPGA for ML training is possible on a Analysis Facility. December
2027

  6. D4.6.

A training is run locally and non-locally from the training and testing
infrastructure. Normalizing training means that researchers can take advantage
of GPU’s for training around the world, or on the infrastructure hosting the
training service, and have the same user experience.

Success Criteria – Metrics:

Metrics are a useful tool to provide management with quantitative insight into
progress toward overarching goals. High-level metrics we expect to be
applicable to the Translational AI area are below:

  1. M3.1.

The number of foundational trainings added to the HSF training library to help
physicists incorporate ML into their workflows.

  2. M3.2.

The number of advanced ML techniques added to a library as exemplars for the
field.

  3. M3.3.

Number of AI / ML models or use cases that are integrated into experimental
infrastructure using the processes developed by the institute.

  4. M3.4.

Number of times the FGPA service at an Analysis Facilities are being used per
quarter.

###  5.4 Data Organization, Management and Access (DOMA)

Given the centrality of ‘data’ to all things the HL-LHC will do, it is not
surprising the investment area of data organization, management, and access is
a key part of all four HL-LHC Computing Gaps. The immense volume of data is
the easiest to conceptualize – starting with the predicted trigger rate, days
of data taking per year, and event size, the HL-LHC experiments can make
leading-order estimates of the raw, processed, and simulated data sets
necessary for the desired scientific scope. Feed these numbers into the
current resource requirement modeling and already the disk requirements needed
outpace the fixed-budget scenario. The problems compound, however, once one
takes into account this data must be moved about the shared national
cyberinfrastructure, analyzed at much higher data rates than today at
facilities, and done using common infrastructure to reduce sustainability
costs.

Figure 9: Schematic showing the DOMA-related components in the production
infrastructure. The components that DOMA started or contributed to in IRIS-HEP
are colored in red. The dashed arrow from IDDS to ServiceX is one of the
planned projects for the next phase.

Together, these challenges in the DOMA area illustrate the fact that HL-LHC
science is not achievable by simply purchasing more servers, disks, and
network switches - but otherwise leaving the approaches untouched. A robust
R&D program is needed across both the production infrastructure and analysis
facilities sub-areas of DOMA building upon the successes in the last 5 years.
For bulk data movement, the entire LHC production infrastructure has migrated
from the niche GridFTP protocol to the industry-standard HTTP, demonstrating
the capability to both deliver from R&D to production and also execute system-
wide changes. The community has planned and executed the first biennial data
challenge, DC21, as part of the Data Grand Challenge (Section 6.2, showing the
ability to coordinate on the global scale and provide integrative milestones
for technology projects. Finally, during IRIS-HEP, the Coffea-Casa analysis
facility platform was developed; Coffea-Casa provides a “first look” location
for new analysis system services to intersect with users. The areas where the
DOMA team has been active during IRIS-HEP are shown in Figure 9.

#### Specific Challenges and Opportunities

In the remaining R&D period between now and the HL-LHC, we see opportunities
in the DOMA area continuing work in preparing the coordinated
cyberinfrastructure or HL-LHC data volumes, transitioning to newer
technologies used by industry and elsewhere in scientific computing
infrastructures, and delivering data to analysis facilities.

Scaling up data volumes to HL-LHC: There is a need for innovation,
development, and integration in nearly every service in the bulk data transfer
infrastructure used by the LHC community to prepare it for the HL-LHC data
volumes. Opportunities include benchmarking the existing software on available
R&D testbeds such as the National Research Platform (NRP) [54], utilizing
next-generation networks such as FABRIC [55], and leveraging engineered
network paths as in the ESNet SENSE project [56]. The community has the
opportunity to utilize these testbeds as a proving ground for techniques
between the proposed biennial global data challenges (Section 6.2) and as part
of any regional / US-specific exercises.

Technology transitions for bulk data movement: Several of the technologies
historically used for the bulk data movement stack are at end-of-life, causing
difficulty in the sustainability of the infrastructure. For example, the WLCG-
wide tape archival protocol, Storage Resource Management (SRM), is based on a
custom transport layer (not Transport Layer Security, TLS, which is ubiquitous
across the Internet) and uses a remote procedure call protocol, SOAP, which is
long out of favor. This has created a new opportunity to reconsider parts of
the ecosystem - for example, the community has already replaced the use of the
niche GridFTP protocol with the industry-standard HTTP - resulting in
potential leaps in sustainability through leveraging industry standards.
During the next 5 years, we see the following potential transitions necessary
for the community:

  * •

Moving the authorization scheme from identity-mapping based (mapping a
credential to a local identity) to capability based (a credential is tied to a
specific action). Capability-based schemes provide for more fine-grained
authorization - only the actions intended for a job must be sent along with
it. This transition requires coordinated changes across the software stack and
integration in the ecosystem.

  * •

Migrating bulk data movement to IPv6. IPv4 is considered a legacy protocol by
the IETF meaning future innovations in the networking layer will be
exclusively done for IPv6; the transition to IPv6 for all bulk traffic should
be completed. For example, packet extensions that enable monitoring of network
flows on ESNet are only defined for IPv6.

Delivering data to analysis facilities: The hallmark of HL-LHC analysis will
be the size of the dataset (event count and volume), the need to deliver
initial results at lower latency, and the use of a broader range of “data
science” and machine learning tools from outside physics. The data volume and
complex environment is expected to cause users to move from laptop-scale
machines to analysis facilities that specialize in high-data-rate I/O, ML-
support services (such as inference services), user interfaces (such as
pseudo-interactive task-based computing or notebooks), and advanced data
management services. These facilities, either from new approaches as in
Section 5.5 or built on top of existing, batch-oriented analysis facilities,
will need advanced data delivery and management services.

For example, an important trend in HEP analysis is the movement to Python
tools and columnar analysis. In addition to giving analyzers experience with
popular data science tooling, columnar analysis has the potential to unlock
vectorization and performance benefits by defining operations on columns as
opposed to individual events. To successfully leverage these analysis
techniques, services are needed to filter, transform, and deliver columns to
workloads running at analysis facilities.

HEP analysis has historically been a strongly filesystem-based activity: a
user will take a dataset – often just a set of files in a directory – and
apply simple transforms and filtering, resulting in a reduced set of output
files in a new dataset. Users often make only simple changes, such as adding a
few derived values to the event, meaning each of these derived datasets have a
high level of overlap. Further, if a column was missing or incorrect in the
original derivation, the user has to repeat the process from scratch. The
system is wasteful of both disk space and – more importantly – physicist time.
There are modern data management systems that provide the ability to do common
database techniques such as virtual views and joins without needing to convert
the ecosystem to a RDBMS; there is opportunity to both tap into a rich vein of
industry investment (helping with the G4 (Sustainability) gap) in addition to
making physicists more productive. As the management evolves, data delivery
will similarly need to change to effectively schedule and pull the data from
the wider distributed cyberinfrastructure into the facility and transform the
data into the desired format for data management. This will require both
services in the analysis facility and at the network layer (leveraging the NRP
and SENSE work) to ensure proper prioritization and effective caching.

Integration point: Finally, the institute structure itself provides a unique
opportunity. The DOMA ecosystem for the LHC and HL-LHC is large, complex, and
slow-moving, making it difficult for independent smaller-scale research
projects (such as those in the NSF CSSI Elements program) to deliver their
work to production. The Institute’s DOMA strategic area would be large enough
to serve as an intellectual hub, connecting between innovative ideas coming
into the NSF ecosystem, R&D testbeds at the national scale, and the production
cyberinfrastructure for the HL-LHC.

#### Current Approaches and Development Roadmap

The DOMA area can be partitioned into investments into the production
infrastructure, focused primarily on the HL-LHC Computing Gaps for
sustainability (G4), scalability (G2), and raw resource requirements (G1), and
the analysis systems investments, focused on sustainability (G4) and analysis
at scale (G2). Below we outline existing approaches in each area.

Production Infrastructure Projects

Scaling the CI to HL-LHC data rates
Funding Scenarios: Medium (reduced scope), High
Description: From the beginning of Run 3 to the first full year of HL-LHC, the
expected data volume moved per year by the LHC experiments will rise by 20
fold. To prepare for these data rates, the community has defined a set of
biennial data challenges as described in Section 6.2. The data challenges are
meant as capstones & milestones; the majority of the effort is in the
preparation in the years leading up to the challenge. This project would work
on the engineering and integration of community-wide research projects to
demonstrate the readiness of the cyberinfrastructure for the HL-LHC scale.
Current and Potential Future Activities:

  * •

Demonstrating scaling of reference platforms: The various software reference
platforms (for IRIS-HEP, this is XRootD on top of a POSIX filesystem) used for
bulk data transfer must be shown to scale well before trying load tests on
production systems. NSF has funded a number of experimental storage (such as
NRP, providing filesystems) and network (such as FABRIC, a 1Tbps platform
between San Diego and New York) resources. By leveraging external R&D
testbeds, the community is able to attempt tests at HL-LHC scales without
needing to “buy ahead” at production facilities.

  * •

Leveraging engineered network paths: By segmenting HL-LHC bulk data traffic
onto specially-engineered network paths, the network connectivity providers
(for US LHC, this is ESNet) have the ability to isolate user specified data
flows, enhancing their accountability and at the same time allowing for
prioritization, segmentation, and specialized quality-of-service techniques.
ESNet’s SENSE project [56] provides a mechanism for engineering the paths end-
to-end; however, this functionality needs to be integrated into the LHC’s data
management software (Rucio) and shown to work across multiple sites.

  * •

Organizing US regional exercises: The WLCG-wide data challenges provide a
biennial global synchronization point for facilities and technologies. These
require significant effort and coordination making it difficult to incorporate
less mature technologies or regional concerns. We foresee a need to have US-
specific regional exercises focused on technologies the US LHC is investing in
(such as the Rucio / SENSE integration) and challenges such as integration
with US HPC sites. An institute-scale DOMA strategic area, cross-cutting the
HL-LHC experiments, would be uniquely situated to coordinate such exercises.

Scaling the cyberinfrastructure to close the 20x gap between today’s and HL-
LHC’s expected transfer rates is one of the most high-impact activities of the
DOMA team.

Authorization technology overhaul for the distributed infrastructure
Funding Scenarios: Medium, High
Description: Capability-based authorization provides a powerful new paradigm
for asserting authorizations on a distributed infrastructure. Each experiment
now has a token issuer service that asserts a specific action the bearer
(hence the term “bearer token”) can perform in the experiment’s distributed
resources; IRIS-HEP has worked to ensure the storage systems used by the LHC
community interpret these assertions in an interoperable manner.

However, significant work remains to ensure all the software used by the
community has been adopted to acquire, manage, and utilize these new
credentials. These technologies remain a headline feature planned for the next
data challenge to allow sufficient time for a complete transition by the start
of the HL-LHC.
Current and Potential Future Activities:

  * •

Coordinating evolution of token usage & profiles: While the intellectual
approach to capability-based tokens was done in 2017 by the SciTokens project,
significant coordination effort was required to expand the idea to cover all
LHC use cases and to gain traction in the community. Common interpretations of
authorization schemes is a strict requirement for establishing trust when
moving data between sites; the DOMA strategic area has the opportunity to
continue working with the WLCG authorization working group as the profiles
evolve and are updated with technologies.

Now the foundational work is done on the capability language, the community
needs to better define the token acquisition and exchange workflows, for
example, designing how they move from data management system (Rucio) to file
transfer system (FTS) to storage endpoint.

  * •

Engineering a reference implementation: The IRIS-HEP DOMA area provided a
reference implementation for token-based authorization embedded in the XRootD
server software. This is used by several LHC facilities and provides an
endpoint for storage and middleware developers to use for testing their own
independent implementations.

  * •

Provide leadership and support: New technologies always pose a challenge for
system administrators, providing expertise and a direction helps to ease the
burden and make changes better welcomed by the community.

Organized delivery of data to running production workflows
Funding Scenarios: High
Description: In the first four years of IRIS-HEP, the DOMA area invested in
the Intelligent Data Delivery Service (IDDS) which provided the ability to do
data-centric workflows within the PanDA workflow management system. IDDS was
used to help implement “disk carousels” (minimizing the use of disk buffers by
the ATLAS experiment) and manage large-scale hyperparameter optimization runs;
in impacts beyond HEP, IDDS was adopted by LSST to manage large-scale data
processing. Given the significant raw cost of disk storage – by some
estimates, HL-LHC dedicated disk storage is more expensive than CPU – tight
management of disk buffers has the opportunity to reduce their required size,
allowing the community to rely more on less-expensive archival tiers.
Current and Potential Future Activities:

  * •

Integration between IDDS and ServiceX: Currently the column delivery service
(ServiceX) for analysis facilities is entirely dependent on the input datasets
being on disk and being transferred then processed. By having ServiceX call
out to IDDS to do the data processing, there’s the opportunity to use this
technology to stream datasets from tape into an analysis facility, extract
columnar data needed for an analysis without needing to persist the entire
dataset on disk.

Analysis Projects

Delivering columnar data with ServiceX
Funding Scenarios: Medium, High
Description: Columnar analysis is a relatively new and growing approach to
physics analysis. ServiceX is an service designed to run inside analysis
facilities to create and cache columns to meet analysts’ data delivery needs.
ServiceX was designed from the beginning to read experiment specific data
formats and write selected events and properties to common columnar data
formats used in industry (in addition to the HEP-native “ROOT” format).
Current and Potential Future Activities:

  * •

Broaden the set of usable data formats: Analysis approaches are ever-evolving
as the compute models for HL-LHC are refined. ServiceX has focused on the
compact, analysis-oriented data format; however, new ML techniques often
include the use of low-level objects that might be in more raw data
(potentially on tape, needing IDDS integration). By building as broad a set of
use cases as possible, ServiceX can help increase the impact of future
analysis facilities.

  * •

Integration with data management tools: The service can currently either
deliver data to a S3-like object store; both provide quite simplistic tools
for the management of data. When combined with industry techniques – for
example, delivering to SkyHook (below) – there users will be able to better
utilize those tools.

  * •

Integrate with analysis preservation systems: One reason for the durability of
“a dataset as a collection of files” is in its simplicity: the only data
service needed to help reproduce a workflow is a filesystem. This is not
necessarily true when a series of complex services is used to in the
processing chain, including the case of delivering a column and integrating
into into an existing dataset (which potentially implies the delivery service
needs to be captured by the analysis preservation). To help sustain the
ability to preserve analysis, commonly-used tools such as RECAST will need to
understand/presserve analyses that were run with ServiceX.

Leveraging Industry Data Management for HEP
Funding Scenarios: Medium (reduced scope), High
Description: Analysis datasets will grow significantly and users will demand
higher event rates to improve the time-to-insight. The Skyhook project is an
investment into emerging data processing architectures where additional
computational resources and hardware accelerators are available in the
networking and storage layer. The service is leveraging commonly used data
access libraries to “push down” structured queries into these layers to reduce
data movement and to create views of datasets without creating copies. A
significant part of the Skyhook project is the management of metadata that
enables the establishment of views as well as versioning and branching of
views (similar to versioning and branching in git-based source code
management). Skyhook-enabled storage systems are a natural place for
maintaining this metadata and servicing it to other tools in convenient
formats. The project is leveraging data and query specifications of successful
open-source projects such as Apache Arrow, Parquet, Substrait, and Apache
Iceberg. The Skyhook project was able to upstream a Ceph extension to the
Apache Arrow project, allowing Apache Arrow dataset interface queries to be
pushed down and distributed into Ceph storage servers storing Parquet files.
Other parts of the Skyhook project involve coordinated processing of Apache
Arrow streams of particle physics data across multiple Bluefield SmartNICs
using Substrait as query specification and processing status and offloading
comparisons of genomes to hard drives.
Current and Potential Future Activities:

  * •

Integrate with file-based data management systems: File-based data management
systems like Apache Iceberg provide functionality typically associated with
databases without losing the filesystem-based approach that LHC is heavily
invested in. SkyHook is interested in utilizing the using ADL Benchmark
library [57] as a way to compare Iceberg’s performance with the to results
presented by Graur et al. at VLDB’22 [58] and using py.iceberg as a mechanism
to implement versioned views.

  * •

Alternate data modeling languages: With such complex data as in LHC events,
modeling and querying is not trivial and maps poorly to common languages like
SQL; we aim to evaluate the new query and data modeling languages (such as
Malloy) and compare it to results presented in [58].

  * •

Substrait integration: Substrait is quickly becoming the established way to
specify query plans independent of the original query language. It appears to
be a promising way to create a composable data management stack that can unify
multiple query approaches. For example, it would be able to help specify views
and name cached queries and connect the LHC analysis stack to future data
processing, management, and storage systems being developed in industry and
wider CS R&D.

Joint Area Projects

Coffea-Casa, an Analysis Facility Platform for the HL-LHC (Joint with Analysis
Systems, Facilities R&D)
Funding Scenarios: Medium, High
Description: One lesson learned from the first four years of IRIS-HEP is the
difficulty in keeping multiple projects at different levels of maturity
aligned toward a goal. Coffea-Casa is a prototype analysis facility which aims
to be a common integration point for a broad set of technologies and provide
them with a first exposure to a user base of analysts.

Coffea-Casa starts with a base of the “Coffea” processing framework for low
latency columnar analysis and has a modular approach for adding other services
such as ServiceX, SkyHook, or scale-out of tasks into a traditional batch
system. The facility provides an interactive experience for physicists that’s
closer to working on a laptop as opposed to a traditional batch system-based
facility. The facility adopts an approach that allows transforming existing
computing facilities into composable systems using Kubernetes as the enabling
technology. Kubernetes not only enables rapid deployment of services by
developers via the DevOps methodology but also serves as a common language for
service orchestration (see Section 5.5), enabling the services to be easily
duplicated across multiple Coffea-Casa instances.

Within IRIS-HEP, Coffee-Casa is used to execute the exercises of the IRIS-HEP
Analysis Grand Challenge (Section 6.1) and has deployments for the U.S. CMS &
U.S. ATLAS physicist community.
Current and Potential Future Activities:

  * •

Maturing Coffea-Casa to production: As the analysis facility concepts and
services mature, we are working to expand the user base to help gain the
experience necessary for at-scale analysis for the HL-LHC. We will continue to
utilize Kubernetes as a technology fabric for additional services and enabling
portability. Coffea-Casa will grow and scale along with the increasingly
complex set of Analysis Grand Challenge exercises. The integrations with real
CMS & ATLAS analyses will help us tune and benchmark for HL-LHC data rates.
Further, as the prototype facilities scales from today’s dozens of users to
potentially hundreds, we expect challenges requiring improved resource
management within these facilities.

  * •

Strengthening the integration across DOMA R&D: As the integration point for
analysis-oriented work in the IRIS-HEP Analysis Systems and DOMA areas, there
is a continued need for Coffea-Casa to deploy new versions and data delivery
services. Integrations with existing projects need to be strengthened (such as
having ServiceX authentication credentials be auto-generated on login or
scaling data caching services) and starting integrations with new technologies
such as bearer token support for authorizing global data access.

  * •

Develop approach for enabling ML-based analysis: Machine learning techniques
have a long history of use within HEP. However, they have rarely grown to the
size where their needs need explicit planning in the site infrastructure; as
ML rapidly grows across science and industry, the interest in the LHC
community has grown as well. We will investigate approaches offering an
interactive training environment with GPU access (more difficult than CPU
scheduling as there are few GPU devices in the facilities) and pursue
opportunities to integrate with national-scale GPU / ML training resources.
Additionally, industry has built several new tools for deployment on
Kubernetes – such as MLflow – to manage hyperparameter optimization tracking
and advanced ML pipelines. Finally, multiple analyses have already begun to
include ML inference into their workload; these would benefit from facility-
local inference services to accelerate throughput.

  * •

Analysis preservation: The addition of new service types (data management, ML
inference services, task-based computing) creates new paradigms that need to
be reflected in analysis preservation systems. Preservation strategies need to
be devised and integrated with other community projects.

  * •

Coordinating the growing network of LHC analysis facilities: IRIS-HEP is not
the sole entity investigating analysis approaches for HL-LHC; there are
several similar efforts globally. There’s a role for a future software
institute to serve as an intellectual hub, ensure coordination and knowledge
sharing between international projects, NSF-funded efforts, and the U.S. LHC
facilities.

The prototype Coffea-Casa facility has proven to be a valuable “meeting point”
for IRIS-HEP’s analysis R&D vision and the team serves as an intellectual hub
inside and outside the institute.

Core data streaming with XRootD and XCache (Joint with OSG-LHC)
Funding Scenarios: Medium, High
Description: The XRootD software framework is foundational for efforts across
the LHC. It is used as a reference platform in IRIS-HEP for data streaming and
bulk data transfer, as the basis for data federations in CMS, and, in its
“XCache” configuration, as a data caching service for both production and
analysis. Sustainability of this platform is essential.
Current and Potential Future Activities:

  * •

Evolving bulk data transfer: As a multi-protocol server, XRootD implements
both the proprietary “xrootd” protocol and the standard HTTPS. It is used by
over half the U.S. LHC facilities for HTTP-based data transfer between sites.
As the technology stack for bulk data transfer evolves – refining HTTP-based
data movement, scaling the data rates, and implementing technology – this
platform will continue to lead the community.

  * •

Core development, integration, and delivery: Beyond bulk data transfer, XRootD
is critical software for the LHC community. It is a collaboration with
contributions from DOE labs, universities, and CERN; there is a role for the
DOMA and OSG-LHC strategic areas to make contributions (core development,
testing, integration, software delivery) to ensure the investments in this
software are sustained. Having an institute-class investment from NSF like
IRIS-HEP participate in these activities would help the collaboration provide
a broader stakeholder setup for the NSF cyberinfrastructure community.

  * •

Exploring event delivery: XRootD’s multi-protocol framework includes an RPC
and streaming layer which provides for a way to scalable, load-balanced data
delivery. The SkyHook project has shown the value of Apache Arrow as a high-
performance, interoperable data format; however, it is currently limited to
being used as part of Ceph (reasonably common, but nowhere near universal at
LHC facilities). There is potential in using XRootD as a way to deliver high-
level, structured data in Apache Arrow format while leveraging the existing
authentication and streaming features of the software.

#### Impact and Success Criteria

DOMA has constituted a historical strength of the US university community.
Particularly, in the last 5 years, the existing IRIS-HEP team has held
leadership positions within the WLCG in this area and has designed and
executed the transition from GridFTP to HTTP-TPC and began the transition to
tokens. Investing in DOMA as part of an NSF-funded software institute would
keep the HL-LHC community well-aligned with the activities and priorities of
the broader NSF Office of Advanced Cyberinfrastructure (OAC); DOMA is uniquely
situated to leverage existing distributed testbed resources in its program of
work. Existing teams are in place for services like ServiceX and Skyhook and
able to build on the success of the IRIS-HEP project. Having DOMA as a
strategic area balances continuity – completing projects that will be in
progress at the end of IRIS-HEP – with leveraging an existing productive team
to start new projects. Further, the team will be able to help sustain critical
projects such as XRootD widely used across the community.

The high-impact outcomes for the production activities in DOMA will revolve
around the successful participation in, and execution of, the biennial data
challenges coordinated in part with WLCG. The capstone of this effort will be
the 2027 data challenge (approximately Year 4 of the next phase of IRIS-HEP)
where we will aim to demonstrate bulk data transfers at 100% of the expected
HL-LHC data rates. Another significant milestone will occur for DC23, when the
majority of transfers will be run using token-based authorization. For DC25,
we expect the remainder of the infrastructure will be token-based and the
community will be leveraging from network services to improve accountability,
monitoring, and management of data transfers.

For analysis-related activities, the expected outcomes of the DOMA strategic
area will be around the usage of new paradigms of analysis facilities and the
associated data delivery and management services. We expect these to be the
most common analysis environment at the start of HL-LHC and for the Institute
to demonstrate data rates to analyses at HL-LHC scale, as defined by the
Analysis Grand challenge, by 2026. ServiceX is expected to be the primary way
for users to ingest official experiment datasets into their analysis
environments, Skyhook or similar services will provide a way to manage data
(virtual views, joins of columns) from within the environment, and for
analyses to have access to production-quality, user-facing services for ML
inference.

HL-LHC Computing Gap Impact: The DOMA area is expected to make an impact on
all four defined HL-LHC computing gaps. Most of the resources will be
dedicated toward G2 (Scalability) as part of the benchmarking of different
software components and the participation in regional data exercises and
global data challenges. This work will provide a better understanding of our
network flows and help increase the use of network services to increases the
manageability of our data transfers translates on a more efficient use of our
resources. The G1 (Raw resource usage) gap will be tackled through the
investment in data delivery via IDDS which helps manage the disk buffers used
for annual data (re-)processing and reduce the total amount of online disk
needed for HL-LHC. For G3 (Analysis at scale), DOMA will put new services and
techniques in production a interactive facilities, combining “local laptop-
like” responsiveness with datasets at the HL-LHC scale.

Finally, for G4 (Sustainability), the DOMA area leverages industry protocols
(HTTP, JWT-formatted bearer tokens), methodologies (Kubernetes-based service
orchestration), and technologies (Apache Arrow) wherever possible. By helping
the LHC community use broader ecosystems instead of developing technologies
in-house, we minimize the LHC-specific pieces to those areas where the
community is truly unique.

Success Criteria – Milestones & Deliverables:

The milestones and deliverables outlined below focus on the first three years
of an Institute. These high-level items would be expanded and later years
filled in as part of the execution of any project.

  1. D4.1.

Coffea-Casa used as part of one production analysis facilities in both ATLAS
and CMS. December 2023.

  2. D4.2.

All the U.S. LHC T2s in the US support bearer tokens on their Storage Elements
for Third Party Copy transfers. December 2023

  3. D4.3.

Rucio/SENSE integration is included as part of the DC23. March 2024

  4. D4.4.

Successful execution of DC23, meeting its data transfer and technology goals.
March 2024

  5. D4.5.

ServiceX used for physics analyses as part of Coffea-Casa. March 2024.

  6. D4.6.

Demonstrate analyses running at 200Gbps as part of the Analysis Grand
Challenge. December 2024

  7. D4.7.

Demonstrate the capability of XRootD to scale beyond 400Gbps. June 2025

Success Criteria – Metrics:

Metrics are a useful tool to provide management with quantitative insight
about progress toward overarching goals. High-level metrics we expect to be
applicable for the DOMA area are below:

  1. M4.1.

Demonstrate data rates as a percentage of the expected HL-LHC rates. Goals for
this metric are defined by the DGC in the timelines set out in Section 6.2.

  2. M4.2.

Number of sites participating in the Rucio/SENSE testbed. Goal is 4 by DC23
and 8 by March 2025.

  3. M4.3.

Percentage of U.S. LHC facilities supporting bearer tokens for data transfer.
Goal is $>50\%$ as part of DC23 and for all sites by DC25.

  4. M4.4.

Maximum transfer rate demonstrated by the XRootD reference platform as a
percentage of the desired HL-LHC rate for Tier-2s (400Gbps sustained). Goal is
200Gbps as part of DC23 and 400Gbps by DC25.

###  5.5 Facilities R&D and Integration

_Facilities R &D_ broadly refers to activities related to the exploration and
innovation of systems, services and physical infrastructure that provide
platforms suitable for HL-LHC service environments and runtime ecosystems.
These can be purely local facilities (platforms deployed within a local area
network) or distributed, in the sense of interoperating services over wide
area networks (the “Grid” or nowadays, distributed platforms). Two IRIS-HEP
research areas (Analysis Systems and DOMA) have opted for software containers
and cloud-native application management methods (Helm charts and GitOps) to
standardize deployments. Driven primarily by DOMA software development and
Analysis Grand Challenge scalability requirements, flexible strategies
employing cloud-native technologies were tried out in the Scalable Systems
Laboratory (SSL).

Indeed the concept of facility “substrates” using Kubernetes, a strategy
pioneered in IRIS-HEP blueprint meetings, “k8s-hep meetups”, and in WLCG
Kubernetes workshops, are gaining popularity in production LHC analysis
facilities as they offer reproducible application deployment and improved
reliability of operation. The scope of facility R&D needed for the
capabilities and scale of the HL-LHC, however, extends well beyond the
substrate and includes several technological areas involving a diversity of
data storage systems, innovative networks, and services for continuous
integration and operation. Figure 10 illustrates the relationships between
infrastructure, platforms and higher level services supporting production
workflows.

Figure 10: Schematic of illustrating role of Facilities R&D in the production
and analysis ecosystem. CI engineers prepare and operate infrastructure and
services in support of advanced data delivery, analysis, and other services.

_Facilities Integration_ refers to activities relating to the installation,
configuration, and operation of development and pre-production IRIS-HEP
software components. Within IRIS-HEP these include Coffea-Casa, ServiceX,
Skyhook, and other services. Other research cyberinfrastructure services
finding application in the ecosystem (such as declarative processing
frameworks, caching services, container image registries) likewise need to be
integrated into production systems. Where possible development services are
deployed in realistic contexts, alongside the experiment’s existing systems
and production infrastructure. This is the most immediate path to
understanding which components in the overall cyberinfrastructure present the
most significant scalability challenges. Collaboration between IRIS-HEP
software developers and experienced U.S. LHC Tier-2 and Tier-3 systems
administrators at universities is essential to achieve sustainable systems
through the lifetime of the HL-LHC. This extends, as well, to distributed
software developers in the US LHC Operations program as well as laboratory
staff at the Tier-1 centers.

The IRIS-HEP Scalable Systems Laboratory is a Kubernetes DevOps platform for
DOMA, Analysis Systems and Innovative Algorithms. The SSL hosts infrastructure
for CoDaS-HEP training events and Analysis Grand Challenge pipeline testing
and tutorials.

#### Specific Challenges and Opportunities

Over the past ten years, WLCG production computing infrastructure has remained
largely static, managed with decades-old deployment and management techniques.
Tier-1 and Tier-2 facility infrastructure, continuously in operation, has been
able to cope with the event processing rates and storage requirements
necessary to conduct the successful physics programs of the LHC experiments to
date. This won’t be true in the HL-LHC era unless those facilities change. For
example, data access and delivery from distributed data sources to analysis
platforms will require leveraging new capabilities in the network, in
computational storage systems, in facility resource managers, and service
orchestration frameworks. As we develop new capabilities to confront the data
rates and processing scales of the HL-LHC, methods for infrastructure
management and scaling become critical. The facility infrastructure itself
must evolve.

This was recognized during the first phase of IRIS-HEP with the SSL providing
two vital roles: a testbed infrastructure to incubate DOMA and analysis system
prototypes to be evaluated by end-user physicists at LHC Run2 scales, and a
challenge-environment for systems administrators and CI engineers to flexibly
and adaptively create the needed infrastructure platforms. Already during LHC
Run3, methods employed building the SSL have provided analysis facilities with
a blueprint for infrastructure creation and operation. It is vital that SSL
inspired facility R&D, together with pre-production service integration,
continue during the next phase of IRIS-HEP. Considerable experience within
IRIS-HEP and from integration with existing production services helps identify
the near and longer-term challenges and opportunities for this area in the
Institute.

Evolving Infrastructure for Next Generation Tier-2 and Analysis Facilities:
Over the course of the next five years, the community will transform the
historic roles of Tier-2 and end-user analysis facilities in the WLCG
computing hierarchy to adapt to different sets of needs. For example, as
opportunistic resources from leadership and national scale HPC centers are
increasingly harnessed by the LHC experiments to fill the resource gaps (even
during Run3 but more significantly for Run4), the Tier-2 centers, which
currently are workhorses for centrally organized simulation and data
derivation tasks, may increasingly become vital actors of the analysis and
data delivery ecosystem. At present, they are not readily adaptable to take on
those challenges. Similarly, end-stage physics analysis will shift from the
laptop (or small-scale institutional clusters) to shared analysis facilities,
hosting analysis and data access services capable of handling HL-LHC scale
processing and data rates. Practically speaking, we may see these facilities
blending roles and sharing responsibilities as frameworks for orchestrating
services provide new agility for resources in the ecosystem.

To create, optimize, and reliably operate diverse services (such as analysis
processing frameworks and data reformatting services) with sustained
capabilities, and to ensure toolset adoption by the current Run3 physics
groups, a sustained research and integration effort is essential. This will
involve working at a nexus of disparate communities: LHC physicists, LHC (&
HL-LHC) software developers, research cyberinfrastructure innovators, national
and leadership HPC technical staff, the Research End User Group [59] in the
Cloud Native Computing Foundation (CNCF) [60].

Distributed Infrastructure Management: DOMA systems intrinsically rely on
distributed services to implement, for example, data delivery to analysis and
production facilities. XCache networks deployed at multiple Tier-1 and Tier-2
sites for ATLAS and CMS are a prime example, essentially providing the
equivalent of a commercial content delivery network (CDN) but with one
important distinction: In the WLCG context, these services are mostly managed
across administrative domains by multiple teams, leading to delays in software
updates and in some cases inefficiency of operation. Lacking a seamless DevOps
pipeline for the entire distributed platform, DOMA development teams are
limited in their ability to advance new capabilities or respond quickly to
bugs and security issues across the platform. Additionally, as analysis
facilities scale to HL-LHC rates, we anticipate “scaling off-site” to handle
bursty workloads during high periods of analysis activity. The off-site
sources of processing cycles may come from Tier-2 production facilities, cloud
resources, and potentially HPC centers (which are GPU-rich). A few groups have
addressed this gap in capability through central management of services with
techniques such as employed by NSF’s National Research Platform
(https://nationalresearchplatform.org/) which uses a single administrative
domain (with full privileges) to combine resources from over 50 sites. In a
similar manner, during the first five years of IRIS-HEP the SSL team partnered
with the NSF SLATE project (https://slateci.io/) to develop a federated
operations model (“FedOps”) which requires only user privileges at Kubernetes-
equipped clusters to deploy and operate edge services. With this approach an
XCache network spanning all US ATLAS Tier-2 facilities, and extending to half
a dozen more in Europe, are managed by one person allowing rapid updates as
the underlying xrootd software evolves. During Run3 and LS3, the lead up
period to HL-LHC, an opportunity exists to expand the number of services
managed in this fashion, evolve the delivery tools with sustainable frameworks
engineered with the broader CNCF community, and strengthen the operational
policies and needed security models (work to be done with OSG-LHC).

Integration with Production Services during Run3: Software developed in the
Institute eventually makes contact with the production ecosystem of the
experiments and on production infrastructure. The sooner this happens the more
quickly challenges are exposed and opportunities identified. To that end we’ve
made certain that software graduated from the SSL that is ready for production
deployment can have its entire lifecycle neatly packed up and reproduced
elsewhere, such as at Tier-2s, national labs (Tier-1s), institutional
clusters, HPC centers providing Kubernetes hosting platforms, and even within
suitably equipped network testbeds (such as FABRIC). For example, ServiceX has
been integrated with the production distributed data management service for
ATLAS and CMS, essentially optimizing the data lookups, and exercised with
production storage endpoints and production caches. Additionally, we have co-
located the Coffea-Casa analysis facility software with shared Run3 analysis
centers in production (both at Tier-2 and analysis facilities) to understand
resource sharing, software access, identity management, data delivery, and
user file system integration. The added bonus is side-by-side proximity to
hundreds of physicists conducting analysis on data from LHC Run2 and Run3 thus
simplifying the introduction of innovative capabilities from IRIS-HEP to new
communities of users, gaining early feedback, increasing adoption.

#### Current Approaches and Development Roadmap

Facility R&D Across the Ecosystem
Funding Scenarios: High
Description: The Scalable Systems Laboratory is a unique resource in our field
as it provides a flexible and diverse DevOps platform shared by IRIS-HEP
software teams, developers from ATLAS and CMS, CI engineers from related NSF
cyberinfrastructure projects, and LHC systems administrators. Part of its
appeal is the context in which it sits, being (administratively) co-located
with existing production LHC Tier-3 and Tier-2 facilities and infrastructure
supporting a number of OSG-LHC related services (such as hosted compute entry
points). It has proved to be an invaluable resource for not only IRIS-HEP
software development and service deployment testing, but also a number of
Kubernetes-deployable research cyberinfrastructure services including a
network analytics service for WLCG PerfSONAR meshes, a network visualization
service, an analytics service for ATLAS software and detector conditions data
distribution, an XCache analytics service for CMS, scalability testing for
FuncX (Functions-as-a-Service) by its core development team, a REANA test
deployment, tests of containerized Rucio services, a notebook service for the
annual CODAS-HEP training event, and a persistent notebook portal to GPU
resources serving the ATLAS machine learning community.
Current and Potential Future Activities:

  * •

Continued operation of the reference SSL: Providing a testbed infrastructure
remains vital as IRIS-HEP software components continue to mature and operate
with other services in the ecosystem. During the first five years of IRIS-HEP
we established development and “production” SSL cluster testbeds to fit the
cadence and stability requirements of the DOMA and Analysis Systems
development teams. As these services come into production, this work will
continue apace as developer sprints respond to user and operational feedback,
new use cases and alternative strategies.

  * •

Constructing shared Tier-3 analysis facilities: participate with the broader
WLCG community in identifying best practices, infrastructure management
patterns, and recipes forged on the SSL leading to reliable and scalable
service deployments capable of supporting hundreds of users at Run3 scales in
the near term, but with proven capabilities to scale up to Run4 during LS3.

  * •

Evolving Tier-2 infrastructure management: retrofitting Tier-2 facilities with
substrates and higher level infrastructure management tools and services,
utilizing industry standard solutions where possible. The major focus will be
placed on sustainability and key capabilities needed for stable, continuous
Run3 operation while introducing new services targeted for HL-LHC
demonstrators and challenge problems. Resources from across the Tier-2 complex
will be needed in coordinated fashion for grand challenge scalability and
other proof-of-concept demonstrators.

The IRIS-HEP SSL informs the design of next generation WLCG Tier-2 centers and
shared Tier-3 analysis facility infrastructure.

  * •

Capturing facility patterns and blueprints: Creation and curation of GitOps
charts and repository actions to provide reliability and reproducibility such
that both infrastructure and applications can be easily replicated across
facilities.

  * •

Identifying infrastructure (and service) bottlenecks: a critical activity
during data and analysis grand challenge exercises, and related demonstrator
proof-of-concept exercises planned by the experiments, are load tests of
representative software workloads that accurately mimic planned infrastructure
and HL-LHC use cases. This will require instrumenting services and
infrastructure with the needed monitoring hooks, streaming the resulting
metrics to analytics dashboards, identifying and characterizing any
bottlenecks that might exist, and measuring overall performance.

  * •

Exploring potential cost savings with ARM processors: valuation and
quantitative assessment of ARM processor technologies for HL-LHC will be a
community-wide effort during the next few years. The US ATLAS Tier-1 will
provide the ATLAS PanDA workload management system with an ARM testbed capable
of testing a diverse set of simulation, reconstruction and analysis tasks.
IRIS-HEP may leverage these resources to test performance and applicability
for key services, for example ServiceX transformer performance on ARM. A
related task will be providing infrastructure for the OSG-LHC software team to
build images for ARM processors and test needed variants of the OSG-LHC
middleware (coordinated work with OSG-LHC).

  * •

Improve resource sharing across tasks on facilities: It is well known that
default resource allocation mechanisms offered by Kubernetes do not match the
full spectrum of fair share scheduling capabilities for a diverse platform
that may be simultaneously providing CPUs for ServiceX transformers, Dask
workers for Coffea-Casa, HTCondor job slots for traditional batch processing.
This must be dealt with during Run3 for shared Tier-3 analysis facilities
which are providing new and old analysis environments.

  * •

Strategies for evolving storage system organization and access: As described
in the DOMA strategy above, a promising approach to accelerate access to data
objects is to “push” a portion of the query evaluation down into the storage
infrastructure, taking advantage of processors and potentially accelerators at
that layer. Skyhook provides this capability for Ceph-based object storage
systems. Measurements at scale scale and assessment of performance impacts for
ServiceX transformers with Skyhook will need to be conducted for
representative analysis queries and processing pipelines.

  * •

Future storage system R&D activities: As described in the preceding DOMA
strategy, closing the storage gap may indeed present the greatest computing
challenge for the HL-LHC era. Over the next few years, efforts within IRIS-HEP
and from across the LHC computing community will inform priorities for
facility R&D activities in the Institute relating to promising storage
technologies and approaches. For example, creation of heterogeneous storage
systems for efficient analysis data access using distributed, asynchronous
object stores (DAOS) and RNTuple formats will be explored in proof-of-concept
exercises. Performance comparisons of processing rates with POSIX file systems
over NVMe drives, distributed dCache pools with spinning disk (and accessed
via xrootd protocol), and off-the-shelf object stores for both RNTuple and
columnar data formats will provide valuable guidance to the community. The
Institute will need to engage broadly and participate directly in specific
demonstrators driven by the experiments.

  * •

Transparent storage quality of service and “tiered storage” models: In many
cases cost savings can be found if the expected workloads are better matched
to the performance characteristics of the underlying storage systems. A
transparent data quality-of-service that feeds this information to DOMA
clients and to job workload management systems may achieve this. For example,
a service which sends tiny log files to object stores, detector simulation
input data to filesystems backed by spinning disk (very low I/O is required
for those tasks), and analysis outputs to fast NVMe-backed file systems for
speedy iterative access. Information collected would permit higher level
services to move data to cold storage based on access patterns: very
infrequently accessed data (“ice cold data”) could migrate to machines
programmatically switched off, something akin to Amazon Glacier, reducing
overall facility costs. Facility managers would use analytics gathered in a
given year to plan purchases for the following year, organizing procurement of
storage types based on the experiment’s projected needs by access category.

Managing Infrastructure over Wide Area Networks to Achieve Scale
Funding Scenarios: Medium, High
Description: Over the past several years significant advances have been made
in remotely managing coordinated sets of services to more easily innovate
systems while reducing operational cost. This has evolved our view of
distributed computing resources from a cooperating grid of computing sites
towards scalable research platforms offering programmatic capabilities for
service orchestration, with dramatic potential impacts given the increased
versatility of access. In the next phase of IRIS-HEP we anticipate making
facility infrastructure advances for both data delivery and analysis
processing incorporating these ideas.
Current and Potential Future Activities:

  * •

Reducing site storage requirements through caching: Currently, both ATLAS and
CMS employ data delivery methods involving differing access and caching
architectures using XRootD technology which have direct implications on
facility management and storage costs. Central tools for management of the
global system can be used to introduce new capabilities and updates which
strengthen the infrastructure. Current storage capacity requirements at WLCG
processing sites can potentially be reduced using caches and knowledge of data
placement and access history. Adding storage-less sites to the environment
obviously would expand the pool of available CPUs and facilitate access to
opportunistic resources. Additionally, existing WLCG sites with sufficient WAN
bandwidth capacity could potentially retire their storage endpoints altogether
and offer increased CPU capacity, trading storage costs (both operational
labor and equipment) for additional CPU. As part of this activity we will
stress-test caching networks with diverse analysis payloads at scale and
provide performance and operational feedback to the DOMA development team.

  * •

Distributed workers to scale up analysis capacity: Coffea-Casa analysis
facilities to date have been deployed only on single-site, Tier-3 or Tier-2
clusters. For potential scale up to HL-LHC volumes, and to reduce waiting
times on existing facilities in production, one could package and deploy
lightweight service endpoints providing potentially an unlimited supply of
Dask workers to a central home base (the “Casa”). Work is needed to understand
the interplay and tradeoffs of providing centralized login servers, home
servers, and local user storage services vs cloned stand-alone systems. The
potential gains could be significant to reduce costs of analysis facilities
(reducing their number) while vastly improving performance. Managing a
coordinated network of lightweight worker providers, or “Coffea-Pots”, across
administrative domains is possible with FedOps techniques we’ve employed in
the SSL.

  * •

Embedding advanced filtering services in the network: We have begun exploring
deployments of ServiceX inside the FABRIC testbed which opens up new
strategies for data delivery and cost assessments in distributed environments.
For example, by placing ServiceX on the server-side, at the data source, one
can reduce bandwidth consumption between analysis clients and storage systems
by combining filtering and reformatting functions at the source rather than
copying complete files over the network. Significant impact could be made with
implications for cost reductions of both network capacity and storage, both
being expensive components of facility infrastructure. In Q2 2023, as part of
the NSF FAB project (FABRIC Across Borders), a FABRIC “node” will be put into
production at CERN, co-located with datasets hosted by the Tier-0. We will
explore benefits of server-side ServiceX in operation at the Tier-0 while
providing fast delivery of Run3 samples to production analysis facilities in
the U.S. The exercise will provide an estimate of potential bandwidth
reductions on the costly transatlantic link.

  * •

Upgrading the WLCG software distribution and detector conditions data caching
network with modern software: We have begun investigations of alternate
approaches for managing cache-friendly data at WLCG sites, such as using the
Varnish (https://www.varnish-software.com/) software. Alternate data
distribution methods would aim to decrease the total cost to operate WLCG
sites, helping close the overall resource gap.

Integration with Experiment Environments on Production Infrastructure
Funding Scenarios: High
Description: Throughout the development process, from prototypes to pre-
production software and systems, there are points of integration with the
larger software ecosystems of the experiments that must be designed, developed
and iteratively tested. This requires significant and frequent contact among
the development teams and with systems staff from the facilities. In addition,
to perform meaningful tests with live environments and at meaningful scales
for grand challenge demonstrators, production-scale resources must be
marshalled and fair-share scheduled with on-going operations.
Current and Potential Future Activities:

  * •

Facility integration of pre-production services: In the first phase of IRIS-
HEP, providing efficient interfaces between ServiceX and Rucio, the LHC data
management service, was a significant challenge and required several
optimizations to reduce latencies in queries to the central production
service. As Rucio evolves with its own development roadmap this integration
work will need to continue. As discussed in the DOMA strategic plan,
integration of ServiceX with iDDS provides an opportunity to scale-up
transformer processor capacity, and conversely, provide PanDA brokered
analysis tasks with a columnar reformatting service. There are potential
performance benefits from improved integration of ServiceX with XCache that
can only be explored in production settings. Infrastructure and deployments
for ancillary services like MinIO, the storage service for ServiceX formatted
data outputs, including utilities to manage these outputs are needed.
Discovering and removing experiment-specific dependencies (overlooked from
early development prototypes) making them suitable for use by multiple
experiments is often done in this phase. Supporting operation and user access
to pre-production instances of DOMA services and analysis facilities for early
adopting communities, including first draft user guides, requires expert
knowledge from both domains.

  * •

Preparing demonstrator resources at scale: Our approaches to date have been
driven by IRIS-HEP developer priorities, analysis grand challenge goals,
facility R&D opportunities, and a desire to find a sustainable path forward
for creating declarative, reproducible cyberinfrastructure capable of scaling
out to wherever resources may be found. In addition to IRIS-HEP data and
analysis grand challenge exercises, the LHC experiments are planning
additional demonstrators as input to computing technical design reports
(TDRs). These TDRs will include a prioritization of R&D projects to complete
during Long Shutdown 3 (S3) of the LHC. To inform those decisions, R&D
projects for TDRs must provide a program of work towards the Run4 software and
systems releases. They must include risks and effort estimates and proof-of-
concept demonstrators which will lead to estimates on impacts for resource
utilization (e.g. CPU time, GPU time, RAM, local disk and class of service,
tape, and networks). IRIS-HEP will work with the computing and software
management teams to align goals, configurations, data sets, and platforms to
be used for these impact studies.

  * •

Advanced integration with HPC facilities: Even as HPC resources are viewed as
a prime opportunity to close the processing gap during the course of HL-LHC,
coupling the experiment’s data and workload management services to those
facilities will remain a significant challenge as high energy physics is only
one of a broad collection of science-drivers served by those facilities. A
number of challenges have been identified in a recent workshop
(https://indico.cern.ch/event/1183995/), among them data delivery into and
from these facilities. HPCs are likely to support Kubernetes platforms within
their infrastructure, even beyond edge platforms for hosting user services,
potentially providing additional options to incorporate those resources into
the LHC computing environment. Working with OSG-LHC, the SSL group will
contribute to strategies to efficiently interface experiment systems with the
resource “APIs” of HPC facilities.

  * •

Infrastructure for training and on-boarding: To ensure community adoption and
maximize user feedback to developers, it is essential to continue to provide
training infrastructure for IRIS-HEP software tutorial events, experiment on-
boarding events to analysis facilities, and advanced training activities
sponsored by the Insitute, such as the annual Computational and Data Science
Training for High Energy Physics school (CoDaS-HEP). Where experiment
dedicated resources cannot be used (e.g. production Tier-3 analysis centers),
the SSL can provide the necessary notebooks, CPU and GPU resources for diverse
groups of young researchers during these events.

#### Impact and Success Criteria

Equipping IRIS-HEP with the Scalable Systems Laboratory not only accelerated
the pace of development of DOMA and Analysis Systems, and their integration
with experiment production services and facility systems, but due to its
versatility provided a unique resource in the field of high energy physics
computing. The SSL has hosted instances of the ATLAS Distributed Computing
Analytics Platform (https://analytics.mwt2.org), OSG Compute Entry points
(providing access to cluster resources from over 20 universities), and
provided a DevOps environment for a diverse community of CI engineers. Clearly
the SSL facility and team assembled to operate it has been an important
contributor to the overall NSF cyberinfrastructure ecosystem. The SSL
implemented approaches for creating and managing flexible and reproducible
infrastructure that others in the WLCG community are adopting. Maintaining an
agile, open facility environment to incubate ideas, try out service prototypes
in context, explore and cultivate new patterns for infrastructure delivery,
and to confront the integration and scalability challenges over the next five
years will be vital to the success of the Institute.

It is important to note that while specific resources (dedicated clusters) and
infrastructure have been assembled and labeled as “the SSL” in the first five
years of IRIS-HEP, facility R&D is of course broadly scoped and distributed
across the community, receiving contributions from software developers, CI
engineers, systems administrators and researchers from many universities,
national laboratories and CERN. In the Institute we have the opportunity to
bring convergence to approaches from many sources that will close the gaps to
meeting HL-LHC computing resource needs and capabilities.

Success Criteria – Milestones & Deliverables:

As for other areas in an Institute, the milestones and deliverables outlined
below focus on the first three years. These high-level items would be expanded
in later years as part of the execution of any project. Note however that at
the time of writing, many proof-of-concept demonstrators and facility R&D
topics are being planned by the wider community during this time frame,
potentially impacting deliverables described below.

  1. D5.1.

Establish infrastructure and orchestration services necessary for a sustained
reference SSL. December 2023

  2. D5.2.

With DOMA, ServiceX deployed inside FABRIC at CERN. December 2023

  3. D5.3.

Releases of Analysis Systems pipelines supporting distributed analysis
deployed on SSL. Continuous

  4. D5.4.

Provide a deployment package for upgraded software distribution and conditions
data caching servers. May 2024

  5. D5.5.

With AS, evaluate distributed workers for Coffea-Casa. May 2024

  6. D5.6.

Curate and publish production Tier-3 deployment patterns including integration
of traditional systems in use during Run3 and forward-looking analysis
systems. December 2024

  7. D5.7.

With AS, support demonstration of running a full analysis suitable that uses
machine learning. December 2024

  8. D5.8.

Evaluate technologies, services and infrastructure management patterns for
next generation LHC Tier-2 facilities and publish for the community. December
2025

Success Criteria – Metrics:

Metrics are a useful tool to provide management with quantitative insight
about progress toward overarching goals. High-level metrics we expect to be
applicable for the Facility R&D and Integration area are below:

  1. M5.1.

Number of deployed analysis facilities in production operation. Goal is 6.

  2. M5.2.

Number of sites providing distributed workers to an analysis facility in
production operation. Goal is 3.

  3. M5.3.

Number of storage-less sites in sustained production. Goal is 2.

  4. M5.4.

Number of sites with upgraded software distribution and detector conditions
data caching servers in production. Goal is 7.

###  5.6 Fabric of distributed high-throughput computing services

The LHC depends on a global, distributed cyberinfrastructure to process, move,
and store data and that infrastructure will need to evolve to meet the needs
of the HL-LHC. The OSG Consortium provides the OSG Fabric of Services, a
tightly-integrated set of services which provides for the cyberinfrastructure
of the existing LHC experiments within the US.

Established in 2005, the OSG Consortium operates a fabric of distributed High
Throughput Computing (dHTC) services in support of the National Science &
Engineering community. The research collaborations, campuses, national
laboratories, and software providers that form the consortium are unified in
their commitment to advance open science via these services.

The OSG Consortium is not a legal entity but a coordinated set of stakeholders
working toward a common goal. A strategic area – the “OSG-LHC” - is needed to
make contributions to the OSG Consortium to cover the existing
cyberinfrastructure needs of the LHC, ensure the U.S. LHC facilities remain
operational, and integrate the U.S. LHC facilities with the worldwide
infrastructure.

Within the national distributed High Throughput Computing infrastructure, the
LHC community leads in the scale and complexity of the distributed system.
Through its participating in the broader OSG Consortium, work performed and
knowledge gained for the LHC community has a broader impact than LHC and HEP.
During IRIS-HEP, the OSG-LHC team has remained closely aligned with the
Partnership to Advance Throughput Computing (PATh). The latter has a scope of
all of NSF’s Science and Engineering community, ensuring innovations done by
OSG-LHC have a broad national footprint. The OSG-LHC helps the HL-LHC with
both the scalability of its cyberinfrastructure and its sustainability by
sharing it with multiple domains.

Over the past 15 years, the OSG Consortium has provided a stable foundation of
software and common services to meet the HEP community’s needs. An important
aspect that is managing change in a rapidly changing world of software. For
example, the OSG-LHC has led the transition of the entire national distributed
high-throughput computing (dHTC) infrastructure from the niche “Grid Security
Infrastructure” (GSI) identity-based authentication model to industry-standard
bearer tokens implementing capabilities. When the Globus organization
announced it would stop supporting the Globus Toolkit, the OSG-LHC team helped
establish the Grid Community Forum and forked the toolkit to be the Grid
Community Toolkit (GCT). During the first four years of IRIS-HEP, a major
accomplishment was retiring the use of the GCT and replacing it with updated
services.

OSG-LHC also integrates with rest of the institute. For example, through
interaction with the Facilities R&D area, OSG-LHC began to add containers as a
“first-class citizen” for software distribution; both DOMA and Analysis
Systems groups make use of the OSG-LHC container registry; and DOMA’s work to
implement bearer tokens or HTTP-TPC within the XRootD software is distributed
in turn by OSG-LHC. Thus, software not only flows from the developers out to
the communities but the ideas from within the institute also affect OSG-LHC’s
approaches. Managing the full software lifecycle is a critical component of
any distributed CI and an important role the OSG-LHC will continue to play in
the future.

#### Specific Challenges and Opportunities

##### Rise of the container-native facilities

Container-focused (“Cloud Native”) compute facilities are a generational
change in how sites are operated and deliver capacity to the LHC experiments;
see Section 5.5 for an overview of the rapid change in the
cyberinfrastructure. OSG-LHC has significant expertise in delivering services
as well-integrated software packages shipped in the operating system’s default
packaging format. This historically was the desired mechanism for most sites
that deploy the distributed services. Container-focused sites (largely
deployed on top of Kubernetes) prefer services be delivered in containers,
agnostic to the packaging or even the container OS, and have a higher-level
service orchestration language for how services should be deployed. OSG-LHC
has started to deliver services using these formats but there are still
opportunities; for example, containers allow services to be delivered without
needing superuser privileges on the host and container registries provide the
ability to do daily scans of OSG-LHC containers for known security
vulnerabilities.

Not all sites have an even adoption of newer technologies. There is a need
amongst the cyberinfrastructure – not specific to any experiment – around
Kubernetes community and knowledge building, including training and workforce
development.

##### Increasing diversity of hardware resources

The LHC community has long benefited from a period of relatively homogeneous
resources – for well over a decade, all its pledged resources were x86 cores
with 2GB of RAM. With the advent of GPUs, this has slowly started to change
(although at a rate limited by the paucity of large-scale GPU workflows). The
change to heterogeneous resources has a potential to accelerate as more ARM-
based servers are released on the market an become competitive with capacity
based on x86.

##### Integration with HPC, ML, and non-WLCG resources

A common thread throughout the LHC lifetime is the fact its minimum hardware
needs were met through its dedicated resources. During Run 2 and 3, additional
HPC resources from outside WLCG were heavily utilized but these resources were
never critical-path for the science program. For HL-LHC, to meet projected
flat budgets, these non-WLCG resources will become essential. As part of its
experiment-agnostic fabric, the OSG-LHC has the opportunity to help integrate
NSF leadership class and ACCESS-allocated resources into the experiments’
fabrics.

A new type of resource and service expected for the HL-LHC will be machine
learning inference services. Users have long used inference as part of their
analysis but the complexity of the models are expected to increase rapidly
through the years. Instead of trying to host sufficient resources within a
LHC-specific facility, integration with a national-scale investment for
inferencing would deliver value to the community.

##### Operating Production Services

Unlike other strategic areas which have a mixture of projects in various
stages of the software development lifecycle, OSG-LHC’s purpose is to operate,
maintain, and evolve production services. OSG-LHC must ensure the OSG
Consortium’s fabric of services remains operational and delivers value to the
U.S. LHC Operations programs. This includes all parts of the lifecycle,
including deprecation and obsolescence – for example, OSG-LHC will retire the
OSG 3.6 software release series along with the June 2024 end-of-life of
Enterprise Linux 7. OSG-LHC will help the community to bridge from today’s
cyberinfrastructure to HL-LHC’s.

#### Current Approaches and Development Roadmap

Broadening Service Delivery
Funding Scenarios: Medium, High
Description: Services today are delivered as packages for the standard
platform on the WLCG (Enterprise Linux 7, 8, or later on the x86
architecture). As the community progresses to the HL-LHC, the needs will
become more heterogeneous (including ARM servers and the HPC platforms
available as part of the NSF coordinated cyberinfrastructure in the next five
years) and expect alternate delivery mechanisms (including container images
and the service packaging such as Helm charts).
Current and Potential Future Activities:

  * •

Support for Heterogeneous Architectures: Evolve the existing software
compilations and repositories to also support non-x86 architectures including
ARM.

  * •

Container-Native Service Delivery: Beyond individual packages, create and
maintain repositories of service descriptions for newer technology platforms
like Kubernetes. Work with the community to develop and implement best
practices such as reducing privileges inside the running containers.

  * •

HPC Integration: The OSG-LHC has developed a variety mechanisms for
integrating resources into the OSG Fabric of Services. This includes both the
“Hosted CE”, which integrates compute resources over SSH connections, and
supporting XRootD endpoints on using unprivileged containers, giving access to
storage resources at HPC centers.

  * •

Hosting Institute container images: other areas of the Institute produce
software outputs in the form of container images. Working closely with the
Securing an Open and Trustworthy Ecosystem for Research Infrastructure and
Applications (SOTERIA) project, OSG-LHC provides a national
cyberinfrastructre-focused home for IRIS-HEP images.

Maintaining the OSG Fabric of Services
Funding Scenarios: Low, Medium, High
Description: Through the OSG-LHC, the OSG Consortium maintains services
essential to the operations of the U.S. LHC facilities and their integration
into the global cyberinfrastructure. Shared services include resource usage
accounting, endpoint registration, and monitoring of service status.
Current and Potential Future Activities:

  * •

Packaging: The OSG Software team, as part of the OSG-LHC contribution to the
Consortium, maintains a curated software stack for facilities to use for
services like compute Entrypoints (CEs), storage endpoints, and the worker
node runtime client.

  * •

Resource and Service Registration: OSG-LHC maintains an authoritative list of
services part of the OSG Consortium and their association to administrators
(necessary for communications and security responses). We aim to integrate
with the CILogon-based Single Sign On (SSO) to ensure administrators can login
to the infrastructure with their local credentials. The same mechanisms can be
extended to help simplify workflows for new Analysis Facilities and other
services such as container registries for ATLAS analysis users.

  * •

International Cyberinfrastructure Integration: Facilities have a requirement
to report usage to the WLCG and report monitoring to ensure the U.S. LHC
organizations are delivering on their commitments. OSG-LHC manages accounting
and monitoring services and ensures they are adopted to changes coming from
the WLCG.

Operational Cybersecurity
Funding Scenarios: Low, Medium, High
Description: Like any cyberinfrastructure, the LHC facilities, OSG Fabric of
Services, and the user population are resources continuously targeted by
hackers. The OSG Security area provides effort for operational security,
responsible for security exercises and policy, responding to threats, and
scanning services for vulnerabilities.
Current and Potential Future Activities:

  * •

Coordination with global cybersecurity teams: Like data transfer, having
functional cybersecurity requires interoperating with other infrastructures.
The OSG-LHC cybersecurity team is the bridge between U.S. LHC efforts and EGI,
CERN, and WLCG security teams.

  * •

Improved monitoring of software artifacts and container-based services: Many
of the security controls were designed when services consisted of packages
deployed on physical hosts. The majority of services, however, run inside
containers and are transient – meaning logs are likely lost unless they are
retained inside a central logging service. Security controls need to be
adopted and policies need to be updated to provide guidance to facilities on
similar approaches.

Network Monitoring
Funding Scenarios: Medium, High
Description: Whether for bulk data movement or for streaming events to
consumers, network performance is critical for the LHC community. The OSG-LHC
Network Monitoring Area helps develop and operate technologies for monitoring
performance and understanding network usage patterns at the application level.
Current and Potential Future Activities:

  * •

Packet monitoring: Historically, network operators have had little insight
into the usage of their networks by the LHC community which makes long-term
planning more difficult. Typically, traffic between LHC sites are all labelled
as “LHC” without regards to whether they are actually LHC traffic (versus
other experiments), user versus production, or high priority versus low
priority. The network monitoring team is working to have transfer services
annotate packet header so network providers have richer metadata.

  * •

Leadership of the WLCG Network Monitoring task force: Networking is another
point where keeping the international community aligned is key. Currently,
OSG-LHC co-leads the WLCG Network Monitoring task force and is well-positioned
to guide network transitions between now and HL-LHC.

#### Impact and Success Criteria

OSG-LHC is unique among the Institute strategic areas in that its primary goal
is to ensure smooth operations of the cyberinfrastructure for the LHC, with
services meeting defined SLAs; “success” during this period is providing a
operational roadmap to the HL-LHC era. However, there are expected outcomes
and milestones beyond operations. For example, the area would finish the
migration of services to Kubernetes and start supporting Kubernetes-only (or,
at least, Kubernetes-centric) sites. The current OSG-LHC provides leadership
in running production services on Kubernetes and we expect these new and
converted facilities will rely heavily on its expertise. OSG-LHC will also be
essential to the deployment of next-generation network monitoring, enabling
network providers to introspect HL-LHC flows.

In its current incarnation, OSG-LHC consists of a team who have led the
evolution of the OSG over the past 15 years, meaning there is a wealth of
accumulated expertise in running a fabric of distributed high-throughput
computing services and deep technical knowledge of how the current system
operates. It is an essential element of any future Institute. Without it, the
U.S. LHC operations programs would need to pick up the activities and it is
unlikely that separate implementations by experiments would be as cost-
effective as a common solution. Having the OSG-LHC be part of the future
cyberinfrastructure for the HL-LHC will ensure key continuity of operations
from now until the HL-LHC era.

##### HL-LHC Computing Challenge Impact

: OSG-LHC will primarily have impact in the G4 (Sustainability) and G2
(Scalability) computing challenges. The approach of the OSG-LHC strategic area
–- contributing to the wider OSG Consortium -– improves the sustainability of
the cyberinfrastructure by having a shared common layer across NSF Science and
Engineering domains. We see this as being mutually beneficial: OSG-LHC will
use the commons that have contributions from multiple projects and other
projects will benefit from the unique scalability experience of the LHC/HL-LHC
community. OSG-LHC will provide the foundational infrastructure for the HL-LHC
and is thus responsible for ensuring the cyberinfrastructure can meet HL-LHC’s
scalability goals.

Success Criteria - Milestones:

  1. D.1.

Use of network flow monitoring as part of the next data challenge exercise;
see Section 6.2. March 2024

  2. D.2.

Successful use of tokens for data transfers during the next data challenge
exercise (joint with DOMA). March 2024

  3. D.3.

Retirement of the OSG 3.6 along with the end-of-life of RHEL7. July 2024

  4. D.4.

Remove last requirements and usage of GSI / X.509 security from the OSG Fabric
of Services at the end of Run 3. January 2026

Success Criteria - Metrics:

  1. M.1.

Percent of critical OSG-LHC services meeting their agreed-upon SLAs. Goal is
100% of SLAs met, measured quarterly.

  2. M.2.

Number of IRIS-HEP and LHC container images hosted in the OSG-LHC container
registry. Goal steady increase, measured quarterly.

###  5.7 Training and Workforce Development

People are the key to successful software. The community is currently building
hardware upgrades and planning for an HL-LHC era which will start collecting
data at the end of this decade, and then acquire data for at least another
decade. People, working together, across disciplines and experiments, over
several generations, will be the critical foundation underlying sustainable
software.

#### Specific Challenges and Opportunities

Training support for software-related activities in HEP has historically been
uneven. Although most universities do provide some relevant computer science
and software engineering courses, and many now provide introductory “data
science” courses, many HEP graduate students and postdocs are not required to
take these classes as part of the curriculum. As students enter the research
phase of graduate student training, many recognize the value of such classes,
but are no longer in a position to take the classes easily. No “standard”
recommendations exist for incoming students, either for HEP experiments or the
HEP field as a whole. Some universities are developing curricula for STEM
training in general or “certificate” programs for basic data science or
software training, but these are by no means yet universal. The result has
been that the graduate student and postdoc population historically had a
diverse spectrum of relevant skills.

To address this, IRIS-HEP worked with HSF to develop a community vision five
years ago (Figure 11) for a progression of training activities which, if
implemented by the community, would create a new generation of software-
enabled scientists. The ultimate goal was to invest today in the young
students and postdocs who would become faculty leaders driving the research
agenda in the HL-LHC era. IRIS-HEP has been working with the community to
implement this vision. A sustainable framework for software training will be
an important community legacy from IRIS-HEP and position the community to
succeed maximally in the HL-LHC era.

Figure 11: The HSF vision for software training in particle physics, from
generally required basic software skills through advanced developer training,
including eventual mentoring to contribute to research software projects.

#### Current Approaches and Development Roadmap

##### Basic Curriculum:

Working with the HSF training group over the past few years, a complete basic
curriculum (the lowest tier in Figure 11) has been implemented [61] at a level
appropriate for HEP students. It includes material developed by The
Carpentries and additional modules tailored to HEP beginners. Using this
curriculum, the IRIS-HEP software training group and collaborators have
organized more than 30 training events and trained over 1,500 students and
postdocs. The material has also been adopted by the DUNE experiment for
software training activities as well as the USCMS PURSUE summer student
program [62], which focuses on broadening participation. Driven by a community
of more than 50 motivated educators, the training modules are open-source and
collaborative in nature. They span the spectrum from robust basic software
skills to advanced topics like machine learning on GPUs. Sustainability has
been the centerpiece of the approach, given that there are hundreds of new
entrants in the community every year. Different tools and platforms, like
GitHub, have enabled technical continuity, collaboration, and nurtured the
sense to develop reproducible and reusable software. A significant effort has
been devoted to ensuring that the basic software curriculum is taught
frequently enough that most students can be trained very early in their
research careers.

Following the approach of The Carpentries, each training module corresponds to
a training webpage, including verbose descriptions, key-point summaries, and
exercises to make training interactive. Recordings for all of our workshops
are available, and several of our lessons are accompanied by dedicated videos
targeting individual learners. As of 2022, the basic curriculum is considered
complete and in production use by the community. A first set of reusable
training modules on more advanced topics is also becoming available. (See
Figure 12 for examples, and the HSF Training Center [63] for the full set.)

Figure 12: Some elements of the HSF training curriculum. The full set can be
found on the HSF Training Center website. [63] Figure 13: Growing the training
group. Four different cumulative metrics are overlaid: the number of
registrations in our events, the number of unique commits in our repositories
(excluding commits to the framework developed by The Carpentries), the number
of educators registered in the HSF Training community, and the number of
unique commit authors (again excluding commits attributed to The Carpentries).
The absolute values of each metric are shown in the legend. Figure 14:
Participants in the CoDaS-HEP 2022 summer school.

##### Expert-level workshops and mentoring:

While the basic training is typically sufficient for physicists involved
primarily in small-group data analysis, many community members become
Cyberinfrastructure professionals and will be involved in larger-scale
software development activities for their experiments. Because of the long
time scale of the experiments, imparting sustainable software development best
practices is crucial. In addition, more advanced software and data science
skills are often a strong asset when career evolution takes people to industry
or other academic research domains. Offering orientation about different
career paths for Cyberinfrastructure professionals will be an important aspect
of sustaining the demand for software experts.

Some elements of more advanced training have been put in place. For example,
the intensive week-long Computational and Data Science for High Energy Physics
(CoDaS-HEP) summer school was begun in 2017 with the aim of teaching more
advanced skills such as parallel programming, data science tools and
techniques, machine learning technology and methods, code performance
evaluation and collaborative use of git and GitHub. The school consists of
both lectures and hands-on sessions and brings together young scientists from
various HEP experiments (not only LHC) with experts from HEP and beyond. A
“cohort” experience is also created by bringing the participants (that have
similar interests) together in a single location for a week. The school was
originally developed as part of an older NSF-funded project (PHY-1521042) and
participant support has also been provided by an NSF CyberTraining award
(OAC-1829729). Many instructors have, however, typically come from among the
personnel funded by IRIS-HEP. The school was not run in 2020 and 2021 due to
COVID, but began again in 2022 and is expected to run annually going forward.

Finally, it is important to note that many advanced concepts, such as
architectural design decisions, cannot be easily imparted by teaching alone
but must be practiced deliberately under the guidance of mentors that have
experience developing research software used by others. For this purpose, the
IRIS-HEP Fellows program was created. Fellowships provide an intense and
personalized form of developer training. During a period of several months,
Fellows work on a project of their choosing, closely mentored by an expert
from the field. By contributing to a HEP software project, Fellows obtain
first-hand experience in collaborative software development and put their
software knowledge into practice. Since 2019, more than 100 fellowships have
been awarded. The success of the IRIS-HEP Fellows program funded by NSF has
also attracted additional funding from private foundations and industry to
extend the program. For example, twenty students from Ukraine were supported
in 2022 and the Simon institute has supported around 5-8 international fellows
every year since 2020. We see already that some of the CoDaS-HEP attendees and
Fellows remain engaged with HEP research software development as they progress
in their careers.

##### Plans for the next phase of IRIS-HEP:

The current IRIS-HEP project has been instrumental in advancing the vision of
a training pipeline shown in Figure 11. It is, however, not yet complete: in
the HL-LHC era, we want every student entering our experiments to be fully
conversant in basic software skills, with many possessing the more advanced
software skills needed to create and evolve research software in HEP.
Moreover, the available training progression will be widely recognized, and
sustainability mechanisms will be established. For example, previous
“students” of training activities may return as instructors for the next
cohorts. The Fellows program will be widely recognized as an “on-ramp” for
students to not only build skills, but also get connected to experiment and
community research software activities. Finally, the students we train in the
coming years include the faculty, staff scientists and research software
engineers of the HL-LHC era.

###  5.8 LHCb & LHC Run 5

Lastly, we describe the additional strategic interest for ongoing
collaborative efforts within IRIS-HEP regarding LHCb, another LHC experiment
supported from the US primarily by the NSF. The upgrade plan for LHCb is out-
of-phase with respect to ATLAS and CMS: major upgrades happened for LHC Run 3
and are planned for LHC Run 5. For example, between LHC Runs 2 and 3 LHCb
underwent a major upgrade: its hardware trigger was removed and the first
level trigger (Hlt1) now processes 30 MHz of beam crossings, about 5 TB/s of
data. Extending several aspects of the software infrastructure will be
important for LHCb in Run 4 and critical for Run 5, and may serve as models or
starting points for software that is valuable to other experiments during the
HL-LHC era.

As of Run 3, Hlt1 executes on GPUs with one instance of Allen[64] running on
each GPU. Currently, every GPU receives complete events from an event building
unit (an x86 CPU server) and handles several thousand events at once. Raw
detector data is copied to the GPU, the full Hlt1 sequence is processed on the
GPU; only selection decisions and objects used for the selections, such as
tracks and primary vertices, are copied back to the CPU. The data rate between
the two x86 server farms (the Hlt1 farm and the Hlt2 farm) is, therefore,
reduced by a factor 30–60. Allen runs inside Gaudi in production. The native
monitoring software inside Allen has an interface similar to that of Gaudi so
the aggregated histograms are propagated back to the host, eventually giving
them to Gaudi. In the future, LHCb plans to use Allen for the second level
trigger as well. This requires additional work. An x86 compilation of Allen
can run on the WLCG. In that case, the event loop is steered by Gaudi, and
Allen is called one event at a time. While Allen is not the only approach to
using GPUs for reconstruction and event selection entirely within GPUs, it
demonstrates that this is possible. Other experiments might consider adopting
it, or integrating some of its features into their GPU software. As GPUs
become part of the WLCG, it will be important to build infrastructure to allow
Allen to use these resources effectively. The environment will be very
different from that of the online system, so the new infrastructure for using
heterogeneous resources will take dedicated work. The same infrastructure
could be designed to be common to all experiments using WLCG resources.

LHCb developed PyConf, to make Gaudi application configuration safer, cleaner,
and simpler to debug. It was originally developed as a general pythonic
functional framework for configuring complex CPU workflows, but it works for
GPU workflows as well. At the moment, this package is specific to the LHCb
version of Gaudi, but it should be possible to generalize for other users of
Gaudi (such as ATLAS). This could be an appropriate effort for IRIS-HEP.

Recently, LHCb demonstrated that an end-to-end deep neural network (DNN) can
provide better efficiency and a lower false positive rate for finding primary
vertices (PVs) using simluated LHCb Run 3 data. It starts with track
parameters and produces “target histograms” that a simple heuristic algorithm
scans to extract PV positions and estimated longitudinal resolutions. Two
separate DNNs are trained, then joined with extra channels added in the
intermediate layer to allow the overall algorithm to learn additional
additional details. The first consists of fully connected layers. It mimics
the calculation of a heuristic kernel density estimator (KDE) that is a one-
dimensional representation of where tracks intersect each other. The second is
a convolutional neural network. It starts with KDEs and produces target
histograms. Several CNN architectures produce statistically indistinguishable
results. The preferred architecture is resembles the U-Net architecture [65]
developed for image segmentation in medical applications. The KDE-to-
histograms algorithm has been adapted to find PVs from simulated ATLAS data,
and initial results are promising. Extending the LHCb algorithm to find
secondary vertices and understanding how to best adapt the algorithm for use
by ATLAS and CMS is a promising avenue for IRIS-HEP.

LHCb has deployed an older version of its KDE-to-histograms algorithm in its
CPU software stack. Work is in progress now to deploy the full end-to-end
algorithm inside Allen. NVIDIA’s newest GPUs feature tensor cores in addition
to CUDA cores. The former are optimized for the matrix multiplications that
characterize DNN inference engines. On an RTX-A6000, they nominally provide
310 TFLOPS performance compared to the CUDA cores’ 39 TFLOPS single precision
performance. The tensor cores are not yet used by Allen. If the end-to-end
inference engine can be run using the tensor cores, this will provide much
better use of the silicon. A future direction for research will be studying
how different DNN architectures most effectively use tensor cores. As many
architectures can provide the same physics performance, identifying those that
provide the best bang per buck will be important as we (HEP computing) think
about how to replace other types of heuristic algorithms with DNNs.

A natural follow-on to developing DNNs to replace heuristic PV-finding
algorithms is developing DNNs to replace aspects of Kalman filters. In an
oversimplified picture, Kalman filters have three functions: (i) they describe
trajectories, (ii) they describe the covariance matrices along the
trajectories, and (iii) they provide quality metrics describing how well the
hits used to construct the trajectories match the projected trajectories. In
the same way that the algorithm for finding PVs was broken into two steps
using domain expertise, the Kalman filter problem will need to be broken into
steps and each step addressed separately before the pieces can be put back
together again. DNNs do not need to provide better physics performance than
heuristic algorithms – they need to execute more quickly by taking better
advantage of the silicon available in GPUs. As the number of tracks per event
increases during HL-LHC, this will become more and more important. While the
details would change from one experiment to another, this is another area
where a cross-experiment effort could produce a significant advance.

The work described in this section overlaps strongly with the work described
in Sections 5.2 and 5.3. The PyConf work discussed in this section may help
address some of the issues related to the increasing diversity of hardware
resources in the WLCG discussed in Section 5.5.

##  6 Grand Challenges for the HL-LHC

Starting in 2020, IRIS-HEP has introduced the concept of a “Grand Challenge”
within the institute to focus one or more areas on a long-term, large-scale
goal part of the institute’s overall vision. A Grand Challenge differs from a
more traditional milestone or deliverable by its scale (often requiring cross-
cutting teams working together), a multi-year timeframe, and the fact the
entirety of the approach may not be known upfront. The challenges are executed
through a series of increasingly difficult exercises coordinated throughout
the community. The currently defined grand challenges are:

  * •

Analysis Grand Challenge (AGC): The AGC aims to execute realistic analyses at
the scale and complexity envisioned by the HL-LHC using a set of tools,
facilities, and services developed within IRIS-HEP as an exemplar. The AGC
team coordinates an annual workshop to demonstrate current progress against
the goals and to update the vision and approach as necessary.

  * •

Data Grand Challenge (DGC): The DGC for IRIS-HEP is realized as a set of
global data challenges coordinated with the WLCG. These challenges, occurring
biennially, bring the entire global community together to demonstrate