# HPAC-ML: A Programming Model for Embedding ML Surrogates in Scientific
Applications

Zane Fink1, Konstantinos Parasyris2, Praneet Rathi1 Giorgis Georgakoudis2,
Harshitha Menon2, Peer-Timo Bremer2  0000-0002-1882-6578 0000-0002-8258-9693
0000-0002-9108-6422 0000-0001-6542-3555 0000-0003-4707-9580
0000-0003-4107-3831 1University of Illinois Urbana-Champaign
{zanef2<EMAIL_ADDRESS>2Lawrence Livermore National Laboratory
{parasyris1, georgakoudis1, harshitha<EMAIL_ADDRESS>
###### Abstract

The end of Dennard scaling and the slowdown of Moore’s Law led to
heterogeneous architectures benefiting machine learning (ML) algorithms. These
hardware advancements and the development of intuitive domain-specific
languages have made ML more accessible, democratizing innovation. ML models
surpass traditional approximation limits, broadening opportunities and
evolving from statistical to complex function modeling. Consequently,
scientific applications leverage ML models for enhanced execution speeds.
However, integrating ML models remains manual and complex, slowing the
adoption of ML as an approximation technique in modern applications.

We propose an easy-to-use directive-based programming model that enables
developers to describe the use of ML models in scientific applications. The
runtime support, as instructed by the programming model, performs data
assimilation using the original algorithm and can replace the algorithm with
model inference. Our evaluation across five benchmarks, testing over 5000 ML
models, shows up to 83.6$\times$ speed improvements with minimal accuracy loss
(as low as 0.01 RMSE).

##  I Introduction

Predictive simulation is a cornerstone of scientific progress in several
fields ranging from quantum chemistry to national security, where the speed of
simulations directly influences the pace of discovery. Until recently,
hardware and software innovations drove progress with increased simulation
speeds. However, the deceleration of hardware technology advancements, marked
by the end of Dennard scaling and the slowing of Moore’s Law, poses a
significant challenge to maintaining this pace of acceleration.

Approximate computing — a promising direction in the post-Moore computing era
— reduces application accuracy for increased performance and energy
efficiency. Approximations can be application-specific, such as Newton’s
method [1] and polynomial approximations [2], or general-purpose techniques
such as loop perforation [3, 4], precision scaling [5, 6], and memoization [7,
8]. More recently, the widespread adoption of specialized accelerators in HPC
and the growing prominence of Machine Learning (ML) has led to the exploration
of using Neural Network (NN) based surrogate models to accelerate scientific
simulations. One of the key benefits of NN-based approximations is that NN are
universal function approximators: with sufficient data and a sufficiently
complex network, they can approximate any function. Furthermore, neural
networks offer a rich space of architectures such as MLPs, CNNs, and RNNs,
providing flexibility and adaptability, thus making them a versatile and
efficient option for many scientific applications. As a result, NN surrogates
have achieved wide success in several domains, including radiative transfer
calculations [9], computational fluid dynamics [10], and Molecular Dynamics
(MD) simulations, demonstrating at least 10$\times$ speedup with low error.

Despite ML’s performance benefits, collecting training data and integrating
such a model in a scientific application is a manual, complicated, and error-
prone process. Since scientific applications are usually developed in high-
performance languages such as C/C++ and FORTRAN, ML models must be invoked
within the programming language’s ecosystem. This requires application and ML
expertise. For example, transforming data to the tensor format expected by ML
models requires intimate knowledge of data transformations, increasing the
complexity of model integration in scientific codes.

Therefore, the adoption and efficient use of ML models in scientific
applications is slow, hindering the exploration of their applicability to
diverse applications and situations. The development complexity of such an ML-
enhanced application has gone unchecked. With no additional support to ease
the integration of ML models in scientific applications, the adoption rate of
ML in HPC is slow, and the HPC community is missing out on potential benefits.
Moreover, progress in understanding and overcoming the limitations is hindered
by the engineering efforts required to embed models in applications.

We present hpac-ml, a directive-based programming model that enables easy use
of ML models in scientific applications. The programming model provides
primitives to invoke a model or collect data during application execution to
be used during offline training of machine learning models. hpac-ml provides
succinct extensions to describe transformations between application and ML
model memory layout. The runtime system can schedule execution on both CPUs
and GPUs, transparently handling data transfers. We extend HPAC [11, 12] and
thus rely on the mature LLVM infrastructure. As an inference engine, the
runtime system uses Torch (C++ interface to PyTorch). Thus, the proposed
approach can digest the latest advancements in compilation and AI/ML
technologies.

The programming model facilitates our core mechanism to explore the trade-offs
between model size, model architecture, performance benefits, and accuracy
degradation in 5 HPC benchmarks. Our evaluation presents how well-defined
programming abstractions can help automate the search for appropriate model
architectures. Instead of manually exploring these trade-offs, we employ
Bayesian Optimization to search the space of possible machine learning models,
tracking both performance gains and accuracy loss across all benchmarks.
Briefly, our contributions are the following:

  * •

hpac-ml, an annotation-based programming model for embedding ML surrogates in
scientific applications. Application developers annotate parts of their
application to replace them with a surrogate model. hpac-ml enables developers
to describe the data flow between application memory ML data layouts using
succinct syntax. This scheme supports supervised machine learning workflows
while maintaining modularity and separation of concerns: no single person is
required to have both application expertise and experience with ML
infrastructure. The same source code can collect training data and later load
a trained surrogate model that replaces the annotated code region accelerating
the application.

  * •

An implementation of the hpac-ml programming model in the production-level
Clang/LLVM compiler with a runtime system that uses industry-standard
infrastructure such as Torch for inference and HDF5 for data collection. The
implementation is open-source software available to the
community111https://github.com/LLNL/HPAC.

  * •

A thorough evaluation on a representative suite of GPU-based HPC mini-apps and
benchmarks. Overall, we find up to $83.6\times$ speedup while maintaining
output quality with RMSE as low as $0.001$, while the layout transformations
add negligible overhead.

  * •

hpac-ml enables a large-scale exploration of the trade-offs between model
size, inference speed, and output quality loss. Across thousands of models
tested in our evaluation, we derive insightful observations. For instance, for
ParticleFilter we identify models with better accuracy than the existing
approximation that yield an end-to-end speedup of $9\times$.

##  II Background and Related Work

Approximate computing (AC) increases computing power and energy efficiency.
Several AC hardware and software strategies demonstrate its utility. Hardware
approximation techniques [13, 14, 15, 16, 17, 18, 19, 20, 21, 22] specialized
hardware components or modify underlying hardware to accelerate computations
while introducing controlled errors. For instance, approximate
adders/multipliers [13, 14, 15, 16, 17] perform add and multiply operations
approximately. Software approximation techniques focus on algorithmic and
software modifications that introduce approximations to gain performance.
Examples include loop perforation [23, 24], which skips certain loop
iterations, memoization [25, 26, 27], which caches previous previous results,
and significance based computing [28, 29, 30] which assign scores to
computations in respect to their contribution to the final output quality.
Additionally, there are several works that use Neural Networks (NNs) to
approximate code regions, accelerating computations. Grigorian et al. [31]
proposed BRAINIAC that used both precise and NN-based approximate accelerators
to achieve significant performance gains. McAfee and Olukotun proposed EMEURO
[32], a NN-based emulation and acceleration platform. Moreau et al. [33]
presented a technique for neural acceleration on an FPGA. Esmaeilzadeh et al.
[34] introduced a programming model that enabled users to offload approximable
code regions to Neural Processing Units (NPUs). Ho et al. [35] proposed
Tensorox to exploit the half-precision tensor cores available on GPUs to
perform approximate computations. These works have focused primarily on
leveraging hardware neural networks as accelerators for approximable programs,
and hence their applicability for integrating ML libraries into scientific
codes is limited. Many frameworks integrate AC techniques into codes. These
include HPAC [11], HPAC-Offload [12], ApproxHPVM [36], ACCEPT [37], and GREEN
[38]. Only HPAC-Offload and SAGE support AC on GPUs. While these frameworks
provide valuable support for several approximate computing techniques, they
don’t support surrogate models, ultimately hindering the adoption of ML models
in HPC applications.

HPAC allows accurate and approximate versions of the same algorithm to coexist
in the same binary by viewing them as different _execution paths_. The path
taken is dynamically decided at runtime based on activation criteria for
approximation. If the activation criteria are unmet, then the _accurate
execution path_ is taken, and the original code is executed. Otherwise, the
_approximate execution path_ , which implements the approximation, is taken.
HPAC provides data-flow semantics[39] to the developer that describe which
application memory regions are accessed by the accurate execution path.

Recently, Dong et al. [40] proposed Auto-HPCnet, which uses a client-server
protocol to enable NN-based surrogate models, losing integrated support and
incurring communication costs between the client and server. Myers et al. [41]
developed Python bindings as a bridge between AMR simulation data and data
science software. They provide a flexible approach for representing simulation
data and supporting routines written in Python; however, they don’t exploit
the latest advances in state-of-the-art ML libraries such as PyTorch [42],
TensorFlow [43], ONNX [44], and LBANN [45]. These ML frameworks allow for a
trained model to be encoded in a language-agnostic JIT-able model formats,
such as TorchScript JIT. The model can be loaded using C++ or Fortran and
invoked to perform inference at application execution time.

Our framework extends HPAC and provides a unified approach to seamlessly use
NN surrogate models within an application. The approach separates concerns and
can be used as a core mechanism, as done in this work, to build automation and
study trade-offs between model accuracy, application performance, and
application accuracy.

##  III hpac-ml Programming Model

Supervised machine learning consists of three phases: data collection,
training, and inference. The current practice of using NN models in
applications follows multiple manual, error-prone sequential steps. Further,
it requires rare interdisciplinary expertise. For example, computer scientists
instrument scientific applications for data collection and provide the data to
ML experts who build and train NN models. These models are integrated into the
application by computer scientists. Domain scientists (e.g., physicists) then
evaluate the validity of the ML-enhanced application’s results. Simply put,
there are too many handshakes in this workflow.

The approach does not scale, limiting scientific throughput. Thus hpac-ml,
facilitates smooth integration of ML surrogates in applications by providing
primitives to capture the developer’s intent to accelerate the application via
NN model usage. hpac-ml’s design abstracts the engineering support required to
use ML models in applications and insulates the user from knowing _how_ the
software system works. Instead, ML and domain scientists can focus on building
more accurate models and validating the ML-enhanced application results.

Figure 1: hpac-ml overview.

###  III-A hpac-ml Design

The main challenge in abstracting ML use is twofold. First, the application
developer needs primitives to specify the source code location, often called
the code region, to be replaced by the ML model. Further primitives are
required to define the application data that the code region consumes and
produces. Yet, describing only the coarse data flow of a code region is not
sufficient. The application data layout and the layout required by modern NN
inference engines differ. Thus, a generic approach needs to provide primitives
to the developer to describe the application data layout and flow at the
boundaries of the replaced code regions.

Specifically, from the application code perspective, we need abstractions for
two key components: a data bridge layer and an execution control layer. The
data bridge layer handles the mismatch between application data and ML
frameworks, which expect dense, contiguous inputs and produce dense,
contiguous outputs. The execution control layer is responsible for either
performing data collection or model inference. The hpac-ml approach, depicted
in Figure 1, integrates the requirements of both the application and surrogate
models in a framework-agnostic way.

####  III-A1 Data Bridge

The data bridge connects two distinct memory spaces: the application memory
space and the tensor memory space, supporting memory access patterns common in
scientific applications. For instance, computational fluid dynamics and finite
difference methods employ a stencil access pattern, where the value at each
gridpoint is updated based on the values of neighboring gridpoints.
Unstructured mesh codes use indirection and pointer-chasing to resolve the map
between application data structures and the computational models used in
simulations, thereby efficiently bridging the gap between their geometrical
representation and the algorithms that operate on them. ML applications such
as clustering compare all points in some space to the same set of queries. In
such cases, it is not enough to simply wrap application memory arrays in ML
library tensors – there is a difference between how data are accessed by the
application and how data are organized in memory. Thus, the data bridge uses
two concepts to handle each component separately: a tensor functor and a
tensor map.

The tensor functor is an abstraction that lets the user specify how individual
data elements are accessed in the application memory space and how these
elements should be organized to form a tensor entry in the tensor memory
space. Through a process we call memory concretization, the tensor map applies
the tensor functor to application memory, completing the bridge between the
application and tensor memory spaces and mapping memory between them. For
example, in Figure 1 (left), the developer describes a tensor representing a
stencil and a tensor mapping. At execution time hpac-ml (Figure 1 middle,
right) uses both the tensor map and the tensor functor in Steps 1 and 2 to
extract tensors representing a 5-point stencil from a contiguous memory grid.
The reverse operation transforms the data layout from the model to the
application layout (Steps 5 and 6 in Figure 1 middle and right).

####  III-A2 Execution Control

The execution control layer leverages existing HPAC primitives. Briefly, when
using HPAC the developer annotates a code region as approximate and specifies
the data flow — the memory locations used as inputs by the annotated region
and the memory locations of the computed outputs — of the code region. With
this information, HPAC generates two separate execution paths: an _accurate
execution path_ with the original application code and the _approximate
execution path_ implementing the approximation.

hpac-ml extends the accurate execution path to perform data collection during
application execution time as presented in Figure 1 (middle). Before executing
the accurate execution path, hpac-ml applies the tensor functors to the
region’s inputs and maps them into concrete tensors (Steps 1 and 2). These
tensors are cached in memory. Then, hpac-ml invokes the accurate execution
path, and the application computes the output values. Before giving execution
control back to the application, hpac-ml transforms the output memory
locations using functors and tensor mapping operations (Steps 5 and 6).
Lastly, hpac-ml stores this data into a persistent storage database and
continues execution.

The persistent storage database can be accessed by the ML expert to train a NN
approximation of the accurate execution path. Once the ML experts produce a
model of sufficient accuracy they can store it in the filesystem and expose
the model path to hpac-ml. In such a case, as presented in Figure 1 (right),
during application execution, hpac-ml reads the model representation and uses
the model as the approximate execution path of HPAC. Before invoking the
model, hpac-ml uses developer-defined tensor functors and mapping operations
provided by the developer to transform data between the application data
layout and model layout.

To summarize, the execution control of hpac-ml is responsible for either
performing data collection or model inference, whereas the data bridge spans
the gap between data layouts.

⬇

#pragma approx tensor functor(ifnctr: @\label{ex_functor_decl}@ \ [i, j, 0:5]
= ( ([i-1, j], [i+1, j], \ [i, j-1:j+2]))

#pragma approx tensor functor(ofnctr: @\label{ex_functor_decl_opt}@ \ [i, j,
0:1] = ([i, j]))

⬇

#pragma approx tensor map(to: @\label{ex_map_to_clause_line}@ \
ifnctr(t[1:N-1, 1:M-1]))

#pragma approx tensor map(from: @\label{ex_map_from_clause_line}@ \
ofnctr(tnew[1:N-1, 1:M-1]))

⬇

#pragma approx ml(predicated:true) in(t) out(tnew) @\label{ex_inference_call}@
\ db(”/path/data.h5”) model(”/path/model.pt”)

do_timestep(t, tnew);

\parswap(t, tnew);

Figure 2: hpac-ml example program. This example replaces the main computation
of a 2-D stencil code with NN surrogate inference. green, highlights the
declaration of tensor-functors, red, highlights the declaration of tensor
maps, blue, highlights the declaration of a region from which hpac-ml will
either collect data from or completely replace with a NN model.

###  III-B hpac-ml Example

The hpac-ml programming model provides concrete realizations of the data
bridge and execution control abstractions. Figure 2 depicts an example where
the directive on Line LABEL:ex_functor_decl creates a tensor functor named
`ifnctr` and applies it to application memory on Line
LABEL:ex_map_to_clause_line, mapping application memory to the tensor memory
space. On Line LABEL:ex_map_from_clause_line, a mapping from the tensor memory
space to the application memory space is created. The tensor functor
declaration has two main components: a left-hand side (LHS) and a right-hand
side (RHS).

The LHS defines an abstract slice that describes a single element access for
some symbolic values `i, j`. In Figure 2, the LHS declares a 3-dimensional
tensor with two spatial dimensions and one feature dimension. The user
specifies this shape using $i$ and $j$ and the range $0$:$5$. The names $i$
and $j$ are not declared in the program; they are considered symbols for any
values $i$ and $j$ may take when the functor is applied to data. The RHS
component specifies the origin of each feature without associating the indices
with any data. Applying a tensor functor to data arrays moves them between the
application and tensor memory spaces, completing the data bridge. Line
LABEL:ex_map_to_clause_line of Figure 2 does this, where the user maps data
`to` the tensor space by applying `functr` to the array `t`. This is where the
range of values are assigned to the symbols of the `ifnctr` functor. Here, $i$
goes from $1$ to $N-1$; $j$ is similarly concretized and mapped to the range
$1:M-1$. Simply put, the tensor map performs a sweep over memory, and the
functor describes the accesses at each point in the sweep.

Finally, the developer uses the `ml` clause with the `predicated` modifier
with the condition set to `true` on Line LABEL:ex_inference_call. Thus, hpac-
ml uses the model stored under `"path/model.pt"` and performs NN approximation
for every instantiation of the region. The `ml` clause accepts a boolean
expression evaluated at every invocation of the region. The developer can
expose this parameter, for example, to the command line interface and toggle
between inference and data collection without requiring recompilation. For
explicit static control of model inference or data collection, the developer
can use `infer` or `collect` to enable inference and data collection for all
instantiations of the code region.

###  III-C hpac-ml Programming Model Grammar

To further understand the programming model, we show the full syntax of the
`tensor functor` and `tensor map` directives along with the `ml` clause in
Figure 3.

The tensor functor directive (Figure 3 top) is powered by the symbolic slice
specifier, or `ss-specifier`, that implements the standard slice syntax with
extensions: symbolic constants and expressions that use them. The symbolic
constant (`s-constant`) terminal serves as a placeholder within the tensor
`ss-specifier`, adapting its value to match the data dimensions provided when
the functor is applied to memory. These constants can be used in arithmetic
expressions with regular integer expressions to describe different offsets and
access patterns involving symbolic constants. For example, in the slice
specifier `[i, j, 0:5]` taken from the RHS of example 2 the symbols `i` and
`j` correspond to an `s-constant` whereas `0` and `5` correspond to integer
expressions.

The `tensor map` (Figure 3 middle) transforms data from the application layout
`to` the ML data layout or `from` the ML layout to the application as
described by the `direction-specifier`. The functor application expression
(`fa-expr`) applies a declared tensor functor identified by the `decl-functor-
id` to a list of user-specified ranges (`map-target-list`). The list contains
a sequence of `map-target` expressions. Every `map-target` initially contains
a reference to some array (the array to be transformed by the functor) and a
series of concrete-slice-specifiers (`cs-specifier`). A concrete specifier may
only contain expressions that use either integer constants or refer to
declared integer variables. The `cs-specifier` expresses ranges of possible
values. For example, `[1:N]` describes a range starting from `1` and going to
exclusive `N` with a step of `1`.

⬇

#pragma approx tensor functor(decl-functor-id:

ss-specifier = (ss-specifier))

ss-specifier ::= ’[’ s-slice, … ’]’

s-slice ::= s-expr [ : [s-expr] [ : [s-expr]]]

s-expr ::= s-expr | s-constant | int-expr

⬇

#pragma approx tensor map(direction-specifier:

fa-expr)

fa-expr ::= decl-functor-id ’(’ map-target-list ’)’

map-target-list ::= map-target …

map-target ::= array-ref

’[’ cs-specifier | map-target ’]’

cs-specifier ::= c-slice [, c-slice]…

c-slice ::= c-expr [ : [c-expr] [ : [c-expr]]]

c-expr ::= c-expr | int-expr

direction-specifier ::= ’to’ | ’from’

⬇

#pragma approx ml(ml-mode [:bool-expr]) [in(mapped-memory)

out(mapped-memory)] [inout(mapped-memory)] model(string-expr) database(string-
expr) if(bool-expr)

structured-block

mapped-memory ::= fa-expr | mapped-target-list

mapped-target-list ::= array-ref [, mapped-target-list]

ml-mode ::= ’infer’ | ’collect’ | ’predicated’

Figure 3: Syntax for the tensor functor declaration (top), tensor mapping
(middle), and ml (bottom) clauses of the hpac-ml programming model.

The `approx` directive combined with the `ml` clause specify regions of
application code to replace with NN surrogate models. The syntax of the
`approx ml` clause is shown in Figure 3 (bottom). The `ml-mode` is a keyword
that accepts `infer,` `collect,` and `predicated`. `infer` instructs hpac-ml
to perform NN inference — replacing the wrapped structured block with the
surrogate model stored at the path provided in the `model` clause. The
`collect` clause instructs hpac-ml to run the structured block and store the
inputs and outputs in a file under the location specified by the `database`
clause. Finally, the `predicated` clause performs model inference or data
collection depending on the boolean expression `ml-cond`. This enables the
dynamic toggling of data collection and model inference. The
`mapped``-``target``-``list` is a list of references to arrays mapped using
the `tensor` `map` clause. The mapped target lists can be designated as input
to the NN/structured block (`in`), output from the NN/structured block
(`out`), or both (`inout`).

##  IV hpac-ml Implementation

The hpac-ml compiler and runtime system implement the data bridge and
execution control layers. The hpac-ml compiler extends the HPAC compiler
implemented in Clang/LLVM v17.

###  IV-A Data Bridge

Figure 4: Steps of the hpac-ml compiler runtime system performed during the
data mapping of ifnctr (example code 2).

hpac-ml implements the data bridge by closely co-designing the compiler and
the runtime system. The runtime system implements custom data structures and
the compiler performs a sequence of calls to runtime functions to declare,
allocate, and fill those structures appropriately.

The parser, semantic analysis, and abstract syntax tree (AST) of Clang are
extended to support the `tensor` directive with its functor, map, and update
clauses. These clauses require new expression types in the AST for multi-
dimensional slices that support symbolic expressions. After parsing and
semantic analysis of the `functor` clause in the `tensor` directive, code
generation instantiates a tensor. In code generation, a functor is effectively
a structure representing views or slices on top of application memory
locations. Until the functor is applied to memory, the pointer to application
memory is undefined and symbolic expressions remain abstract. As such, code
generation generates two descriptors to be used by the runtime library one for
the LHS and one the RHS of the functor. Each represents an `ss-specifier`
(Figure 3).

The descriptors are a collection of slices. For example, `([i-1, j],` `[i+1,
j],` `[i, j-1:j+2])` contains three slices. Every slice describes a view of
memory through some pointer that will be provided when the functor is mapped
to a memory location. A structure describes a slice and contains fields for
the number of dimensions, shape, and strides. Symbolic constants
(`s-constant`), such as `i, j`, are encoded in an AST node during parsing and
semantic analysis to be assigned a unique identifier. The hpac-ml code
generator refers to them to compute possible ranges of values for these
symbolic constants. The compiler bundles the LHS and RHS expressions and
associates them with the `functor-id` provided by the clause.

The tensor map clause creates a view of some user-specified application
memory. Supporting the `map` clause requires a sequence of steps to determine
the base pointer, shape, and strides for converting user memory to a tensor.
Figure 4 illustrates these steps for the example in Figure 2. We demonstrate
the process for `map(ifnctr(t[1:N-1, 1:M-1]))`, where `ifnctr` corresponds to
the LHS `[i, j, 0:5]` and RHS `([i-1, j], [i+1, j], [i, j-1:j+2])`

During _Symbolic Shape Extraction_ , the code generator extracts a _shape_
vector that contains one offset per dimension to apply to the base pointer,
and the number of elements to retrieve. Figure 2 has three slice descriptors,
each referring to $2$-dimensional data. For every slice, code generation
associates the functor dimensions with the dimensions of the provided ranges
in the `map` clause. It then computes the offset of the first element in the
slice relative to the first element in the base pointer, taking into account
the minimum value of the range. In this example, this corresponds to
`t[1][1]`. Thus the slice `[i-1][j]` will access the memory at location
`[0][1]` and the computed offset is $(-1,0)$. Further, it computes the number
of elements accessed by the slice.

Next follows _Symbolic Shape Resolution_ , which computes the shape of the
resulting tensor when a slice is applied to memory. It computes the start
index, end index, and stride for every dimension. Symbolic shape resolution
adds a dimension of size 1 to slices with one element; for slices with more
than one element (e.g., the slice `[i,j-1:j+2]` has three elements), symbolic
shape resolution adds a new dimension sized with the number of elements.

Following, the code generation moves to _Tensor Wrapping_ , in which it
combines the per-slice descriptors computed during _symbolic shape resolution_
with the offsets computed during _symbolic shape extraction_. It adds the
offsets to each dimension’s start and end ranges. This step creates a data
structure compatible with most tensor libraries (Torch, TensorFlow, NumPy), as
every slice now describes a view of some memory region. So far no memory
copies are needed: code generation creates lightweight wrappers around
existing memory. Next, the hpac-ml backend materializes these slices as
library-specific tensors. Each RHS slice has been applied to application data,
yielding one tensor for each.

These RHS tensors are combined in the _Tensor Composition_ step to create a
single LHS tensor. If more than one dimension was added in the symbolic shape
resolution, they are flattened to create a tensor whose shape matches the LHS.
Then, the RHS tensors are concatenated, and the mapping is complete.

The runtime system handles mapping in both directions between the application
and tensor memory spaces. When the tensor map clause uses the `from`
specifier, the runtime creates a mapping from the tensor memory space to the
application memory space. That involves the symbolic shape extraction,
resolution, and tensor wrapping steps, which create a collection of tensors
wrapping user data, without requiring the tensor reshaping and composition
steps.

###  IV-B Execution Control

hpac-ml extends HPAC’s `approx` directive parsing and semantic analysis to
recognize the `ml`, `model`, and `database` clauses. HPAC captures the
associated code statement following the `approx` directive and outlines it in
a new function. The wrapped region is replaced by a call to the HPAC runtime
system and the compiler forwards a function pointer to the outlined function
to the runtime system.

hpac-ml extends HPAC’s runtime API entry point to accept two file paths given
as C-strings, pointing to files. The first file, given by the `database`
clause, points to a storage location to use during data collection, while the
second file refers to a model given in the `model` clause. Code generation for
the `ml` clause checks the `ml-mode` identifier. When the identifier is set to
`infer`, code generation instructs the runtime to perform model inference
throughout the application run. When `ml-mode` is set to `collect`, the
compiler instructs the runtime to collect data throughout the application
execution. For the `predicated` identifier, the compiler generates code to
evaluate a boolean expression passed as a second optional argument to the `ml`
clause. When the argument expression evaluates to `false`, the runtime
performs data collection; when it is set to `true` the runtime performs
inference.

At execution time, the runtime system can access the accurate execution path
by invoking the function outlined by the compiler. hpac-ml utilizes this
during data collection. Briefly, the runtime system maps the memory regions
annotated as inputs with the `in` clause to tensors using the data bridge and
stores them into an HDF5 file [46]. Once the inputs are stored, the outlined
region is invoked. When it finishes, the outputs provided to the `out` clause
are mapped from memory and stored in the file. The output HDF5 file contains
an independent HDF5 group for every annotated region. Within each group hpac-
ml stores the following datasets:  1. inputs gathered by the data bridge;  2.
outputs gathered by the data bridge; and  3. execution time of the wrapped
code region.

Storing both the data and the execution time of the outlined region in the
HDF5 file supports ML development workflows in the architecture search/model
training phase of supervised machine learning. Developers can load the
database using user-friendly HDF5 interfaces, such as `h5py` in Python. They
can then use the stored inputs and outputs to create training/validation
datasets for model training. The included runtime information can be used to
assess performance-accuracy trade-offs without executing the application.

When the application instructs the hpac-ml runtime system to perform
inference, the backend loads the model file if it has not already been loaded.
The implementation uses Torch, the C++ API of PyTorch, to load models. Once
the model is loaded, hpac-ml uses the data bridge to transform data from the
application space to the tensor space and calls the model inference engine.
The inference output tensor is converted back to the application layout
through the data bridge.

##  V Evaluation

Table I: The benchmarks used to evaluate hpac-ml. Benchmark |  Description
---|---
MiniBUDE [47] |  Executes virtual screening in molecular docking, assessing poses to predict ligand-protein binding energy using an empirical forcefield. QOI: Ligand-protein binding energy for each pose. Metric: MAPE
Binomial Options [48] |  Iteratively calculates the price for a portfolio of American stock options at multiple time points before expiration. QOI: The computed prices. Metric: RMSE
Bonds [49] |  Calculates bond valuations and interest payments for fixed-rate bonds with a flat forward curve. QOI: The accrued interest for each bond. Metric: RMSE
MiniWeather [50] |  Simulates atmospheric dynamics through essential weather and climate modeling equations, emphasizing buoyant force impacts. QOI: Simulation state variables (density, $x$ momentum, $y$ momentum, and potential temperature) at each gridpoint. Metric: RMSE
ParticleFilter [51] |  Statistical estimation of a target object’s location given noisy measurements. QOI: The location of the object. Metric: RMSE

hpac-ml provides an easy-to-use approach to collect training data and use a
model within an application. In this section, we evaluate the complexity
required to annotate the benchmarks, the one-time overhead of data collection,
and finally, the performance overheads induced by our runtime implementation
to invoke the inference engine.

Table I lists the evaluated CUDA benchmarks and the Quantity of Interest (QoI)
for each. The evaluation platform is equipped with one $64$-core AMD Epyc 7763
CPU and $4\times$ NVIDIA A100 GPUs, each with $40$GB of memory and $108$ SMs.
We use CUDA 11.8, PyTorch 2.2.0, and compile all benchmarks with the hpac-ml
compiler.

###  V-A Code Complexity of Using hpac-ml

For every benchmark we aggressively annotate application regions with hpac-ml
to maximize potential performance gains. We use Lines of Code (LoC) as a
metric of the complexity of using hpac-ml. We use `clang-format` with the LLVM
style to format the sources and report the additional Lines of Code required
by hpac-ml. Table II details the total LoCs for every benchmark and LoC
required to use hpac-ml along with the number of directives.

Table II: Application source code impact of hpac-ml.

| Total LoC |  hpac-ml LoC |  hpac-ml Directives
---|---|---|---
MiniBUDE | 539 | 8 | 4
Binomial Options | 455 | 8 | 4
Bonds | 2248 | 9 | 4
MiniWeather | 905 | 3 | 3
ParticleFilter | 1025 | 5 | 4

Overall, we observe an average LoC increase of less than $2\%$ across all
benchmarks. MiniBUDE, Binomial Options, Bonds, and ParticleFilter each take 4
directives: input/output tensor functor declarations, a tensor mapping for the
input, and the `approx ml` directive. MiniWeather is an iterative solver using
the same memory for an iteration’s input/output, so we use the `inout` clause.
In summary, a developer should expect to add up to two directives for the
inputs and outputs to declare the tensor functor and the mapping between data
layouts. An additional directive is required to annotate the code to be
replaced by the model. Using the same source code, hpac-ml can perform data
collection and model inference. In cases of identical memory layouts across
application variables, functors can be re-used, further reducing code
complexity.

###  V-B Data Collection

Table III: Data collection overhead.

| Original Runtime | Runtime With Data Collection | Collected Data Size (MB)
---|---|---|---
MiniBUDE | 23.15s | 23.25s | 29
Binomial Options | 2.02s | 2.21s | 268
Bonds | 0.96s | 1.18s | 303
MiniWeather | 0.65ms | 28.92ms | 19
ParticleFilter | 3.43 ms | 3.51 ms | 852

Following best practices [52], we split the application datasets distributed
with each benchmark into $2$ different sets: a _training/validation_ set and a
_test_ set. The training/validation set is used for model training, and the
test set evaluates the models when deployed with hpac-ml.

We execute the hpac-ml enhanced applications with the _training/validation_
inputs and set the condition in the `ml(predicated:bool-expr)` clause to
false, instructing the hpac-ml runtime system to collect data for future model
training. Collected data are stored in HDF5 files, a suitable format for ML
engineers to consume. Effectively, inner dimensions correspond to the
applications’ input-output tensors and the outer dimension corresponds to a
unique data identifier of this set in the data collection ensemble. Data
collected are directly readable by builtin torch data loaders.

Table III details the overhead of capturing data for model training. This
overhead ranges between $1.01\times$ and $44.6\times$ over a normal
application run. Typically, this overhead is amortized, as ML experts reuse
those data when searching for a suitable model. This overhead is worthwhile
after training such a model and significantly accelerating the application.

###  V-C Nested Bayesian-driven Model Exploration

hpac-ml facilitates easy use and deployment of different surrogate models.
Additionally, the trade-offs between application accuracy and performance are
application/model-specific. The model’s architecture (i.e., how layers and
nodes are structured and connected) and training hyperparameters, such as the
_learning rate_ , greatly affect the the model’s inference speed, prediction
accuracy, and overall fidelity. Exploring all possible model architectures and
hyperparameters is practically impossible, since they form an intractable,
combinatorial space. Hence, we devise an automated, guided approach to
generate efficient models, leveraging application domain expertise to sub-
select candidate ML architectures and Bayesian Optimization for searching
architecture and training (hyper-)parameters.

We define a neural architecture space for each benchmark to search for well
performing models. We confine the model architecture based on our application
domain expertise, e.g. we use convolutional layers to capture spatial
information in the video frames of ParticleFilter. Table V details each
benchmark’s neural architecture search space, while Table V defines the space
of feasible hyperparameter values.

Table IV: Search space used for neural architecture search The _Feature
Multiplier_ parameter denotes a factor that reduces the neuron count across
hidden layers.

MiniBUDE |  Binomial Options, Bonds
---|---
Num. Hidden Layers | $[2,12]$ | Hidden 1 Features | $[5,512]$
Hidden 1 Size | $64,128,\ldots,4096$ | Hidden 2 Features | $[0,512]$
Feature Multiplier | $[0.1,0.8]$ |  |
MiniWeather | ParticleFilter
Conv. Layer 1 Kernel Size | $[2,8]$ | Conv. Kernel Size; Conv. Stride | $[2,14]$
Conv. Layer 1 Output Channels | $[4,8]$ | Maxpool Kernel Size | $[1,10]$
Conv. Layer 2 Kernel Size | $[0,6]$ | FC 2 Size | $[0,128]$

Table V: Search space used for BO hyperparameter tuning.

_Learning Rate_ : $[1\times 10^{-4},1\times 10^{-2}]$ |  _Weight Decay_ : $[1\times 10^{-4},1\times 10^{-1}]$
---|---
_Dropout_ : $[0,0.8]$ |  _Batch Size_ : $[32,512]$

Briefly, we employ Bayesian Optimization (BO) [53] to identify efficient model
architectures and respective hyperparameters. BO streamlines model selection
by using Gaussian Processes to predict the performance of an objective
function, focusing on areas with potential improvement. Through an iterative
process involving an acquisition function, BO efficiently explores and
exploits the search space to find optimal solutions. This method is
particularly effective for tasks like neural network tuning, where evaluations
are costly [54, 55].

We use a nested, two-level, multi-objective BO for neural architecture search
and hyperparameter tuning. The outer level selects a neural architecture with
the objectives to jointly minimize inference latency _and_ validation error.
Model architectures found on the Pareto front are further tuned in the inner
level Bayesian search that tunes training hyperparameters to minimize further
the model’s validation error.

Figure 5: End-to-end application speedup and error of hpac-ml enhanced
applications.

The outer-level search is executed for $100$ iterations, with early stopping
when it doesn’t find a faster or more accurate model for five consecutive
trials. The inner level tunes training hyperparameters for $30$ iterations.
Faithful to good practice, we train/evaluate all models _only_ using the
_training/validation_ data from the data collection phase.

We implement this approach as a workflow using Parsl [56] and employ the
Adaptive Experimentation platform [57] for the Bayesian search. The search
convergence is fast, typically earlier than 16 trials. Overall, our automated
method explored $5130$ models: MiniBUDE $990$, Binomial Options $900$, Bonds
$540$, MiniWeather $780$, and ParticleFilter $1920$.

###  V-D Performance Benefits from hpac-ml Model Deployment

After model search and training, we integrate hpac-ml using the lowest
validation error models generated by the nested BO search (cutoff is RMSE $<$
10) and evaluate the application’s error as observed at the final computed
QoI. Modifications in the application source code are minor: one needs to set
the `cond` in the `ml(infer:<cond-expr>)` to `True` and add the `model(string-
expr)` clause specifying the path of the generated model.

We evaluate hpac-ml on two metrics: end-to-end speedup over the original GPU-
accelerated application and quality loss. We run each configuration 20 times
and use the (arithmetic) mean runtime yielding statistically significant
results. Speedup is relative to end-to-end application runtime and includes
all required data transfers and transformations. Quality loss is computed
using the QoI for each application and measuring how much the DNN surrogate
changes this QoI.

Figure 5 shows results of speedup and validation error across the lowest
validation error models for all programs. Overall, deploying models with hpac-
ml substantially decreases application execution time. We observe up to
$83.6\times$ speedup, with a geometric mean of maximum possible speedup of
$13.0\times$. Regarding validation error, (see figure 5-top, BO indeed finds
several models below the cutoff of RMSE $<$ 10 we set. Yet, the interpretation
of whether that RMSE results in acceptable QoI is application-specific and we
discuss it in the next section, which discusses per-program results.

Figure 6: Proportion of time required for each of the primary operations
performed by hpac-ml in inference mode: mapping between tensors and memory and
performing inference.

Further, figure 6 shows a breakdown of execution time within the hpac-ml
runtime library, divided in time spent mapping tensors (to/from memory) and
inference time with the fastest model, which is delegated to the ML backend.
The overhead of hpac-ml is between $0.01\%$ and $8\%$, compared to the latency
of the inference engine, which depends solely on the evaluation of the model
by the ML backend.

##  VI Observations

We discuss the speedup and QoI accuracy results using the hpac-ml programming
model to extend programs for surrogate model execution and the extensive ML
model search campaign it made possible. We center the discussion around _four
observations_ that summarize key insights.

Figure 7: ParticleFilter results for the models with RMSE $\leq 1$. The black
vertical line marks the RMSE of the original particle filter algorithm. Points
are colored according to their number of parameters relative to the model with
the fewest parameters.

_Observation 1: An ML Model can outperform custom algorithmic approximation in
both execution time and accuracy._ Particlefilter (PF) approximates the
location of a target object, given noisy measurements of that target’s
location and an estimate of the object’s path using a Bayesian framework. The
application outputs the ground-truth object location and the particle filter’s
estimate. The hpac-ml version of PF captures the ground-truth values to create
the training dataset.

Our BO search algorithm finds several models that accelerate the original
application, as shown in Figure 7. Briefly, NN surrogate modeling speeds up
execution between $8.67\times$ and $9.60\times$ (end-to-end) over the original
application implementing the algorithmic approximation. Additionally,
surrogates can reduce the output’s RMSE to as low as $0.12$, whereas the
original approximation’s RMSE is $0.5$. The surrogate model replaces the
entire ParticleFilter algorithm, which includes three distinct GPU kernels,
with a CNN that estimates the object’s location in each video frame. CNN
inference is up to $23.3\times$ faster than the kernels it replaces. While
end-to-end speedup is subject to Amdahl’s law, surrogate modeling speeds up
the application by an order of magnitude. As such, ParticleFilter is an
excellent application for replacing algorithmic approximation with surrogate
modeling approximation and aligns well with the known strengths of CNNs for
image/video tasks.

(a) MiniBUDE

(b) Binomial Options

(c) Bonds

Figure 8: Speedup vs Accuracy. Points are colored according to the number of
parameters relative to the model with the fewest.

(a) Original Simulation

(b) Only model inference simulation

(c) Mixed simulation.

(d) RMSE vs Speedup at timestep $1200$

(e) Per-timestep RMSE

(f) CDF of relative error

Figure 9:  Top shows MiniWeather simulation outputs at timestep 1200. All
plots use the original solution until timestep 1000 because models are trained
on the first 1000. In (b), we replace every timestep with model inference, in
(c) we interleave model inference with the original timestep every other step.
(d) Speedup vs. RMSE: This panel compares speedup against RMSE across various
Original:Surrogate configurations, where ‘Original’ denotes simulation
iterations performed with the original timestep evaluation, and the next
‘Surrogate’ loops are executed using the surrogate model. (e) Timestep vs RMSE
for different configurations. (f) CDF of relative error from the surrogate
model predictions at timestep 1001 versus error at timestep 1010.

_Observation 2: Surrogate models can improve performance by increasing
hardware utilization._ While this motivates the use of surrogate models, we
quantify it more precisely using MiniBUDE. MiniBUDE predicts complex molecular
structures and estimates their binding affinities. Those binding affinities
are estimated by rotating and moving a ligand into different positions, called
poses, and estimating the interaction strength between a ligand in each pose
and a target protein. Its original implementation is a compute-bound molecular
docking mini-app whose kernel achieves $33.5\%$ of the device’s peak compute
throughput and $6.13\%$ of peak memory bandwidth.

By contrast, the fastest model achieves $47.2\%$ of peak compute throughput
and $31.5\%$ of peak memory bandwidth, demonstrating a substantially more
efficient use of hardware resources. MiniBUDE accesses memory with a scattered
pattern, whereas NN inference is implemented by dense linear algebra
operations, for which there are highly optimized vendor software libraries
that execute near-optimally on GPU architectures. Hence, execution of the
surrogate model extracts higher performance both by better alignment with the
hardware architecture features and by reducing computational load.

To quantify the effects of model size on accuracy and speedup, Figure 8(a)
shows the Mean Absolute Percentage Error (MAPE) of QoI and speedup for all
Pareto-optimal models. The largest, slowest model achieves $25.5\times$ end-
to-end speedup with $2.71\%$ MAPE, while the fastest model worsens MAPE to
$6.82\%$ MAPE but increases speedup to $35\times$. While this result of
miniBUDE is expected – larger models reduce speed up but are more accurate –
this is not always the case, as our next observation details.

_Observation 3: Automation complements human ML domain expertise for exploring
the trade-offs of model architecture, fidelity of the computed QoI, model
generalization, and overfitting._ Besides miniBUDE, figure 8(b) shows the
speedup and accuracy of Pareto-optimal models for Binomial Options. Similar to
miniBUDE, there is the same trade-off: larger models are slower and more
accurate. However, the range of speedup of application performance is
different. The smallest model speeds it up by $83.59\times$, with an RMSE of
$0.114$. Conversely, the largest model has significantly less speedup of
$19.36\times$ (4.3$\times$ reduction) but with an RMSE of $0.0111$, an order
of magnitude improvement over the fastest one. Exactly quantifying this trade-
off is greatly facilitated by the automation brought forward by hpac-ml for
applying surrogate modeling to enable this campaign.

Contrasting MiniBUDE and Binomial Options, larger models do not always
increase accuracy for Bonds, as figure 8(c)) shows, The fastest model is the
most accurate, yielding an RMSE of $0.280$ and $7.96\times$ speedup,
surpassing the largest model (second-best for accuracy) with an RMSE of
$0.320$ and a speedup of $6.00\times$. Generally, when larger models are less
accurate than their smaller counterparts, it suggests that overfitting to the
training data has occurred. hpac-ml makes it easier to identify and mitigate
such cases. The standard way to reduce overfitting is to augment the dataset,
either by collecting more data or increasing the complexity of the dataset.
hpac-ml makes data augmentation easy through its built-in data collection
mechanisms.

_Observation 4: In iterative auto-regressive scenarios, the models’ error
propagates across iterations and disturbs the applications’ QoI. hpac-ml
provides primitives to interleave model executions with accurate evaluations,
thereby decreasing error._ MiniWeather serves as a representative benchmark
for atmospheric dynamics simulation programs, a domain where NN surrogate
modeling has been particularly challenging [58]. In this context, even small
inaccuracies introduced by the surrogate approximation can rapidly accumulate,
leading to highly unstable and unrealistic simulations. Our findings, despite
the use of BO search to identify high-fidelity models, align with and support
these previously reported difficulties. In more detail, figures 9(a) and 9(b)
present the visual output of MiniWeather after timestep $1200$ of the original
simulation and of a simulation using the model with the highest fidelity for
all timesteps starting from timestep $1000$. Although the surrogate model
approximation adds little error in most grid locations, this error propagates
and amplifies in later iterations. Figure 9(f) shows the CDF of relative error
in timestep 1001, after the first iteration using surrogate modeling, and
timestep 1010. At timestep 1001, the RMSE is 0.0003. Most error comes from few
domain locations: $80\%$ of domain locations have relative error less than
$0.09$ while $90\%$ have a relative error less than $0.2$. _However, by using
the model for $10$ consecutive times steps, the error distribution shifts to
the right by almost an order of magnitude._ At timestep, 1010, $80\%$ of
domain locations have a relative error of less than 1.25 while $90\%$ have a
relative error of less than 3.04.

hpac-ml can be used to interleave model evaluations with the original
analytical algorithm on different timesteps. Specifying the if clause controls
usage of the surrogate model, enabling it in a user-specified fraction of the
simulation’s timesteps. Figure 9 shows results interleaving accurate time
steps with surrogate inference in hpac-ml to reveal performance and accuracy
trade-offs for several interleaving configurations. Interleaving model
approximations with original outputs significantly decreases error, at the
expense of performance improvement from surrogate modeling. Nonetheless, this
class of computation is especially hard to approximate with surrogates. Figure
9(c) depicts the output of interleaving one original application timestep with
one model timestep. Although the visual result is closer to the accurate one,
the observed difference is still high.

##  VII Conclusion

hpac-ml is a novel pragma-based programming model for embedding ML surrogates
in scientific applications. The design and implementation of hpac-ml and its
key abstractions yield a high-productivity, high-performance method to
integrate ML surrogate models into scientific applications with a few lines of
code. Using hpac-ml, we perform a comprehensive evaluation of $5$ GPU-
accelerated HPC benchmarks testing thousands of models, demonstrating the
versatility of hpac-ml and exploring the trade-offs between model
architecture, accuracy, and performance. hpac-ml bridges the gap between ML
infrastructure and scientific applications, enabling researchers to rapidly
evaluate ML surrogates in applications. This accelerated experimentation
facilitates fundamental research on the interactions between ML and scientific
computing. Application developers who want to improve application performance
in the post-Moore era will find hpac-ml a valuable tool in their belt.

## Acknowledgements

The authors would like to thank the anonymous reviewers for their valuable
comments and helpful suggestions.

The views and opinions of the authors do not necessarily reflect those of the
U.S. government or Lawrence Livermore National Security, LLC neither of whom
nor any of their employees make any endorsements, express or implied
warranties or representations or assume any legal liability or responsibility
for the accuracy, completeness, or usefulness of the information contained
herein. This work was prepared by LLNL under Contract DE-AC52-07NA27344 (LLNL-
CONF-862478) and was supported by both the LLNL LDRD Program under Project No.
22-SI-004. This material is based in part upon work supported by the
Department of Energy, National Nuclear Security Administration, under Award
Number DE-NA0003963. This work used the Delta system at the National Center
for Supercomputing Applications through allocation ASC050025 from the Advanced
Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS)
program, which is supported by National Science Foundation grants #2138259,
#2138286, #2138307, #2137603, and #2138296.

## References

  * [1] A. Galántai, “The theory of newton’s method,” _Journal of Computational and Applied Mathematics_ , vol. 124, no. 1, 2000, numerical Analysis 2000. Vol. IV: Optimization and Nonlinear Equations.
  * [2] D. Funaro, _Polynomial approximation of differential equations_. Springer Science & Business Media, 2008, vol. 8.
  * [3] S. Sidiroglou-Douskos, S. Misailovic, H. Hoffmann, and M. Rinard, “Managing performance vs. accuracy trade-offs with loop perforation,” in _Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering_ , 2011.
  * [4] H. Hoffmann, S. Misailovic, S. Sidiroglou, A. Agarwal, and M. Rinard, “Using code perforation to improve performance, reduce energy consumption, and respond to failures,” 2009.
  * [5] H. Menon, M. O. Lam, D. Osei-Kuffuor, M. Schordan, S. Lloyd, K. Mohror, and J. Hittinger, “ADAPT: Algorithmic differentiation applied to floating-point precision tuning,” in _SC18: International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2018, pp. 614–626.
  * [6] I. Laguna, P. C. Wood, R. Singh, and S. Bagchi, “Gpumixer: Performance-driven floating-point tuning for gpu scientific applications,” in _International Conference on High Performance Computing_ , 2019.
  * [7] A. K. Mishra, R. Barik, and S. Paul, “iACT: A software-hardware framework for understanding the scope of approximate computing,” in _Workshop on Approximate Computing Across the System Stack (WACAS)_ , 2014.
  * [8] G. Tziantzioulis, N. Hardavellas, and S. Campanoni, “Temporal approximate function memoization,” _IEEE Micro_ , vol. 38, no. 4, 2018.
  * [9] A. Pal, S. Mahajan, and M. R. Norman, “Using deep neural networks as cost-effective surrogate models for super-parameterized e3sm radiative transfer,” _Geophysical Research Letters_ , vol. 46, no. 11, pp. 6069–6079, 2019.
  * [10] D. Kochkov, J. A. Smith, A. Alieva, Q. Wang, M. P. Brenner, and S. Hoyer, “Machine learning–accelerated computational fluid dynamics,” _Proceedings of the National Academy of Sciences_ , vol. 118, no. 21, p. e2101784118, 2021.
  * [11] K. Parasyris, G. Georgakoudis, H. Menon, J. Diffenderfer, I. Laguna, D. Osei-Kuffuor, and M. Schordan, “HPAC: evaluating approximate computing techniques on hpc openmp applications,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2021, pp. 1–14.
  * [12] Z. Fink, K. Parasyris, G. Georgakoudis, and H. Menon, “Hpac-offload: Accelerating hpc applications with portable approximate computing on the gpu,” in _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , ser. SC ’23. New York, NY, USA: Association for Computing Machinery, 2023. [Online]. Available: https://doi.org/10.1145/3581784.3607095
  * [13] S. Froehlich, D. Große, and R. Drechsler, “Towards reversed approximate hardware design,” in _2018 21st Euromicro Conference on Digital System Design (DSD)_ , 2018.
  * [14] S. Rehman, W. El-Harouni, M. Shafique, A. Kumar, J. Henkel, and J. Henkel, “Architectural-space exploration of approximate multipliers,” in _2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ , 2016.
  * [15] P. Kulkarni, P. Gupta, and M. Ercegovac, “Trading accuracy for power with an underdesigned multiplier architecture,” in _2011 24th Internatioal Conference on VLSI Design_ , 2011.
  * [16] A. B. Kahng and S. Kang, “Accuracy-configurable adder for approximate arithmetic designs,” in _Proceedings of the 49th annual design automation conference_ , 2012.
  * [17] V. Gupta, D. Mohapatra, S. P. Park, A. Raghunathan, and K. Roy, “Impact: Imprecise adders for low-power approximate computing,” in _IEEE/ACM International Symposium on Low Power Electronics and Design_ , 2011.
  * [18] V. K. Chippa, D. Mohapatra, K. Roy, S. T. Chakradhar, and A. Raghunathan, “Scalable effort hardware design,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 22, no. 9, 2014.
  * [19] H. Esmaeilzadeh, A. Sampson, L. Ceze, and D. Burger, “Architecture support for disciplined approximate programming,” in _Proceedings of the seventeenth international conference on Architectural Support for Programming Languages and Operating Systems_ , 2012.
  * [20] R. Hegde and N. Shanbhag, “Energy-efficient signal processing via algorithmic noise-tolerance,” in _1999 International Symposium on Low Power Electronics and Design_ , 1999.
  * [21] A. Sampson, W. Dietl, E. Fortuna, D. Gnanapragasam, L. Ceze, and D. Grossman, “Enerj: Approximate data types for safe and general low-power computation,” _ACM SIGPLAN Notices_ , vol. 46, no. 6, pp. 164–174, 2011.
  * [22] K. Parasyris, V. Vassiliadis, C. D. Antonopoulos, S. Lalis, and N. Bellas, “Significance-aware program execution on unreliable hardware,” _ACM Transactions on Architecture and Code Optimization (TACO)_ , vol. 14, no. 2, pp. 1–25, 2017.
  * [23] H. Hoffmann, S. Misailovic, S. Sidiroglou, M. Rinard, and A. Agarwal, “Using Code Perforation to Improve Performance, Reduce Energy Consumption, and Respond to Failures,” Tech. Rep. MIT-CSAIL-TR-2009-042, 2009.
  * [24] I. Goiri, R. Bianchini, S. Nagarakatte, and T. D. Nguyen, “Approxhadoop: Bringing approximations to mapreduce frameworks,” in _Proceedings of the Twentieth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)_ , 2015.
  * [25] D. Michie, ““memo” functions and machine learning,” _Nature_ , vol. 218, 1968.
  * [26] G. Keramidas, C. Kokkala, and I. Stamoulis, “Clumsy value cache: An approximate memoization technique for mobile gpu fragment shaders,” in _Workshop on approximate computing (WAPCO’15)_ , 2015.
  * [27] M. Samadi, D. A. Jamshidi, J. Lee, and S. Mahlke, “Paraprox: Pattern-based approximation for data parallel applications,” in _Proceedings of the 19th international conference on Architectural support for programming languages and operating systems_ , 2014.
  * [28] V. Vassiliadis, J. Riehme, J. Deussen, K. Parasyris, C. D. Antonopoulos, N. Bellas, S. Lalis, and U. Naumann, “Towards automatic significance analysis for approximate computing,” in _2016 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)_ , 2016, pp. 182–193.
  * [29] V. Vassiliadis, C. Chalios, K. Parasyris, C. D. Antonopoulos, S. Lalis, N. Bellas, H. Vandierendonck, and D. S. Nikolopoulos, “Exploiting significance of computations for energy-constrained approximate computing,” _International Journal of Parallel Programming_ , vol. 44, no. 5, pp. 1078–1098, 2016.
  * [30] V. Vassiliadis, K. Parasyris, C. Chalios, C. D. Antonopoulos, S. Lalis, N. Bellas, H. Vandierendonck, and D. S. Nikolopoulos, “A programming model and runtime system for significance-aware energy-efficient computing,” _ACM SIGPLAN Notices_ , vol. 50, no. 8, pp. 275–276, 2015.
  * [31] B. Grigorian, N. Farahpour, and G. Reinman, “BRAINIAC: Bringing reliable accuracy into neurally-implemented approximate computing,” in _2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA)_ , 2015.
  * [32] L. McAfee and K. Olukotun, “Emeuro: A framework for generating multi-purpose accelerators via deep learning,” in _2015 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)_. IEEE, 2015, pp. 125–135.
  * [33] T. Moreau, M. Wyse, J. Nelson, A. Sampson, H. Esmaeilzadeh, L. Ceze, and M. Oskin, “Snnap: Approximate computing on programmable socs via neural acceleration,” in _2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA)_. IEEE, 2015, pp. 603–614.
  * [34] H. Esmaeilzadeh, A. Sampson, L. Ceze, and D. Burger, “Neural acceleration for general-purpose approximate programs,” in _2012 45th Annual IEEE/ACM International Symposium on Microarchitecture_ , 2012.
  * [35] N.-M. Ho and W.-F. Wong, “Tensorox: Accelerating GPU Applications via Neural Approximation on Unused Tensor Cores,” _IEEE Transactions on Parallel and Distributed Systems_ , vol. 33, no. 2, pp. 429–443, Feb. 2022.
  * [36] H. Sharif, P. Srivastava, M. Huzaifa, M. Kotsifakou, K. Joshi, Y. Sarita, N. Zhao, V. S. Adve, S. Misailovic, and S. V. Adve, “Approxhpvm: a portable compiler ir for accuracy-aware optimizations.” _Proc. ACM Program. Lang._ , no. OOPSLA, 2019.
  * [37] A. Sampson, A. Baixo, B. Ransford, T. Moreau, J. Yip, L. Ceze, and M. Oskin, “Accept: A programmer-guided compiler framework for practical approximate computing,” _University of Washington Technical Report UW-CSE-15-01 1 (2)_ , 2015.
  * [38] W. Baek and T. M. Chilimbi, “Green: A framework for supporting energy-conscious programming using controlled approximation,” in _Proceedings of the 31st ACM SIGPLAN Conference on Programming Language Design and Implementation_ , 2010, pp. 198–209.
  * [39] K. Parasyris, J. Diffenderfer, H. Menon, I. Laguna, J. Vanover, R. Vogt, and D. Osei-Kuffuor, “Approximate computing through the lens of uncertainty quantification,” in _SC22: International Conference for High Performance Computing, Networking, Storage and Analysis_ , 2022.
  * [40] W. Dong, G. Kestor, and D. Li, “Auto-hpcnet: An automatic framework to build neural network-based surrogate for high-performance computing applications,” in _Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing_ , ser. HPDC ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 31–44. [Online]. Available: https://doi.org/10.1145/3588195.3592985
  * [41] A. Myers, W. Zhang, A. Almgren, T. Antoun, J. Bell, A. Huebl, and A. Sinn, “Amrex and pyamrex: Looking beyond ecp,” 2024.
  * [42] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [43] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin _et al._ , “Tensorflow: Large-scale machine learning on heterogeneous distributed systems,” _arXiv preprint arXiv:1603.04467_ , 2016.
  * [44] J. Bai, F. Lu, K. Zhang _et al._ , “Onnx: Open neural network exchange,” https://github.com/onnx/onnx, 2019.
  * [45] B. Van Essen, H. Kim, R. Pearce, K. Boakye, and B. Chen, “Lbann: livermore big artificial neural network hpc toolkit,” in _Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments_ , ser. MLHPC ’15. New York, NY, USA: Association for Computing Machinery, 2015. [Online]. Available: https://doi.org/10.1145/2834892.2834897
  * [46] The HDF Group, “Hierarchical Data Format, version 5.” [Online]. Available: https://github.com/HDFGroup/hdf5
  * [47] A. Poenaru, W.-C. Lin, and S. McIntosh-Smith, “A performance analysis of modern parallel programming models using a compute-bound application,” in _International Conference on High Performance Computing_. Springer, 2021, pp. 332–350.
  * [48] V. Podlozhnyuk. (2007) Binomial option pricing model.
  * [49] S. Grauer-Gray, W. Killian, R. Searles, and J. Cavazos, “Accelerating financial applications on the gpu,” in _Proceedings of the 6th Workshop on General Purpose Processor Using Graphics Processing Units_ , ser. GPGPU-6. New York, NY, USA: Association for Computing Machinery, 2013, p. 127–136. [Online]. Available: https://doi.org/10.1145/2458523.2458536
  * [50] M. R. Norman and USDOE, “miniweather,” 3 2020. [Online]. Available: https://www.osti.gov//servlets/purl/1631691
  * [51] S. Che, M. Boyer, J. Meng, D. Tarjan, J. W. Sheaffer, S.-H. Lee, and K. Skadron, “Rodinia: A benchmark suite for heterogeneous computing,” in _2009 IEEE International Symposium on Workload Characterization (IISWC)_ , 2009.
  * [52] I. Goodfellow, Y. Bengio, and A. Courville, _Deep Learning_. MIT Press, 2016, http://www.deeplearningbook.org.
  * [53] J. Mockus, _Bayesian Approach to Global Optimization: Theory and Applications_ , ser. Mathematics and its Applications. Kluwer Academic Publishers, 1989, vol. 37.
  * [54] J. Wu, X.-Y. Chen, H. Zhang, L.-D. Xiong, H. Lei, and S.-H. Deng, “Hyperparameter optimization for machine learning models based on bayesian optimization,” _Journal of Electronic Science and Technology_ , vol. 17, no. 1, pp. 26–40, 2019.
  * [55] K. Kandasamy, W. Neiswanger, J. Schneider, B. Poczos, and E. P. Xing, “Neural architecture search with bayesian optimisation and optimal transport,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [56] Y. Babuji, A. Woodard, Z. Li, D. S. Katz, B. Clifford, R. Kumar, L. Lacinski, R. Chard, J. Wozniak, I. Foster, M. Wilde, and K. Chard, “Parsl: Pervasive parallel programming in python,” in _28th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC)_ , 2019.
  * [57] E. Bakshy, L. Dworkin, B. Karrer, K. Kashin, B. Letham, A. Murthy, and S. Singh, “Ae: A domain-agnostic platform for adaptive experimentation,” in _NeurIPS Systems for ML Workshop_ , 2018.
  * [58] M. G. Meena, M. R. Norman, D. M. Hall, and M. S. Pritchard, “Surrogate modeling of subgrid-scale effects in idealized atmospheric flows: A deep learned approach using high-resolution simulation data,” _Authorea Preprints_ , 2023.

Appendix: Artifact Description/Artifact Evaluation

Artifact Description (AD)

##  VIII Overview of Contributions and Artifacts

###  VIII-A Paper’s Main Contributions

  * $C_{1}$

hpac-ml, an annotation-based programming model for embedding ML surrogates in
scientific applications. Application developers annotate parts of their
application to replace them with a surrogate model. hpac-ml enables developers
to describe the data flow between application memory ML data layouts using
succinct syntax.

  * $C_{2}$

An implementation of the hpac-ml programming model in Clang/LLVM v17 with a
runtime system that uses Torch for inference and HDF5 for data collection.

  * $C_{3}$

A thorough evaluation on a representative suite of GPU-based HPC mini-apps and
benchmarks. We find up to $83.6\times$ speedup while maintaining output
quality with RMSE as low as $0.001$.

  * $C_{4}$

hpac-ml enables a large-scale exploration of the trade-offs between model
size, inference speed, and output quality loss. To explore these trade-offs,
we trained thousands of models.

###  VIII-B Computational Artifacts

Our paper produces the following computational artifacts. Before the freeze
date, we will generate permanent DOIs for each.

  * $A_{1}$

Software environment for the hpac-ml compiler/runtime system and experimental
evaluation. https://github.com/ZwFink/hpacml_artifact/tree/main/software_env

  * $A_{2}$

hpac-ml compiler and runtime system implementation.
https://github.com/ZwFink/hpacml/tree/sc24

  * $A_{3}$

hpac-ml benchmark programs.
https://github.com/ZwFink/hpacml_artifact/tree/main/benchmarks

  * $A_{4}$

hpac-ml model training and evaluation infrastructure.
https://github.com/ZwFink/hpacml-experimentation

Artifact ID | Contributions | Related
---|---|---
| Supported | Paper Elements
$A_{1}$ | $C_{1}-C_{4}$ | Tables 1-3
|  | Figures 5-9
$A_{2}$ | $C_{1}-C_{4}$ | Tables 2-3
|  | Figures 5-9
$A_{3}$ | $C_{3}-C_{4}$ | Tables 1-3
|  | Figures 5-9
$A_{4}$ | $C_{3}-C_{4}$ | Figures 5-9

##  IX Artifact Identification

###  IX-A Computational Artifact $A_{1}$

### Relation To Contributions

This artifact contains the software environment used to build the
compiler/runtime system and perform the experiments listed in the paper.
Therefore, this artifact is a requirement for all the artifacts that follow.

### Expected Results

After reproducing this artifact, the hpac-ml compiler/runtime system can be
built and evaluated.

### Expected Reproduction Time (in Minutes)

Setting up the artifact involves cloning the GitHub repository and navigating
to the `software_``env` directory. It is expected to take less than 1 minute.
Use the included installation script to execute the artifact and install the
software environment. This step will take up to 5 hours, depending on the
number of cores used.

### Artifact Setup (incl. Inputs)

#### Hardware

Our hardware evaluation platform has one $64$-core AMD Epyc 7763 CPU and
$4\times$ NVIDIA A100 GPUs, each with $40$GB of memory and $108$ SMs.

Generally, the software environment can be built for NVIDIA GPUs and does not
depend on the interconnect and other platform details.

#### Software

This artifact requires git. Our platform uses git version 2.39.3.
Additionally, either Docker or Apptainer are required for the container. We
have used Docker v26.1.4 and Apptainer v1.3.1. The provided installation
script installs the other required software into the container. All packages
are installed using gcc 11.4.0.

#### Datasets / Inputs

There are no datasets or inputs for this artifact.

#### Installation and Deployment

This artifact is installed and activated with the included `setup.sh` and
`run_container.sh` scripts.

### Artifact Execution

After installation, the software environment can be activated by running the
container with the provided `run_container.sh` script.

### Artifact Analysis (incl. Outputs)

This artifact performs no data analysis.

###  IX-B Computational Artifact $A_{2}$

### Relation To Contributions

This artifact is the implementation of the hpac-ml programming model and
runtime system as described in the paper. Therefore, all contributions are
derived from this implementation. In particular, $C_{3}$ and $C_{4}$ use this
implementation to perform our computational experiments on a target platform.

### Expected Results

Reproducing this artifact means compiling the compiler and runtime system in
the provided container. We have also shared container images with the pre-
built compiler; during installation, the user can decide whether to build the
container image or download an existing image using the installation script.
The hpac-ml compiler can then be used to compile programs using the hpac-ml
programming model; the runtime system implements the runtime operations of the
programming model.

### Expected Reproduction Time (in Minutes)

On our platform, building the compiler/runtime system from scratch using $25$
CPU cores takes about $30$ minutes.

### Artifact Setup (incl. Inputs)

#### Hardware

Our hardware evaluation platform is equipped with one $64$-core AMD Epyc 7763
CPU and $4\times$ NVIDIA A100 GPUs, each with $40$GB of memory and $108$ SMs.

Generally, the software environment can be built for NVIDIA GPUs and does not
depend on the interconnect and other platform details.

#### Software

The hpac-ml compiler/runtime system use the software environment described in
$A_{1}$. All packages are built and installed in a container by the
installation process. Some packages of particular importance to this artifact
include:

  1. 1.

PyTorch 2.2.0 https://github.com/pytorch/pytorch

  2. 2.

HDF5 1.14.3 https://github.com/HDFGroup/hdf5

  3. 3.

CUDA 11.8.89

A complete list of software packages can be found in the configuration files
associated with $A_{1}$.

#### Datasets / Inputs

The only input required is the number of cores to use for building the
software.

#### Installation and Deployment

To deploy the compiler so it can be used to build programs, run the included
`source` `hpac_``env``.sh` to add the compiler and its associated runtime
libraries to the system path. The container sources this script as part of the
startup process: when using the container, the user need not manually deploy
the compiler.

### Artifact Execution

This artifact is installed by cloning the associated git repository and
running the `setup``.sh` script that builds the hpac-ml compiler and runtime
system.

### Artifact Analysis (incl. Outputs)

The build process produces versions of the `clang` and `clang++` compilers
equipped to ingest code written in the hpac-ml programming model and generate
code as described in the paper.

###  IX-C Computational Artifact $A_{3}$

### Relation To Contributions

This artifact consists of the suite of programs modified to use the hpac-ml
programming model. This includes the benchmark programs gathered from the
original sources and modified to include the hpac-ml annotations. Additional
modifications are added as needed to support the analysis workflow. They are
compiled with the hpac-ml compiler and run under the hpac-ml runtime system to
generate the per-benchmark results shown in the paper.

### Expected Results

The user can navigate to the directory associated with each benchmark to view
its source code and compile the original/approximated version using the
provided makefiles.

### Expected Reproduction Time (in Minutes)

Each benchmark takes less than 30 seconds to build on our platform. To run
each with data collection enabled requires up to $1$ minute for any of the
benchmarks.

### Artifact Setup (incl. Inputs)

#### Hardware

Our hardware evaluation platform is equipped with one $64$-core AMD Epyc 7763
CPU and $4\times$ NVIDIA A100 GPUs, each with $40$GB of memory and $108$ SMs.

Generally, the software environment can be built for NVIDIA GPUs and does not
depend on the interconnect and other platform details.

#### Software

We use the hpac-ml compiler to build each benchmark; thus, the build process
for these benchmarks assumes the software environment created in $A_{1}$ is
activated, and the compiler has been built.

#### Datasets / Inputs

All benchmarks except for Binomial Options and MiniBUDE generate their own
data sets at runtime. For Binomial Options, we use a data set with $16$
million unique options; for MiniBUDE, our dataset includes $16$ million poses.
These datasets are available at https://zenodo.org/records/12751779.

#### Installation and Deployment

Each benchmark is compiled with the hpac-ml programming model using the
makefiles found with each.

### Artifact Execution

The benchmarks can be run on their own by compiling them with the provided
makefiles and running the generated binary. $A_{4}$ implements a workflow that
automates the end-to-end process of training models and deploying them in each
benchmark. To gather training data, each benchmark is annotated with the hpac-
ml programming model, and data collection mode is enabled. Then, the benchmark
is run and the input/output data are captured into an HDF5 database for
offline training. We make all training data available at
https://zenodo.org/records/12751779.

### Artifact Analysis (incl. Outputs)

This artifact does not generate data to be analyzed.

###  IX-D Computational Artifact $A_{4}$

### Relation To Contributions

This artifact performs the neural architecture search driven by nested
Bayesian Optimization described in the paper. The trained models are then
embedded in the application, and the performance and accuracy are reported.
These performance/accuracy values are reported via the plots given in the
paper.

### Expected Results

This artifact produces the following:  1. the model with the best
hyperparameters for a given neural architecture explored during the search;
2. the performance and accuracy that result from embedding each trained model
in the application.  With this artifact, it is possible to reproduce the
figures listed in the table. The results should indicate substantial runtime
speedup with low accuracy loss for the applications when the model is used.

### Expected Reproduction Time (in Minutes)

We make the trained models for all benchmarks available at
https://zenodo.org/records/12751779. Thus, they can be easily embedded in the
applications to assess performance/accuracy trade-offs, skipping the neural
architecture search. To perform the full BO-based neural architecture search
takes between $50$ and $400$ GPU hours for a given benchmark. Assessing the
performance and accuracy trade-offs for a given model takes between 1 and 5
minutes.

### Artifact Setup (incl. Inputs)

#### Hardware

Our hardware evaluation platform is equipped with one $64$-core AMD Epyc 7763
CPU and $4\times$ NVIDIA A100 GPUs, each with $40$GB of memory and $108$ SMs.

Generally, the software environment can be built for NVIDIA GPUs and does not
depend on the interconnect and other platform details.

#### Software

This artifact assumes the container environment is being used. All software
packages used for this artifact are included in the software environment. We
note two packages important for this step:

  1. 1.

Adaptive Experimentation Platform v0.3.1: Performs Bayesian Optimization.
https://ax.dev/

  2. 2.

Parsl v2024.02.26: Orchestrates the workflow steps required for the nested BO-
based neural architectures search. https://parsl-project.org/

#### Datasets / Inputs

The training data for all benchmarks are located at:
https://zenodo.org/records/12751779. These datasets are collected by running
each benchmark with its corresponding input dataset using hpac-ml in data
collection mode. Each data set is split into two parts: a training/validation
set used for training models and hyperparameter tuning and a test set used to
evaluate the trained models on unseen data. Each dataset is large enough to
have sufficient model training data and enough testing data to saturate device
computational resources.

#### Installation and Deployment

The Spack environment introduced in $A_{1}$ should be activated before running
the artifact.

### Artifact Execution

This artifact has two steps: model training and embedding. The model training
step is located at: https://github.com/ZwFink/hpacml-
experimentation/tree/main/model_training; the embedding step is located at:
https://github.com/ZwFink/hpacml-
experimentation/tree/main/benchmark_evaluation.

The provided software performs each step end-to-end. Given the provided
architecture search space configuration file, the model training step
automatically performs the neural architecture search and saves the best
models to the specified output directory for each benchmark. Given trained
models, the benchmark evaluation script embeds each model in the application,
runs it with and without the NN model used in the application, captures the
accurate and approximated output, and calculates end-to-end speedup and output
loss. This script also captures ancillary timing information, such as time
spent on data transfers, model inference, etc.

Each step generates a `csv` file containing the results. In the model
evaluation stage, each benchmark is run $20$ times, with each instance given a
line in the `csv` output file.

The trained models are available for download, allowing you to skip the model
training step. Each directory has a `commands``.sh` script that performs the
corresponding step.

### Artifact Analysis (incl. Outputs)

The information displayed in the plots is derived directly from the output
`csv` files generated by the model evaluation stage for each benchmark. Each
benchmark is run for $20$ iterations: we drop the first two and use the final
$18$ to calculate the arithmetic average values used for speedup: accuracy
results are constant for a given model.

Artifact Evaluation (AE)

###  IX-A Computational Artifact $A_{1}$

### Artifact Setup (incl. Inputs)

To begin, clone the repository https://github.com/ZwFink/hpacml_artifact.
Then, use the provided `setup``.sh` script to build or download or build a
container image using either Docker or Apptainer.

This script also downloads all data necessary for training, and evaluating
surrogate models. Upon completion, the installation creates a file
`run_``container``.sh` that activates the container environment.

### Artifact Execution

The script `setup.sh` installs/compiles all necessary libraries and code into
a container.

### Artifact Analysis (incl. Outputs)

This artifact does not contain any output for analysis.

###  IX-B Computational Artifact $A_{2}$

### Artifact Setup (incl. Inputs)

This artifact is built in the setup for Artifact 1 and installed in the
`hpacml` directory. The container includes the hpac-ml compiler and runtime
system, and sets the modified `clang` and `clang++` executables to the hpac-ml
compiler.

### Artifact Execution

This artifact is executed by compiling programs with the `clang++` executable
and using the `-fapprox` flag to compile programs. Note that the artifact
includes Makefiles and driver scripts that automatically build all benchmakr
programs.

### Artifact Analysis (incl. Outputs)

This artifact does not contain any output for analysis.

###  IX-C Computational Artifact $A_{3}$

### Artifact Setup (incl. Inputs)

The hpac-ml benchmark suite is installed to the `benchmarks` directory during
the setup for Artifact 1. Each benchmark has a `Makefile` to compile the non-
approximated version and a `Makefile.``approx` to compile the approximated
version.

### Artifact Execution

These artifacts are executed in the experimental workflow of Artifact 4.

### Artifact Analysis (incl. Outputs)

This artifact does not itself contain any output for analysis.

###  IX-D Computational Artifact $A_{4}$

### Artifact Setup (incl. Inputs)

This artifact is installed in the container setup in Artifact 1.

### Artifact Execution

This artifact has two components: model training and benchmark evaluation.
Model training performs the nested Bayesian Optimization process outlined in
the paper to explore accuracy/performance trade-offs. This artifact stores the
created models in a directory for each benchmark. Because training for each
benchmark takes between $50$ and $400$ GPU hours, we have included the trained
models in the artifact. These trained models are used for benchmark
evaluation.

For each model created for every benchmark, the benchmark evaluation step does
the following with a Python script:

  * •

Build the accurate version of the benchmark and run it, capturing the runtime
and outputs.

  * •

Build the approximate version of the benchmark and run it, capturing the
runtime and outputs.

  * •

Compute error and speedup and store the results to a CSV file.

  * •

Plot the results to create the accuracy/performance trade-off figures shown in
the paper.

### Artifact Analysis (incl. Outputs)

The CSV files contain model speedup and error, along with the timing of
different events in the hpac-ml runtime system, such as translating between
application memory and NN tensors. The format is explained futher in the
`README` file that accompanies the evaluation.

These files are used to create accuracy/speedup scatterplots, like the ones
shown in the paper. This artifact does this automatically, executing the
entire workflow and outputting a plot.