figurec

# The Flux Operator

Vanessa Sochat ID Aldo Culquicondor  Antonio Ojea  Daniel Milroy ID Google

###### Abstract

Converged computing brings together the best of both worlds for high
performance computing (HPC) and cloud-native communities. In fact, the
economic impact of cloud-computing, and need for portability, flexibility, and
manageability make it not important, but inevitable. Navigating this uncharted
territory requires not just innovation in the technology space, but also
effort toward collaboration and sharing of ideas. With these goals in mind,
this work first tackles the central component of running batch workflows,
whether in cloud or HPC: the workload manager. For cloud, Kubernetes has
become the de facto tool for this kind of batch orchestration. For HPC, the
next-generation HPC workload manager Flux Framework is analogous – combining
fully hierarchical resource management and graph-based scheduling to support
intelligent scheduling and job management. Convergence of these managers would
mean the implementation of Flux inside of Kubernetes, allowing for
hierarchical resource management and scheduling that scales impressively
without burdening the Kubernetes scheduler itself. This paper introduces the
Flux Operator – an on-demand HPC workload manager that is easily deployed in
Kubernetes. The work here highlights design decisions, mapping of components
between environments, experimental features, and shares the results of
experiments that compare performance with an equivalent operator in the space,
the MPI Operator. Finally, discussion closes with a review of challenges
remaining, and hopes for the future for improved technological innovation and
collaboration.

##  1 Introduction

Portability, manageability, and modularity of complex, heterogeneous workflows
is becoming increasingly important for high performance computing (HPC). In
particular, the need for workflows to be extended to cloud environments is a
key component of collaboration across an increasingly diverse set of
computational resources, and a likely solution for “green computing" to ensure
energy efficiency and optimal usage of shared resources [1]. Other demands for
flexibility of compute include the increasing use of internet of things “IoT”
remote devices to conduct research [2, 3], an explosion of hardware available
on cloud platforms [4, 5], and the dynamic addition of external resources [6].
A powerful demonstration of need also comes from a series of events organized
by the European Commission [7] to assemble experts for discussion on
innovation in the “computing continuum," citing a strong need for flexibility
for distributed systems, green and dynamic technologies, and an emphasis on
open source software. The discussion continues with workshops [8] emphasizing
the importance of shaping Europe’s digital future. Given this landscape, any
entity involved in the business of scaled computing will fall behind if these
technological needs are not prioritized [4].

In cloud computing communities, machine learning workloads are also becoming
increasingly important [9, 10, 11, 12], and the cloud container orchestration
technology Kubernetes [13] has become the de facto standard for orchestration
of these workflows. As of June of 2023, the Kubernetes project had
approximately 74,000 contributors, making it the second largest open source
project ever after Linux, and the “most widely used container orchestration
platform in existence” [13]. Outside of the academic community it is the
chosen platform for orchestration, being used at over 70% of Fortune 500
companies [13]. In recent years [14], the growing need for supporting batch
workflows [15] has led to the batch working group. This group works on the
design and implementation of application programming interfaces (APIs) to
enable cloud-native batch workflows and jobs, and provides an interesting
transition of Kubernetes from primarily a stateless, service-oriented
architecture to one that can support states and a desired completion of work.
The first stable release [16] of the Job controller marked an unofficial
declaration of Kubernetes supporting what, at face value, looked like more
traditional workflows from HPC. This development made the idea of deploying
one of the world’s top supercomputers in Kubernetes an achievable goal [17],
and the needs of the cloud computing communities overlapped better with the
needs of HPC than ever before.

In the high performance computing space batch processing has a long history,
and consequently the community has deep expertise [18]. The need to embrace
traditionally more cloud-like features arguably came down to the demands of
the workloads themselves. While a traditional HPC workload might be
embarrassingly parallel, meaning running equivalent, scoped tasks across a
homogeneous set of resources concurrently, modern workflows include the gamut
from simulation to batch processing to artificial intelligence (AI) and
services. A standard workflow is no longer a single run that writes to a
shared filesystem, but rather an assortment of tasks that vary in their needs
for hardware, storage, and running times. Modern workflows are typically
provided via directed acyclic graphs (DAGs) that not only indicate direction
or order, but also utilization of entirely different architectures, services,
and virtualization technologies. Indeed, the high performance computing
community needed the same portability, flexibility, and automation for these
workflows afforded by cloud, and to span both applications and services.

It would be in the best interest of cloud communities to learn from and take
on the best technological innovations from HPC, and vice versa. Thus, this
landscape with overlapping interests was a spark for collaboration, and the
time was right for the convergence of not just these communities, but the
technologies themselves. This collaboration, or the convincing of one
community to engage with the other, is arguably more challenging than the
development work itself. When approaching those on the HPC side, discussions
that suggest using cloud very quickly turn to the matter of performance, costs
[19, 20] and security [21]. Approaching the cloud side, there is often lack of
understanding for how high performance computing technologies might be useful
or needed. In the case of the first, a simple solution that resolves many
concerns is the often forgotten reality that cloud setups can be public or
private. A technology such as Kubernetes could be deployed on-premises. In the
case of the second, arguably the cloud computing community needs convincing
that they too can benefit from the adoption of HPC technologies. Increasing
performance and efficiency by using techniques from HPC, and providing better
transparency of underlying resources by way of low-level performance analysis,
would both lower costs and time to completion. A solution that falls in the
middle would likely bring together the best of both worlds, but could come
with compromises such as paying a performance penalty for flexibility.
Ideally, a converged approach would be able to most effectively use hardware
and improve performance, with increased flexibility offsetting any potential
compromises for said performance.

Anticipating a desired future where cloud and high performance computing
communities are collaborating and developing solutions together, the
challenges stated above can be inspected to understand what is needed for both
sides. First, a solid demonstration is needed that there are benefits for both
cloud and HPC to take on attributes of the other side. For HPC, this means
more modularity, portability, and automation. For cloud, this means more
performant workflows, efficient use of hardware, schedulers, and communication
protocols that span networking, applications, and storage. Secondly, examples
of such technologies must be prototyped that can bring together the best of
both worlds - the performance of HPC and the flexibility of clouds. This
vision, or the space of technologies that exists between cloud and HPC can be
described with the term “converged computing" [22, 23]. In a converged
computing landscape of the future not only will technologies from
traditionally disparate communities be brought together, but traditionally
disparate communities will be united culturally to identify shared goals and
foster deeper, more meaningful collaborations.

While many areas of work can be tackled, it was logical to start with a
workflow manager analogous to Kubernetes in the high performance computing
community, with the common use case of running simulations or machine learning
tasks. The Flux Framework, a novel hierarchical framework for resource
management and scheduling, provides similar abstractions that parallel those
in Kubernetes, including modularity, well-defined developer and user
interfaces, and an ability to integrate and manage different types of
resources. It stands out from other resource managers because of its ability
to manage the exploding complexity of modern workflows described previously.
To start with modularity, in the same way that several components are combined
to create Kubernetes [24], components from Flux Framework [25] are assembled
together to manifest in a workload manager that is referred to simply as
“Flux." This modularity might mean, for example, that a component of Flux
could be used in Kubernetes, or vice versa. For developer interfaces, arguably
a core ingredient of convergence is having common programming languages or
bindings. Kubernetes, as it was developed at Google, chose to use the Go
programming language, also designed at Google [26, 27]. Flux also provides a
rich landscape of language bindings, one of which is Go. These shared
interfaces and modularity make convergence possible.

Given the overlapping need to schedule jobs, the first work in this space was
to integrate the Flux scheduler “Fluxion” as a plugin scheduler for Kubernetes
called “Fluence" [23]. The rationale for this early work was that Flux could
benefit Kubernetes in several ways. Firstly, Flux represents and schedules
resources via directed graphs, which is notably different from the default
Kubernetes scheduler that selects work for nodes based on a feasibility score
[28]. In fact, Flux was created to address significant limitations of
traditional HPC resource managers and schedulers by facilitating workload
portability, handling high throughput job requests, and employing
sophisticated techniques for flexible and fine-grained task placement and
binding [29]. Enabling the Flux scheduler in Kubernetes would bring this same
graph-based and hierarchical resource-aware approach to Kubernetes, and this
early work demonstrated exactly that – improved performance against the
default Kubernetes scheduler [30]. More efficient scheduling was demonstrating
by enabling MPI-based CORAL2 workloads to run and scale in Kubernetes that, by
way of Fluence, avoided pathological resource mappings and resource starvation
[22]. This work also demonstrated a valuable point – that the scheduling
provided by a workload manager must be able to concretely meet the resource
demands of a workflow, but to do so efficiently and effectively to maximally
utilize a set of computational resources.

Aside from the technological benefits that might come from convergence of
these two specific technologies for end- and developer- users, enabling Flux
to run in the cloud would also provide benefits for cloud vendors attempting
to attract a larger HPC customer base. Current products that target HPC
researchers [31, 32, 33, 34] arguably serve as training wheels to help with a
transition to the cloud. Other products that do not deliver a familiar command
line interface would require an on-boarding process. The realization that
workflows could be seamlessly portable by way of Flux, and Flux could serve as
a vehicle for the workflow user-base to move between cloud and HPC, inspired
the next round of work discussed in this paper. By making the full Flux
workflow manager, with all components assembled, available in Kubernetes,
workflow specifications that work on HPC with Flux would also work in a cloud
environment also with Flux. For cloud vendors, the HPC user base could more
easily make a smooth transition to using cloud too.

This paper introduces the Flux Operator [35], the next step in work to explore
integration of a traditional HPC scheduler within Kubernetes. The Flux
Operator is a Kubernetes operator [36] that handles all the setup and
configuration of a fully fledged Flux cluster inside of Kubernetes itself,
allowing the user to efficiently bring up and down an entire HPC cluster for a
scoped set of work that optimizes for important aspects of HPC. This paper
first reviews the design and architecture of the Flux Operator (Section 2),
discussing Kubernetes abstractions for efficient networking and node setup.
Discussion then moves into how these design decisions impact essential needs
such as workflows that use message passing interfaces (MPI), and experimental
features like scaling and elasticity (Section 3). Finally, experimental work
shows the Flux Operator having superior performance over the MPI Operator, the
primary available option at the time for running MPI workflows (Section 4).
The paper concludes with discussion for anticipated future work,
considerations for workflow design, and vision for the future (Section 5).

##  2 Architecture

This section details the architecture of the Flux Operator, first describing
the design and needs of Flux, and how those are satisfied in Kubernetes. From
the standpoint of a software architect, the task of designing the Flux
Operator could be approached as a problem of pattern matching. Knowing that
Kubernetes provides a set of components [37] and application programming
interfaces [38], a key challenge was to assemble the components in a way that
would deploy the full Flux Framework stack running inside of Kubernetes. An
ideal design might aim to achieve the keystone properties of Kubernetes
applications, including but not limited to fault tolerance, load balancing,
and elasticity [39]. Abstractions for storage [40], networking [41], and
workloads [42] could be selected for this design, and with a mindset of
portability, meaning that the software components would be in containers
themselves [43]. The following sections refer to two roles – an operator
developer, or someone that designs and implements the Flux Operator itself,
and an operator user, an individual that installs the operator and uses it.
These architecture sections start with an overview of Kubernetes Operators,
and then describe each component of Flux, and a mapping from traditional bare
metal solutions to abstractions in Kubernetes.

###  2.1 Kubernetes Operators

While individual components such as pods or services [42, 41] could be
individually implemented and created in the Kubernetes cluster, the advent of
programmatic operators [44] in 2016 has hugely simplified this process for the
developer user. A Kubernetes operator serves as a controller for one or more
Kubernetes objects, meaning that a developer can express all of the custom
logic needed for an application of interest in code that is compiled and run
in Kubernetes [45]. The operator implements a custom resource whose behavior
is dictated by a custom resource definition (CRD), a YAML file with a
specification of variables for the controller to use [46]. For the Flux
Operator, this custom resource is called a “MiniCluster” [47]. The basic
design of a controller is a loop, running a reconciliation process until a
cluster reaches a desired state requested by a user via this custom resource
definition YAML specification [36]. This is a declarative model [48] in that
the operator user can specify high level constructs such as the cluster size
and application to run, and they don’t need to know the details of setting up
a Flux cluster, nor do they need to consider orchestration or update of
components. This approach is advantageous in that it exposes only the amount
of detail that is needed for some number of jobs, and the complexity that
would require niche expertise is hidden.

###  2.2 Flux Framework

Flux Framework is a novel, graph-based resource manager that is typically
deployed on-premises at HPC centers [29]. It won an R&D 100 award in 2021 [49,
50], and is currently stated to be the primary scheduler for the upcoming
exascale-class system “El Capitan” at Lawrence Livermore National Laboratory
[51]. It is called a framework because several projects combine together to
form the resource manager known as Flux. A summary of these projects is
described in Table 1, and the interested reader is directed to the learning
guide [52] for an in-depth overview.

Table 1: Flux Framework Projects Project | Description
---|---
flux-core | core services
flux-pmix | flux shell plugin to bootstrap OpenMPI v5+
flux-sched | Fluxion graph-based scheduler
flux-security | security code
flux-accounting | user bank and accounting

While Flux can be described in terms of its modules or components, for the
work here it will described as it is seen in the space of Kubernetes
abstractions.

####  2.2.1 A Flux MiniCluster

##### The Node

Flux is typically deployed across nodes on an HPC cluster, where each node can
be thought of as an addressable compute or storage unit, and as having a
unique network address. Moving into the space of cloud, the physicality of the
server goes away, and instead the basis of a node is a virtual machine.
However, while Kubernetes itself is deployed on nodes, the notable object is
the pod – an abstract slice of a node that is tasked with some unit of work to
run one or more containers, and allocated a particular set of resources [53].
Since the Kubernetes scheduler has no issue slicing up a single node into many
pods, the first task in defining this cluster was to ensure that there was a
mapping of one pod per actual physical node. The reason for this mapping is
due to Flux not being able to detect running on a partial node, which is a
result of its use of the portable harware locality (hwloc) [54] library to
discover resources. The hwloc library can only detect the resources of an
entire host [55], which in the context of two pods running on one node, would
erroneously discover the same set of resources twice, double what is actually
available. In practice, this would mean that Flux could schedule too much work
on a single physical node that, to the resource manager, is seen as two
separate nodes with identical resources. The 1:1 mapping of pods to nodes was
originally achieved by way of a resource specification, a strategy that
required the user to ask for just under the upper limit of CPU and memory
offered by their cloud instance of choice [56]. This strategy was later
improved to not require user expertise by way of rules for pod affinity and
anti-affinity. These are essentially constraints that tells the scheduler to
ensure one pod per node, each with a hostname for Flux [57].

##### The Cluster

While it would be possible to deploy individual pods onto nodes, early
Kubernetes offered further abstractions for sets of pods such as deployments
[58] and Stateful or Replica sets [59, 60], and in 2015, an abstraction called
a Job that was the first of its kind to emulate a traditional HPC job with the
intention to run and complete [61]. As of 2021, the batch working group
introduced the indexed mode addition to Job [62] where the same work could be
done in parallel, expecting 1 to $N$ completions [63]. In that each node of a
simple Flux cluster would be identical aside from subtle differences in
startup, although other abstractions were considered, an indexed job was
ultimately chosen. The indexed job is ideal in that in inherits needed
features from the base Job such as having states, and adds an ability to
create duplicates of the pods. To create pods it uses a batched approach [64],
which is also advantageous to introduce an indexed ordering that ensures the
index 0 is cleaned up last. This allowed us to easily design the operator to
use the index 0 pod as the lead broker, and any scaling up or down of the
cluster (Section 3.1) would never risk deleting the lead broker. Within this
cluster context, given the assignment of one pod to one node, for the
remainder of this paper the terms “node" and “pod" are used interchangeably as
they are mapped to the same resources, memory and CPU.

##### Networking

While it might be thought that the core of Flux is the project “flux-core,”
one of the foundational components of Flux Framework is the scalable tree-
based overlay network, or “TBON” that connects the core modules for scheduling
and resource management. The Flux TBON is a rooted tree [65] that features
leader and follower processes (brokers), each of which is typically mapped to
one node. The leader expects follower brokers to connect to it. Mapping this
design to the indexed job, the role of lead broker can be assigned to index 0,
and the follower brokers to indices 1 through N. Along with being easy to
remember, this design decision allows pods to be created in order of their
index with the lowest first [66], and this setup is ideal to have the lead
broker up earlier for the follower brokers to find. The initial networking of
the cluster is done with ZeroMQ [67], where the follower brokers identify
their place in the cluster by way of their rank in a shared system
configuration file, and then connect to the lead broker on a specific port via
transmission control protocol (tcp) [68]. If the lead broker is not up first,
while the worker will wait to try connecting again, by default the ZeroMQ
library falls back to a tcp default retry timeout that increases exponentially
[69]. In practice this means delayed cluster startup times waiting for the
follower brokers to retry. The scheduler and resource manager combined with
this set of brokers that can communicate to run jobs is called a Flux instance
[70]. A Flux instance can be on the system level, meaning shared by many
users, or owned by a single user. In both cases, the Flux instances handle
user requests for submitting or otherwise interacting with jobs. The instance
itself is hierarchical because it can spawn sub-instances whose resources are
a subgraph of their parent.

The above networking setup can give each pod a unique address that can be
written into the Flux system configuration, and used to identify lead and
follower brokers. For the first naive implementation, the Flux Operator
created the pods, retrieved the IP addresses after creation, and then added
corresponding entries to the “/etc/hosts” file for DNS resolution. Automated
management of the hosts file proved to be a bad design because it required
restarting all the pods. Instead, a later design created a headless service
for the MiniCluster [71], meaning that each pod could be given a label, a key
value pair, that was known to the headless service, and discovering the
labeled pod would add it to the network provided by the service. The headless
service can then also assign a predictable hostname, which is essential for
Flux to identify it. This simplified the creation of the cluster, and allowed
for having the networking ready as soon as the service object was ready. Once
this is done and the brokers have connected over TCP, further communication
for the overlay network is done via ZeroMQ sockets [29]. However, for the
cases of workflows that use a message passing interface (MPI) [72], Flux has
built-in modules for MPI and communication, meaning that Flux simply uses
standard MPI libraries that can rely on sophisticated networking fabrics or
other high speed interconnects [73]. This reliance on cloud hardware has
proven to be a challenge when deploying the Flux Operator to different cloud
providers, and is a focused area of collaborative work.

##### Volume Mounts

In terms of configuration, Flux requires a system configuration file, a ZeroMQ
CURVE certificate used to encrypt communication [74], and some means to start
the brokers. In a traditional HPC setup, this means could be using a systemd
service [75], however in a Kubernetes environment with containers, it means a
start command for the container with conditional logic for the lead vs.
follower brokers. In Kubernetes, all of the above configuration can be
achieved via volume mounts provided via ConfigMap [76] objects. By way of
mounting configuration files and other needed files to each pod container as
read-only volumes, all nodes in the cluster have access to them. These are
mounted at /etc/flux and /flux_operator for configurations and the starting
script, respectively, and the choice of a root path affords discoverability.

The curve certificate presented a bootstrapping design problem, as the
standard way to generate it is usually via Flux itself (ZeroMQ is compiled
within and exposed via the flux keygen command). However, this content was
also required to exist for the read-only volume before starting the pod
container. For the earliest design of the Flux Operator, a one-off certificate
generator container was brought up that ran this key generation command, and
the key was printed to the log to be retrieved by the operator. It could then
be straightforward to write into the ConfigMap to be shared by the MiniCluster
pods. In a later design, by way of collaboration with authors of this paper
following Kubecon Amsterdam ’23 [77] this bootstrapping problem was further
improved by compiling ZeroMQ directly into the Flux Operator, and using cgo
[78] to interact with it directly to generate the certificate content for the
ConfigMap inside the operator. This removed an entire step to generate the
one-off pod, and is a beautiful example of how sharing ideas and collaboration
can lead to improvements in design and functionality.

##### A Flux Container

The libraries and software needed for Flux along with configuration steps must
happen in a common Flux container that is replicated by the indexed Job. This
container would need to come pre-built not only with Flux and needed modules,
but also with any application of interest to be run by the user. This is a
design flaw in that most containerized applications for HPC have not been
built with Flux, and would need to be built again. While the HPC community is
attuned to building and optimizing components for different architectures,
ideally a more modular, cloud-native solution would not require investing time
to do that. Once required software and configuration files are present, setup
continues to create either a single-user or site-wide installation of Flux.
The Flux Operator opts for a single-user design, and enables customization via
variables exposed on the CRD. This customization includes (but is not limited
to) archiving of data, creating multiple users, starting in an interactive
mode, starting a RESTful application programming interface, or creating a
custom batch job. The final component of the container is the “entrypoint" or
the command that is run when the container is started. This varies between the
lead and follower brokers, where the lead broker typically starts with a
command to run a job, and the follower broker starts expecting to connect and
receive work.

##  3 Features

Once it was possible to run and complete a basic workflow (discussed in
Section 4), development thinking moved toward adding desired features for such
a workload manager in Kubernetes. This section will describe early work to
enable scalability and saving state, elasticity and auto-scaling along with
workflow integration. These features for workflows, along with the core design
of the Flux Operator, are considered experimental in the sense that they are
implemented with the goal of testing and improvement in mind. The below
represents a sample of this work, and more experiments can be found in the
examples directory of https://github.com/flux-framework/flux-operator.

###  3.1 Saving State

The goal of experiments to save state would involve starting a Flux
MiniCluster, running some number of jobs, pausing them, saving the state of
the job queue, and then bringing it down to bring up a differently sized
cluster (larger or smaller) to load the jobs into, where they would continue
running. This concept of saving state is similar to forensic container
checkpointing [79], an experimental idea for Kubernetes, and would be useful
for pausing workflows for cost savings or waiting on resource availability.
These experiments varied based on when the queue was paused. In the earliest
tests, job completion was required before saving state, while for later tests,
jobs were stopped mid-run.

In practice saving state meant waiting for the queue, pausing, and then saving
to an archive in a shared volume between two MiniClusters. The first
MiniCluster pods were then waited for to terminate, and the new MiniCluster
was waited for to come up and restore the jobs. It was observed that jobs
would successfully save and load into the new cluster, maintaining job
identifiers and size, however when stopping a running queue, 1-2 jobs could be
lost between the transfer. While the reason for this loss would be interesting
to understand, as it is an experimental prototype for a feature, the work is
beyond the scope of this paper, and akin to other features discussed here,
should be pursued with a compelling research use case. When this time comes,
more analysis would be needed to understand exactly what is going on. The
majority of jobs (e.g., a rough estimate of 9 out of 10) transition nicely,
meaning that a job on the previous queue can get scheduled to a new larger or
smaller cluster. As would be expected, if a job is moved onto a cluster
lacking enough resources, it would logically not be scheduleable. A write-up
and tutorial to reproduce this work is available [80].

While this early work to save state was simple, it was a glimpse into the idea
that scheduled workflows could in fact be moved. In changing the size of the
resources available by way of creating a new MiniCluster, it was the earliest
prototype for what might be called scaling or elasticity, discussed next.

###  3.2 Elasticity

Elasticity can be thought of as automated dynamic scaling [81]. Instead of
making a cluster larger or smaller by way of saving state and loading into a
different size, true elasticity means changing the size of a single cluster,
which in the context of the Flux Operator means that Flux must adapt
dynamically. To accomplish this, ideally Flux needed to have support for
resource dynamism, however it did not. Short of that there was a way – one
that might be considered a hack – to get this to work. The following steps
enable an elastic Flux MiniCluster:

  * •

A max size variable is added, meaning more nodes are defined in the system
configuration file than actually exist.

  * •

Flux is told to create a cluster at a size that is between 1 and this max
size.

  * •

Any change request to the CRD (from a user or API) validates the request, and
updates the indexed job.

  * •

An update to increase in size creates new pods.

  * •

An update to decrease in size terminates pods.

The above also carries the constraints that the cluster cannot be smaller than
one node (only a lead broker) or larger than the initial “maxSize." The larger
indices are terminated first and the operator does now allow reduction to size
zero, so the lead broker is never at risk of deletion – such a request would
delete the entire MiniCluster that relies on it. The reason this works is that
Flux sees the initial set of pods that do not exist as simply being down,
which is happens frequently in a high performance computing environment. When
the nodes are created their corresponding follower brokers start, ping the
lead broker on the port to connect (typically port 8050) and then they
seamlessly join the cluster. From the standpoint of the user, they change the
“size" value in their MiniCluster CRD, apply it, and then see their cluster
grow or shrink. The Flux instance run by the lead broker simply sees a node
come online. On the Kubernetes side, this ability for the indexed job to have
elasticity requires a minimum Kubernetes version of 1.27 [82].

At this point, it needed to be decided what might trigger this change in size.
Elasticity was implemented in two ways, first with an application-driven
approach [83] that required extra permissions to be given to the in-cluster
service account, allowing the application inside the cluster to ask for more
or fewer pods directly. It was then discovered that Kubernetes has autoscaling
APIs intended for this use case. This autoscaling approach is discussed in the
next section.

###  3.3 Autoscaling

In Kubernetes there are two types of scaling - horizontal and vertical.
Horizontal typically refers to adding pods, while vertical refers to adding
resources to existing pods [84]. Both are based on the idea that resources
should change in response to changing workload needs – if cluster or resources
are too big, they are made smaller, and vice versa. In the case of the Flux
Operator the primary interest was horizontal auto-scaling, or changing the
number of pods to dynamically increase or decrease the size of the MiniCluster
to respond to the demands of a workload. This led to a first try to implement
horizontal pod auto-scaling (HPA) [84] using the HorizontalPodAutoscaler API
resource, a cluster API to watch a selected set of pods for resource
consumption and increase or decrease the number depending on utilization. For
the simplest cases, a default autoscaler was first deployed that considers a
metric such as percent CPU usage and uses an algorithm [85] to calculate a
target scale value for the number of pods. This could be tested by running a
CPU intensive simulation [86] to observe the autoscaler adding and removing
pods. However, the approach was not fine-tuned enough to the potential needs
of an application being run by Flux. Instead of an arbitrary decision to add
or remove pods based on CPU, a design more specific to Flux is warranted. As
an example, one design might be that the Flux lead broker makes decisions
about when and how to scale depending on the content of the queue. Another
valid design would be to allow for changing the size of a single running job.
Both of these ideas, and more generally designs for autoscaling, are valid and
prime for future work.

With this in mind, a custom metrics API was implemented [87], meaning
implementing an equivalent API endpoint controller that would be called by an
autoscaler with instructions for how to scale the cluster. This resulted in
the Flux metrics API [88], a standalone API that runs directly from the lead
broker pod and provides decisions about scaling up or down based on the size
or other metrics about the queue. With this API, it was possible to
demonstrate an auto-scaling operation running based on a trigger coming
directly from Flux. More work will be needed to test this setup with real
workflows. In the meantime, more details about this setup and basic elasticity
are available in an external post [86].

One notable feature about the implementation of the autoscaling approaches
described above is that regardless of whether the request comes directly from
a user changing a value in a file or an application or a programmatic
autoscaler, the same internal logic (functions) are used to validate and then
perform the patch.

###  3.4 Multi-Tenancy

Multi-tenancy refers to the ability to support multiple users on the same
resources. This is not a common design in Kubernetes, as ownership of
resources is typically designated by namespaces, custom permissions on
connected resources like storage, and role based access controls (RBAC) [89].
Recognizing these challenges, as an early approach there are several modes of
interaction:

  * •

Single user: the user owns an entire MiniCluster, and uses the default Flux
user in the container

  * •

Multiple users: controlled via PAM [90] authentication

  * •

Multiple users: controlled via RESTFul API access

In anticipation of the last two cases that implement multi-tenancy, a RESTFul
application programming interface (API) [91, 92] was designed that runs from
the lead Flux broker pod, and thus exposes interactions with Flux to submit,
get info for, cancel, and otherwise interact with jobs via Flux Python
bindings exposed by the API. This is made possible by exposing the internal
port that the API is running on via an external NodePort [93] and port
forwarding [94] an external client outside of the cluster can interact with
it.

In all cases of requiring authentication, the Flux RESTful API uses an OAuth2
based approach, storing a database of user identifiers and encoded passwords
[95], and first authenticates by using a base64 encoded username and password
(typical of a basic authentication scheme [96]), and then provides the user
with an expiring token that can be used for further interaction. In the case
of a single Flux user behind a multi-tenant API, this authentication and
authorization happens, and then all users submit jobs to a shared queue. In
the case of more true multi-tenancy with PAM, the custom resource definition
asks for usernames (and optionally, passwords) in advance, and then creates
the accounts on the system that are checked after authentication. The
installation of flux-accounting [97] can then be enabled for the lead broker’s
queue, and use a traditional fair-share algorithm to determine job priority
[97]. This work can be extended with more cloud-native approaches that take
advantage of namespaces and roles, such as is described later (Section 3.6).

###  3.5 Bursting

To complete the early work in autoscaling, the concept of bursting was
considered [98], which means not just extending the size of a local cluster,
but actually extending work to external resources. The bursting work for Flux
would extend this approach to not just deploy external resources, but allow
the lead broker to connect to brokers that are deployed in the other clusters.
As an example, a Kubernetes cluster running on Google Cloud might burst to a
cluster running on Compute Engine (CE), or to a cluster on Amazon Elastic
Kubernetes Service (EKS).

To implement a prototype for bursting, a simple design was chosen first. A
plugin service would be running from the lead broker of a primary cluster, and
the running user would load one or more bursting plugins into it. Each
bursting plugin is targeted to a particular provider (e.g., EC2 or CE). While
there are many ways to trigger a burst, a simple approach of looking for an
attribute “burstable” on a job set to true was chosen first. This request
could be done on the command line. Upon discovery of this attribute, the
bursting service attempts to schedule the job with the plugin. Each plugin is
free to decide if the request is satisfiable by its own custom terms. If the
burst is satisfiable, the job is assigned to the bursting plugin, and the
plugin creates a new cluster or assign the job to an existing cluster. In the
case of creation, the technique of telling the primary cluster that there are
more nodes that are expected (and start down) than there actually are is used,
and assign them namespaced hostnames that will correspond to the bursted
cluster. The calls that are necessary to bring up the second cluster are run,
which might mean deploying Terraform [99] configuration files or creating a
second Kubernetes cluster via API calls, and then the cluster starts just as a
local MiniCluster would. The key difference, however, is that the lead broker
of the primary cluster is exposed as a NodePort [93] service that can be
discovered by the external cluster. The secondary brokers, all followers, then
come up, find their hostnames in the ranked system configuration, and connect
to the lead broker IP address from another cluster. To the user, they simply
see that the nodes are down, and then they come up when the cluster is ready.
Jobs that are scheduled on the primary broker queue that possibly could not
run due to insufficient resources can then run. At the time of this writing,
the main bursting service is implemented along with four bursting plugins for
each of GKE, EKS, CE, and a local burst [100].

Finally, the bursting service is designed to be modular and flexible. Aside
from being able to load different plugins, it allows for customization of the
function provided to select a burstable plugin, to interact with the queue,
and to select jobs. A mock version of a Flux job is also available for
development. The work in bursting is still early, and akin to elasticity, work
on these prototypes should continue to eventually develop more hardened Flux
modules and algorithms for bursting.

###  3.6 Workflow Integration

Running a single MiniCluster to create an isolated Flux cluster in Kubernetes
is a good first step, but not sufficient for real-world use cases of complex
workflows. While it would be possible to shell into or otherwise interact with
the cluster and run a workflow tool that implements Flux as an executor [101,
102, 103], this also does not enable features needed for complex,
heterogeneous workflows that might require different sizes or configurations
of MiniClusters. For this reason, the authors of this paper have started to
think about how to integrate the MiniCluster custom resource definition as a
first-class citizen into workflow tools.

After the Kubecon Amsterdam ’23 presentation, collaborators (including author
AC) were quickly motivated to add the Flux Operator as a job type to the
workflow tool Kueue, a Kubernetes-native job submission operator that handles
managing multi-tenancy of Kubernetes batch jobs that can be produced by a
number of operators [104]. A similar approach is being developed to have a
workflow tool control creation and management of MiniCluster, an idea being
implemented into the Snakemake [101] workflow tool as a Kueue executor plugin
[105]. Defining even an example workflow for high performance computing is a
non-trivial problem, as many codes are either private or not portable. This
initial work with the Flux Operator is hopefully setting the stage for doing
more exploratory work for integrations of this type. Tackling this early
problem is a two-fold challenge to design technologies and inspire more
collaborative opportunities for the HPC community.

##  4 Experiment

The Flux Operator is compared to another state-of-the-art Kubernetes Operator,
the MPI Operator, which at the time of the experiments was considered the main
option in the field for running MPI workloads [106, 107]. The MPI Operator
started as part of the Kubeflow project and defines an “MPIJob” custom
resource [108]. Unlike the Flux Operator that coordinates between brokers with
ZeroMQ, the MPI operator coordinates workers via secure shell (SSH) [109]. It
requires an extra launcher node that serves the sole purpose of coordinating
the workers, and akin to the Flux Operator, uses dedicated hostnames with an
equivalent headless service. This launcher node conceptually could be thought
of as analogous to the lead broker of the Flux instance in that it serves as
an entrypoint for submitting jobs, however the main difference is that the
Flux lead broker is considered part of the cluster to perform work. The MPI
launcher node is not, and in practice this means the user will need to always
incur the costs of an extra node just for the launcher.

###  4.1 Methods

Experiments were conducted on Amazon Web Services Elastic Kubernetes Engine
(EKS) [110], using the hpc6a.48xlarge instance type [111] with the elastic
fiber adapter (EFA) [112]. intending to test the Large-scale Atomic/Molecular
Massively Parallel Simulator (LAMMPS) [113] in a strong-scaling configuration
across cluster sizes and ranks of 64/6016, 32/3008, 16/1504, and 8/752,
respectively (Figure 1). This design was chosen to mirror previous work [30].
LAMMPS is used by the Collaboration of Oak Ridge, Argonne, and Livermore
(CORAL) as a representative scalable science benchmark as part of CORAL-2
[114]. In testing, LAMMPS runs in parallel on MPI ranks (processes) across
nodes, a molecular simulation [115] and problem size $64x16x16$ was chosen
that would adequately test strong scalability across the chosen rank and node
counts. LAMMPS scalability depends on network latency, and the experiment
results report the total wall time recorded by the LAMMPS run as a metric for
performance. A cluster setup that enables lower latency will run faster, and
ideally the simulation should get faster as the number of nodes is increased
with strong scaling. A second metric time of interest is the time for the
launcher to submit and complete a job. For Flux this means timing the flux
submit command that is given the command to run LAMMPS, and for the MPI
operator it means timing the mpirun command that does the same. The final
metric time of interest was the total cluster creation and deletion times,
which can be calculated based on the total runtime of the MiniCluster minus
the LAMMPS total wall time. This time would include each pod preparing the
broker, starting Flux, and networking with the lead broker. The runtime would
ideally decrease across these chosen rank and node sizes.

To ensure the nodes of the cluster are consistent and do not influence
results, experiments were run on the same Kubernetes cluster, and simply used
smaller portions of it. As the MPI Operator requires an extra launcher node,
the maximum cluster size needed (64) was increased by 1, resulting in a size
65 node cluster for these experiments. Finally, a modified version of the MPI
Operator was used [22] to allow it to scale to over 100 MPI ranks.

Figure 1: Experiment design for testing the Flux Operator against the MPI
Operator. A single Kubernetes cluster of size 65 was created (blue outline) to
test subsequently smaller cluster sizes, including 64 pods (3008 ranks), 32
pods (3008 ranks), 16 pods (1504 ranks), and 8 pods (752 ranks). An extra node
(blue) is needed for the MPI Operator launcher to supplement the nodes doing
work (green).

The experiments proceeded as follows. The main Kubernetes cluster of size 65
is first created. Then, for each of the Flux Operator and MPI Operator:

  1. 1.

Launch Job / create MiniCluster for sizes 64, 32, 16, 8

  2. 2.

Run LAMMPS x 20

  3. 3.

Record timings and save all configuration files and logs

For each experiment run, a single “throwaway” run is first performed to pull
the container with Flux and LAMMPS to the node, where it is cached for further
runs [116]. This ensures that time recorded in creating the MiniCluster does
not include pulling the container image, which would have variability
depending on the image size. The experiments are then run in an automated
fashion using Flux Cloud [117], a simple experiment orchestration tool for
running experiments with Flux MiniClusters on Kubernetes. All experiment code,
configuration files, and tagged containers [118, 119] are available [120].

###  4.2 Results

##### Cluster Creation and Deletion

Bringing up and down clusters of sizes 8, 16, 32, and 64 across 20 runs each,
a weak linear scaling was observed (Figure 2) indicating that the indexed job
could efficiently create pods. All sizes were created and ready in under a
minute with variability of approximately 5 seconds. Likely this variability
reflects the slowest or last node to come up, as the cluster is not considered
to be completely up until all nodes are ready. This result was surprisingly
good, as it could have scaled differently.

Figure 2: Combined Flux MiniCluster Creation and Deletion Times across MPI
ranks, corresponding to cluster sizes of 8, 16, 32, and 64 nodes,
respectively. These experiments were run on Amazon Web Services with the
hpc6a.48xlarge instance type and the elastic fiber adapter for networking.

Due to the design of the MPI Operator, there was not a corresponding measure
of creation time appropriate for comparison, so the two are not comparable
side by side.

##### LAMMPS Total Wall Time

A primary time of interest for running LAMMPS is the total wall time, which is
reported by the LAMMPS software itself. Consistently better performance of the
Flux Operator over the MPI Operator was observed, with mean times that are
approximately 5% faster, meaning that the Flux Operator software completed the
same workload more efficiently (Figure 3). While the differences in means for
this experiment are small, this improvement would likely be more prominent for
longer experiments, and could lead to reduction in total costs. Identifying
the underlying reasons for the improved performance is another task suitable
for investigation with performance tools in future work.

Figure 3: Total wall time, as reported by the LAMMPS software, between the
Flux Operator and MPI Operator. The Flux Operator is consistently faster, a
result that could be more impactful for longer running experiments.

##### Launcher Times

Comparing launchers (flux submit for the Flux Operator, and mpirun for the MPI
Operator) there is a slightly larger time difference (Supplementary Figure 5),
where both generally perform well under strong scaling (the time goes down).
What is unknown is whether there is an inflection point at larger scales where
the MPI Operator might plateau or otherwise show a different pattern. The
resources were not available to run these larger experiments at the time, but
these patterns and scaling can be explored in future work.

##### Design Considerations

Anticipating interest in running experiments of this type, where there is
generally some operator in Kubernetes that is going to pull one or more
containers to Kubernetes and perform scoped work, a visualization of the steps
that have salient times are provided, and the the interested reader is
encouraged to think about them (Figure 4).

Figure 4: Times to consider when performing experiments with a Kubernetes
operator, from the point of creating the cluster to deleting it. For a single
cluster without autoscaling (left) many of the operations are one time costs
(blue), while for an auto-scaling cluster that needs to provision new nodes
and pull containers to them, the costs become repeated (green).

In Figure 4 above, there is a distinction between an operator setup that is
using autoscaling (right) vs. not (left), and costs that are incurred once
(blue) vs. repeated (green). Cluster creation generally means the start and
setup of instances, along with any networking and devices that are needed. The
primary difference between the two scenarios is that an autoscaling cluster is
going to be adding new nodes, meaning that the cluster will need to provision
those nodes, and the user will be required to wait. Thus, this update process
for the cluster that requires the user to wait (and pay for the time) becomes
a repeated cost. This also means that a typically one time cost of pulling a
container may occur several times, primarily when new nodes are added. Note
that this diagram assumes experiments running on one cluster. An autoscaling
setup that employs bursting to new clusters would need to consider the
additional time of creating and deleting the new clusters.

It is suggested to the reader to consider these times, along with the
differences between a setup with autoscaling versus one without or potentially
bursting, for future experiments when anticipating costs. Notably, the setup
time for any particular operator could be generally consistent, and variance
has been seen (but is not reported here) in the other steps between cloud
providers. A more critical study and understanding of these times is warranted
for future work.

##  5 Discussion

This work demonstrates improved performance using the Flux Operator for
running an HPC workflow in a cloud-native environment. This early innovation
comes with strengths, limitations, and and desires for future work that
include the topics of workloads and scheduling, storage, tenancy, and cost-
estimation, among others.

A discussion of limitations and further hopes for innovation is needed for
transparency of this work. First, it is a design flaw that the main execution
container is required to have Flux and the application of interest. This means
that any user of the Flux Operator is required to rebuild their container in
full to include Flux, which not only requires the work to do the re-build, but
some limited knowledge of Flux. While container requirements are provided
alongside the documentation [121], this is not good enough. While the
dependencies and complexity exist to enable advanced capabilities, the authors
believe there are approaches that can improve upon this strict requirement.

A next limitation is the creation of the entire MiniCluster using a single
indexed Job. While this is the ideal for the time being, as the indexed Job is
released with core Kubernetes, an eventual refactor to use a JobSet [122]
would be desired, which allows for the same indexed Job to be used, however
with better ownership of assets, and an ability to define different groups of
nodes each as a Replicated Job under the same network namespace. Allowing for
different sets of nodes would not only make it possible to separate logic
between the lead and follower brokers, but also allow for creation of a
MiniCluster with different pod specifications mapped to different resource
needs. JobSet would also allow for better definition of a success policy, or
explicitly saying that the job is completed when the lead broker exits. Author
VS has created a prototype using JobSet, and hopes to continue in this
direction when it is considered for Kubernetes core.

For next steps of work for experimental features, the Flux software itself
needs innovation for the set of the hacks that were implemented. The ability
to scale up and down dynamically without “registering” the non-existent nodes
in the Flux system configuration is a good example, along with a more hardened
approach for bursting that likely comes down to plugins written directly in C
or C++ alongside Flux. Especially the work in bursting is early and exciting,
and will be continued in future work. Notably, the experiment application did
not require use of storage, and while there are several tutorials and examples
for different cloud solutions, this is an entire area of the design that
requires further work and thinking.

Another challenge is that of poor workflow consistency and reproducibility.
While there are scattered workflow tools that are used by HPC centers (e.g.,
national labs), these authors consider much of the HPC community behind with
respect to the reproducible workflow movement. Part of this work moving
forward is to not only identify proxy applications and workflows, but also to
containerize them, and make it quick for an interested party to run them
easily in a cloud environment. Part of this work will not only be
understanding how they work in containers and across a Kubernetes cluster, but
also developing means to assess performance.

An understated challenge in the converged computing space is also culture and
communication. As stated in the introduction, convincing one side to be open
to ideas from the other is a non-trivial task. For basic communication, if
there is discussion between HPC and cloud community members, a simple term
like “node” or “scheduler” can mean something different. This might be tackled
through discussion, and creation of a shared lexicon that allows for talking
about comparable distractions. This introduces a further challenge when
looking at the means for communication. Academic groups tend to write papers
(and industry groups less so), and developing software in research is made
more complex by the publication incentive structure that wants to highlight
new research results [123]. Practically speaking, both the HPC and cloud
communities will need to meet one another half way. This might mean
researchers presenting work at (traditionally) more cloud-oriented conferences
and venues, or cloud developers participating in more traditionally research-
oriented venues. For both, it means distributing knowledge through blogs and
other common mediums. This work calls out to cloud vendors an immense desire
to work together. While it is understandable that there is a primary concern
about direct comparison, there is a path for respectful collaboration,
developing technologies that can work across clouds, and learning from one
another.

These experiments were run on one cloud with a particular networking fabric
and instance type, and at a maximum scale of 64 nodes, which is very small for
traditional HPC. However, the recent work to train large language models [124]
provides a common use case for needing scaled resources, and might allow for
shifting incentives toward that. One of the most challenging decision points
for running experiments of this nature is the sheer size of the space of
potential experiments that could be run. As the goal of this work was to
compare the two operators with an HPC application, the choices made reflect
that goal, and a desire to optimize performance (choosing a configuration to
support low network latency) as much as possible. These same experiments run
on other instance types, interconnects, or even regions could have different
results. Further experiments should be pursued that continue to test scale,
elasticity, and performance in the space of networking, IO, and application
metrics.

The Flux Operator brings several features that could be helpful to more
general Kubernetes workflows. The first is that using a Flux cluster inside of
Kubernetes gets around some of the infamous etcd limits or bottlenecks [125].
Submitting to Flux does not stress Kubernetes application programming
interfaces or etcd, and could scale to hundreds of thousands to potentially
millions of jobs [50, 29]. The second is the hierarchical way of looking at
heterogeneous tasks. Kubernetes would benefit from having more flexibility
about telling tasks where they can go, and then binding them to exactly the
resources needed. This brings up tension between a more manual vs. automated
decision made by the Kubernetes kubelet. The Flux Operator does something that
is not native to Kubernetes to help this issue. By way of allocating a pod to
a node and giving control to Flux, possibly ineffective bindings decisions
that are made by the kubelet can be avoided. The Flux Operator allows Flux, a
workload manager accustomed to making intelligent resource bindings, to take
control. To step back, ideally there should (or could) be a mechanism in
Kubernetes to enable more performance oriented decisions that the kubelet
makes. Having a consistent view of resources that the Kubelet is exporting as
the final truth via cgroups is not necessarily desirable, and there are
several reasons why. The first is that there are many ways to slice up a node,
and a “best" way depends entirely on the application in question. For example,
some applications may perform well given an equal split, while others might be
optimally be broken across sockets. This suggests that granularity on the
level of the socket is needed, which is not currently exposed in Kubernetes. A
concrete example comes from the MuMMi workflow [126] that requires one
dedicated CPU and socket to be close to a PCI express us for a particular GPU
to minimize trading data back and forth between CPU and GPU memory. This level
of granularity is not currently exposed in Kubernetes, nor are there efforts
to understand applications on this level. This is a key area for innovation
and collaboration, and understanding basic design patterns for networking, IO,
and application performance is likely a good start. Ideally applications that
are running in Kubernetes today could be better understood via performance
analysis, and a decision made about if the time required to optimize is worth
to be invested for the potential benefit.

##  6 Conclusion

The popularity and economic clout behind cloud computing presents a challenge
for the high performance computing community – to resist new paradigms of
cloud-native technologies and fall behind, losing talent and customers, or to
embrace it, pursuing technological innovation, and viewing the shift as an
opportunity for collaboration and growth. The latter approach, a movement
identified as “converged computing” is a good path to take, and this work
represents an early start towards that desired future. The work here starts
with one of the highest levels for running workflows – the orchestration tool
– and has thought about the convergence of the HPC workload manager Flux
Framework with the cloud orchestration framework Kubernetes. The Flux Operator
is an example of convergence of technologies in this workload management
space, and has demonstrated superior performance to the current equivalent in
the space. In sharing this process of thinking about design to implementation,
the authors of this paper hope to start discussion with the larger community
that spans cloud and HPC for how to think about working on convergence for
other types of software and technology. This work is a shining example of the
collaboration and fun possible. The sharing of these results at Kubecon
Amsterdam ’23 [77], inspired collaboration, discussion, and excitement about
the converged computing movement. Projects and work are underway to address
gaps that have been identified, and each a collaboration between computer
scientists and cloud developers. The authors of this paper hope that this
early work inspires, and allows for continued discussed for innovation in this
space for not just workloads and scheduling, but also the challenges around it
– storage, tenancy, and cost-estimation, among others. Through collaboration
and creativity of design, pathways can be discovered for moving seamlessly
between the spaces of cloud and HPC. Converged computing is a paradigm shift,
and it’s up to the HPC and cloud communtities to decide to embrace change and
grow from it, or fight it off. This work chooses the first approach, and
embraces it with hopes for a better future, and stronger more reproducible
science.

## Supplementary Figures

Figure 5: Launch time measuring flux submit (the Flux Operator) or mpirun (the
MPI Operator). The Flux Operator is consistently faster, a result that could
be more impactful for longer running experiments.

## Acknowledgements

We would like to thank the entire Flux Framework team for regular discussion
on design points, their immense expertise and experience in this space, and a
safe, fun environment to learn and grow. We give our gracious thank you to
both Amazon Web Services and Google Cloud for their support. This project was
supported by the LLNL-LDRD Program under Project No. 22-ERD-041.

This work was performed under the auspices of the U.S. Department of Energy by
Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. LLNL-
JRNL-855145-DRAFT.

## References

  * Bharany et al. [2022] Salil Bharany, Sandeep Sharma, Osamah Ibrahim Khalaf, Ghaida Muttashar Abdulsahib, Abeer S Al Humaimeedy, Theyazn HH Aldhyani, Mashael Maashi, and Hasan Alkahtani.  A systematic survey on energy-efficient techniques in sustainable cloud computing.  _Sustainability_ , 14(10):6256, 2022.
  * Sadeeq et al. [2021] Mohammed Mohammed Sadeeq, Nasiba M Abdulkareem, Subhi RM Zeebaree, Dindar Mikaeel Ahmed, Ahmed Saifullah Sami, and Rizgar R Zebari.  Iot and cloud computing issues, challenges and opportunities: A review.  _Qubahan Academic Journal_ , 1(2):1–7, 2021.
  * Zhu et al. [2023] Weiqiang Zhu, Alvin Brian Hou, Robert Yang, Avoy Datta, S Mostafa Mousavi, William L Ellsworth, and Gregory C Beroza.  Quakeflow: a scalable machine-learning-based earthquake monitoring workflow with cloud computing.  _Geophysical Journal International_ , 232(1):684–693, 2023.
  * Thompson and Spanuth [2021] Neil C Thompson and Svenja Spanuth.  The decline of computers as a general purpose technology.  https://cacm.acm.org/magazines/2021/3/250710-the-decline-of-computers-as-a-general-purpose-technology/fulltext?mobile=false, March 2021.  Accessed: 2023-8-31.
  * Reuther et al. [2022] Albert Reuther, Peter Michaleas, Michael Jones, Vijay Gadepally, Siddharth Samsi, and Jeremy Kepner.  Ai and ml accelerator survey and trends.  In _2022 IEEE High Performance Extreme Computing Conference (HPEC)_ , pages 1–10. IEEE, 2022.
  * Jena et al. [2022] Biswajit Jena, Gopal Krishna Nayak, and Sanjay Saxena.  High-performance computing and its requirements in deep learning.  _High-Performance Medical Image Processing_ , pages 255–288, 2022\.
  * [7] Digital autonomy in the computing continuum.  https://digital-strategy.ec.europa.eu/en/library/digital-autonomy-computing-continuum.  Accessed: 2023-9-24.
  * [8] Concertation and consultation on computing continuum: From cloud to edge to IoT.  https://swforum.eu/events/concertation-and-consultation-computing-continuum-cloud-edge-iot.  Accessed: 2023-9-24.
  * Spjuth et al. [2021] Ola Spjuth, Jens Frid, and Andreas Hellander.  The machine learning life cycle and the cloud: implications for drug discovery.  _Expert Opinion on Drug Discovery_ , 16(9):1071–1079, 2021.  doi: 10.1080/17460441.2021.1932812.  URL https://doi.org/10.1080/17460441.2021.1932812.  PMID: 34057379.
  * George and Saha [2022] Johnu George and Amit Saha.  End-to-end machine learning using kubeflow.  In _5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)_, pages 336–338, 2022.
  * Kreuzberger et al. [2023] Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl.  Machine learning operations (mlops): Overview, definition, and architecture.  _IEEE Access_ , 2023.
  * Liu et al. [2022] Peini Liu, Gusseppe Bravo-Rocca, Jordi Guitart, Ajay Dholakia, David Ellison, and Miroslav Hodak.  Scanflow-k8s: Agent-based framework for autonomic management and supervision of ml workflows in kubernetes clusters.  In _2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_ , pages 376–385. IEEE, 2022.
  * noa [2023] Project journey report.  https://www.cncf.io/reports/kubernetes-project-journey-report/, June 2023.  Accessed: 2023-8-31.
  * noa [a] Batch working group addition to kubernetes community, a.
  * [15] Maciek Różacki.  Kubernetes evolution: From microservices to batch processing powerhouse.  https://thenewstack.io/kubernetes-evolution-from-microservices-to-batch-processing-powerhouse/.  Accessed: 2023-9-24.
  * kub [2022] Kubernetes 1.26: Job tracking, to support massively parallel batch workloads, is generally available.  https://kubernetes.io/blog/2022/12/29/scalable-job-tracking-ga/, December 2022.  Accessed: 2023-9-24.
  * noa [2022a] Building the #7 global supercomputer with google kubernetes.  https://www.pgs.com/company/newsroom/news/industry-insights--hpc-in-the-cloud/, November 2022a.  Accessed: 2023-8-31.
  * Wikipedia contributors [2023a] Wikipedia contributors.  Batch processing.  https://en.wikipedia.org/w/index.php?title=Batch_processing&oldid=1149313823, April 2023a.  Accessed: NA-NA-NA.
  * Munhoz et al. [2022] Vanderlei Munhoz, Márcio Castro, and Odorico Mendizabal.  Strategies for fault-tolerant tightly-coupled hpc workloads running on low-budget spot cloud infrastructures.  In _2022 IEEE 34th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)_ , pages 263–272. IEEE, 2022.
  * Munhoz and Castro [2022] Vanderlei Munhoz and Márcio Castro.  Hpc@ cloud: A provider-agnostic software framework for enabling hpc in public cloud platforms.  In _Anais do XXIII Simpósio em Sistemas Computacionais de Alto Desempenho_ , pages 157–168. SBC, 2022.
  * Jangjou and Sohrabi [2022] Mehrdad Jangjou and Mohammad Karim Sohrabi.  A comprehensive survey on security challenges in different network layers in cloud computing.  _Archives of Computational Methods in Engineering_ , 29(6):3587–3608, 2022.
  * Milroy et al. [2022] Daniel J Milroy, Claudia Misale, Giorgis Georgakoudis, Tonia Elengikal, Abhik Sarkar, Maurizio Drocco, Tapasya Patki, Jae-Seung Yeom, Carlos Eduardo Arango Gutierrez, Dong H Ahn, et al.  One Step Closer to Converged Computing: Achieving Scalability with Cloud-Native HPC.  In _2022 IEEE/ACM 4th International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)_ , pages 57–70. IEEE, 2022.
  * Misale et al. [2021a] Claudia Misale, Daniel J Milroy, Carlos Eduardo Arango Gutierrez, Maurizio Drocco, Stephen Herbein, Dong H Ahn, Zvonko Kaiser, and Yoonho Park.  Towards standard Kubernetes scheduling interfaces for converged computing.  In _Smoky Mountains Computational Sciences and Engineering Conference_ , pages 310–326. Springer, 2021a.
  * [24] Kubernetes components.  https://kubernetes.io/docs/concepts/overview/components/.  Accessed: 2023-9-24.
  * flu [a] Flux framework.  https://flux-framework.org/, a.  Accessed: 2023-9-24.
  * Honeypot [2022] Honeypot.  Kubernetes: The documentary [PART 1], January 2022.
  * Pike [2012] Rob Pike.  Go at google.  In _Proceedings of the 3rd annual conference on Systems, programming, and applications: software for humanity_ , pages 5–6, 2012.
  * kub [a] Kubernetes scheduler.  https://kubernetes.io/docs/concepts/scheduling-eviction/kube-scheduler/, a.  Accessed: 2023-9-24.
  * Ahn et al. [2020] Dong H. Ahn, Ned Bass, Albert Chu, Jim Garlick, Mark Grondona, Stephen Herbein, Helgi I. Ingólfsson, Joseph Koning, Tapasya Patki, Thomas R.W. Scogland, Becky Springmeyer, and Michela Taufer.  Flux: Overcoming scheduling challenges for exascale workflows.  _Future Generation Computer Systems_ , 110:202–213, 2020\.  ISSN 0167-739X.  doi: https://doi.org/10.1016/j.future.2020.04.006.  URL https://www.sciencedirect.com/science/article/pii/S0167739X19317169.
  * Misale et al. [2021b] Claudia Misale, Maurizio Drocco, Daniel J. Milroy, Carlos Eduardo Arango Gutierrez, Stephen Herbein, Dong H. Ahn, and Yoonho Park.  It’s a Scheduling Affair: GROMACS in the Cloud with the KubeFlux Scheduler.  In _2021 3rd International Workshop on Containers and New Orchestration Paradigms for Isolated Environments in HPC (CANOPIE-HPC)_ , pages 10–16, 2021b.  doi: 10.1109/CANOPIEHPC54579.2021.00006.
  * aws [a] AWS batch, a.  URL https://aws.amazon.com/batch/.
  * aws [b] AWS paralell cluster, b.  URL https://aws.amazon.com/hpc/parallelcluster/.
  * [33] Google batch.  URL https://cloud.google.com/batch/docs/get-started.
  * [34] Azure cyclecloud.  URL https://azuremarketplace.microsoft.com/en-us/marketplace/apps/azurecyclecloud.azure-cyclecloud?tab=Overview.
  * Sochat and Ojea [2023] Vanessa Sochat and Antonio Ojea.  flux-framework/flux-operator, September 2023.  URL https://doi.org/10.5281/zenodo.8312253.
  * Dobies and Wood [2020] Jason Dobies and Joshua Wood.  _Kubernetes operators: Automating the container orchestration platform_.  O’Reilly Media, 2020.
  * noa [b] Kubernetes components.  https://kubernetes.io/docs/concepts/overview/components/, b.  Accessed: 2023-9-1.
  * [38] A P I Changes.  The kubernetes API.  https://kubernetes.io/docs/concepts/overview/kubernetes-api/.  Accessed: 2023-9-1.
  * noa [c] Kubernetes overview.  https://kubernetes.io/docs/concepts/overview/, c.  Accessed: 2023-9-1.
  * noa [d] Storage.  https://kubernetes.io/docs/concepts/storage/, d.  Accessed: 2023-9-1.
  * noa [e] Services, load balancing, and networking.  https://kubernetes.io/docs/concepts/services-networking/, e.  Accessed: 2023-9-1.
  * noa [f] Workloads.  https://kubernetes.io/docs/concepts/workloads/, f.  Accessed: 2023-9-1.
  * noa [g] Containers.  https://kubernetes.io/docs/concepts/containers/, g.  Accessed: 2023-9-1.
  * noa [2022b] Kubernetes operators: what are they? some examples.  https://www.cncf.io/blog/2022/06/15/kubernetes-operators-what-are-they-some-examples/, June 2022b.  Accessed: 2023-9-1.
  * noa [h] Controllers.  https://kubernetes.io/docs/concepts/architecture/controller/, h.  Accessed: 2023-9-1.
  * [46] Custom resources.  https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/.  Accessed: 2023-9-1.
  * noa [i] Flux framework.  https://flux-framework.org/flux-operator/getting_started/custom-resource-definition.html, i.  Accessed: 2023-9-1.
  * Wikipedia contributors [2023b] Wikipedia contributors.  Declarative programming.  https://en.wikipedia.org/w/index.php?title=Declarative_programming&oldid=1174296946, September 2023b.  Accessed: NA-NA-NA.
  * [49] Paul Heney.  2021 R&D 100 award winners announced in Process/Prototyping and Software/Services categories.  https://www.rdworldonline.com/2021-rd-100-award-winners-announced-in-process-prototyping-and-software-services-categories/.  Accessed: 2023-9-1.
  * [50] Lawrence Livermore and Dong H Ahn.  Dong h. ahn.  https://ipo.llnl.gov/sites/default/files/2022-02/Flux_RD100_Final.pdf.  Accessed: 2023-9-28.
  * Trader [a] Tiffany Trader.  Flux supercomputing workload manager hits milestones in advance of supporting exascale science.  https://www.hpcwire.com/2022/03/02/flux-supercomputing-workload-manager-hits-milestones-exascale-science/, a.  Accessed: 2023-9-1.
  * noa [j] Flux framework.  https://flux-framework.readthedocs.io/en/latest/guides/learning_guide.html, j.  Accessed: 2023-9-1.
  * Trader [b] Tiffany Trader.  Flux supercomputing workload manager hits milestones in advance of supporting exascale science.  https://www.hpcwire.com/2022/03/02/flux-supercomputing-workload-manager-hits-milestones-exascale-science/, b.  Accessed: 2023-9-1.
  * [54] Portable hardware locality (hwloc): Version 2.0.  https://www.open-mpi.org/software/hwloc/v2.0/.  Accessed: 2023-9-24.
  * ibm [2023] IBM documentation - portable hardware locality.  https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=affinity-portable-hardware-locality, February 2023.  Accessed: 2023-9-24.
  * [56] Resource management for pods and containers.  https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/.  Accessed: 2023-9-1.
  * [57] Assigning pods to nodes.  https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/.  Accessed: 2023-9-1.
  * [58] Kubernetes Api Conventions and Spec Section.  Deployments.  https://kubernetes.io/docs/concepts/workloads/controllers/deployment/.  Accessed: 2023-9-1.
  * [59] StatefulSets.  https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/.  Accessed: 2023-9-1.
  * Section [a] Spec Section.  ReplicaSet.  https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/, a.  Accessed: 2023-9-1.
  * job [a] Adding of job to kubernetes, a.
  * ind [2021] Introducing indexed jobs.  https://kubernetes.io/blog/2021/04/19/introducing-indexed-jobs/, April 2021\.  Accessed: 2023-9-1.
  * Section [b] Spec Section.  Jobs.  https://kubernetes.io/docs/concepts/workloads/controllers/job/, b.  Accessed: 2023-9-1.
  * [64] Batched pod creation for indexed job.  URL https://github.com/kubernetes/kubernetes/blob/4619f7e9d9d64a0ae0b17270496ca3f7c67d24e8/pkg/controller/job/job_controller.go#L1557-L1564.
  * [65] flux-config-bootstrap(5) — flux-core documentation.  https://flux-framework.readthedocs.io/projects/flux-core/en/stable/man5/flux-config-bootstrap.html.  Accessed: 2023-9-1.
  * [66] Indexed job enhancement.  https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2214-indexed-job.  Accessed: 2023-9-1.
  * Hintjens [2013] P. Hintjens.  _ZeroMQ: Messaging for Many Applications_.  Oreilly and Associate Series. O’Reilly Media, Incorporated, 2013.  ISBN 9781449334062.  URL https://books.google.com/books?id=KWtT5CJc6FYC.
  * Wikipedia contributors [2023c] Wikipedia contributors.  Transmission control protocol.  https://en.wikipedia.org/w/index.php?title=Transmission_Control_Protocol&oldid=1172867047, August 2023c.  Accessed: NA-NA-NA.
  * [69] 0MQ api.  http://api.zeromq.org/4-2:zmq-setsockopt.  Accessed: 2023-9-24.
  * [70] Flux framework glossary.  https://flux-framework.readthedocs.io/en/latest/glossary.html.  Accessed: 2023-9-1.
  * [71] Kubernetes headless service.  https://kubernetes.io/docs/concepts/services-networking/service/.  Accessed: 2023-9-1.
  * Wikipedia contributors [2023d] Wikipedia contributors.  Message passing interface.  https://en.wikipedia.org/w/index.php?title=Message_Passing_Interface&oldid=1154658999, May 2023d.  Accessed: NA-NA-NA.
  * flu [b] 13/simple process manager interface v1 — flux 0.13.0 documentation.  https://flux-framework.readthedocs.io/projects/flux-rfc/en/latest/spec_13.html, b.  Accessed: 2023-9-1.
  * [74] CurveZMQ.  http://curvezmq.org/page:read-the-docs.  Accessed: 2023-9-1.
  * [75] Product Signing (gpg) Keys.  Chapter 10. managing services with systemd red hat enterprise linux 7\.  https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/chap-managing_services_with_systemd.  Accessed: 2023-9-24.
  * con [a] ConfigMaps.  https://kubernetes.io/docs/concepts/configuration/configmap/, a.  Accessed: 2023-9-1.
  * CNCF [Cloud Native Computing Foundation] [2023] CNCF [Cloud Native Computing Foundation].  Enabling HPC & ML workloads with the latest kubernetes job features- michał woźniak & vanessa sochat, May 2023.
  * [78] cgo command - cmd/cgo - go packages.  https://pkg.go.dev/cmd/cgo.  Accessed: 2023-9-1.
  * for [2022] Forensic container checkpointing in.  https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/, December 2022.  Accessed: 2023-9-24.
  * [80] Flux operator - saving state.  https://flux-framework.org/flux-operator/tutorials/state.html.  Accessed: 2023-9-1.
  * Thurgood and Lennon [2019] Brandon Thurgood and Ruth G. Lennon.  Cloud computing with kubernetes cluster elastic scaling.  In _Proceedings of the 3rd International Conference on Future Networks and Distributed Systems_ , ICFNDS ’19, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450371636.  doi: 10.1145/3341325.3341995.  URL https://doi.org/10.1145/3341325.3341995.
  * [82] Elastic indexed jobs.  URL https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/3715-elastic-indexed-job.
  * [83] Basic elasticity for the flux operator.  URL https://flux-framework.org/flux-operator/tutorials/elasticity.html#basic-example.
  * [84] Horizontal pod autoscaling.  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/.  Accessed: 2023-9-1.
  * [85] Horizontal pod autoscaling - algorithm.  https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#algorithm-details.  Accessed: 2023-9-1.
  * Vanessa Sochat [2023a] Vanessa Sochat.  Elasticity and the flux operator.  https://vsoch.github.io/2023/elasticity/, May 2023a.  Accessed: 2023-9-1.
  * Vanessa Sochat [2023b] Vanessa Sochat.  Writing a custom metrics API.  https://vsoch.github.io/2023/autoscaling-v2/, May 2023b.  Accessed: 2023-9-2.
  * flu [c] flux-metrics-api: An experiment to make a custom metrics API (for kubernetes) just for a flux instance, c.  URL https://github.com/converged-computing/flux-metrics-api.
  * [89] Multi-tenancy.  https://kubernetes.io/docs/concepts/security/multi-tenancy/.  Accessed: 2023-9-2.
  * [90] Susan Lauber.  An introduction to pluggable authentication modules (PAM) in linux.  https://www.redhat.com/sysadmin/pluggable-authentication-modules-pam.  Accessed: 2023-9-2.
  * flu [d] flux-restful-api: Flux RESTful API provided by FastAPI, d.  URL https://github.com/flux-framework/flux-restful-api.
  * Sochat and Nguyen [2023] Vanessa Sochat and Khoi Nguyen.  flux-framework/flux-restful-api, September 2023.  URL https://doi.org/10.5281/zenodo.8312273.
  * [93] NodePort service.  https://kubernetes.io/docs/concepts/services-networking/service/.  Accessed: 2023-9-2.
  * [94] Use port forwarding to access applications in a cluster.  https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/.  Accessed: 2023-9-2.
  * [95] Flux operator CRD user definition.  https://flux-framework.org/flux-operator/getting_started/custom-resource-definition.html#users.  Accessed: 2023-9-2.
  * Reschke [2015] Julian Reschke.  RFC 7617: The ’basic’ HTTP authentication scheme.  https://datatracker.ietf.org/doc/html/rfc7617, September 2015.  Accessed: 2023-9-2.
  * flu [e] flux-accounting: bank/accounting interface for the flux resource manager, e.  URL https://github.com/flux-framework/flux-accounting.
  * Drako [2022] Dean Drako.  The need for speed: From electric supercars to cloud bursting for design.  In _Proceedings of the 2022 International Symposium on Physical Design_ , pages 1–1, 2022.
  * [99] Terraform by HashiCorp.  https://www.terraform.io/.  Accessed: 2023-9-24.
  * flu [f] flux-burst: Remote bursting for flux using a plugin framework (prototype in python), f.  URL https://github.com/converged-computing/flux-burst.
  * Mölder et al. [2021] Felix Mölder, Kim Philipp Jablonski, Brice Letcher, Michael B Hall, Christopher H Tomkins-Tinch, Vanessa Sochat, Jan Forster, Soohyun Lee, Sven O Twardziok, Alexander Kanitz, Andreas Wilm, Manuel Holtgrewe, Sven Rahmann, Sven Nahnsen, and Johannes Köster.  Sustainable data analysis with snakemake.  _F1000Res._ , 10:33, January 2021.
  * Di Tommaso et al. [2017] Paolo Di Tommaso, Maria Chatzou, Evan W Floden, Pablo Prieto Barja, Emilio Palumbo, and Cedric Notredame.  Nextflow enables reproducible computational workflows.  _Nat. Biotechnol._ , 35(4):316–319, April 2017\.
  * Colonnelli et al. [2021] Iacopo Colonnelli, Barbara Cantalupo, Ivan Merelli, and Marco Aldinucci.  StreamFlow: Cross-Breeding cloud with HPC.  _IEEE Transactions on Emerging Topics in Computing_ , 9(4):1723–1737, October 2021.
  * kue [2022] Run a flux MiniCluster.  https://kueue.sigs.k8s.io/docs/tasks/run_flux_minicluster/, February 2022.  Accessed: 2023-9-2.
  * [105] snakemake-executor-kueue: A snakemake executor plugin to enable running jobs with kueue on kubernetes.
  * [106] Erik Jacobs.  How to use kubeflow and the MPI operator on OpenShift.  https://cloud.redhat.com/blog/how-to-use-kubeflow-and-the-mpi-operator-on-openshift.  Accessed: 2023-9-2.
  * Biliaiev [2021] Mykola Biliaiev.  Enabling openmpi workloads on bare-metal infrastructure using kubernetes.  2021\.
  * mpi [1] MPI training (MPIJob).  https://www.kubeflow.org/docs/components/training/mpi/, January 1\.  Accessed: 2023-9-2.
  * mpi [a] getOrCreateSSHAuthSecret code, a.  URL https://github.com/kubeflow/mpi-operator/blob/6bce22d1ab3a42e7ea52ef573afbc662a49c18a5/pkg/controller/mpi_job_controller.go#L875.
  * [110] Amazon web services EKS.  URL https://aws.amazon.com/eks/.
  * [111] Amazon EC2 hpc6a instances.  URL https://aws.amazon.com/ec2/instance-types/hpc6a/.
  * [112] Elastic fiber adapter on AWS.  URL https://aws.amazon.com/hpc/efa/.
  * Thompson et al. [2022] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in ’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton.  LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.  _Comp. Phys. Comm._ , 271:108171, 2022.  doi: 10.1016/j.cpc.2021.108171.
  * [114] Amazon web services EKS.  URL https://asc.llnl.gov/coral-2-benchmarks.
  * van Duin et al. [2001] Adri C T van Duin, Siddharth Dasgupta, Francois Lorant, and William A Goddard.  ReaxFF: A reactive force field for hydrocarbons.  _J. Phys. Chem. A_ , 105(41):9396–9409, October 2001.
  * [116] Images.  https://kubernetes.io/docs/concepts/containers/images/.  Accessed: 2023-9-2.
  * Sochat [2023] Vanessa Sochat.  converged-computing/flux-cloud: Flux Cloud Release v0.2.0.1, September 2023.  URL https://doi.org/10.5281/zenodo.8311879.
  * flu [g] Flux operator container for kubecon experiments, g.
  * mpi [b] MPI operator optimized container, b.  URL https://github.com/converged-computing/operator-experiments/pkgs/container/mpi-operator-experiment/124341626?tag=feb-2023.
  * kub [b] Kubecon 2023 flux operator experiments, b.  URL https://github.com/converged-computing/operator-experiments/tree/main/aws/lammps/kubecon/run6.
  * con [b] Flux operator container requirements.  https://flux-framework.org/flux-operator/development/developer-guide.html?h=container#container-requirements, b.  Accessed: 2023-9-2.
  * job [b] jobset: JobSet: An API for managing a group of jobs as a unit, b.  URL https://github.com/kubernetes-sigs/jobset.
  * Merow et al. [2023] Cory Merow, Brad Boyle, Brian J Enquist, Xiao Feng, Jamie M Kass, Brian S Maitner, Brian McGill, Hannah Owens, Daniel S Park, Andrea Paz, Gonzalo E Pinilla-Buitrago, Mark C Urban, Sara Varela, and Adam M Wilson.  Better incentives are needed to reward academic software development.  _Nat Ecol Evol_ , 7(5):626–627, May 2023.
  * Shen et al. [2023] Yiqiu Shen, Laura Heacock, Jonathan Elias, Keith D Hentel, Beatriu Reig, George Shih, and Linda Moy.  ChatGPT and other large language models are double-edged swords.  _Radiology_ , 307(2):e230163, April 2023.
  * Larsson et al. [2020] Lars Larsson, William Tärneberg, Cristian Klein, Erik Elmroth, and Maria Kihl.  Impact of etcd deployment on kubernetes, istio, and application performance.  _Softw. Pract. Exp._ , 50(10):1986–2007, October 2020.
  * Di Natale et al. [2019] Francesco Di Natale, Harsh Bhatia, Timothy S Carpenter, Chris Neale, Sara Kokkila-Schumacher, Tomas Oppelstrup, Liam Stanton, Xiaohua Zhang, Shiv Sundram, Thomas R W Scogland, Gautham Dharuman, Michael P Surh, Yue Yang, Claudia Misale, Lars Schneidenbach, Carlos Costa, Changhoan Kim, Bruce D’Amora, Sandrasegaram Gnanakaran, Dwight V Nissley, Fred Streitz, Felice C Lightstone, Peer-Timo Bremer, James N Glosli, and Helgi I Ingólfsson.  A massively parallel infrastructure for adaptive multiscale simulations: modeling RAS initiation pathway for cancer.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , number Article 57 in SC ’19, pages 1–16, New York, NY, USA, November 2019. Association for Computing Machinery.