Type Universes as Allocation Effects

Paulette Koronkevich
William J. Bowman

University of British Columbia

In this paper, we explore a connection between type universes and memory
Type universe hierarchies are used in dependent type theories to ensure
consistency, by forbidding a type from quantifying over all types.
Instead, the types of types (universes) form a hierarchy, and a type can only
quantify over types in other universes (with some exceptions), restricting
cyclic reasoning in proofs.
We present a perspective where universes also describe where values are
allocated in the heap, and the choice of universe algebra imposes a structure
on the heap overall.
The resulting type system provides a simple declarative system for reasoning
about and restricting memory allocation, without reasoning about reads or writes.
We present a theoretical framework for equipping a type system with
higher-order references restricted by a universe hierarchy, and conjecture that
many existing universe algebras give rise to interesting systems for
reasoning about allocation.
We present 3 instantiations of this approach to enable reasoning about
allocation in the simply typed $\lambda$-calculus: (1) the
standard ramified universe hierarchy, which we prove guarantees termination of
the language extended with higher-order references by restricting cycles in the
heap; (2) an extension with an impredicative base universe, which we
conjecture enables full-ground references (with terminating computation but
cyclic ground data structures); (3) an extension with universe
polymorphism, which divides the heap into fine-grained regions.
(This is a fresh perspectives submission.)

§ INTRODUCTION

Many type systems have been designed to reason about memory.
But what does it mean to reason about memory?

A large amount of work is dedicated to safety, ensuring updates,
aliasing, etc., do not cause bugs when reading and writing from memory.
For example, L3 [1] has a type system that enforces safety with strong updates,
updates that may change the type of already allocated locations on the heap.
L3's type system focuses on the safety of reads in the presence of these strong
updates, and thus focuses solely on the problem of aliasing.
Rust [12] implements a similar system, enabling reasoning about
aliasing in the presence of concurrency.

Some type systems enable reasoning about deallocation.
Region type-and-effect systems [19] were originally designed
for static fine-grained memory management, and Rust follows in this tradition.

Much of this work reasons about memory indirectly, from how references to memory are used.
In Rust, allocation and deallocation is inferred from usage—when a value is
created, or copied, it is allocated, and it is deallocated when the value is no longer owned.
Regions are also inferred through individual reference reads and writes.
Any heap structure, such as the stack of regions structure [19] or
stratified regions [4, 3], is also inferred through reads
and writes.
L3 turns out to be terminating, which is enforced indirectly because of the
linearity usage requirement on references, but not because of any direct
restriction on structure of the heap.

We present a theoretical framework for designing type systems to enable
directly reasoning about allocation, by giving a declarative
description of allocation.
By declarative, we mean that each type declares where values of this
type are allocated on the heap, and parameters of the type system
declare what heap shapes and dependencies are allowed, in contrast to
these other type systems where allocation is inferred from usage of heap
For the time being, we do not consider reasoning about aliasing, although we
conjecture that existing designs that track aliasing could be integrated into
our type systems.

Our framework is based on a type universe hierarchy, where the
underlying hierarchy can be changed to result in different heap structures.
Type universes are used to eliminate inconsistencies in dependent type
theories, which can arise when the type of types (the universe Type>),
has type Type>.
This, and similar patterns, can lead to inconsistencies by admitting cyclic
A standard solution to this is to stratify universes, each one a member of the next.
Instead of considering the Type> to be in universe Type>, dependent type
theories use a universe hierarchy to assign a level to each universe, so
$\<Type>_i$ is in universe $\<Type>_{i+1}$, ensuring consistency.
In our framework, each type has a universe, but the universe describes where
values are allocated, and the universe hierarchy enforces a heap structure for
all programs.
By changing the universe hierarchy, or level algebra, we impose different heap structures.

We first present an instantiation of our framework with a standard predicative
universe hierarchy, which enforces a stratified, acyclic heap.
Because the heap is acyclic, we prove this language with higher-order references is
The language is simplified compared to similar work using stratified regions via a
type-and-effect system [4, 3].
Our language does not require tracking of reads or writes of individual
references, nor region inference.
We also present the proof of termination, which relies on a semantic notion of
garbage collection based on universe level.
We conjecture that the semantics might be lead to a declarative syntax for

We then abstract this type system to present the main parameters that change
the underlying universe algebra to create different heap structures.
We give two instantiations of this framework with existing type universe
hierarchies, resulting in type systems that enforce different heap structures.
In particular, we design a language with a heap with one level where cycles can
occur, based on universe hierarchies with a single impredicative base
We conjecture the language is still terminating and yields
full-ground references [15].
We also explore an extension with universe polymorphism, which allows
a user to name regions of the heap and results in a system with more
fine-grained regions and conjecture could be used for static memory

In short, we present a fresh perspective on viewing type universes as a
system for simple, direct, declarative reasoning about
allocation—when and where allocation occurs, and which allocations a
computation depends on.

§ TYPE UNIVERSES FOR ACYCLIC HEAPS

We now study how cycles in the heap can be created with
higher-order references through a pattern called
Landin's Knot [10].
We show how to eliminate such cycles in a typed language equipped
with a universe type hierarchy.
In the type system we present, Landin’s Knot is ruled out in a purely
declarative way, without any reference usage tracking.

Preventing cycles in the heap is useful to achieve various language design and
implementation goals.
Reference counting, used to implement memory management, cannot handle (strong)
cycles in the heap [13].
Certain cycles through the heap can lead to unrestricted recursion, which one
might instead desire to control through only explicit recursive constructs.
For example, dependent type theories require strong normalization for
decidability of type checking, shown for example by Jutting [9],
and termination might be desired to ensure for fairness in concurrent
settings [4].

Unfortunately, the common wisdom is that higher-order references introduce
cycles in the heap that are difficult to prevent.
Landin's Knot, a pattern that uses a function capturing a mutable reference
and updates that reference to contain the function itself, is a typical example
of these kinds of cycles.
This pattern is illustrated through the following diverging program.

id = (lam> x . x)     : Nat> →Nat>

r = new> id     : Ref> (Nat> →Nat>)

f = (lam> x . (deref> r)  x)     : Nat> →Nat>

r := f;

f  0

Three expressions $id$, $r$, and $f$ are defined in this program.
The expression $id$ is the identity function for natural numbers, and
$r$ is a reference initialized with $id$.
The second function $f$ captures the reference $r$, then expects a natural
number $x$ and and applies the function stored in $r$ to $x$.
The program diverges because it updates the reference $r$ to store the function
$f$, enabling unrestricted recursion in $f$.
The function $f$ calls the function stored in $r$, which is now $f$ itself, and
the program diverges.

We visualize the heap and dependencies on mutable references to better illustrate
the cause of this cycle.
In the following diagram, the reference $r$ initially points to a memory
location storing the identity function, and the function $f$ depends on $r$.
A reference pointing a memory location uses the arrow $|->$,
with the location and contents denoted by a box.
Code dependent on a reference uses the arrow $->$ to distinguish from pointing
to memory locations.

\[
\begin{tikzpicture}[
squarednode/.style={rectangle, draw=black, very thick, minimum size=5mm},
\node[]                 (ref)                       {$r$};
\node[]                 (fun)        [left=of ref]  {$\<lam> x . (\textbf{!}~ r) \ x$};
\node[squarednode]      (cell)       [right=of ref] {$\<lam> x . x$};

\draw[->]  (fun.east) -- (ref.west);
\draw[|->] (ref.east) -- (cell.west);
\end{tikzpicture}
\]

After updating $r$ to $f$, the problematic cycle comes from the combination of the
dependency arrow for $f$ on $r$ and $r$ pointing to $f$.

\[
\begin{tikzpicture}[
squarednode/.style={rectangle, draw=black, very thick, minimum size=5mm},
\node[]                 (ref)                       {$r$};
\node[squarednode]      (cell)       [right=of ref] {$\lambda x . (\textbf{!}~ r) \ x$};

\draw[|->] (ref.east) -- (cell.west);
\draw[->]  (cell.north) to [out=150,in=30] (ref.north);
\end{tikzpicture}
\]

To prevent such cycles, we need to prevent $r$ from containing functions that
depend on $r$.
The first diagram had no cycles, and updates to $r$ with other functions that are not
like $f$ has no such cycles: for example, updating $r$ to contain the
function $\lambda x . 0$.

\[
\begin{tikzpicture}[
squarednode/.style={rectangle, draw=black, very thick, minimum size=5mm},
\node[]                 (ref)                       {$r$};
\node[]                 (fun)        [left=of ref]  {$\lambda x . (\textbf{!}~ r) \ x$};
\node[squarednode]      (cell)       [right=of ref] {$\lambda x . 0$};

\draw[->]  (fun.east) -- (ref.west);
\draw[|->] (ref.east) -- (cell.west);
\end{tikzpicture}
\]

To use a type universe hierarchy for preventing cycles,
we consider $r$ and functions with any dependencies on $r$ to be in the
same universe.
The values that $r$ can store will be at a separate universe, so that
$r$ cannot be updated with any value from $r$'s universe,
preventing cyclic dependencies.
A type universe hierarchy similarly prevents circular reasoning in proofs encoded in
dependent type theory, where $\<Type>_i$ is considered to be of type $\<Type>_{i+1}$,
and not of type $\<Type>_i$.

To prevent other potential cycles, we also have to prevent $r$ from pointing
to any expressions in a higher universe than $r$ since such expressions
could depend on $r$.
This additional restriction imposes an acyclic heap by stratifying the heap such that
references can only point “down” in the heap.

This diagram may cause déjà vu for those familiar with models of languages with
mutable references, where a similar diagram is also illustrated by
Ahmed [2].
The stratification introduced in Ahmed's work is used in the model to break the
type-world circularity, which we discuss in more detail in <Ref>.
We reflect this stratification into the syntax with a type universe hierarchy
for a stratified (and thus acyclic) heap.
Since functions are also allocated on the heap, our diagram here also includes
functions and their dependencies on the heap.

The key kinding rule in our type universe hierarchy for a stratified heap is
the rule for determining the universe level of reference types.
To require references to point “downward” in the heap, the universe level of a
reference type must be one level higher than the type it stores.
Here we use $::$ to indicate kinding a type.

τ:: Type>_i
τ :: Type>_i+1

Given a type $\tau$ in universe $\<Type>_i$, then the type
$\refty{\tau}$ is in universe $\<Type>_{i+1}$.
For example, given $id$'s type is in universe $\<Type>_0$,
then $r$'s type is in universe $\<Type>_1$.

To complete the design of these stratified higher-order references,
we also consider how to determine the universe level of a function type,
since functions are also allocated on the heap.
The typical predicative function kinding rule in dependent type theories is
essentially as follows, where a function's universe level is the maximum of the
universe levels of its input and output types.

τ_1 :: Type>_i

τ_2 :: Type>_j

k ≥i, j
τ_1 →τ_2  :: Type>_k

However, since functions are allocated as closures with code and environment,
the universe information of the environment is lost with this kinding rule.
Without considering a function's environment, the functions $id$ and $f$ are
in the same universe.
But having $id$ and $f$ in the same universe violates the stratified heap
invariant, as $f$'s closure contains $r$.
The two functions $id$ and $f$ differ as closures, since $id$ requires no
environment, but the function $f$'s environment depends on the reference $r$.
This means that the function $id$ and $f$, despite having the same input and
output types, will be in different universes based on their environment.
The universe level of a function type is not only influenced by the
input and output types, but also the types captured in its environment!

To implement this, we include an annotation on the function arrow.
When viewing a function type in isolation, the only information available is the
universes of the input and output types, and not the (maximum) universe of the
We check that the universe annotation is consistent with the universe of the
input and output types.
We also separately check the annotation against the environment when the
type is used to check a term.

τ_1 :: Type>_i

τ_2 :: Type>_j

k ≥i, j
τ_1τ_2k  :: Type>_k

The types $\funty{\<Nat>}{\<Nat>}{1}$ and $\funty{\<Nat>}{\<Nat>}{0}$ are both
well kinded, but describe different kinds of functions: the former are
permitted to capture references in universe $\<Type>_1$, while the latter cannot
capture references at all.
In our example, $f$'s type would be $\funty{\<Nat>}{\<Nat>}{1}$ since it captures
the universe $\<Type>_1$ reference $r$.
Then $r$ cannot be updated to $f$, since $f$ is in the same universe $\<Type>_1$,
preventing the cycle.

We now present the full language , where PR stands for
predicative references, inspired by predicative universe hierarchies.
The type system is essentially standard, with a type universe hierarchy to enforce
the heap stratification.
The type system is syntax directed, as is the kinding system, creating a simple,
declarative system for higher-order references with termination without complicated
resource tracking.

In <Ref>, we present the syntax of ,
a simply typed $\lambda$-calculus with higher-order references.
The language includes base types, $\<Nat>$ and $\<Unit>$, denoted by the
metavariable $n$ and expression $\unite$ respectively.
The rest of the syntax is standard, with $\<nsnew>$, $\<nsderef>$, and $\coloneq$ for
initializing, dereferencing, and updating references.
The only exception is the annotation on the function type $\funty{\tau}{\tau}{k}$,
where $k$ indicates the universe of the function type and can be read as where
to allocate a function of this type on the heap.

In <Ref>, we present the typing rules of .
The rules are standard, except for the function case.
Here we check (or could infer) the annotation on the function type against the
current context, $k \geq \<max>(\Gamma,\tau_1,\tau_2)$.
This side condition requires that the level $k$ of the function type is greater
than or equal to the levels of variables in $\Gamma$ and the levels of
types $\tau_1$ and $\tau_2$.
Note that the context $\Gamma$ could
contain more variables (and thus universe levels) than the function actually captures,
but through weakening the context one can easily determine the smallest level of a
function type by only the variables captured.

The function type annotation relies on kinding the input and output types and
the types in $\Gamma$ to determine the maximum universe level.
We present the kinding of types in <Ref>.
The simple base types $\<Nat>$ and $\<Unit>$ are of $\<Type>_0$ as expected.
The universe level of a function is determined by the annotation, given that this
annotation is greater than or equal to the input and output type universe levels.
And finally, a reference type is one level higher than the level of the
type it stores.

In , our example of Landin's Knot is not well typed, which
we demonstrate through type and kind annotations rather than a large derivation tree.

id = (lam> x . x)      : Nat>Nat>0

r = new> id     : Ref> (Nat>Nat>0) :: Type>_1

f = (lam> x . (deref> r)  x)     : Nat>Nat>1 :: Type>_1     $\<Type>_1$ due to $r$

r := f     type error, expected: $\funty{\<Nat>}{\<Nat>}{0}$     actual: $\funty{\<Nat>}{\<Nat>}{1}$

§.§ Proof of Termination

The proof of termination for uses the standard logical relations
technique, an introduction of which can be found in standard textbooks [5].
Our relation models types as sets of normalizing expressions.
We prove all well-typed expressions in are in the set associated
with their type in <Ref>.
To be in the set associated with its type, an expression must step to a value, which
allows us to conclude that all well typed expressions step to a value, terminate.

To model a language with mutable references, one has to model the heap, since
expressions access and modify the heap.
The model of a heap is often referred to as a world, and is modelled as a
finite map from locations to sets of values modelling types.
Since each location is mapped to a set of all possible values, the world
represents any possible concrete execution heap.

However, we run into a circularity in reasoning when modelling heaps as worlds, known
as the type-world circularity.

Type     =     World →Set of Terms

World     =     Loc ⇀Type

To model a type, we take in a world and produce a set of terms representing
that type.
The world maps locations to types and must also take in a world,
resulting in the following equation with an inconsistent cardinality, that is, the
set must contain itself: $\textit{World} = \textit{Loc} \rightharpoonup (\textit{World} \rightarrow \textit{Set of Terms})$.

Step-indexed logical relations were introduced by Ahmed [2]
to model languages with mutable references, where the type-world circularity is
eliminated using a decreasing number $k$.
Using step-indexing, the equations become:

Type_k+1     =     World_k →Set of Terms

World_k     =     Loc ⇀Type_k

Unrolling $\textit{World}_k$, we have the following consistent equation
$\textit{World}_{k} = \textit{Loc} \rightharpoonup (\textit{World}_{k-1} \rightarrow \textit{Set of Terms}$.
In previous work by Ahmed, the decreasing metric $k$ is determined by
the number of steps of reduction that an expression takes.
Given an expression $e \in \textit{Type}_{k+1}(W_{k})$, the expression $e$ steps
“safely”, without type errors, for $k+1$ steps.
Ahmed hints that there is a way to stratify types based on syntax, but chooses this
alternative semantic approximation in order to model arbitrarily quantified types.

We take the opposite approach to Ahmed and stratify the types based on syntax.
The decreasing metric in the type-world equations is instead based on the
universe level of a type.
The resulting equations are also slightly different, where the universe level matches
the world level, and a world level is increased by one based on the universe level of
the mapped types.

Type_i     =     World_i →Set of Terms

World_i+1     =     Loc ⇀Type_i

Unrolling $\textit{World}_{i+1}$, we have the following consistent equation
$\textit{World}_{i+1} = \textit{Loc} \rightharpoonup (\textit{World}_{i} \rightarrow \textit{Set of Terms})$.
These equations correspond closely to our kinding rule for references,
where reference type $\refty{\tau}$ is of $\<Type>_{i+1}$, given $\tau$ is $\<Type>_i$.
The type $\<Type>_{i+1}$ has access to $\textit{World}_{i+1}$, that is, locations mapping to
$\<Type>_i$, just as the reference at $\<Type>_{i+1}$ has access to $\tau$
at $\<Type>_i$.
One may also notice the absence of $\textit{World}_0$, where locations would be mapped to
$\<Type>_{-1}$, and this corresponds to the idea that values at $\<Type>_0$ do not
need access to the heap.
Another interpretation is that these values are valid in any world, because they
do not access or update the heap, and thus any world at any level
can stand in for $\textit{World}_0$.

Since the decreasing metric is included in the syntax, the resulting
logical relation reads as a simpler version of a step-indexed logical relation,
since each expression no longer needs to be paired with a step index $k$.
In <Ref>, we present the value relation for reference and function types.
The resulting sets are simply sets of values.
The value relation indexes the world at the same level as the
universe, avoiding the type-world circularity and ensuring that values are
not accessing locations beyond their world level.

The set of values for a reference type $\refty{\tau}$ is indexed by a number $i+1$,
since the universe level of $\refty{\tau}$ is always $i + 1$ for $\tau$'s level $i$.
The world associated with the type $\refty{\tau}$ is also at level $i + 1$.
Then, the set of values for type $\refty{\tau}$ is all the locations in the current
world that map to the set of values associated with $\tau$,$\mathcal{V}|[ \tau |]_{i}$.
The level of worlds associated with such values is necessarily lower than that
of the current world $W_{i+1}$, since the set is indexed by $i$.

The set of values for a function type $\funty{\tau}{\sigma}{k}$ relies on
additional definitions presented in <Ref>.
The first definition is world extension $\sqsupseteq_k$, which
describes a future world $W'_k$ with respect to the world $W_k$.
World extension is necessary because a function can be applied later in a
future heap, and so the model must include function values valid in
future worlds.
The relation guarantees that $W_k'$ has as many
locations as $W_k$ at the same types, but may have additional locations allocated.

The next definition lowers a world $W_k$ to level $i$.
Lowering a world is necessary since the values in the relation
$\mathcal{V}|[ \tau |]_{i}$ rely on a world indexed at the same level $i$.
However, the available world $W'_k$ is at level $k$, which is potentially
incompatible with level $i$.
We use the lower operation to remove all locations that map to types with levels
higher than level $i-1$.

The resulting world $\lw{W_k'}{i}$ is a $\textit{World}_i$ since
there are no locations mapping to types with level greater than $i-1$.
This lower operation can be considered a form of semantic garbage collection.

The final definition lifts a world $W_k$ to a world that is
an intersection of worlds at all possible levels, denoted as
$\mathbf{W} = \forall i . \textit{World}_i$.
Such a world $\mathbf{W}$ is used in the expression relation, since an expression
may allocate and access levels higher than its universe level $i$.
The lift operation can be considered a “cast” from a world with types at
universe level $k-1$ allocated to a world that can allocate at higher levels.
The current allocated locations in the resulting world are the same as
$W_k$, and any other locations at higher levels than $k$ have yet to be
allocated, hence denoted as $\emptyset$.

The value relation is defined over a fully annotated function type, where the
universe levels of both the input type $\tau$ and output type $\sigma$ are known.
Kinding types is easy as shown in <Ref>, and the levels are
necessary for indexing the relations for $\tau$ and $\sigma$ correctly.
The set contains functions that, given any value $v$ in the value relation for the
input type $\tau$, the body of the function $e$ with the parameter $x$ substituted
with $v$ is in the expression relation for $\sigma$.

The expression relation defined in <Ref> describes the set of terminating
expressions associated with a type $\tau$ at universe level $i$.
At a high level, the set consists of expressions that step to a value $v$ that is
in the value relation for $\tau$ at level $i$, that is $\mathcal{V}|[ \tau |]_{i}$.
The expression relation is defined over a world $\mathbf{W}$ at any level because
during evaluation, an expression may allocate and access levels higher than its
universe level $i$, as long as the final value does not depend on them.

There are three worlds $\mathbf{W}$, $\mathbf{W'}$, and $\mathbf{W''}$
in the expression relation.
The world $\mathbf{W}$ is considered the model of the minimum or initial
concrete heap needed for evaluation.
However, expressions can be evaluated in heaps larger than the initial heap, which
is why the expression relation includes the future world $\mathbf{W'}$.
Then, evaluation is under a concrete heap $h'$ realizing world $\mathbf{W'}$,
denoted by $\hwt{h'}{\mathbf{W'}}$.
The concrete heap $h'$ contains the same locations as $\mathbf{W'}$,
but maps each location to a single value from the value relation mapped
by $\mathbf{W'}$.
There is a final concrete heap at the end of evaluation $h''$, and there must exist
a future world $\mathbf{W''} \sqsupseteq \mathbf{W'}$ related
to $h''$.
Finally, the value $v$ resulting from evaluation is in the value relation for
type $\tau$, with the final world $\mathbf{W''}$ lowered to $i$ since $v$ does not
rely on any part of the heap higher than $i$.
Lowering also maintains the stratification invariant since values such as
functions will be guaranteed not to access the heap at levels greater than $i$,
and thus can be “deallocated”.

We prove terminating by proving the fundamental lemma, which relies on a
substitution $\gamma$ respecting the context $\Gamma$, as defined in <Ref>.

If  $\tyjudg{\Gamma}{e}{\tau}$ and $\tau :: \<Type>_i$, then
$\forall \gamma \in \mathcal{G}|[ \Gamma |](\mathbf{W}) . \gamma(e) \in \mathcal{E}|[ \tau |]_i(\mathbf{W})$.

This theorem states that if an expression $e$ is well typed at type $\tau$ with
universe level $\<Type>_i$, then $e$ is in the expression relation for $\tau$, $e$ steps to a value $v$ at type $\tau$.
This theorem also applies to open terms by using a substitution $\gamma$.

The key takeaways from this proof is that a syntactic hierarchy of types is able to
resolve the type-world circularity by ensuring that there are no cycles in the heap,
and such a language is terminating.
We need a notion of garbage collection with the lower operation $\lw{-}{i}$,
which potentially allows for a language design where either the garbage collection
is explicit in the syntax, or can be inferred by the type system similar to
region type-and-effect systems.

§.§ Interpretations and Our Framework

We conclude our presentation of with some intuitive
interpretations of the type system, although these connections are not yet
We also present our general framework for instantiating the type system with
alternative type universe hierarchies, and present additional instantiations in
subsequent sections.

One interpretation of is as a region type-and-effect system, where each
universe level is considered as one very large region.
All pure values are in region (level) 0, references to level 0 and functions that
close over references are in region (level) 1, and so on.
Instead of separate typing and kinding judgments, one could perhaps combine them
into one type-and-effect judgment as $\Gamma \vdash e : (\tau , \textbf{region } i)$,
where the effect $\textbf{region } i$ describes allocation in region $i$ and is
determined by kinding $\tau$.
Additionally, if a program is made up of values that do not exceed
region $n$ (level $n$), then any regions above $n$ can be safely deallocated.

The universe levels can also be interpreted as an allocation effect.
Values of types in universe $\<Type>_0$ are allocated at region 0 of the heap.
Values of types in universe $\<Type>_1$ are allocated in region 1 of the heap, where
values have access to region 0, but not any regions above, and so on
for each additional region.
Simply based on the type, one knows where a value of a certain type
will be allocated on the heap.
The structure of the heap is maintained for all programs because values are
allocated in their appropriate regions, as enforced by the type hierarchy.

Where a function is allocated is dependent on what parts of the heap the function
relies on.
Interestingly, a function may allocate and update new references and still
be considered at $\<Type>_0$, allocated in region 0, for example:

ex :  Nat>Nat>0

ex = (lam> x . r = new> 3 ; !r  +  x)

One can consider the reference $r$ as completely “private” to the function $ex$,
and $r$ does not affect the existing structure of the heap nor does $r$ ever leave
the function's scope.
Under this effect system, functions that do not leak effectful values can be considered
“pure” in the sense they do not depend on any part of the heap and thus can be used in
any arbitrary heap.
Furthermore, the allocations performed inside $ex$ are essentially invisible after
the computation ends, and can freely be collected or optimized away.

There may also be an interpretation of as a coeffect system [16],
since the function typing rule is dependent on the context.
In coeffect systems, a function type is annotated with an effect derived from
the current context, and variables in the context are annotated with their effects.
A possible coeffect judgment for function typing is as follows,
assuming $\tau$ has level $i$.

Γ, x :^i τ⊢e : σ
k = max-effect(Γ)
Γ⊢xτe : τσk

For our more general framework, we abstract the design of
's type system into a few “knobs” that can be adjusted to create different
The abstraction leads to two parameters that can be adjusted between designs.

* The highlighted side condition for typing a function, in this equation is $k \geq \<max>(\Gamma,\tau_1,\tau_2)$.

* The relationship between the universe level of a reference type and the type it stores. In , a reference universe level was always one level higher.

What these equations create is essentially the desired algebra
of the universe levels, the values of the levels and operations over them.
In , the algebra consists of natural numbers, a $\geq$ operator,
and a successor operator for the level of reference types.

We now explore how to adjust these parameters for alternative designs.

§ A UNIVERSE HIERARCHY WITH IMPREDICATIVITY

One type universe hierarchy already in use in dependent type theories
allows one universe level to be impredicative, and the rest of the levels are
predicative [11].
The impredicative level allows propositions to quantify over their own level, propositions can be sound with cyclic definitions, with some limitations.
We conjecture similar reasoning can be used to distinguish between “sound” and
“unsound” cycles in the memory heap.
Many data structures are encoded using cycles, so we believe there is a
correspondence between “sound” cycles for data and “unsound” cycles
resulting in nontermination.

We conjecture that full-ground references [15], references
to base data and other full-ground references, can be encoded using an
impredicative universe level.
The idea is that references containing base types will remain at the same level as
the base universe $\<Type>_0$.
Functions are not included as base types for full-ground references, and will be
at universe $\<Type>_1$ and above.
When a reference stores a function, the reference is no longer full-ground, and
is subject to the stratified (predicative) part of the universe hierarchy.

Adding an impredicative universe level is easily done by tweaking our two parameters.
For the function case, the equation is changed from $\geq$ to $>$, resulting in a
system where pure functions and functions that close over full-ground references
are at $\<Type>_1$.
Functions that close over other higher-order references continue up the
universe hierarchy similarly to .

$k > \<max>(\Gamma,\tau_1,\tau_2)$

Without this change to $>$, Landin's Knot would be well typed, since both the function
$id$ and the function that closes over a reference $f$ would be in the same universe.

For reference types, the universe of a reference type now depends on the level
of what it stores.
When storing a type at base universe $\<Type>_0$, the reference universe level remains
the same.
For any types above level 0, the reference universe level is one higher.

τ:: Type>_0
τ :: Type>_0
τ:: Type>_i
i ≠0
τ :: Type>_i+1

The resulting type system has full-ground references with sound cycles at
$\<Type>_0$, but unsound cycles that result in non-termination are still prevented
by the stratification imposed by predicativity.
We conjecture that this system is still terminating and has the ability to create
cyclic data.

The resulting system will likely have a stratified structured heap as seen in ,
with cycles occurring in level 0 of the heap.

Here we have a $\<Type>_0$ level of the heap where base data can be
allocated along with references to base data, and thus can encode a cyclic list.

§ TYPE UNIVERSE LEVEL POLYMORPHISM

Abstracting explicit levels has been the subject of many research papers on
the design of type universe hierarchies.
Universe polymorphism allows expressions to be polymorphic with respect to
universe levels, to eliminate the redundancy of defining the same expressions at
different universe levels.
Current implementations of dependent type theories use three main techniques
for universe polymorphism: explicit quantification for universe levels
as done in Agda [18],
user constraints as done in Coq [17], and
constraint solving based on typical ambiguity originally developed by
Huet [8] and Harper and Pollack [6] and also
implemented in Coq.
Recent work by Hou et al. [7] influenced by
McBride's crude-but-effective stratification [14] has
found that an alternative system with an explicit displacement operator is
a simple and effective mechanism for universe polymorphism.

We conjecture that universe polymorphism allows us to “name” regions of the heap,
resulting in a type system similar to region type-and-effect systems.
To implement McBride's crude-but-effective stratification version of universe
polymorphism, our language needs an explicit displacement operator on both expressions
and types.
Then, a value typically in one “region” of the heap can be lifted to another
region using the displacement operator.

Most base values exist in region 0 (in universe $\<Type>_0$), and having
such a large region that can never be deallocated is poor for fine-grained memory
We then use the displacement operator to distinguish values into different
For example, given a term $e$ of type $\tau$, suppose we would like to allocate
$e$ not in region 0 but in some region we call $\beta$.
We use the explicit displacement operator $\Uparrow_\beta$ to do this,
an operator also defined over types that updates the kind level accordingly.

Γ⊢e : τ
Γ⊢⇑_βe : ⇑_βτ
τ:: Type>_α
⇑_βτ:: Type>_α+β

Without adjusting the equations , we can now allocate
values in different levels of the stratified heap.

When no expression relies on a particular region (level) $\beta$,
any values in and references to level $\beta$ can be safely deallocated.
Higher levels are considered separate regions, and allow for fine-grained
region-like memory management.
The advantages of this system with a lifting operator is that the system is still
completely syntax directed, whereas some region type-and-effect systems need to
infer the regions based on reads and writes.

However, because we haven't changed the equations from ,
the resulting regions still have a stratified structure.
We conjecture that a more advanced system is possible, where the relationship
between a reference universe level and the universe level it stores can be
specified through some other relation besides successor.

τ:: Type>_α
τ :: Type>_β

To obtain the stratified variant, $\mathcal{R}$ specifies that $\beta = \alpha+1$.
However, to obtain different heap shapes, for example a shape where values allocated
in region $\alpha$ are completely separate from references to $\alpha$, we could
specify $\alpha \ \# \ \beta$ to mean that $\alpha$ and $\beta$ are disjoint.
Alternatively, certain values and references to these values could be allocated
in the same region by specifying $\alpha = \beta$.
With this abstraction, different heap shapes could be formed simultaneously.
The pictured heap below has distinct regions $\alpha$, $\beta$, and $\gamma$, where
references in $\alpha$ are stratified, references in $\beta$ are in $\beta$, and
$\gamma$ exists to contain references separate from values in $\alpha + 1$.

This kinding rule is no longer syntax directed, and
may result in a type universe polymorphism system more similar to Coq, where
the universes are inferred using constraint solving.
Alternatively, a user could declare universes (regions), and constraints between them.
For example, the following constraints could be specified to create the heap
visualized above, with the assumption that reference types bump the universe level by 1
when no constraints are given.

Type>_α  #  Type>_β  #  Type>_γ

Type>_β+ 1 = Type>_β

Type>_α+ 2 = Type>_γ

These constraints specify that three regions, $\alpha$, $\beta$, and $\gamma$
are all disjoint.
However, references to values in region $\beta$ are still in $\beta$,
hence the constraint $\<Type>_{\beta + 1} = \<Type>_{\beta}$.
Additionally, references to level $\alpha + 1$, $\<Type>_{\alpha+2}$, will be
in $\<Type>_{\gamma}$, since the reference $t$ is in $\<Type>_{\gamma}$.

§ CONCLUSION

What we've studied in this paper is the connection between type universe hierarchies
and memory allocation.
By studying an example of non-termination using higher-order references, we found
a simple mechanism to prevent such cycles by equipping a type system with a
type universe hierarchy.
The universe hierarchy imposes a structure on the heap for all programs, and
we found that the kind of a type describes where values can be allocated on the heap.
We are able to distill the essence of this type system to two parameters,
one describing the relationship of a function's type level to the levels it captures,
and another describing the relationship between a reference type level and what
level it stores.
These parameters give us a theoretical framework to create type systems that
enforce different heap designs by changing the algebra of the universe hierarchy.
We believe the simplicity and power of these systems with a universe hierarchy
provide a fresh foundation for new language designs for low-level memory reasoning.

§ ACKNOWLEDGMENTS

We are extemely grateful for Ohad Kammar's guidance, enthusiasm, and patience
while discussing this project, and the countless hours you spent trying
to teach us categorical semantics for local state, which helped us immensely
with our own semantics.
We are also grateful to Max S. New, who provided early intuitions into this work and
the model more generally, as well as a proof of termination for higher-order
references with closed functions.
We are also thankful to Ugo Dal Lago for pointing us to the stratified regions work
[4, 3].
We also would like to thank all who engaged with our work in presentations at the
workshops HOPE and PNW PLSE, and other informal venues.
Your enthusiasm, suggestions, and critiques are all a part of our work more than
you know, and we appreciate your contribution to the scientific process.
This material is based upon work supported by the Defense Advanced Research Projects
Agency (DARPA) and Naval Information Warfare Center Pacific (NIWC Pacific) under
Contract No. NN66001-22-C-4027.
Any opinions, findings and conclusions or recommendations expressed in this material
are those of the author(s) and do not necessarily reflect the views of DARPA or NIWC

[1]
Ahmed, A., Fluet, M., Morrisett, G.: L3: A linear language with locations
(2007). 10.1007/11417170_22

[2]
Ahmed, A.J.: Semantics of Types for Mutable State. Ph.D. thesis (2004),

[3]
Amadio, R.M.: On Stratified Regions. Asian Symposium on Programming Languages
and Systems (APLAS) (2009). 10.1007/978-3-642-10672-9_16

[4]
Boudol, G.: Typing termination in a higher-order concurrent imperative
language. Information and Computation  (2010). 10.1016/j.ic.2009.06.007

[5]
Crary, K.: Logical relations and a case study in equivalence checking. In:
Pierce, B.C. (ed.) Advanced Topics in Types and Programming Languages,
chap. 6. MIT Press (2005)

[6]
Harper, R., Pollack, R.: Type checking with universes. Theoretical Computer
Science  (1991). 10.1016/0304-3975(90)90108-t

[7]
Hou (Favonia), K.B., Angiuli, C., Mullanix, R.: An order-theoretic analysis of
universe polymorphism. In: Symposium on Principles of Programming Languages
(POPL) (2023). 10.1145/3571250

[8]
Huet, G.: Extending the Calculus of Constructions with Type:Type (1988),
<http://pauillac.inria.fr/ huet/PUBLIC/typtyp.pdf>

[9]
Jutting, L.: Typing in pure type systems. Information and Computation  (1993).

[10]
Landin, P.J.: The mechanical evaluation of expressions. The Computer Journal
(1964). 10.1093/comjnl/6.4.308

[11]
Luo, Z.: An Extended Calculus of Constructions. Ph.D. thesis, University of
Edinburgh (1990),

[12]
Matsakis, N.D., Klock, F.S.: The Rust language. In: High Integrity Language
Technology (2014). 10.1145/2663171.2663188

[13]
McBeth, J.H.: Letters to the editor: on the reference counter method.
Communications of the ACM  (1963). 10.1145/367593.367649

[14]
McBride, C.: Crude but effective stratification (2011),

[15]
Murawski, A.S., Tzevelekos, N.: Algorithmic Games for Full Ground References.
International Colloquium on Automata, Languages, and Programming (ICALP)
(2012). 10.1007/978-3-642-31585-5_30

[16]
Petricek, T., Orchard, D., Mycroft, A.: Coeffects: a calculus of
context-dependent computation. In: International Conference on Functional
Programming (ICFP) (2014). 10.1145/2628136.2628160

[17]
Sozeau, M.: Polymorphic universes (Coq reference manual),

[18]
The Agda Development Team: Universe levels (Agda reference manual),

[19]
Tofte, M., Talpin, J.P.: Region-based memory management. Information and
Computation  (1997). 10.1006/inco.1996.2613


# HEP Software Foundation Community White Paper Working Group – Software
Development, Deployment and Validation

HEP Software Foundation a Benjamin Couturier a,1 Giulio Eulisse b Hadrien
Grasland a Benedikt Hegner b Michel Jouvin c Meghan Kane d Daniel S. Katz e
Thomas Kuhr f David Lange a,1 Patricia Mendez Lorenzo e Martin Ritter a,1
Graeme Andrew Stewart a and Andrea Valassi CERN, Geneva, SwitzerlandLAL,
Université Paris-Sud and CNRS/IN2P3, Orsay, FranceSoundcloud, Berlin,
GermanyUniversity of Illinois Urbana-Champaign, Champaign, Illinois,
USAFakultät für Physik, Ludwig-Maximilians-Universität München, München,
GermanyPrinceton University, Princeton, USAPaper Editor

###### Abstract

The High Energy Physics community has developed and needs to maintain many
tens of millions of lines of code and to integrate effectively the work of
thousands of developers across large collaborations. Software needs to be
built, validated, and deployed across hundreds of sites. Software also has a
lifetime of many years, frequently beyond that of the original developer, it
must be developed with sustainability in mind. Adequate recognition of
software development as a critical task in the HEP community needs to be
fostered and an appropriate publication and citation strategy needs to be
developed. As part of the HEP Software Foundation’s Community White Paper
process a working group on Software Development, Deployment and Validation was
formed to examine all of these issues, identify best practice and to formulate
recommendations for the next decade. Its report is presented here.

|  | HSF-CWP-2017-13
---|---|---
|  | December 21, 2017

##  1 Scope

As part of the process to layout the HEP Software Foundation’s Community White
Paper Roadmap HSF-CWP-2017-01 (1) we formed a group to study issues around
software development, deployment and validation. The starting point for the
group was to recognise that modern High Energy Physics experiments are large
distributed collaborations comprising up to a few hundred people actively
writing software. It is therefore central that the processes and tools used
for development are streamlined to ease the task of contributing code and to
facilitate collaboration between geographically separated peers. At the same
time we must properly manage the whole project, ensuring code quality,
reproducibility and maintainability with the least effort possible. Making
sure this happens is largely a continuous process, and shares a lot with non-
HEP specific software industries.

The group is therefore interested in tracking and promoting solutions for the
following areas:

  * •

Distributed development of software components, including the tools and
processes required to do so (code organisation, documentation, issue tracking,
artifact building) and the best practices in terms of code and people
management.

  * •

Software quality, including aspects such as modularity and reusability of the
developed components, sustainability of the development effort, architectural
and performance best practices.

  * •

Deployment of software and interaction with operations teams.

  * •

Validation of the software both at small scales (e.g., best practices on how
to write a unit test) and larger ones (large scale validation of data produced
by an experiment).

  * •

Software licensing and distribution, including their impact on software
interoperability.

  * •

Recognition of the significant contribution that software makes to High Energy
Physics as a field.

The goal of this document is to highlight the challenges the field will face
in the next 5 to 10 years, and to provide a strategy on how address them.
Adopting best practices from non-HEP communities and adapting them to HEP-
specific needs will be key for the success of the effort.

##  2 Main challenges in the next 5-10 years

The challenges can be divided into two broad categories: those generic to
large-scale distributed software development and those specific to the HEP
environment.

The HEP-specific challenges are those derived from the fact that HEP is a
large, inhomogeneous community with multiple sources of funding, mostly formed
of people belonging to small university groups and some larger laboratories.
Software development effort within an experiment usually encompasses a huge
range of experience and skills, from a few more or less full time experts to
many physicist programmers with little formal software training. In addition,
the community is split between different experiments that often diverge in
timescales, size and resources.

Another peculiarity of HEP is that the experiment software is usually divided
in two separate use cases, production (data acquisition, data reconstruction
or simulation) and user analysis, whose requirements and life-cycles are
completely different. The former is very carefully managed in a central and
slow moving manner, following the schedule of the experiment itself. The
latter is much more dynamic and strongly coupled with conferences or article
publication timelines. Finding solutions which adapt well to both cases is not
always obvious or even possible.

Principle areas of challenge are:

##### Effective use of developer effort

  * •

The large fraction of short term students and collaborators found in our field
requires a process for evaluating software contributions to common projects
and for assuring not just their quality, but also their maintainability.

  * •

For the same reason, we need to minimise the time required for people to begin
to contribute effectively.

##### Knowledge sharing

  * •

There is a strong need for knowledge sharing, including effective training.
This is required by groups that are geographically separated, but also amongst
different fields, with different skills, languages and cultures. E.g., some
students might be able to track new trends better than older colleagues, but
they might lack vision; software engineers might provide better
implementations than postdocs, but may lack a complete understanding of the
broader picture for the experiment needed to set priorities.

  * •

There is an historical and stubborn compartmentalisation between different
experiments, so propagating knowledge horizontally across experiments is
particularly hard.

##### Reusing code effectively

  * •

While striving towards common solutions, the group recognises that a single
platform or set of tools will not fit all use cases, but we emphasise that
generic best practices and recommendations can be made. A top-down approach,
where a given tool or solution is mandated has not proven to be viable in the
past. However, if we can identify success stories and make sure they can be
extended (if needed) and advertised as possible general solutions, duplicate
efforts or developments are more likely to be avoided.

##### Validation of complex output

  * •

Efficient validation of the complex outputs of HEP code, involving inputs from
highly specialised experts and unique sources, is hard to do, especially
taking into account stochastic processes and long tails.

##### Data, code and knowledge preservation

  * •

Long term maintainability of the code to handle data that was obtained decades
before as required by the lifetime of a HEP experiment.

##### Code lifecycle and deprecation

  * •

In a highly demanding software environment such as the HEP communities, there
is great pressure to adopt updated versions of tools and libraries for
performance or design benefits. However, there is little corresponding
pressure to deprecate old components that may have been used for years,
causing a bloat of the active software stack and provoking crises when
abandoned upstream libraries start to fail to build. Hence there is a need to
engage in effective, smooth and continuous lifecycle management of the
codebase.

##### Hardware and software evolution

  * •

Hardware trends are now towards increasing processing capacity through
concurrency (many cores and vectorisation) rather than any increase in serial
processing capacity. This is at odds with the traditional HEP software
workflows, which have largely been serial. Memory latencies and a deep memory
and storage hierarchy have become key drivers of performance. Increasingly
specialised processors are also arriving in the marketplace, which are not
naturally suited to HEP codes. Adapting to this new hardware environment and
maintaining the efficiency of HEP code becomes increasingly hard.

  * •

New innovations in software development usually come from outside of the HEP
community. These can provide significant advantages in exploiting new hardware
or aiding efficient distributed development and such technologies need to be
tracked, investigated and deployed.

##### Software deployment

  * •

As HEP software today runs in widely heterogeneous environments, from
traditional grid computing sites to High Performance Computing sites,
including the gamut of academic and commercial cloud services, there is a
challenge to uniformly provide the appropriate software in all of these
environments. End user analysis code also needs to be managed and distributed
into these environments. CVMFS 7310920 (2) has been hugely successful here,
however it does not cover all cases and a viable common solution may emerge
and be adopted more widely.

##### Recognition of software development

  * •

Writing high quality software is a skilled activity that greatly contributes
to the success of the field. Ensuring that software developers receive
recognition for their contributions will help retain people in the field and
help their career development.

##  3 Meeting the Challenges

The previous section introduced a brief summary of the main challenges this
working group has identified. In the next paragraphs we both expand on these
and identify viable existing solutions and areas where more investigation is
required.

###  3.1 Effective use of developer effort

####  3.1.1 Design documentation and training

For developers to be able to contribute effectively to experiment software it
is first of all necessary for them to have a clear idea of how to write code
that will mesh effectively with the framework or the rest of the software
stack. Thus it is essential that developers can access clear documentation,
with good examples as to how to model their code. Effective contact with
experts should be available so that good design choices are made from the
outset.

Once a clear picture of the framework is available to new developers the first
task of the contributor should be to make their intentions clear to others. A
common technique is to use a request for comments document which outlines the
design and allows other developers to provide feedback before substantial
coding effort is expended. Such documents can be accompanied by demonstrator
or prototype code to better illustrate the purpose of the of the changes. This
approach allows documentation and testing to be incorporated into the
development itself and not left as an afterthought. This model was used by the
Concurrency Forum ConcurrencyForum (3) in the past and was very successful in
selecting the most promising approaches for the community and disseminating
knowledge amongst different experiments.

The task for developers who might contribute a general purpose piece of code
that is used across experiments is more challenging as active solicitation of
input will need to be sought. However, if experiment documentation is open to
the HEP community then different experimental boundary conditions will be more
easily understood and translated into the design and interface of a software
product.

In general, good software design principles should be followed and thus
generic training in software design, but with a focus on the HEP problem
space, is a valuable investment for the community. We note that several
computing schools exist already, but that overall only a small fraction of HEP
students attend them as they are considered quite specialist. Universities, in
both their undergraduate and graduate programmes, play an important role too
in core skills training (and in some institutes also in relevant advanced
topics), but this can be uneven. Effort should be made to incorporate more
generic training into the education programmes of the experiments; centering
these activities around some of the larger labs will also allow sharing of the
training effort across different experiments.

####  3.1.2 Standard development tools

To maximise the productivity of developers it is very beneficial to use
development tools that are not HEP-specific. There are many open-source
projects and tools that are of similar scale to large experiment software
stacks and standard tools are usually well documented. Many simple questions
can be answered using standard resources, such as Stack Overflow.

Insofar as an experiment requires customisation of tools, documentation and
training are evidently important. However, the costs of such customisation can
be easily outweighed by the cost of maintenance and training, as well as a
possible loss of generic skills for users.

For source control in particular, the open-source world has moved strongly
towards distributed version control as superior to centralised solutions. The
advantages for the individual developer are numerous, including independence
from central services and greater freedom to experiment. Finished code
enhancements can be proposed using a pull request workflow so that high
quality contributions are made with the central repository receiving pristine
patches. We note that the community has generally chosen to move to git. This
standardisation is very welcome as it also brings an alignment with many open-
source projects and commercial organisations.

The HEP current community is widely using CMake for the builds of software
packages. This is an open-source system dedicated to building, testing, and
installing software that captures the specific build specifications for each
particular package in a platform independent way. It is highly regarded
because of its modular and scalable characteristics and is also very widely
used outside HEP with a lot of support available. CMake’s use of plugin
modules allows scope for sharing between HEP experiments in the cases of
building HEP specific code, though at the moment probably the best use is not
yet made of this feature in our community. CMake can orchestrate the builds of
large numbers of packages (e.g., the LCG releases of over 300 packages), but
can become quite specific when doing so and thus more limited. The HSF
Packaging group found Spack HSF-TN-2016-03 (4, 5) to be a promising more
general solution for the task of building an HEP software stack, however, it
currently has a relatively small user base, centered on HPC, and might just
become a niche product with little traction in the wider development
community. More generic tools, such as Homebrew or Conda are interesting but
may not have all the features needed by HEP experiments. Thus this area still
requires close attention to active developments in the open-source world.

For reporting problems, tracking bug fixes and planning improvements to the
software issue trackers should be used. The best of these offer multi-level
grouping of issues and provide direct support for useful aspects of agile
development discussed below. Optimal choices will offer good integration with
source code management (e.g., GitHub Issue Tracker, Jira, GitLab Issue
Tracker) and the choice should usually be made in parallel with an source code
management tool.

####  3.1.3 Development environment

It is important that developers can focus on the design and implementation of
the code and do not have to spend a lot of time on technical issues. Clear
procedures and policies must exist to perform administrative tasks in an easy
and quick way. This starts with the setup of the development environment, for
example by having a minimal set of requirements on the operating system and
the installed packages. Supporting different platforms not only allows the
developers to use their machine directly for the development, it also provides
a check of portability basically for free.

Other typical workflows are the update of versions or the submission of code
for review. It decreases the turnaround time and reduces frustration if code
quality criteria and policies are checked as early as possible, ideally on
commits to the local repository or even by the editor. Modern IDEs, which
offer good integration with modern development tools, can be particularly
valuable for novice coders. They provide support for developers in navigating
the code base as well as catching elementary errors. It would be of great
benefit for the community to ensure that standard project templates (e.g., HSF
Project Template HSFProjectTemplate (6)) work with minimal configuration with
standard IDEs, such as Eclipse, XCode, KDevelop, Qt Creator, etc. For large
experiment code bases, the scaling of these IDEs to many millions of lines
requires investigation.

Finally, support is important. A system that enables developers to get quick
and competent advice from experts will improve their productivity.

####  3.1.4 Social coding sites

The advantages of social coding sites are that they allow developers to share
and discuss code through very streamlined and highly functional web
interfaces. Distributed version control systems are key enablers for these
sites. There are commercial choices, such as GitHub or BitBucket, that can
serve a community well; alternatively a privately hosted service, such as
CERN’s GitLab instance, may mesh more easily with other related services. We
note that “bridging” code between different service platforms has been
developed by a number of experiments and could be shared across the community.
Smaller projects would probably benefit from the increased exposure and larger
development community from being hosted on GitHub. For attribution and credit,
GitHub also offers a more standard platform that can be helpful to a
developer’s career. However, the key requirement here is the ability to
discuss and revise code changes in a recorded fashion as they become part of
the code repository.

####  3.1.5 Continuous integration

Proper testing of changes to code should always be done in advance of a
request being accepted. Continuous integration, where merge or pull requests
are built and tested in advance is now standard practice in the open-source
community and in industry. HEP software should not be different. Not only
should continuous integration run unit and integration tests, it can also
incorporate code quality checks and policy checks that will help improve the
consistency and quality of code at low human cost.

For running continuous integration and build orchestration many options exist
(Jenkins, Gitlab CI, Travis, Bamboo) and the particular choice of which one to
use is a relatively pragmatic decision, with only a very limited impact on the
ability to cooperate around software.

####  3.1.6 Code review

Incorporation of code into the code base should usually also involve sign off
from at least one relevant expert who can check for architectural violations,
anti-patterns or even just simple coding errors. Code review should be seen as
a process of dialogue between the developer and the reviewer (community
expert) and it should be firm, but friendly.

As well as a way to validate code, reviews can also be a way for newcomers to
learn about the architecture of a project. As part of a mixed experience
reviewing team, fresh eyes on a piece of software can produce original
comments, but the discussions occurring as a side effect of the review
effectively helps new developers understand the issues at stake.

####  3.1.7 Agile development

HEP is a community where the developer of a piece of software is also likely
to be the ‘customer’ as well. However, despite this, many useful ideas can be
taken from the various agile methodologies, in particular with regard to
solving only the problem at hand and keeping development cycles as short as
possible to find iterative and evolutionary solutions. Agile development
methodologies for physicists, and how to select the most appropriate one,
might be best taught by example rather than as formalism.

One aspect of agile methodology that would serve HEP rather well is the
concept of a user story that describes the purpose of a system, rather than
requirements capture that was used in the past. Such a method is more flexible
in adapting to changing requirements, which can be quite a common situation in
HEP, especially in the analysis domain.

HEP would benefit from adopting agile retrospectives. A retrospective is a
formalised way for contributors to meet regularly to discuss ways to improve
their software development process. Since a retrospective occurs regularly,
there is a chance to catch issues affecting the development process as they
arise. Issues can be tracked and mitigated over time rather than ballooning
into daunting barriers. Additionally, the focus is on learning from failures
quickly, without placing blame. This welcomes feedback, positive or negative,
toward the goal of improving the software development process.

Retrospectives can be structured as needed to suit HEP, but the standard
approach lends itself nicely to solving current problems since the last
retrospective:

  * •

Gather data. What went well? What didn’t go well? What needs to be improved?
Each person who has input should have a chance to succinctly state their
observations.

  * •

Discuss top issues in prioritised order. Based on the feedback of the group,
discuss top issues to reach an understanding about the root causes.

  * •

Assign action points. Based on the pain points discussed, assign action items
that are small enough in scope that they can be resolved before the next
retrospective.

This setting of information sharing and collaboration can yield clear
productivity benefits. For example, it is possible that a developer
experiences issues that slow down their productivity without being aware of
it. Perhaps the developer became accustomed to the build time being very high
and thinks of it is as given constraint. In a retrospective, a discussion
about build times may arise along with known solutions for reducing the build
times. This will elicit awareness about the issue, a solution, and a
subsequent gain in productivity.

Agile development is very well supported by the modern development
infrastructure outlined above.

###  3.2 Knowledge sharing

####  3.2.1 Documentation

As was mentioned above, proper training and documentation is key to
efficacious use of developer effort. For documentation that has to be
specific, favoured solutions would have a low barrier of entry for
contributors, but also allow and encourage the review of material.
Consequently it is very useful to host documentation sources in a repository
with a similar workflow to code and to use an engine that translates the
sources into modern web pages. An example would be using the Jekyll engine, as
the main HSF website currently does, or using services such as Read The Docs.

Traditionally HEP has often used Doxygen for reference documentation,
especially given its emphasis on C++. New documentation tools are also
available that integrate Doxygen with better solutions in other languages,
e.g., the python Sphinx documentation tool can use Breathe to integrate C++
and python documentation. Sites such as Read The Docs also take much of the
pain out of producing an attractive documentation site.

####  3.2.2 Training

Code development, as a critical and transferable skill, is well recognised
inside the HEP software community. Various schools exist that train students
in good coding practice, but only a small minority of contributors to HEP code
attend these. There is therefore scope for broader and more accessible
training that experiments can play a role in encouraging collaborators to
attend. Training events can start with generic material, shared between
experiments, then move to separate experiment specific topics. This would be
especially useful for experiments sharing a home laboratory and provides an
additional benefit when different groups have managed to converge on common
components.

Training and documentation are intimately linked and good training materials
can be studied offline by students in their own time.

As much use as possible should be made of generic resources that already
exist; there is a role here for the HSF to curate useful training materials
(where their ongoing usefulness and validity needs to be managed) and
encourage cooperation with other training initiatives, such as Software
Carpentry SoftwareCarpentry (7).

####  3.2.3 Seminars and discussion forums

These are important to the community, in particular to discuss new
technologies or tools at an early stage of development. A good example is the
Software Technology Forum SoftwareTechForum (8). These forums also allow
questions to be posed about problems that are encountered and identifying
existing software that may provide a solution (thus avoiding re-inventing the
wheel yet again).

In terms of knowledge sharing among developers, solutions such Stack Overflow
are excellent for establishing communication between software developers and
publishing answers on the web. In fact, this solution is used by many
developers to find solutions to the daily problems they face. It seems that in
general HEP software developers are consumers of these general help lists and
we encourage the community to take more active role in contributing as well.
Where a smaller community does not have the critical mass for their own Stack
Exchange service other products, like Discourse, can be installed on-premises.

In addition, attempts have also been made to set up general software
discussion mailing lists in the past. However, it seems that after some
initial flurry of activity these are not really very successful. They may be
too far away from most developers day-to-day concerns and there is probably
some reluctance to pose questions to a wide list. Though being a fast and
attractive solution for discussions about a particular issues they are not as
viable for long term or broad knowledge sharing. There may be some use in
trying alternatives, such as a Slack channel, at least to prototype with low
investment.

####  3.2.4 Workshops and community reviews

Hosting workshops on specific topics is a particularly useful way to bring
together experts from different experiments. These have an essential role in
introducing new ideas to the community and providing an environment in which
people can meet and interact. The HSF has also organised community meetings on
aspects of future HEP software development that are of particular interest to
the community in a strategic sense. Two such meetings have been organised so
far: one on Geant V simulation HSF-GeantV-Report (9), and the other on the
analysis ecosystem HSF-Analysis-Report (10). These meetings can provide very
valuable feedback to developers on the priorities of the experiments and the
most critical features to develop and support. Organising further meetings on
such topics provides important strategic feedback on development.

####  3.2.5 Conferences

HEP has a strong tradition of software and computing conferences (ACAT and
CHEP are good examples). These continue to be a critical part of the lifecycle
of software and of the community that supports it, offering recognition of
work done as well as dissemination of knowledge about available solutions that
can, hopefully, be adopted by others. Although they focus on completed pieces
of work, they complement other activities well and provide an important space
for social interaction.

It would be beneficial for the field as a whole to increase its participation
in general conferences and software focused events. This would help generate
relationships and collaborations outside the direct field with experts who may
have interesting observations and suggestions.

####  3.2.6 Journal publication

Journals disseminate information to the wider community in a permanent way and
so form an important part of the collective memory of the community. Journals
also have another significant role to play in recognition of software work,
which is discussed at greater length in Section 3.9.

###  3.3 Reusing code effectively

Effective reuse of code requires good design, documentation and support. Good
design starts from a good understanding of the problem to be solved, with an
appreciation of the important external interface to provide to clients. APIs
should be stable, with scope to change the internal details of an
implementation as needed. Training software developers in good design
principles will certainly help in this regard; however, often the reuse
potential of code is only discovered post-facto and then some advice and
expertise in refactoring will be useful for the longevity of the package. Here
Agile methodologies can help to focus on the key areas of responsibility and
avoid over designing interfaces for imagined situations.

There is scope to improve the training in these areas within the HEP community
as discussed above.

In all cases good documentation for code will greatly help adoption (the same
solutions apply here as were discussed above) and this provides a key aspect
of user support. Utilising popular standard development tools encourages
collaboration and provides issue tracking capabilities.

####  3.3.1 Software Licenses

All code should have a clear copyright holder, which gives the legal ownership
of the software. Applying a license specifies how the software can be used by
others. Open-source licenses are now widely accepted by the HEP experiments,
which we believe is the correct approach. Open source licenses fall into
various categories

  * •

Copyleft licenses mandate that any public releases with changes to the code
must be licensed under the same terms and, in addition, that any software
which uses the copyleft code must also itself be distributed under a copyleft
license (which is why these licenses are sometimes referred to as ‘viral’).
The General Public License, GPL, is the best known of these.

  * •

Weak copyleft licenses (such as the Lesser General Public License, LGPL or the
Mozilla Public License, MPL) remove the latter copyleft via use requirement,
but still mandate that direct modifications to the package itself be
redistributed under the same terms.

  * •

Permissive licenses (Apache, MIT, BSD) generally allow for public
modifications to be made without requiring them to be released under the same
terms as the original code.

There is a modern report from the HSF on licensing topics HSF-TN-2016-01 (11)
and we do not repeat the discussion here. Instead we encourage code authors to
ensure that copyright and license for code is well established at the
beginning of the project, including the status of contributions to the code.
The implications of the chosen license should be properly understood,
especially in so far as it affects contributors to the code base and users of
the code who may incorporate it into a larger software stack, together with
their own code (the GPL has a fairly profound effect in this regard).

###  3.4 Validation of complex outputs

The outputs of HEP software packages can be extremely complex and thus
difficult to validate. This is especially true of the simulation of modern
high energy physics detectors, which are incredibly complex machines and need
to be accurately modeled to understand their performance for rare decays and
weak signals. Up to now most experiments have developed their own
infrastructure for comparing results. Comparison of distributions is a
relatively well understood topic in statistics, so having standard tools that
help compare outputs from different versions of software would help save
effort between experiments and offers scope for improving the quality of our
validation.

###  3.5 Data and code preservation

The longevity of code in HEP is becoming increasingly important. Large
experiments have multi-decade active lifetimes, with “data preservation” now
considered an important topic after an experiment has finished taking data.
Data taken in the early years of the experiment may need to be reanalysed
years afterwards with older software that has not been built for a long time.
Software virtual machines or containers can help a great deal in preserving
the working environment, including the source code itself. Should changes need
to be made to software to correct for a long dormant bug or add a necessary
new feature, having a virtual machine or container with the source code and
build environment, in addition to any binary products, is important. Here the
use of standard tools, as encouraged above, can help, with the state of the
code and externals being encapsulated in a source repository tag. This is
becoming a prerequisite to publication in some journals, so being able to
trace the software used to produce some results, all the way through to an
analysis, is crucial. Experiments need to strongly encourage their members to
work in a preservable way.

In the design of code, relying on standard language features is more
sustainable than utilising external libraries that may become deprecated. Of
course, libraries that do tasks well are preferable to inferior home grown
implementations and achieving some uniformity in the community will help for
longer term support issues when they arise.

For further details on this topic we refer to the Data Preservation Community
White Paper group HSF-CWP-2017-06 (12).

###  3.6 Code deprecation

The complexity of HEP software and the generic nature of many of the sub-tasks
that need to be performed mean that many external libraries are essential to
our workflows. It would be completely unfeasible and a waste of effort to
reproduce such code internally. However, there are often many solutions to the
same problem and picking the one that has the best long term prospects or
becomes the most widely adopted is not possible a priori. Sometimes externals
will simply be eclipsed by alternatives and sometimes they will change their
APIs in backward-incompatible ways. The problem is often exacerbated by the
fact that the HEP-specific client may itself be only in maintenance mode and
the original author may not even work in the field anymore.

In these cases evolving away from deprecated code is required. This can be a
labor intensive operation, but often many aspects of it are automatable. There
is a good example in the refactoring of C++, which is a hard problem, but with
the development of Clang and full knowledge of the abstract syntax tree (AST)
of the code, there are significant possibilities for using this knowledge to
tackle much of the drudge work of refactoring. There have been successful
examples of using Clang in this way ClangMR41342 (13) and we believe this is
an area where investment could be very profitable, both specifically for HEP
and as a contribution to the wider open-source community. Tools to further
help validate the updated code can be shared between experiments, e.g.,
support for any migration to new versions of ROOT Brun1996 (14) or Geant4
Agostinelli2003 (15).

Changing software versions and evolving platforms is a natural part of the
lifecycle of production environments and can be helped by migration to more
self-contained deployment models, e.g., by using containers instead of a site
provided environment.

###  3.7 Hardware and software evolution

The evolution of processors in recent years have given rise to highly parallel
architectures (from many core CPUs to GPUs); in addition, thanks to the push
coming from the mobile market, viable low-power solutions are also now common
(e.g., ARM architectures).

####  3.7.1 Effective exploitation of parallel hardware

Implementation and optimisation of software on parallel architectures requires
different skills, tools and training (generally more advanced) than the ones
required for the serial world. In order to allow physicists to effectively
exploit parallel architecture, means are required to insulate end user
programmers from many of the low-level difficult details of parallel
programming.

Many specific details of parallel architectures can be masked by the adoption
of appropriate libraries. Generally HEP problems fit better into a task based
scheduling pattern, rather than to an optimiser of kernels run in tight loops.
The community has utilised Intel’s Threaded Building Blocks (TBB)
Pheatt:2008:ITB:1352079.1352134 (16, 17) successfully in many areas. Should
TBB continue to develop and be a good fit for our problems this common
solution also helps solve the problem of resource balancing between different
parts of the code. Should the solutions adopted become fragmented then overall
resource balancing becomes much harder and may become difficult to solve.
Evolution of the C++ standard will most likely bring more concurrency
primitives into the language, however, so some evolution here is to be
expected and should be planned for. This would mean appropriate zero cost
abstractions when possible.

####  3.7.2 Achieving performance portability

The fact that different parallel architectures (e.g., CPUs and GPUs) have
different sweet-spots for performance makes compile time performance
portability extremely difficult to obtain. This is true across architectures,
but also between different generations of the same architecture. Past
abstraction techniques might not work and development needs to target specific
hardware for best performance. However, it is strongly advisable that the
design of the code separates physics logic from performance optimisation. This
helps to avoid premature optimisation and a harmful reduction in code
maintainability. Options for runtime optimisation would also be possible
(e.g., fat libraries with code for various vector architectures) and may help
to avoid an explosion in build target architectures. Such techniques are an
area where common community investigation would be worthwhile, especially when
common tools are used in multiple projects.

When aiming for cross-architecture performance portability, one important
concern is data layout, as different architectures have different preferences
in this respect. For example, GPUs perform best when memory accesses from
different threads are coalesced (as it optimises memory bus utilisation),
whereas CPUs perform best when writes from different threads are relatively
far apart in RAM (to avoid the cache trashing effect known as “false
sharing”). These differences can be abstracted away through the use of data
structure generation mechanisms, which are still in their early days but are
begining to be offered by some programming environments such as Kokkos
CarterEdwards20143202 (18), and some domain specific languages, such as those
used in the context of the H.E.S.S. Observatory HESS (19) and in the Cherenkov
Telescope Array reconstructions CTA (20).

####  3.7.3 Tracking of new software technologies and tools

Compared to when HEP started as a field, the importance and ubiquity of
software development has increased hugely. If in the late 80s and 90s many of
the tools and practices were actually started inside our field itself, or at
least within academia, it is now true that a large amount of development
happens outside the traditional HEP field. Starting from the contributions of
“bedroom coders”, to the scale of large corporations like Google, Apple, or
Facebook, a large number of high quality tools and libraries are now available
as open-source projects for everyone to use. While this affords the field
great opportunities, the rate at which these tools and libraries change is
something that necessitates a continuous attention and an in-depth scrutiny of
their potential, as well as managing inevitable deprecations. As was pointed
out above, pooling community knowledge here will make optimal choices in new
technology more likely.

####  3.7.4 Dealing with hardware diversity

The increasing number of different architectures, operating systems and
compilers with different versions give rise to a large number of possible
platforms that combine these parameters. Experiment software should be tested
on a large range of platforms to ensure a competitive and effective use of
these new technologies, which requires good test cases and striving to
minimise the operational and human costs of multiple platforms. Use of open-
source orchestration toolkits for managing these stacks can help a lot.
Jenkins has received much attention, but alternatives, such as Travis, exist
and there may be scope for a community evaluation. Aspects such as application
independence, adaptation and modularity (a plugin-based structure) are
important. Basic platform types can share a lot of common infrastructure
(e.g., x86_64, AArch64 and PowerPC) that minimise the cost of adding new
flavors if common components are used.

There is an important link here when deciding what level of validation is
considered acceptable on each platform, especially if platforms do not produce
binary identical results. In order to evaluate and prepare for the
incorporation of such new platforms having early access is extremely
important. CERN’s TechLab and OpenLab projects help provide that, but there is
a need to ensure that the whole community can benefit and more sites could be
involved in this activity.

###  3.8 Software deployment

Deployment of HEP software has been revolutionised by the widespread adoption
of CERN Virtual Machine File System (CVMFS) 7310920 (2) for flexible and
ubiquitous access to a common filesystem namespace with efficient distribution
of content that can be strongly cached at multiple levels. We expect that
CVMFS will remain the mainstay of software distribution to most computing
sites in the foreseeable future.

However, CVMFS cannot cover all possible use cases. Sites that have no
outbound network access (common in the HPC world) or are mobile and perhaps
temporarily disconnected (e.g., developer laptops) require additional
distribution mechanisms. Here, containerisation offers a great deal of promise
as a way to isolate the software environment from the underlying hardware and
to conveniently provide all the needed runtime components. This is
particularly useful when other constraints drive the base operating system
away from the standard one adopted and validated by the experiment. At the
moment Docker is favored as being exceptionally convenient for development,
with automated conversion to other container implementations (e.g., Shifter or
Singularity) being possible. There should be no reason to deeply bind anything
in the install process into any particular container implementation.

Containers also play a role in data and code preservation as already noted.
While containers are architecture specific, there is currently no serious risk
that, e.g., x86_64 platforms will be unavailable in the foreseeable future.

###  3.9 Recognition of software development

Proper recognition of the contribution that software makes to the success of
High Energy Physics is a key element of sustaining that effort and
establishing software as a ‘first class citizen’ in the field. As an academic
discipline, HEP primarily recognises publications in refereed academic
journals and the citations of those publications as indicators of merit.
Therefore we strongly encourage authors of software to submit publications of
that software to such journals, of which there are now many SSI2017 (21). It
is also equally important that publications from the field appropriately cite
the software that they have used.

The correct level of granularity of software versions for publication will
need to be judged by authors themselves, as well as reviewers. Significant
improvements in algorithms, in either machine resource consumption or in
physics performance, would certainly merit a new publication, as might a major
release of software with significant new features. There is also a strong case
for having an overall publication that encompasses a major piece of combined
software used by an experiment. One example is an article on an experiment’s
offline software stack (that itself includes citations of the other software
used within the combined software). Such publications would then be highly
citable in physics papers, analogous to the common citations of the detector
design papers.

Currently, if HEP software has an article written about it, it will tend to be
in the conference proceedings of, e.g., CHEP or ACAT. Although these are
refereed, in general the barrier for entry is low, and as such they are not
considered as prestigious as refereed journals as the quality is mixed.
Without some reorganisation of the community, and a more strict refereeing
process, it is unlikely that conference proceedings will become the journals
of choice for recognition in our field. However, conference journals (e.g.,
Journal of Physics: Conference Series) usually do not allow resubmission of
articles to other journals without extended content. Software authors should
be aware of these issues when deciding where best to publish.

There are also now interesting projects to directly publish software itself,
for example to Zenodo Zenodo (22) or figsharefigshare (23), as has been
recommended by the FORCE11 Software Citation Working Group 10.7717/peerj-cs.86
(24). We are very encouraging of such efforts, and believe it would be useful
for HEP software authors to participate, but at the moment the actual
recognition accrued from such a publication is unclear. Through its work in
reviewing papers, the HEP community could choose to suggest to authors that
they cite the software they use, whether directly or via software papers. Over
time, this small effort on the part of reviewers could be sufficient to change
community practice regarding software credit.

For further discussion on this topic see also the Careers and Training
Community White Paper HSF-CWP-2017-02 (25).

##  4 Roadmap for the next 10 years

In the areas covered by this white paper HEP must endeavor to be as responsive
as possible to developments outside of our field. In terms of hardware and
software tools there remains great uncertainty as to which platforms will
offer the best value for money in a decade. It therefore behooves us to be as
generic as possible in our technology choices, retaining the necessary agility
to adapt to this uncertain future.

Our vision is characterised by HEP being current with technologies and
paradigms that are dominant in the wider software development world,
especially for open-source software, which we believe to be the right model
for our community. In order to achieve that aim we propose that the community
establishes a development forum that allows for technology tracking and
discussion of new opportunities. The HSF can play a key role in marshalling
this group and in ensuring its findings are widely disseminated. In addition,
having wider and more accessible training for developers in the field, that
will teach the core skills needed for effective software development, would be
of great benefit.

Given our agile focus, it is better to propose here projects and objectives to
be investigated in the short to medium term, alongside establishing the means
to continually review and refocus the community on the most promising areas.
The main idea is to investigate new tools as demonstrator projects where clear
metrics for success in reasonable time should be established to avoid wasting
community effort on initially promising products that fail to live up to
expectations.

###  4.1 Short term projects (1-2 years and ongoing activities)

  * •

Establish a common forum for the discussion of HEP software problems. This
should be modeled along the lines of the Concurrency Forum ConcurrencyForum
(3), which was very successful in establishing demonstrators and prototypes
that were used as experiments started to develop multi-threading frameworks.

  * •

Continue the HSF working group on packaging, with more prototype
implementations based on the stronger candidates identified so far.

  * •

Provide practical advice on how to best set up new software packages,
developing on the current project template work and working to advertise this
within the community.

  * •

Work with HEP experiments and other training projects to provide accessible
core skills training to the community. This training should be experiment
neutral, but could be usefully combined with the current experiment specific
training. Specifically this work can build on and collaborate with recent
highly successful initiatives like the LHCb StarterKit 1742-6596-898-8-082054
(26) and ALICE Juniors ALICEJuniors (27) and with established generic training
initiatives such as Software Carpentry SoftwareCarpentry (7).

  * •

Strengthen links with software communities and conferences outside of the HEP
domain, presenting papers on the HEP experience and problem domain. SciPy, SC
(informally known as Supercomputing), RSE Conference, and Workshop on
Sustainable Software for Science: Practice and Experiences (WSSSPE) would all
be useful conferences to consider.

  * •

Write a paper that looks at case studies of successful and unsuccessful HEP
software developments and draws specific conclusions and advice for future
projects.

###  4.2 Medium term projects (3-5 years)

  * •

Prototype C++ refactoring tools, with specific use cases in migrating HEP
code.

  * •

Prototyping of portable solutions for exploiting modern vector hardware on
heterogenous platforms.

  * •

Develop tooling and instrumentation to measure software performance,
especially in the domain of concurrency. This should primarily aim to further
the developments of existing tools, e.g., igprof Eulisse:2005zz (28), rather
than developing new ones.

  * •

Develop a common infrastructure to gather and analyse data about experiments’
software, including profiling information and code metrics, to ease sharing of
this information across experiments.

  * •

Develop tool(s) that allow developers to understand the usage of their
software and to be credited for this usage, possibly via using these metrics
associated with software citation.

  * •

Undertake a feasibility study of a common toolkit for statistical analysis
that would be of use in regression testing for experiment’s simulation and
reconstruction software.

It would be highly beneficial for the group to continue to function, through
the means of the forum proposed above, and to have at least one meeting a year
devoted to reviewing our activities and updating our knowledge of best tools
and practices.

## References

  * (1) The HEP Software Foundation  “A Roadmap for HEP Software and Computing R&D in the 2020s”, 2017  arXiv:1712.06982 [physics.comp-ph]
  * (2) J. Blomer et al.  “The Evolution of Global Scale Filesystems for Scientific Software Distribution”  In _Computing in Science Engineering_ 17.6, 2015, pp. 61–71  DOI: 10.1109/MCSE.2015.111
  * (3) “Concurrency Forum”  URL: http://concurrency.web.cern.ch/
  * (4) Elizabeth Sexton-Kennedy, Benedikt Hegner and Brett Viren  “HSF Packaging Working Group Report”, 2016  URL: http://hepsoftwarefoundation.org/notes/HSF-TN-2016-03.pdf
  * (5) Todd Gamblin et al.  “The Spack Package Manager: Bringing Order to HPC Software Chaos”  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , SC ’15  Austin, Texas: ACM, 2015, pp. 40:1–40:12  DOI: 10.1145/2807591.2807623
  * (6) “Common HSF Tools Repository”  URL: https://github.com/HEP-SF/tools
  * (7) “Software Carpentry”  URL: https://software-carpentry.org
  * (8) “Software Technology Forum”  URL: https://indico.cern.ch/category/7857/
  * (9) Gloria Corti et al.  “HEP software community meeting on GeantV R&D Panel Report”, 2016  URL: http://hepsoftwarefoundation.org/assets/GeantVPanelReport20161107.pdf
  * (10) HEP Software Foundation  “HEP Analysis Ecosystem Workshop Report”, 2017  URL: hthttp://hepsoftwarefoundation.org/assets/AnalysisEcosystemReport20170804.pdf
  * (11) Michel Jouvin et al.  “Software Licence Agreements HSF Policy Guidelines”, 2016  URL: http://hepsoftwarefoundation.org/notes/HSF-TN-2016-01.pdf
  * (12) The HEP Software Foundation  “HEP Software Foundation Community White Paper Working Group – Data and Software Preservation”, 2017
  * (13) Hyrum Wright et al.  “Large-Scale Automated Refactoring Using ClangMR”  In _Proceedings of the 29th International Conference on Software Maintenance_ , 2013  URL: https://research.google.com/pubs/pub41342.html
  * (14) R. Brun and F. Rademakers  “ROOT: An object oriented data analysis framework”  In _New computing techniques in physics research V. Proceedings, 5th International Workshop, AIHENP ’96, Lausanne, Switzerland, September 2-6, 1996_ A389, 1997, pp. 81–86  DOI: 10.1016/S0168-9002(97)00048-X
  * (15) S. Agostinelli  “GEANT4: a simulation toolkit”  In _Nucl. Instrum. Meth._ A506, 2003, pp. 250–303  DOI: 10.1016/S0168-9002(03)01368-8
  * (16) Chuck Pheatt  “Intel®; Threading Building Blocks”  In _J. Comput. Sci. Coll._ 23.4 USA: Consortium for Computing Sciences in Colleges, 2008, pp. 298–298  URL: http://dl.acm.org/citation.cfm?id=1352079.1352134
  * (17) “Intel Threading Building Blocks”  URL: https://www.threadingbuildingblocks.org/
  * (18) H. Edwards, Christian R. Trott and Daniel Sunderland  “Kokkos: Enabling manycore performance portability through polymorphic memory access patterns” Domain-Specific Languages and High-Level Frameworks for High-Performance Computing  In _Journal of Parallel and Distributed Computing_ 74.12, 2014, pp. 3202–3216  DOI: https://doi.org/10.1016/j.jpdc.2014.07.003
  * (19) “High Energy Stereoscopic System Observatory”  URL: https://www.mpi-hd.mpg.de/hfm/HESS/
  * (20) “The Cherenkov Telescope Array observatory”  URL: https://www.cta-observatory.org/
  * (21) “Sustainable Software Institute: In which journals should I publish my software?”  URL: https://www.software.ac.uk/which-journals-should-i-publish-my-software
  * (22) “Zenodo”  URL: https://zenodo.org
  * (23) “figshare”  URL: https://figshare.com
  * (24) Arfon M. Smith, Daniel S. Katz and Kyle E. Niemeyer  “Software citation principles”  In _PeerJ Computer Science_ 2, 2016, pp. e86  DOI: 10.7717/peerj-cs.86
  * (25) The HEP Software Foundation  “HEP Software Foundation Community White Paper Working Group – Careers & Training”, 2017
  * (26) Albert Puig and LHCb Starterkit Team  “The LHCb Starterkit”  In _Journal of Physics: Conference Series_ 898.8, 2017, pp. 082054  URL: http://stacks.iop.org/1742-6596/898/i=8/a=082054
  * (27) H. Beck  “The Junior Community in ALICE” Presented at EPS conference, 2017  URL: https://indico.cern.ch/event/466934/contributions/2589553/attachments/1489205/2314059/EPS-Juniors-v6.pdf
  * (28) G. Eulisse and Lassi A. Tuura  “IgProf profiling tool”  In _Computing in high energy physics and nuclear physics. Proceedings, Conference, CHEP’04, Interlaken, Switzerland, September 27-October 1, 2004_ , 2005, pp. 655–658  URL: http://doc.cern.ch/yellowrep/2005/2005-002/p655.pdf