[1]Joosep Pata

[1]National Institute of Chemical Physics and Biophysics (NICPB), Rävala pst
10, 10143 Tallinn, Estonia 2]European Center for Nuclear Research (CERN), CH
1211, Geneva 23, Switzerland 3]University of California San Diego, La Jolla,
CA 92093, USA

# Scalable neural network models and terascale datasets for particle-flow
reconstruction

<EMAIL_ADDRESS>Eric Wulff<EMAIL_ADDRESS>Farouk Mokhtar
<EMAIL_ADDRESS>David Southwick<EMAIL_ADDRESS>Mengke Zhang
<EMAIL_ADDRESS>Maria Girone<EMAIL_ADDRESS>Javier Duarte
<EMAIL_ADDRESS>*  [  [

###### Abstract

We study scalable machine learning models for full event reconstruction in
high-energy electron-positron collisions based on a highly granular detector
simulation. Particle-flow (PF) reconstruction can be formulated as a
supervised learning task using tracks and calorimeter clusters or hits. We
compare a graph neural network and kernel-based transformer and demonstrate
that both avoid quadratic memory allocation and computational cost while
achieving realistic PF reconstruction. We show that hyperparameter tuning on a
supercomputer significantly improves the physics performance of the models. We
also demonstrate that the resulting model is highly portable across hardware
processors, supporting Nvidia, AMD, and Intel Habana cards. Finally, we
demonstrate that the model can be trained on highly granular inputs consisting
of tracks and calorimeter hits, resulting in a competitive physics performance
with the baseline. Datasets and software to reproduce the studies are
published following the findable, accessible, interoperable, and reusable
(FAIR) principles.

##  1 Main

One of the main approaches for event reconstruction at the Large Hadron
Collider (LHC) is currently based on the particle-flow (PF) algorithm [1, 2,
3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13], which combines measurements from
different subdetectors to produce a holistic particle-based description of the
entire event. Given the planned High Luminosity LHC (HL-LHC) program, as well
as possible future experimental programs such as the Future Circular Collider
(FCC), it is necessary to develop and evaluate a computationally efficient and
physically optimal evolution of PF reconstruction.

Among various approaches, there has been considerable interest and development
of machine learning (ML)-based reconstruction methods, including for global
event description. To summarize recent developments, a recipe for a per-
particle loss was given in Ref. [14]. In Ref. [15], a graph neural network
(GNN)-based approach was demonstrated to reconstruct events with high particle
multiplicity accurately. This approach was successfully applied in the Compact
Muon Solenoid (CMS) experiment [16, 17]. Recently, Ref. [18], demonstrated
that hypergraph networks can improve jet reconstruction. In parallel,
clustering using ML has been demonstrated for high-granularity calorimeter
reconstruction [19].

Figure 1: A conceptual overview of the MLPF approach based on tracks, hits and
clusters.

One of the critical challenges for PF reconstruction is the highly granular
nature of the data: events can comprise hundreds of thousands of heterogeneous
measurements in various detector subsystems. This motivates studying models
that can scale to large input multiplicities and efficiently process full
events or batches of events. To support such studies, it is beneficial to
establish open realistic simulated datasets with sufficient granularity to
test various approaches.

In this paper, we describe an open dataset of electron-positron
($\mathrm{e}^{+}\mathrm{e}^{-}$) collision events at a center of mass energy
$\sqrt{s}=380\,\text{GeV}$ with full GEANT4 simulation, suitable for PF
reconstruction, available in the EDM4HEP [20] format for future studies.
Furthermore, we test two types of scalable ML models as benchmarks that can
process full events consisting of tens of thousands of measurements, while
avoiding memory allocation or computations that scale quadratically with the
input size. Using an input of charged particle tracks and clusters of
calorimeter energy deposits, we minimize a particle-based loss function and
monitor physics metrics that quantify event reconstruction performance during
training. We report the results of an extensive hyperparameter optimization
(HPO) of the GNN-based model performed using high performance computing (HPC)
resources. We then evaluate the model’s physics and computational performance,
as well as its portability and quantization compatibility. We find that the
GNN-based model can reconstruct physics events with a higher degree of
fidelity compared to the baseline, and its computational cost scales linearly
with the size of the input event. Moreover, we demonstrate the portability of
the model to several computational hardware processors from Nvidia, AMD, and
Intel Habana. The model demonstrates high-fidelity physics reconstruction
performance when trained on a lower-level dataset consisting of tracks and raw
calorimeter hits, providing further evidence that such heterogeneous and
granular datasets can be reconstructed at scale using ML. Finally, we identify
steps for future development and publish the code and datasets following the
findable, accessible, interoperable, and reusable (FAIR) principles [21, 22,
23] for reproducibility and future development.

The setup follows the MLPF approach previously used in Refs. [15, 16, 17], and
applies a physics-inspired ansatz to simplify the event-based reconstruction
loss to a particle-based classification and regression loss. The approach is
described in more detail in the Methods section.

We note that in the recent hypergraph-based reconstruction method [18], the
strict particle-to-element association is completely avoided, and the
associations are instead made an optimization target for an intermediate
model. This hypergraph reconstruction approach shows excellent physics
performance on small jet-based samples, but further work is needed to extend
it to full events with large particle multiplicities and to demonstrate
computational feasibility on realistic datasets.

We have not yet specified the specific neural network (NN) structure of the
reconstruction model $\Phi(X)$. It could be implemented with a simple
feedforward network over the individual feature vectors of each input element,
i.e. $\phi(x)\rightarrow y^{\prime}$. However, such a model would be unable to
consider correlations between related input elements, such as tracks and
calorimeter clusters (or hits) arising from a single particle.

An early approach proposed for full-event reconstruction previously used a GNN
with a learnable graph structure between the input nodes [15]. This is
conceptually similar to a transformer with full self-attention, which has also
recently been thoroughly investigated [18].

However, it is well known that the evaluation of full self-attention can be
computationally demanding and prohibitive for input multiplicities
${\geq}10^{4}$. Such input multiplicities can be easily reached when
considering all tracks and calorimeter clusters in future high luminosity
collider events, or if calorimeter clustering is avoided by using tracks and
calorimeter hits even in the case of a future lepton collider, as shown below.
It is possible that a future successful ML-based reconstruction approach will
exploit the local nature of the problem by first pre-partitioning the event
and running the full attention-based model only on subsets. However, this
requires careful implementation to avoid artifacts from partitioning and
subsequent stitching.

Here, we study models that can naturally scale to very large input
multiplicities by avoiding pairwise memory allocations or computational
operations. Increasing the context lengths for ML models also receives
considerable attention in the literature because of its wide range of
applications [24, 25], and our approaches are based on existing research:

  * •

The first model we study uses a dynamically learned graph structure [15], but
avoids a full quadratic allocation or computation by using a learnable binning
based on locality sensitive hashing (LSH) in each graph building layer [16,
17], inspired by the reformer architecture [26].

  * •

The alternative model we test is a transformer in which the softmax self-
attention layer is approximated using fast attention via positive orthogonal
random features [27]. The complete model for PF reconstruction is then
implemented based on stacked layers of self-attention and feedforward
networks, with the number of layers being a hyperparameter.

The models can be implemented using simple matrix operations, directly in
native TensorFlow.

Having defined NN models for PF reconstruction that avoid $\mathcal{O}(N^{2})$
memory allocation and computational scaling while retaining the ability to
treat large input sets and consider associations between all input elements,
we now train and test them on a realistic dataset. We generate
$\mathrm{e}^{+}\mathrm{e}^{-}$ collision events with Pythia8 (v8.306) [28] and
carry out a complete detector simulation with GEANT4 (v11.0.2) using the
Key4HEP framework (v2023-01-15) [29]. In particular, we use the Compact Linear
Collider (CLIC) detector model [30, 31], along with the Marlin reconstruction
code [32], and the Pandora package [33, 34, 35] for a baseline PF
implementation. The CLIC detector model is chosen because it is publicly
available, well documented and realistic. The model implementation is not
specific to the detector.

The CLIC detector model is based on the CMS detector at CERN. It features a
superconducting solenoid with an internal diameter of 7 m, providing a
magnetic field of 4 T in the center of the detector. Silicon pixel and strip
trackers, the electromagnetic (ECAL) and hadron calorimeters (HCAL) are
embedded within the solenoid. Each subdetector is divided into a barrel and
two endcap sections. The ECAL is a highly granular array of 40 layers of
silicon sensors and tungsten plates. The HCAL is built from 60 layers of
plastic scintillator tiles, read out by silicon photomultipliers, and steel
absorber plates. The muon system surrounding the solenoid consists of 6 and 7
layers of resistive plate chambers interleaved with yoke steel plates in the
endcap and barrel respectively. Two smaller electromagnetic calorimeters,
LumiCal and BeamCal, cover the very forward region of the detector on either
side of the interaction point [30, 31].

Collision events are generated with different physics processes to test the
out-of-distribution performance of the model. In particular, we use Pythia8 to
generate ${\simeq}10^{6}$ $\mathrm{t\overline{t}}$, $\mathrm{ZH}$,
$\mathrm{WW}$ each, and ${\simeq}2\times 10^{6}$ $\mathrm{q\overline{q}}$
events. Furthermore, we generate ${\simeq}7\times 10^{5}$ of
$\mathrm{t\overline{t}}$ PU10 events with a hadronic overlay
$\mathrm{g}\mathrm{g}\to\mathrm{q\overline{q}}$ interactions, corresponding to
an average of 10 additional interactions per event, known as pileup (PU), to
test the stability under varying conditions. The datasets are augmented with
single $\mathrm{e}^{\pm}$, $\mathrm{\mu}^{\pm}$, $\mathrm{K^{0}_{L}}$,
$\mathrm{\pi^{0}}$, $\mathrm{\pi^{\pm}}$, neutron, and photon particle gun
samples, with a uniform energy distribution in $E\in[1,100]\,\text{GeV}$,
generated using the ddsim package, $\simeq 10^{6}$ events each. The
$\mathrm{q\overline{q}}$, $\mathrm{t\overline{t}}$ and single particle gun
datasets were split in an 80/20 ratio to form train/test samples, while the
$\mathrm{ZH}$, $\mathrm{WW}$ and $\mathrm{t\overline{t}}$ PU10 datasets were
never used in training.

The datasets with generator particles; reconstructed tracks, hits and
calorimeter clusters; as well as reconstructed particles from the baseline
Pandora algorithm are saved in the EDM4HEP format, including all the relevant
associations. Overall, the size of the dataset is approximately 2.5 TB before
preprocessing to the ML-specific format using the tfds library [36]. The raw
datasets in EDM4HEP format, along with the scripts and configurations to
generate the data, are available at [37].

There are about 50–150 tracks or calorimeter clusters per event, while the
multiplicity for raw calorimeter hits is considerably larger at 5–$15\times
10^{3}$ per event. We analyze the datasets at the level of the tracks and
calorimeter clusters to study the physics performance of models, while
foregoing the Pandora-based clustering allows us to test the scalability of
the model to high input multiplicities, as reconstructing events from raw hits
is a much more challenging task with an ab-initio ML-based model. The input
elements and target particles have different underlying signatures in
different samples, depending on the physics process, therefore the models will
have to demonstrate out-of-distribution generalization.

Jets are defined by clustering particles with the generalized $k_{\mathrm{T}}$
algorithm ($R=0.7,p=-1$) for $\mathrm{e}^{+}\mathrm{e}^{-}$ colliders [38,
39]. We also evaluate the ${\vec{p}}_{\mathrm{T}}^{\kern 1.0pt\text{miss}}$
and total 3D momentum of the reconstructed and generated particles. The
missing transverse momentum vector ${\vec{p}}_{\mathrm{T}}^{\kern
1.0pt\text{miss}}$ is calculated as the negative vector sum of the transverse
momenta of all particles in an event, and its magnitude is denoted as
$p_{\mathrm{T}}^{\text{miss}}$. We use these quantities to assess the
performance of MLPF and the baseline PF reconstruction with respect to the
ground truth defined by the generator particles.

The input features for the tracks consist of the track $p_{\mathrm{T}}$, the
pseudorapidity of the particle momentum (track tangent) $\eta$, the azimuthal
angle of the particle momentum $\phi$, the track goodness-of-fit $\chi^{2}$
and the number of degrees of freedom $N_{\mathrm{dof}}$, the slope of the
track $\tan{\lambda}=p_{z}/\sqrt{p_{x}^{2}+p_{y}^{2}}$, the signed impact
parameter $D_{0}$ in the $xy$ plane, the curvature of the track $\Omega=1/R$,
the position of the track along the $z$-axis $Z_{0}$ with respect to the
reference point at $(0,0,0)$ [40]. For calorimeter clusters, the features
consist of $E_{\mathrm{T}}$, $\eta$, $\phi$, $E_{\mathrm{ECAL}}$,
$E_{\mathrm{HCAL}}$, Cartesian spatial coordinates $x$, $y$, $z$; the number
of hits and the cluster size, measured in standard deviations of the hit sets
in $x$, $y$, and $z$.

##  2 Results

Figure 2: The validation loss, jet and $p_{\mathrm{T}}^{\text{miss}}$
resolution of the GNN model before and after hypertuning, evaluated using 10
trainings, each trained on four Nvidia A100 devices in a distributed data-
parallel manner. We show the evolution of the validation loss as well as the
jet and $p_{\mathrm{T}}^{\text{miss}}$ resolutions during the full training
process, where only the random seed differs in each run. The shaded regions
show the standard deviation of the metrics across the runs.

An extensive HPO was performed for both the GNN\- and transformer-based model
using the JURECA supercomputer [41]. JURECA is a pre-exascale modular
supercomputer operated by Jülich Supercomputing Centre at Forschungszentrum
Jülich. The system consists of a flexible Data Centric (DC) module, based on
the Atos BullSequana XH2000. It has, among others, 192 accelerated compute
nodes with four NVIDIA A100 GPUs and two AMD EPYC 7742 CPUs each. We employ a
similar approach to HPO as in Ref. [42], using Bayesian Optimization in
combination with the ASHA [43] algorithm. The optimization was performed in a
distributed manner on 96 GPUs spread over 24 compute nodes, consuming roughly
7000 and 5000 GPU hours for the optimization of the transformer and the GNN
model respectively. The performance improvements achieved from HPO are
presented in Fig. 2. We find that the optimized versions of both the GNN and
the transformer significantly outperform the unoptimized versions. The
optimized version of the GNN significantly outperforms the transformer,
although both use a similar number of trainable parameters (${\simeq}5\times
10^{6}$).

The final GNN model is trained separately in two different input modes,
visualized on Fig. 1:

  1. (1)

using charged particle tracks and calorimeter clusters—this tests the physics
performance of the model under a typical PF scenario, where a form of
clustering is applied beforehand—and

  2. (2)

using charged particle tracks and raw calorimeter hits—this tests the
scalability of the model in extreme cases of high input multiplicity when an
explicit clustering step is omitted.

Due to the computational cost, the hyperparameters are identified using the
cluster-based model (1) and reused for the hit-based model (2). We plan to
follow up on hypertuning of model (2) in a subsequent work. The relevant
evaluation criterion for the model is the reconstruction of jets and
$p_{\mathrm{T}}^{\text{miss}}$ compatible with those at the generator level, a
target that is not explicitly trained for, but can be reached by minimizing
the particle-level loss. The physics performance in event-level quantities is
summarized in Fig. 3.

Figure 3: Overall jet (top) and total 3-momentum (bottom) response in the
$\mathrm{ZH}$, $\mathrm{WW}$ and $\mathrm{t\overline{t}}$ PU10 validation
samples that were never used in training.

We use $\mathrm{ZH}$, $\mathrm{WW}$ and $\mathrm{t\overline{t}}$ PU10 events
to evaluate out-of-distribution performance. We evaluate the jet response by
clustering reconstructed particles into jets as described above, by matching
the reconstructed and generator-level jets, and computing the ratio of
reconstructed to generator-level jet $p_{\mathrm{T}}$. In all samples, the
fraction of reconstructed jets from MLPF is the same or higher than for PF,
with generally an improved jet response width, quantified by the interquartile
range (IQR) and median compared to the rule-based baseline. This improvement
over the baseline was not observed before hyperparameter tuning. We also
evaluate the total 3-momentum response for either PF or MLPF particles, and
find that the MLPF model improves both the median and IQR of the total
3-momentum response distributions for all samples.

We further quantify the evolution of the median and IQR of jets (total
3-momentum) in bins of generator-level jet $p_{\mathrm{T}}$ (generated total
3-momentum) on the $\mathrm{t\overline{t}}$ with 10 PU interactions
($\mathrm{t\overline{t}}$ PU10) sample in Fig. 4.

Figure 4: The median and interquartile range of the jet (left) and total
3-momentum (right) response in the $\mathrm{t\overline{t}}$ PU10 sample.

The baseline PF and proposed MLPF algorithms behave qualitatively similarly,
with improved response IQR at higher gen-jet $p_{\mathrm{T}}$ (total
3-momentum), while the MLPF algorithm consistently outperforms the baseline PF
on this sample.

We train version (2) of the model with significantly higher input element
multiplicity, where each event consists of ${\simeq}10^{4}$ tracks and raw
calorimeter hits. This is to test the computational scalability of the
approach, and to study the physics performance using lower-level inputs. To
improve the physics response, single-particle gun samples are now included in
the training set. The training is carried out on a single node at the LUMI HPC
center on four AMD MI250X devices with two accelerator processors each. The
model is currently trained for 10 epochs with the same procedure as described
above, with the results shown in Fig. 5.

Figure 5: The evolution of the IQR and median of the jet and
$p_{\mathrm{T}}^{\text{miss}}$ response in the $\mathrm{t\overline{t}}$
sample, using tracks and raw calorimeter hits as input, trained on eight AMD
MI250X accelerator chips on the LUMI HPC. Also shown are the final jet and
total 3-momentum response distributions after 10 epochs of training.

Throughout the training, despite using a much lower level training input, the
jet and $p_{\mathrm{T}}^{\text{miss}}$ IQR and median trend towards the
optimal values of 0 and 1 respectively. In Fig. 5, we see that the hit-based
MLPF has comparable, currently slightly worse reconstruction performance
compared to the baseline PF when using raw calorimeter hits. This can partly
be attributed to not using an optimized set of hyperparameters, and not
training for a sufficient number of epochs—both limited by computational
throughput of the training and motivating further research effort into high-
performance training.

An important factor in the development of scalable ML-based full event
reconstruction models is improving the computational throughput in future
deployment scenarios. Therefore, we compare the inference scalability of the
rule-based PF implementation on CPU and the ML-based implementation on GPU
with respect to increasing input multiplicity, with the results summarized in
Fig. 6. Inference scaling tests of the GNN model are tested on a small, 8 GB
consumer GPU (Nvidia RTX2070S) to emulate a resource-constrained scenario in
edge deployment.

First, the rule-based version is run with 25, 50, 100 and 200 generated
$\mathrm{\pi^{-}}$ particles per event for 100 events, three times each. The
relative runtime of the particle-flow module in Key4HEP is measured, and
compared with the 25-particle case. We find that the runtime of the baseline
increases nonlinearly with increasing particle multiplicity, and memory
segmentation faults occur for more than approximately 200 particles per event.

Similarly, the GNN model is run with a varying input size on an 8 GB consumer
GPU multiple times to average over random fluctuations. Both the batch size
$B$, i.e., the number of batched events ($B\in\\{1,2,4,8,16\\}$) and the event
size $N$, i.e., the number of input elements per event ($N=256n$ for
$n\in[1,20]$) are varied independently. The relative runtime with respect to
the $N=256,B=16$ case scales approximately linearly with $N$. We note that
$B\gg 1$ is required to saturate the GPU, but this is highly model and device
specific.

We confirm that the ML-based reconstruction based on the LSH-binned GNN for
full event reconstruction avoids the quadratic scaling present in a typical
rule-based full event reconstruction algorithm. It is likely that additional
effort will be able to significantly improve the performance of both
algorithms. For example, in CMS, a $k$-dimensional tree [44] is used to avoid
quadratic scaling [11]. With aggressive quantization, the throughput of ML
models can be significantly improved with a negligible performance degradation
[45].

At this point, it is not meaningful to compare the absolute throughput of the
rule-based model on a CPU and the ML-based model on a GPU, as neither method
is particularly optimized for throughput, and the comparison is strongly
affected by the specific choice of hardware.

The training scalability is tested, with results presented in the rightmost
plot in Fig. 6, on three different HPC centers with different accelerator
hardware: Nvidia H100 GPUs from Flatiron Institute’s CoreSite cluster [46],
AMD MI250 GPUs from the LUMI supercomputer [47], and Intel Habana Gaudi HPUs
from the Voyager supercomputer [48]. The LUMI supercomputer features GPU nodes
with 64-core AMD Trento CPUs and four AMD MI250X cards, each card consisting
of two accelerator chips. Voyager is an NSF-funded supercomputer with 42
first-generation Intel Habana Gaudi (training) nodes, each with eight cards,
two first-generation Intel Habana Goya (inference) nodes, a 400 GbE Arista
switch, and 3 PB of Ceph file system available at the San Diego Supercomputer
Center located at the University of California San Diego. For the training
tests, there are some differences in the configuration on the HPCs. For the
AMD processors, multi-card training was implemented with a mirrored worker
configuration, while for the Nvidia and Habana processors, Horovod [49] was
used. Events were zero-padded to a regular size of 512 elements per events. A
batch size of 250 events per device was used for the Nvidia and AMD
processors, while a batch size of 100 per device was used for the Habana
processors. We observe nearly linear scaling or better for all processors. The
improved scaling for the Habana processors can be explained by the all-to-all
non-blocking intra-node network connection, where each processor has a 100 Gb
network connection to every other processor [48].

Figure 6: Relative timing of the baseline PF (left) and MLPF algorithms
(middle), illustrating the scaling with respect to input particle or element
(track, cluster, or hit) multiplicity. We also demonstrate the scaling of the
training performance across multiple devices on a single machine (right) on
Nvidia, AMD, and Habana processor cards from the CoreSite, LUMI, and Voyager
supercomputers, respectively. For the multi-device scaling test, Horovod was
used on CoreSite and Voyager, while the TensorFlow MirroredStrategy was used
on LUMI. The batch size was adjusted to fit a single device and the dataset
was fully cached in RAM.

##  3 Discussion

In this paper, we proposed a new multi-terabyte dataset of high-energy
electron-positron collisions in a future detector concept, simulated with
GEANT4 and stored in EDM4HEP format, featuring increased realism and
granularity for particle-flow (PF) reconstruction studies. We stressed the
question of model scalability on large-scale datasets as a research direction,
and investigated and compared two alternative models featuring approximately
linear scaling. Through extensive high performance computing (HPC)-enabled
hyperparameter tuning, we found that a graph neural network (GNN)-based model
outperforms a transformer-based one, as well as the rule-based baseline PF
algorithm. We evaluated the physics performance in detail with event-level
metrics and find that both the interquartile range (IQR) and median of the jet
and total 3-momentum response distributions are improved by the GNN model. We
demonstrated that the model supports and benefits from mixed-precision
training. Furthermore, we tested the computational scalability of the model,
finding that the runtime of the GNN-based model scales approximately linearly
with increasing input size, while the rule-based baseline PF algorithm scales
nonlinearly. Finally, we demonstrate that training the GNN model on a dataset
consisting of tracks and raw calorimeter hits is feasible and results in
physics performance comparable to the baseline, confirming that explicit
clustering can in principle be avoided in favor of direct particle
reconstruction. This training, carried out on the LUMI HPC featuring AMD GPUs,
and other scaling tests done on Nvidia GPUs and Intel Habana processing units
demonstrate effective portability across vendors.

Further research is possible in several directions. First, having demonstrated
improved physics performance through large-sample training in the Compact
Linear Collider (CLIC) detector samples used in this paper, it would be useful
to repeat this exercise using the simulation from detectors that are taking
data in Run 3 of the Large Hadron Collider (LHC), e.g. CMS, to study the
performance of the reconstruction in real data. Second, we are currently using
a simple particle-based loss, while the use of contrastive-adversarial
learning methods may allow to account for event-level discrepancies more
effectively [50]. Third, the hypergraph model [18] shows promising physics
performance but currently only supports small input sequences. It may be
interesting to extend the hypergraph construction over dynamically binned
events using the locality sensitive hashing (LSH) approach. It may also be
interesting to construct features using semi-supervised learning from real
data, to reduce the reliance on simulated datasets for supervised learning.
Furthermore, in large language model (LLM) research, large-context models are
continuously improving, and it may be interesting to apply the latest
developments such as FlashAttention [51, 52] on the models in this paper. For
improving throughput on e.g. CPUs, quantization to int8 has shown promise, and
it may be interesting to investigate if this can be repeated for the type of
models proposed for PF reconstruction. Finally, it is important to integrate
the proposed machine learning (ML)-based reconstruction models into
reconstruction frameworks such as CMSSW and Key4HEP, to ensure that further
development and testing can be performed under realistic circumstances.

##  4 Methods

The input to the model is a set of input detector elements: charged tracks and
calorimeter clusters (or alternatively, raw calorimeter hits), each described
by a feature vector $\mathbf{x}\in X$. The target of the model is a set of
stable particles, each described by a particle type, charge, and continuous
four-momentum values, combined to form a target feature vector $\mathbf{y}\in
Y$. The goal of the model is, for each event, to reconstruct the set $Y$,
given the set $X$, i.e. $\Phi(X)\rightarrow Y^{\prime}\simeq Y$. To make the
set-to-set translation tractable with ML, and to be able to treat particle
multiplicities in the range of $|X|,|Y|\simeq 10^{4}$ with a single model
pass, each target particle is associated with a single unique input element.
The target particles are assigned to input elements by an injective, non-
surjective function based on a physics prior. The general approach is based on
the object condensation (OC) [14], which includes a potential between inputs
and outputs. Here, we only associate charged particles to tracks and neutral
particles to the highest-energy calorimeter cluster (or hit), as a tradeoff
between expressiveness and computational cost. The event reconstruction loss
can be written as a sum over the input elements in each event

$L(Y,Y^{\prime})=\sum_{i}L_{\mathrm{cls}}(y_{i},y^{\prime}_{i})+L_{\mathrm{reg}}(y_{i},y^{\prime}_{i}).$
(1)

Here, $L_{\mathrm{cls}}(y_{i},y^{\prime}_{i})$ is a classification loss
between the predicted and target particle type and charge labels. The
classification task is imbalanced, so we use the focal loss for particle type
classification, which assigns greater weight to samples that are difficult to
classify. Similarly, $L_{\mathrm{reg}}(y_{i},y^{\prime}_{i})$ is a regression
loss for the momentum components, where we use the Huber loss to reduce the
effect of outliers.

Figure 7: Top: one layer of the LSH-based GNN. Bottom: one layer of the
kernel-based self-attention, figure adapted from [27].

The LSH-based GNN is constructed as follows. We define a graph building layer,
which receives a set of input feature vectors $\mathbf{x}\in X$, assigns each
input vector to a bin with a learnable function $f(\mathbf{x})\rightarrow b\in
B$, such that the number of bins is much smaller than the input multiplicity
$|B|\ll|X|$. In each bin of average size $N_{B}=|X|/|B|$, a fully connected
graph is built between all elements of that bin using a learnable function
$g(x_{i},x_{j})\rightarrow w_{ij}$. Therefore, the output of the graph
building layer is a learnable association matrix $W=w_{bij}$ of size
$|B|N_{B}^{2}$, avoiding the allocation of a matrix of size $|X|^{2}$. This
matrix can be used for further GNN operations, such as message passing. The
graph structure defined by the association matrix consists of $|B|$ disjoint
graphs of $N_{B}$ elements each, which is not a major limitation because the
graph building layers can be stacked multiple times, each building different
disjoint graphs. From a physics point of view, it is reasonable to expect that
nearby (for some learnable definition of neighborhood) inputs should have a
stronger association than input elements in highly separated parts of the
detector. This graph building layer can be implemented using elementary matrix
operations in a fully batched and differentiable way using TensorFlow [53,
54]. This is similar to manually partitioning and restitching the event, but
achieved directly in the neural network (NN) model using fully differentiable
operations, rather than with a manual heuristic. The GNN is based on stacked
layers of graph building and convolution, with the number of layers being a
configurable hyperparameter.

The alternative kernel-based transformer model avoids quadratic scaling using
the following approach. For $N$ elements, given queries
$Q\in\mathbb{R}^{N{\times}d_{q}}$ and keys $K\in\mathbb{R}^{N{\times}d_{k}}$,
the attention mechanism encodes a value matrix
$V\in\mathbb{R}^{N{\times}d_{v}}$ as

$\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\left(\frac{QK^{\intercal}}{\sqrt{d_{k}}}\right)V.$
(2)

Here, the $\mathrm{softmax}(QK^{\intercal})$ operation creates a full
$N{\times}N$ matrix. As in Ref. [27], we define a transformation
$\psi(\mathbf{x})\rightarrow\mathbf{x^{\prime}}$ that transforms an input
feature map $\mathbf{x}$ using predetermined random projections to a new
feature space $\mathbf{x^{\prime}}$. For a sufficiently large number of random
projections, attention can be approximated as

$\mathrm{Attn}(Q,K,V)\simeq Q^{\prime}(K^{\prime\intercal}V)$ (3)

where $Q^{\prime}$ and $K^{\prime}$ are the query and key matrices after the
random feature mapping $\psi$, respectively. Allocation of the entire
$N{\times}N$ matrix is avoided, since the order of operations is changed to
first multiply keys with values and then subsequently with queries. In the
special case of self-attention, $Q$, $K$, and $V$ are all derived from $X$
through a linear layer, and the self-attention mechanism can be seen as an
analogy to graph building and message passing. A visual overview of the
supervised learning setup, as well as the two alternative model
configurations, is shown in Fig. 7.

Several steps were taken to ensure efficient training on large-scale datasets.
All relevant matrix operations for the GNN implementation support efficient
batched application using native TensorFlow kernels. Given the varying size of
events, the model implements variable-sized batching, supporting batching
events at fixed batch size increments, with the batch size inversely
proportional to the size of the events in the batch. This was necessary to
reduce the amount of zero-padding, and to enable automatic kernel compilation,
so that a limited number of kernels have to be compiled for each bucket size
increment. Since the total size of the datasets can reach several terabytes,
efficient larger-than-RAM training is enabled through dynamic dataset loading
and interleaving using the TensorFlow datasets library. The model supports
mixed-precision training in bfloat16 and float16. The training in bfloat16
results in loss values that are stable and largely compatible with that of
float32, while we find that the dynamic loss scaling for float16 results in
NaN gradients and does not converge. At the time of writing, software support
for bfloat16 in TensorFlow is limited, such that the operations are not placed
on tensor cores and the use of bfloat16 does not result in increased training
throughput, while training with float16 results in moderate speedups of 30–40%
for the GNN model. Improving training throughput using bfloat16 is the focus
of future work.

## Data and software availability

The datasets are available in [37], the results in [55], and the software used
for analysis in [56].

## Authorship statement

The authors contributed according to the contributor roles taxonomy (CRediT)
categories as follows. Joosep Pata: conceptualization, methodology, software,
validation, data curation, writing (original draft), funding acquisition,
resources. Eric Wulff: methodology, software, data curation, writing (review
and editing), resources. Farouk Mokhtar: methodology, software, investigation,
supervision. David Southwick: software, investigation. Mengke Zhang:
validation, investigation. Maria Girone: funding acquisition, resources.
Javier Duarte: conceptualization, software, resources, funding acquisition,
supervision, writing (review and editing).

## Acknowledgements

Eric Wulff and David Southwick were supported by CoE RAISE. The CoE RAISE
project has received funding from the European Union’s Horizon 2020 – Research
and Innovation Framework Programme H2020-INFRAEDI-2019-1 under grant agreement
no. 951733.

Joosep Pata was supported by the grant PSG864 of the Estonian Research
Council.

Javier Duarte, Farouk Mokhtar, and Mengke Zhang were supported by U.S.
Department of Energy (DOE) Award Nos. DE-SC0021187 and DE-SC0021396 (FAIR4HEP)
and U.S. National Science Foundation (NSF) Cooperative Agreement OAC-2117997
(A3D3). Farouk Mokhtar was also supported by a UCSD HDSI fellowship and an
IRIS-HEP fellowship through NSF Cooperative Agreement OAC-1836650. The use of
Intel Habana processors and additional support for Javier Duarte was
facilitated by NSF Award No. 2005369 (Voyager). We also thank Amit Majumdar,
Mahidhar Tatineni, Paul Rodriguez, and Marty Kandes at the San Diego
Supercomputer Center and Leonid Goldgeisser, Charles Chen, Chen Levkovich, and
Chenna Bayapureddy at Intel Habana for technical support.

The authors gratefully acknowledge the computing time granted through JARA on
the supercomputer JURECA [41] at Forschungszentrum Jülich. We acknowledge the
Estonian Scientific Compute Infrastructure and NICPB for awarding this project
access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking,
hosted by CSC (Finland) and the LUMI consortium through Estonian Scientific
Compute Infrastructure.

Datasets and software published following the findable, accessible,
interoperable, and reusable (FAIR) principles support open, transparent, and
reusable research, and therefore we thank our colleagues at the relevant
collaborations for releasing the Key4HEP, Marlin, EDM4HEP and CLICdp software
to the wider community, and encourage the wider adoption of the FAIR
principles.

## References

  * [1] CELLO Collaboration, “An analysis of the charged and neutral energy flow in $\mathrm{e^{+}e^{-}}$ hadronic annihilation at 34 GeV, and a determination of the QCD effective coupling constant”,  Phys. Lett. B 113 (1982) 427, 10.1016/0370-2693(82)90778-X.
  * [2] ALEPH Collaboration, “Performance of the ALEPH detector at LEP”,  Nucl. Instrum. Meth. A 360 (1995) 481, 10.1016/0168-9002(95)00138-7.
  * [3] DELPHI Collaboration, “Performance of the DELPHI detector”,  Nucl. Instrum. Meth. A 378 (1996) 57, 10.1016/0168-9002(96)00463-9.
  * [4] ZEUS Collaboration, “Measurement of the diffractive structure function F2(D(4)) at HERA”,  Eur. Phys. J. C 1 (1998) 81, 10.1007/s100520050063, arXiv:hep-ex/9709021.
  * [5] ZEUS Collaboration, “Measurement of the diffractive cross-section in deep inelastic scattering using ZEUS 1994 data”,  Eur. Phys. J. C 6 (1999) 43, 10.1007/PL00021606, arXiv:hep-ex/9807010.
  * [6] A. Bocci, S. Lami, S. Kuhlmann, and G. Latino, “Study of jet energy resolution at CDF”,  Int. J. Mod. Phys. A 16S1A (2001) 255, 10.1142/S0217751X01006632.
  * [7] A. L. Connolly, “A Search for Supersymmetric Higgs Bosons in the Di-tau Decay Mode in $p\bar{p}$ Collisions at 1.8 TeV”.  PhD thesis, UC Berkeley, 2003.  10.2172/15017134.
  * [8] CDF Collaboration, “Measurement of $\sigma(p\bar{p}\to Z).{\rm Br}(Z\to 2\tau)$ in $p\bar{p}$ collisions at $\sqrt{s}=1.96$ TeV”,  Phys. Rev. D 75 (2007) 092004, 10.1103/PhysRevD.75.092004.
  * [9] D0 Collaboration, “Measurement of $\sigma(p\bar{p}\to Z+X)$ Br($Z\to\tau^{+}\tau^{-}$) at $\sqrt{s}=1.96\,\text{TeV}$”,  Phys. Lett. B 670 (2009) 292, 10.1016/j.physletb.2008.11.010, arXiv:0808.1306.
  * [10] CMS Collaboration, “The CMS Experiment at the CERN LHC”,  JINST 3 (2008) S08004, 10.1088/1748-0221/3/08/S08004.
  * [11] CMS Collaboration, “Particle-flow reconstruction and global event description with the CMS detector”,  JINST 12 (2017), no. 10, P10003, 10.1088/1748-0221/12/10/P10003, arXiv:1706.04965.
  * [12] ATLAS Collaboration, “Jet reconstruction and performance using particle flow with the ATLAS detector”,  Eur. Phys. J. C 77 (2017) 466, 10.1140/epjc/s10052-017-5031-2, arXiv:1703.10485.
  * [13] H1 Collaboration, “Measurement of charged particle multiplicity distributions in DIS at HERA and its implication to entanglement entropy of partons”,  Eur. Phys. J. C 81 (2021), no. 3, 212, 10.1140/epjc/s10052-021-08896-1, arXiv:2011.01812.
  * [14] J. Kieseler, “Object condensation: one-stage grid-free multi-object reconstruction in physics detectors, graph and image data”,  Eur. Phys. J. C 80 (2020), no. 9, 886, 10.1140/epjc/s10052-020-08461-2, arXiv:2002.03605.
  * [15] J. Pata et al., “MLPF: Efficient machine-learned particle-flow reconstruction using graph neural networks”,  Eur. Phys. J. C 81 (2021), no. 5, 381, 10.1140/epjc/s10052-021-09158-w, arXiv:2101.08578.
  * [16] CMS Collaboration, J. Pata et al., “Machine Learning for Particle Flow Reconstruction at CMS”, in  20th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, L. Teodorescu, J. Goh, E. Brondolin, and J. Ngadiuba, eds., volume 2438, p. 012100.  2023\.  arXiv:2203.00330.  10.1088/1742-6596/2438/1/012100.
  * [17] F. Mokhtar et al., “Progress towards an improved particle flow algorithm at CMS with machine learning”, 3, 2023.  10.48550/arXiv.2303.17657.
  * [18] F. A. Di Bello et al., “Reconstructing particles in jets using set transformer and hypergraph prediction networks”,  Eur. Phys. J. C 83 (2023), no. 7, 596, 10.1140/epjc/s10052-023-11677-7, arXiv:2212.01328.
  * [19] CMS Collaboration, S. Bhattacharya et al., “GNN-based end-to-end reconstruction in the CMS Phase 2 High-Granularity Calorimeter”, in  20th International Workshop on Advanced Computing and Analysis Techniques in Physics Research, L. Teodorescu, J. Goh, E. Brondolin, and J. Ngadiuba, eds., volume 2438, p. 012090.  2023\.  arXiv:2203.01189.  10.1088/1742-6596/2438/1/012090.
  * [20] F. Gaede et al., “Edm4hep and podio-the event data model of the key4hep project and its implementation”, in  Eur. Phys. J. Web Conf., C. Biscarat et al., eds., volume 251, p. 03026.  2021\.  10.1051/epjconf/202125103026.
  * [21] M. D. Wilkinson et al., “The FAIR guiding principles for scientific data management and stewardship”,  Sci. Data 3 (2016) 160018, 10.1038/sdata.2016.18.
  * [22] Y. Chen et al., “A FAIR and AI-ready Higgs boson decay dataset”,  Sci. Data 9 (2022) 31, 10.1038/s41597-021-01109-0, arXiv:2108.02214.
  * [23] J. Duarte et al., “FAIR AI Models in High Energy Physics”, 12, 2022. arXiv:2212.05081. Submitted to _Mach. Learn. Sci. Technol_.
  * [24] T. Brown et al., “Language models are few-shot learners”, in  Advances in Neural Information Processing Systems, H. Larochelle et al., eds., volume 33, p. 1877.  Curran Associates, Inc., 2020.  arXiv:2005.14165.
  * [25] H. Touvron et al., “LLaMA: Open and efficient foundation language models”, 2023. arXiv:2302.13971.
  * [26] N. Kitaev, Łukasz Kaiser, and A. Levskaya, “Reformer: The efficient transformer”, in  ICLR 2020.  2020\.  arXiv:2001.04451.
  * [27] K. Choromanski et al., “Rethinking attention with performers”, in  ICLR 2021.  2020\.  arXiv:2009.14794.
  * [28] C. Bierlich et al., “A comprehensive guide to the physics and usage of PYTHIA 8.3”,  SciPost Phys. Codeb. (2022) 10.21468/SciPostPhysCodeb.8, arXiv:2203.11601.
  * [29] G. Ganis, C. Helsens, and V. Völkl, “Key4hep, a framework for future HEP experiments and its use in FCC”,  Eur. Phys. J. Plus 137 (2022), no. 1, 149, 10.1140/epjp/s13360-021-02213-1, arXiv:2111.09874.
  * [30] CLIC Collaboration, “CLICdet: The post-CDR CLIC detector model”, CLIC Detector and Physics Study Note, 2017\.
  * [31] CLICdp Collaboration, “A detector for CLIC: main parameters and performance”, 12, 2018. arXiv:1812.07337.
  * [32] F. Gaede, “Marlin and LCCD: Software tools for the ILC”,  Nucl. Instrum. Meth. A 559 (2006) 177, 10.1016/j.nima.2005.11.138.
  * [33] J. S. Marshall and M. A. Thomson, “The Pandora software development kit for particle flow calorimetry”,  J. Phys. Conf. Ser. 396 (2012) 022034, 10.1088/1742-6596/396/2/022034.
  * [34] J. S. Marshall, A. Münnich, and M. A. Thomson, “Performance of particle flow calorimetry at CLIC”,  Nucl. Instrum. Meth. A 700 (2013) 153, 10.1016/j.nima.2012.10.038, arXiv:1209.4039.
  * [35] J. S. Marshall and M. A. Thomson, “The Pandora software development kit for pattern recognition”,  Eur. Phys. J. C 75 (2015) 439, 10.1140/epjc/s10052-015-3659-3, arXiv:1506.05348.
  * [36] “TensorFlow Datasets, a collection of ready-to-use datasets”. https://www.tensorflow.org/datasets.
  * [37] J. Pata et al., “Simulated datasets for detector and particle flow reconstruction: CLIC detector”, 2023.  10.5281/zenodo.8260741.
  * [38] M. Cacciari, G. P. Salam, and G. Soyez, “The anti-$k_{\mathrm{T}}$ jet clustering algorithm”,  JHEP 04 (2008) 063, 10.1088/1126-6708/2008/04/063, arXiv:0802.1189.
  * [39] M. Cacciari, G. P. Salam, and G. Soyez, “FastJet user manual”,  Eur. Phys. J. C 72 (2012) 1896, 10.1140/epjc/s10052-012-1896-2, arXiv:1111.6097.
  * [40] T. Kramer, “Track parameters in LCIO”, Technical Report LC-DET-2006-004, 2006.  10.3204/PHPPUBDB-2955.
  * [41] Jülich Supercomputing Centre, “JURECA: Data Centric and Booster Modules implementing the Modular Supercomputing Architecture at Jülich Supercomputing Centre”,  J. Large-Scale Res. Facilities 7 (2021), no. A182, 10.17815/jlsrf-7-182.
  * [42] E. Wulff, M. Girone, and J. Pata, “Hyperparameter optimization of data-driven AI models on HPC systems”,  Journal of Physics: Conference Series 2438 (Feb, 2023) 012092, 10.1088/1742-6596/2438/1/012092.
  * [43] L. Li et al., “Massively parallel hyperparameter tuning”,  CoRR abs/1810.05934 (2018) arXiv:1810.05934.
  * [44] J. L. Bentley, “Multidimensional binary search trees used for associative searching”,  Commun. ACM 18 (1975) 509, 10.1145/361002.361007.
  * [45] T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8 (): 8-bit matrix multiplication for transformers at scale”, 2022\.  10.48550/arXiv.2208.07339.
  * [46] Flatiron Institute, “CoreSite Cluster”, 2023\.  https://www.simonsfoundation.org/2022/11/14/new-flatiron-institute-supercomputer-the-most-power-efficient-ever-built/.
  * [47] LUMI Consortium, “LUMI Supercomputer”, 2023\.  https://www.lumi-supercomputer.eu/.
  * [48] R. E. Amaro et al., “Voyager – An Innovative Computational Resource for Artificial Intelligence & Machine Learning Applications in Science and Engineering”, in  PEARC ’23: Practice and Experience in Advanced Research Computing, S. Cleveland et al., eds.  2023\.  10.1145/3569951.3597597.
  * [49] A. Sergeev and M. D. Balso, “Horovod: fast and easy distributed deep learning in TensorFlow”, 2018. arXiv:1802.05799.
  * [50] T. Huang et al., “Learning to measure the point cloud reconstruction loss in a representation space”, in  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12208–12217.  2023\.  10.1109/CVPR52729.2023.01175.
  * [51] T. Dao et al., “FlashAttention: Fast and memory-efficient exact attention with IO-awareness”, in  Advances in Neural Information Processing Systems, S. Koyejo et al., eds., volume 35, p. 16344.  Curran Associates, Inc., 2022.  arXiv:2205.14135.
  * [52] T. Dao, “FlashAttention-2: Faster attention with better parallelism and work partitioning”, 2023. arXiv:2307.08691.
  * [53] M. Abadi et al., “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems”, 2015\. arXiv:1603.04467.
  * [54] TensorFlow Developers, “TensorFlow”, 2023.  10.5281/zenodo.7753622.
  * [55] J. Pata et al., “MLPF results on the simulated CLIC dataset, v2023.09”, September, 2023.  10.5281/zenodo.8328683, https://doi.org/10.5281/zenodo.8328683.
  * [56] J. Pata et al., “jpata/particleflow: MLPF training with CLIC simulation”, August, 2023.  10.5281/zenodo.8290119, https://doi.org/10.5281/zenodo.8290119.


$\displaystyle q_{p,x}(x_{i},z_{k},t)$ $\displaystyle=\dfrac{V_{p}}{\Delta
x_{bin}\Delta
z_{bin}}\displaystyle\sum_{l=1}^{N_{p}}u_{p}^{(l)}(t)\;I_{(l)}^{(i,k)}(t)$
(13a) $\displaystyle q_{p,z}(x_{i},z_{k},t)$
$\displaystyle=\dfrac{V_{p}}{\Delta x_{bin}\Delta
z_{bin}}\displaystyle\sum_{l=1}^{N_{p}}w_{p}^{(l)}(t)\;I_{(l)}^{(i,k)}(t).$
(13b)

where $q_{p,x}$, $q_{p,z}$ are the streamwise and spanwise components of the
particle flux vector $\boldsymbol{q}_{p}$. $V_{p}=(\pi/6)D^{3}$ denotes the
volume of a single spherical particle and $u_{p}^{(l)}$, $w_{p}^{(l)}$ are the
streamwise and spanwise component of the particle velocity vector
$\boldsymbol{u}_{p}^{(l)}$ of particle $l$, respectively. The bin centers are
located at $x_{i}=((i-1)+i)\Delta x_{bin}/2$ and $z_{k}=((k-1)+k)\Delta
z_{bin}/2$. $I_{(l)}^{(i,k)}(t)$ is an indicator function that is unity if the
$l$-th particle is centered in the $(i,k)$-th bin, i.e.

$I_{(l)}^{(i,k)}(t)=\begin{cases}1\quad\;\parbox[t]{260.17464pt}{if
$(i-1)\Delta x_{bin}\leq x_{p}^{(l)}(t)\leq i\Delta x_{bin}\\\
\;\wedge\;(k-1)\Delta z_{bin}\leq z_{p}^{(l)}(t)\leq k\Delta
z_{bin}$}\\\\[11.62495pt] 0\quad\text{else.}\end{cases}$ (14)

Note that the particle flux is an integral measure and as such does not
reflect the inhomogeneous distribution of particle transport intensity over
the wall-normal direction. Let us therefore additionally introduce the
streamwise particle flux density as

$\langle{\phi u_{p}}\rangle_{xz}(y_{j},t)=\frac{V_{p}}{L_{x}L_{z}\Delta
y_{bin}}\sum_{l=1}^{N_{p}}u_{p}^{(l)}(t)\;I_{(l)}^{(j)}(t),$ (15)

with the wall-normal bin centers $y_{j}=((j-1)+j)\Delta y_{bin}/2$ ($1\leq
j\leq N_{y,bin}$) and the indicator function $I_{(l)}^{(j)}(t)$ defined as

$I_{(l)}^{(j)}(t)=\begin{cases}1\quad\;\parbox[t]{260.17464pt}{if $(j-1)\Delta
y_{bin}\leq y_{p}^{(l)}(t)\leq j\Delta y_{bin}$}\\\\[11.62495pt]
0\quad\text{else}.\end{cases}$ (16)

In a similar way as in the previous definitions, we have subdivided the wall-
normal box length into $N_{y,bin}$ bins with however smaller bin height
$\Delta y_{bin}\approx 0.5D$, this time spanning over the entire box length
$L_{x}$ and width $L_{z}$. By definition, the following relation holds
(Lobkovsky et al., 2008; Chiodi et al., 2014):

$\langle{q_{p,x}}\rangle_{xzt}=\int_{y=0}^{L_{y}}\langle{\phi
u_{p}}\rangle_{xzt}\mathrm{d}y.$ (17)

## References

  * Adrian (2007) R. J. Adrian.  Hairpin vortex organization in wall turbulence.  _Phys. Fluids_ , 19(4):041301, 2007.  doi:10.1063/1.2717527.  URL https://doi.org/10.1063/1.2717527.
  * Adrian and Marusic (2012) R. J. Adrian and I. Marusic.  Coherent structures in flow over hydraulic engineering surfaces.  _J. Hydraul. Res._ , 50(5):451–464, 2012.  doi:10.1080/00221686.2012.729540.  URL https://doi.org/10.1080/00221686.2012.729540.
  * Adrian et al. (2000) R. J. Adrian, C. D. Meinhart, and C. D. Tomkins.  Vortex organization in the outer region of the turbulent boundary layer.  _J. Fluid Mech._ , 422:1–54, 2000.  doi:10.1017/S0022112000001580.  URL https://doi.org/10.1017/S0022112000001580.
  * Allen (1968) J. R. L. Allen.  The nature and origin of bed-form hierarchies.  _Sedimentology_ , 10(3):161–182, 1968.  doi:10.1111/j.1365-3091.1968.tb01110.x.  URL https://doi.org/10.1111/j.1365-3091.1968.tb01110.x.
  * Anderson (2019) W. Anderson.  Non-periodic phase-space trajectories of roughness-driven secondary flows in high-${R}e_{\tau}$ boundary layers and channels.  _J. Fluid Mech._ , 869:27–84, 2019.  doi:10.1017/jfm.2019.244.  URL https://doi.org/10.1017/jfm.2019.244.
  * Bagherimiyab and Lemmin (2018) F. Bagherimiyab and U. Lemmin.  Large-scale coherent flow structures in rough-bed open-channel flow observed in fluctuations of three-dimensional velocity, skin friction and bed pressure.  _J. Hydraul. Res._ , 56(6):806–824, 2018.  doi:10.1080/00221686.2017.1416686.  URL https://doi.org/10.1080/00221686.2017.1416686.
  * Bagnold (1956) R. A. Bagnold.  The flow of cohesionless grains in fluids.  _Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences_ , 249(964):235–297, 1956.
  * Barros and Christensen (2014) J. M. Barros and K. T. Christensen.  Observations of turbulent secondary flows in a rough-wall boundary layer.  _J. Fluid Mech._ , 748:R1, 2014.  doi:10.1017/jfm.2014.218.  URL https://doi.org/10.1017/jfm.2014.218.
  * Bauer et al. (2021) C. Bauer, Y. Sakai, and M. Uhlmann.  DNS of plane open channel flow: Scaling near the free-slip boundary and evidence of very-large-scale motions.  _in preparation_ , 2021.
  * Bradshaw (1987) P. Bradshaw.  Turbulent secondary flows.  _Ann. Rev. Fluid Mech._ , 19(1):53–74, 1987\.  doi:10.1146/annurev.fl.19.010187.000413.  URL https://doi.org/10.1146/annurev.fl.19.010187.000413.
  * Cameron et al. (2008) S. M. Cameron, V. I. Nikora, and S. E. Coleman.  Double-averaged velocity and stress distributions for hydraulically-smooth and transitionally-rough turbulent flows.  _Acta Geophysica_ , 56(3):642–653, 2008.  doi:10.2478/s11600-008-0028-8.  URL https://doi.org/10.2478/s11600-008-0028-8.
  * Cameron et al. (2020) S. M. Cameron, V. I. Nikora, and M. J. Witz.  Entrainment of sediment particles by very-large-scale motions.  _J. Fluid Mech._ , 888:A7, 2020.  doi:10.1017/jfm.2020.24.  URL https://doi.org/10.1017/jfm.2020.24.
  * Casey (1935) H. J. Casey.  _Über Geschiebebewegung_.  Eigenverlag der Preussischen Versuchsanstalt für Wasserbau und Schiffbau, Berlin, 1935.
  * Chan-Braun et al. (2011) C. Chan-Braun, M. García-Villalba, and M. Uhlmann.  Force and torque acting on particles in a transitionally rough open-channel flow.  _J. Fluid Mech._ , 684:441–474, 2011.  doi:10.1017/jfm.2011.311.  URL https://doi.org/10.1017/jfm.2011.311.
  * Chiodi et al. (2014) F. Chiodi, P. Claudin, and B. Andreotti.  A two-phase flow model of sediment transport: transition from bedload to suspended load.  _J. Fluid Mech._ , 755:561–581, 2014.  doi:10.1017/jfm.2014.422.  URL https://doi.org/10.1017/jfm.2014.422.
  * Chung et al. (2018) D. Chung, J. P. Monty, and N. Hutchins.  Similarity and structure of wall turbulence with lateral wall shear stress variations.  _J. Fluid Mech._ , 847:591––613, 2018.  doi:10.1017/jfm.2018.336.  URL https://doi.org/10.1017/jfm.2018.336.
  * Coleman and Nikora (2011) S. E. Coleman and V. I. Nikora.  Fluvial dunes: initiation, characterization, flow structure.  _Earth Surface Processes and Landforms_ , 36(1):39–57, 2011.  doi:10.1002/esp.2096.  URL https://doi.org/10.1002/esp.2096.
  * Colombini (1993) M. Colombini.  Turbulence-driven secondary flows and formation of sand ridges.  _J. Fluid Mech._ , 254:701–719, 1993.  doi:10.1017/S0022112093002319.  URL https://doi.org/10.1017/S0022112093002319.
  * Colombini and Parker (1995) M. Colombini and G. Parker.  Longitudinal streaks.  _J. Fluid Mech._ , 304:161–183, 1995.  doi:10.1017/S0022112095004381.  URL https://doi.org/10.1017/S0022112095004381.
  * Culbertson (1967) J. K. Culbertson.  Evidence of secondary circulation in an alluvial channel.  _US Geological Survey Professional Paper_ , pages D214–D216, 1967\.
  * Del Álamo and Jiménez (2003) J. C. Del Álamo and J. Jiménez.  Spectra of the very large anisotropic scales in turbulent channels.  _Phys. Fluids_ , 15(6):L41–L44, 2003.  doi:10.1063/1.1570830.  URL https://doi.org/10.1063/1.1570830.
  * Del Álamo et al. (2006) J. C. Del Álamo, J. Jiménez, P. Zandonade, and R. D. Moser.  Self-similar vortex clusters in the turbulent logarithmic region.  _J. Fluid Mech._ , 561:329–358, 2006.  doi:10.1017/S0022112006000814.  URL https://doi.org/10.1017/S0022112006000814.
  * Edelsbrunner and Mücke (1994) H. Edelsbrunner and E. P. Mücke.  Three-dimensional alpha shapes.  _ACM Transactions on Graphics (TOG)_ , 13(1):43–72, 1994.
  * Exner (1925) F. M. Exner.  Über die wechselwirkung zwischen wasser und geschiebe in flüssen.  _Akad. Wiss. Wien Math. Naturwiss. Klasse_ , 134(2a):165–204, 1925.
  * Flores and Jiménez (2006) O. Flores and J. Jiménez.  Effect of wall-boundary disturbances on turbulent channel flows.  _J. Fluid Mech._ , 566:357–376, 2006.  doi:10.1017/S0022112006001534.  URL https://doi.org/10.1017/S0022112006001534.
  * Flores and Jiménez (2010) O. Flores and J. Jiménez.  Hierarchy of minimal flow units in the logarithmic layer.  _Phys. Fluids_ , 22(7):071704, 2010.  doi:10.1063/1.3464157.  URL https://doi.org/10.1063/1.3464157.
  * Flores et al. (2007) O. Flores, J. Jiménez, and J. C. Del Álamo.  Vorticity organization in the outer layer of turbulent channels with disturbed walls.  _J. Fluid Mech._ , 591:145, 2007.  doi:10.1017/S0022112007008506.  URL https://doi.org/10.1017/S0022112007008506.
  * Franklin and Charru (2011) E. d. M. Franklin and F. Charru.  Subaqueous barchan dunes in turbulent shear flow. part 1. dune motion.  _J. Fluid Mech._ , 675:199–222, 2011.  doi:10.1017/S0022112011000139.  URL https://doi.org/10.1017/S0022112011000139.
  * Goldstein and Tuan (1998) D. Goldstein and T.-C. Tuan.  Secondary flow induced by riblets.  _J. Fluid Mech._ , 363:115–151, 1998.  doi:10.1017/S0022112098008921.  URL https://doi.org/10.1017/S0022112098008921.
  * Gyr and Schmid (1997) A. Gyr and A. Schmid.  Turbulent flows over smooth erodible sand beds in flumes.  _J. Hydraul. Res._ , 35(4):525–544, 1997.  doi:10.1080/00221689709498409.  URL https://doi.org/10.1080/00221689709498409.
  * Hamilton et al. (1995) J. M. Hamilton, J. Kim, and F. Waleffe.  Regeneration mechanisms of near-wall turbulence structures.  _J. Fluid Mech._ , 287(1):317–348, 1995.  doi:10.1017/S0022112095000978.  URL https://doi.org/10.1017/S0022112095000978.
  * Hinze (1967) J. O. Hinze.  Secondary currents in wall turbulence.  _Phys. Fluids_ , 10(9):S122–S125, 1967.  doi:10.1063/1.1762429.  URL https://doi.org/10.1063/1.1762429.
  * Hinze (1973) J. O. Hinze.  Experimental investigation on secondary currents in the turbulent flow through a straight conduit.  _Applied Scientific Res._ , 28(1):453–465, 1973\.  doi:10.1007/BF00413083.  URL https://doi.org/10.1007/BF00413083.
  * Hwang and Lee (2018) H. G. Hwang and J. H. Lee.  Secondary flows in turbulent boundary layers over longitudinal surface roughness.  _Phys. Rev. Fluids_ , 3(1):014608, 2018.  doi:10.1103/PhysRevFluids.3.014608.  URL https://doi.org/10.1103/PhysRevFluids.3.014608.
  * Ikeda (1981) S. Ikeda.  Self-formed straight channels in sandy beds.  _J. Hydraul. Div._ , 107(4):389–406, 1981.
  * Jeong and Hussain (1995) J. Jeong and F. Hussain.  On the identification of a vortex.  _J. Fluid Mech._ , 285:69–94, 1995.  doi:10.1017/S0022112095000462.  URL https://doi.org/10.1017/S0022112095000462.
  * Jeong et al. (1997) J. Jeong, F. Hussain, W. Schoppa, and J. Kim.  Coherent structures near the wall in a turbulent channel flow.  _J. Fluid Mech._ , 332(185-214):188, 1997.  doi:10.1017/S0022112096003965.  URL https://doi.org/10.1017/S0022112096003965.
  * Jiménez (1998) J. Jiménez.  The largest scales of turbulent wall flows.  _CTR Annual Research Briefs_ , pages 137–154, 1998.
  * Jiménez (2004) J. Jiménez.  Turbulent flows over rough walls.  _Annu. Rev. Fluid Mech._ , 36:173–196, 2004.  doi:10.1146/annurev.fluid.36.050802.122103.  URL https://doi.org/10.1146/annurev.fluid.36.050802.122103.
  * Jiménez (2013a) J. Jiménez.  Near-wall turbulence.  _Phys. Fluids_ , 25(10):101302, 2013a.  doi:10.1063/1.4824988.
  * Jiménez (2013b) J. Jiménez.  How linear is wall-bounded turbulence?  _Phys. Fluids_ , 25(11):110814, 2013b.  doi:10.1063/1.4819081.  URL https://doi.org/10.1063/1.4819081.
  * Jiménez (2018) J. Jiménez.  Coherent structures in wall-bounded turbulence.  _J. Fluid Mech._ , 842, 2018.  doi:10.1017/jfm.2018.144.  URL https://doi.org/10.1017/jfm.2018.144.
  * Jiménez and Moin (1991) J. Jiménez and P. Moin.  The minimal flow unit in near-wall turbulence.  _J. Fluid Mech._ , 225:213–240, 1991.  doi:10.1017/S0022112091002033.  URL https://doi.org/10.1017/S0022112091002033.
  * Jiménez and Pinelli (1999) J. Jiménez and A. Pinelli.  The autonomous cycle of near-wall turbulence.  _J. Fluid Mech._ , 389:335–359, 1999.  doi:10.1017/S0022112099005066.  URL https://doi.org/10.1017/S0022112099005066.
  * Jiménez et al. (2001) J. Jiménez, M. Uhlmann, A. Pinelli, and G. Kawahara.  Turbulent shear flow over active and passive porous surfaces.  _J. Fluid Mech._ , 442:89–117, 2001.  doi:10.1017/S0022112001004888.  URL https://doi.org/10.1017/S0022112001004888.
  * Jiménez et al. (2004) J. Jiménez, J. C. Del Álamo, and O. Flores.  The large-scale dynamics of near-wall turbulence.  _J. Fluid Mech._ , 505:179–199, 2004.  doi:10.1017/S0022112004008389.  URL https://doi.org/10.1017/S0022112004008389.
  * Karcz (1967) I. Karcz.  Harrow marks, current-aligned sedimentary structures.  _The Journal of Geology_ , 75(1):113–121, 1967\.  doi:10.1086/627235.  URL https://doi.org/10.1086/627235.
  * Kawahara et al. (2012) G. Kawahara, M. Uhlmann, and L. van Veen.  The significance of simple invariant solutions in turbulent flows.  _Ann. Rev. Fluid Mech._ , 44:203–225, 2012.  doi:10.1146/annurev-fluid-120710-101228.  URL https://doi.org/10.1146/annurev-fluid-120710-101228.
  * Kevin et al. (2017) K. Kevin, J. P. Monty, H. Bai, G. Pathikonda, B. Nugroho, J. M. Barros, K. T. Christensen, and N. Hutchins.  Cross-stream stereoscopic particle image velocimetry of a modified turbulent boundary layer over directional surface pattern.  _J. Fluid Mech._ , 813:412–435, 2017.  doi:10.1017/jfm.2016.879.  URL https://doi.org/10.1017/jfm.2016.879.
  * Kevin et al. (2019a) K. Kevin, J. Monty, and N. Hutchins.  Turbulent structures in a statistically three-dimensional boundary layer.  _J. Fluid Mech._ , 859:543–565, 2019a.  doi:10.1017/jfm.2018.814.  URL https://doi.org/10.1017/jfm.2018.814.
  * Kevin et al. (2019b) K. Kevin, J. Monty, and N. Hutchins.  The meandering behaviour of large-scale structures in turbulent boundary layers.  _J. Fluid Mech._ , 865:R1, 2019b.  doi:10.1017/jfm.2019.131.  URL https://doi.org/10.1017/jfm.2019.131.
  * Kidanemariam (2015) A. G. Kidanemariam.  _The formation of patterns in subaqueous sediment_.  PhD thesis, Institute for Hydromechanics, Karlsruhe Institute for Technology (KIT), 2015.  URL https://doi.org/10.5445/KSP/1000054986.
  * Kidanemariam and Uhlmann (2014a) A. G. Kidanemariam and M. Uhlmann.  Direct numerical simulation of pattern formation in subaqueous sediment.  _J. Fluid Mech._ , 750, 2014a.  doi:10.1017/jfm.2014.284.  URL https://doi.org/10.1017/jfm.2014.284.
  * Kidanemariam and Uhlmann (2014b) A. G. Kidanemariam and M. Uhlmann.  Interface-resolved direct numerical simulation of the erosion of a sediment bed sheared by laminar channel flow.  _Int. J. Multiphase Flow_ , 67:174–188, 2014b.  doi:10.1016/j.ijmultiphaseflow.2014.08.008.  URL https://doi.org/10.1016/j.ijmultiphaseflow.2014.08.008.
  * Kidanemariam and Uhlmann (2017) A. G. Kidanemariam and M. Uhlmann.  Formation of sediment patterns in channel flow: minimal unstable systems and their temporal evolution.  _J. Fluid Mech._ , 818:716–743, 2017.  doi:10.1017/jfm.2017.147.  URL https://doi.org/10.1017/jfm.2017.147.
  * Kidanemariam et al. (2013) A. G. Kidanemariam, C. Chan-Braun, T. Doychev, and M. Uhlmann.  Direct numerical simulation of horizontal open channel flow with finite-size, heavy particles at low solid volume fraction.  _New J. Phys._ , 15(2):025031, 2013.  doi:10.1088/1367-2630/15/2/025031.  URL https://doi.org/10.1088/1367-2630/15/2/025031.
  * Kidanemariam et al. (2021) A. G. Kidanemariam, M. Scherer, and M. Uhlmann.  Open channel flow over evolving subaqueous ripples.  _under review for publication in J. Fluid Mech._ , 2021.
  * Kim et al. (1987) J. Kim, P. Moin, and R. Moser.  Turbulence statistics in fully developed channel flow at low reynolds number.  _J. Fluid Mech._ , 177:133–166, 1987.  doi:10.1017/S0022112087000892.  URL https://doi.org/10.1017/S0022112087000892.
  * Langlois and Valance (2007) V. Langlois and A. Valance.  Initiation and evolution of current ripples on a flat sand bed under turbulent water flow.  _Eur. Phys. J. E. Soft Matter_ , 22(3):201–208, 2007.  doi:10.1140/epje/e2007-00023-0.  URL https://dx.doi.org/10.1140/epje/e2007-00023-0.
  * Lobkovsky et al. (2008) A. E. Lobkovsky, A. V. Orpe, R. Molloy, A. Kudrolli, and D. H. Rothman.  Erosion of a granular bed driven by laminar fluid flow.  _J. Fluid Mech._ , 605:47–58, 2008.  doi:10.1017/S0022112008001389.  URL https://doi.org/10.1017/S0022112008001389.
  * Lozano-Durán et al. (2012) A. Lozano-Durán, O. Flores, and J. Jiménez.  The three-dimensional structure of momentum transfer in turbulent channels.  _J. Fluid Mech._ , 694:100, 2012.  doi:10.1017/jfm.2011.524.  URL https://doi.org/10.1017/jfm.2011.524.
  * Lozano-Durán et al. (2016) A. Lozano-Durán, M. Holzner, and J. Jiménez.  Multiscale analysis of the topological invariants in the logarithmic region of turbulent channels at a friction reynolds number of 932.  _J. Fluid Mech._ , 803:356, 2016.  doi:10.1017/jfm.2016.504.  URL https://doi.org/10.1017/jfm.2016.504.
  * Mazzuoli and Uhlmann (2017) M. Mazzuoli and M. Uhlmann.  Direct numerical simulation of open-channel flow over a fully rough wall at moderate relative submergence.  _J. Fluid Mech._ , 824:722–765, 2017.  doi:10.1017/jfm.2017.371.  URL https://doi.org/10.1017/jfm.2017.371.
  * McLelland et al. (1999) S. J. McLelland, P. J. Ashworth, J. L. Best, and J. R. Livesey.  Turbulence and secondary flow over sediment stripes in weakly bimodal bed material.  _J. Hydr. Eng._ , 125(5):463–473, 1999.  doi:10.1061/(ASCE)0733-9429(1999)125:5(463).  URL https://doi.org/10.1061/(ASCE)0733-9429(1999)125:5(463).
  * Medjnoun et al. (2020) T. Medjnoun, C. Vanderwel, and B. Ganapathisubramani.  Effects of heterogeneous surface geometry on secondary flows in turbulent boundary layers.  _J. Fluid Mech._ , 886:1–36, 2020.  doi:10.1017/jfm.2019.1014.  URL https://doi.org/10.1017/jfm.2019.1014.
  * Mejia-Alvarez and Christensen (2010) R. Mejia-Alvarez and K. T. Christensen.  Low-order representations of irregular surface roughness and their impact on a turbulent boundary layer.  _Phys. Fluids_ , 22(1):015106, 2010.  doi:10.1063/1.3291076.  URL https://doi.org/10.1063/1.3291076.
  * Meyer-Peter and Müller (1948) E. Meyer-Peter and R. Müller.  Formulas for bed-load transport.  In _IAHSR 2nd meeting, Stockholm, appendix 2_. IAHR, 1948.
  * Mizuno and Jiménez (2013) Y. Mizuno and J. Jiménez.  Wall turbulence without walls.  _J. Fluid Mech._ , 723:429–455, 2013.  doi:10.1017/jfm.2013.137.  URL https://doi.org/10.1017/jfm.2013.137.
  * Motoori and Goto (2019) Y. Motoori and S. Goto.  Generation mechanism of a hierarchy of vortices in a turbulent boundary layer.  _J. Fluid Mech._ , 865:1085–1109, 2019.  doi:10.1017/jfm.2019.76.  URL https://doi.org/10.1017/jfm.2019.76.
  * Motoori and Goto (2021) Y. Motoori and S. Goto.  Hierarchy of coherent structures and real-space energy transfer in turbulent channel flow.  _J. Fluid Mech._ , 911, 2021.  doi:10.1017/jfm.2020.1025.  URL https://doi.org/10.1017/jfm.2020.1025.
  * Müller and Studerus (1979) A. Müller and X. Studerus.  Secondary flow in an open channel.  _Proceedings of the 18th International Congress of IAHR_ , 3:19–24, 1979.
  * Nezu (2005) I. Nezu.  Open-channel flow turbulence and its research prospect in the 21st century.  _J. Hydr. Eng._ , 131(4):229–246, 2005.  doi:10.1061/(ASCE)0733-9429(2005)131:4(229).  URL https://doi.org/10.1061/(ASCE)0733-9429(2005)131:4(229).
  * Nezu and Nakagawa (1984) I. Nezu and H. Nakagawa.  Cellular secondary currents in straight conduit.  _J. Hydr. Eng._ , 110(2):173–193, 1984.
  * Nezu and Nakagawa (1989) I. Nezu and H. Nakagawa.  Self forming mechanism of longitudinal sand ridges and troughs in fluvial open-channel flows.  In _Proceedings of the 23rd Congress_ , pages B65–B72. IAHR Madrid, 1989.
  * Nezu and Nakagawa (1993) I. Nezu and H. Nakagawa.  _Turbulence in open channels_.  IAHR/AIRH Monograph. Balkema, Rotterdam, The Netherlands, 1993.
  * Nezu and Rodi (1985) I. Nezu and W. Rodi.  Experimental study on secondary currents in open channel flow.  In _Proceedings of the 21st Congress_ , pages 115–119. IAHR Melbourne, 1985.
  * Nikuradse (1926) J. Nikuradse.  _Untersuchung über die Geschwindigkeitsverteilung in turbulenten Strömungen_.  PhD thesis, Georg-August-Universität Göttingen, 1926.
  * Peskin (2002) C. S. Peskin.  The immersed boundary method.  _Acta numerica_ , 11:479–517, 2002.  doi:10.1017/S0962492902000077.  URL https://doi.org/10.1017/S0962492902000077.
  * Pinelli et al. (2010) A. Pinelli, M. Uhlmann, A. Sekimoto, and G. Kawahara.  Reynolds number dependence of mean flow structure in square duct turbulence.  _J. Fluid Mech._ , 644:107–122, 2010.  doi:10.1017/S0022112010001849.  URL https://doi.org/10.1017/S0022112010001849.
  * Pope (2000) S. B. Pope.  _Turbulent Flows_.  Cambridge University Press, Cambridge, 5th edition, 2000.
  * Prandtl (1926) L. Prandtl.  Über die ausgebildete Turbulenz, Verhandlungen des II.Internationalen Kongresses für Technische Mechanik 1926.  In W. Tollmien, H. Schlichting, and H. Görtler, editors, _Ludwig Prandtl - Gesammelte Abhandlungen zur angewandten Mechanik, Hydro-und Aerodynamik: in 3 Teilen, (published 1961)_ , pages 736–751. Springer-Verlag, 1926.
  * Roy et al. (2004) A. G. Roy, T. Buffin-Belanger, H. Lamarre, and A. D. Kirkbride.  Size, shape and dynamics of large-scale turbulent flow structures in a gravel-bed river.  _J. Fluid Mech._ , 500:1–27, 2004.  doi:10.1017/S0022112003006396.  URL https://doi.org/10.1017/S0022112003006396.
  * Sakai (2016) Y. Sakai.  _Coherent structures and secondary motions in open duct flows_.  PhD thesis, Institute for Hydromechanics, Karlsruhe Institute for Technology (KIT), 2016.  URL http://dx.doi.org/10.5445/KSP/1000082066.
  * Scherer et al. (2020) M. Scherer, A. G. Kidanemariam, and M. Uhlmann.  On the scaling of the instability of a flat sediment bed with respect to ripple-like patterns.  _J. Fluid Mech._ , 900:A1, 2020.  doi:10.1017/jfm.2020.423.  URL https://doi.org/10.1017/jfm.2020.423.
  * Schoppa and Hussain (2002) W. Schoppa and F. Hussain.  Coherent structure generation in near-wall turbulence.  _J. Fluid Mech._ , 453:57–108, 2002.  doi:10.1017/S002211200100667X.  URL https://doi.org/10.1017/S002211200100667X.
  * Shvidchenko and Pender (2001) A. B. Shvidchenko and G. Pender.  Macroturbulent structure of open-channel flow over gravel beds.  _Water Resour. Res._ , 37(3):709–719, 2001.  URL https://doi.org/10.1029/2000WR900280.
  * Soulsby et al. (1997) R. L. Soulsby, R. J. S. Whitehouse, et al.  Threshold of sediment motion in coastal environments.  In _Pacific Coasts and Ports’ 97: Proceedings of the 13th Australasian Coastal and Ocean Engineering Conference and the 6th Australasian Port and Harbour Conference; Volume 1_ , page 145. Centre for Advanced Engineering, University of Canterbury, 1997.
  * Speziale (1982) C. G. Speziale.  On turbulent secondary flows in pipes of noncircular cross-section.  _Int. J. Eng. Science_ , 20(7):863–872, 1982\.  doi:10.1016/0020-7225(82)90008-8.  URL https://doi.org/10.1016/0020-7225(82)90008-8.
  * Speziale (1987) C. G. Speziale.  On nonlinear k-l and k-$\varepsilon$ models of turbulence.  _J. Fluid Mech._ , 178:459–475, 1987.
  * Stroh et al. (2016) A. Stroh, Y. Hasegawa, J. Kriegseis, and B. Frohnapfel.  Secondary vortices over surfaces with spanwise varying drag.  _J. Turbul._ , 17(12):1142–1158, 2016.  doi:10.1080/14685248.2016.1235277.  URL https://doi.org/10.1080/14685248.2016.1235277.
  * Stroh et al. (2020a) A. Stroh, K. Schäfer, P. Forooghi, and B. Frohnapfel.  Secondary flow and heat transfer in turbulent flow over streamwise ridges.  _Int. J. Heat and Fluid Flow_ , 81:108518, 2020a.  doi:10.1016/j.ijheatfluidflow.2019.108518.  URL https://doi.org/10.1016/j.ijheatfluidflow.2019.108518.
  * Stroh et al. (2020b) A. Stroh, K. Schäfer, B. Frohnapfel, and P. Forooghi.  Rearrangement of secondary flow over spanwise heterogeneous roughness.  _J. Fluid Mech._ , 885, 2020b.  doi:10.1017/jfm.2019.1030.  URL https://doi.org/10.1017/jfm.2019.1030.
  * Tamburrino and Gulliver (1999) A. Tamburrino and J. S. Gulliver.  Large flow structures in a turbulent open channel flow.  _J. Hydraul. Res._ , 37(3):363–380, 1999.  doi:10.1080/00221686.1999.9628253.  URL https://doi.org/10.1080/00221686.1999.9628253.
  * Tamburrino and Gulliver (2007) A. Tamburrino and J. S. Gulliver.  Free-surface visualization of streamwise vortices in a channel flow.  _Water Resour. Res._ , 43(11), 2007.  doi:10.1029/2007WR005988.  URL https://doi.org/10.1029/2007WR005988.
  * Toh and Itano (2005) S. Toh and T. Itano.  Interaction between a large-scale structure and near-wall structures in channel flow.  _J. Fluid Mech._ , 524:249, 2005.  doi:10.1017/S002211200400237X.  URL https://doi.org/10.1017/S002211200400237X.
  * Türk et al. (2014) S. Türk, G. Daschiel, A. Stroh, Y. Hasegawa, and B. Frohnapfel.  Turbulent flow over superhydrophobic surfaces with streamwise grooves.  _J. Fluid Mech._ , 747:186, 2014.  doi:10.1017/jfm.2014.137.  URL https://doi.org/10.1017/jfm.2014.137.
  * Uhlmann (2005) M. Uhlmann.  An immersed boundary method with direct forcing for the simulation of particulate flows.  _J. Comput. Phys._ , 209(2):448–476, 2005.  doi:10.1016/j.jcp.2005.03.017.  URL https://doi.org/10.1016/j.jcp.2005.03.017.
  * Uhlmann et al. (2007) M. Uhlmann, A. Pinelli, G. Kawahara, and A. Sekimoto.  Marginally turbulent flow in a square duct.  _J. Fluid Mech._ , 588:153–162, 2007.  doi:10.1017/S0022112007007604.  URL https://doi.org/10.1017/S0022112007007604.
  * Uhlmann et al. (2010) M. Uhlmann, G. Kawahara, and A. Pinelli.  Travelling-waves consistent with turbulence-driven secondary flow in a square duct.  _Phys. Fluids_ , 22(8):084102, 2010.  doi:10.1063/1.3466661.  URL https://doi.org/10.1063/1.3466661.
  * van Rijn (1984) L. C. van Rijn.  Sediment transport, part i: bed load transport.  _J. Hydr. Eng._ , 110(10):1431–1456, 1984.  URL https://doi.org/10.1061/(ASCE)0733-9429(1984)110:10(1431).
  * Vanderwel and Ganapathisubramani (2015) C. Vanderwel and B. Ganapathisubramani.  Effects of spanwise spacing on large-scale secondary flows in rough-wall turbulent boundary layers.  _J. Fluid Mech._ , 774:1–12, 2015.  doi:10.1017/jfm.2015.292.  URL https://doi.org/10.1017/jfm.2015.292.
  * Vanderwel et al. (2019) C. Vanderwel, A. Stroh, J. Kriegseis, B. Frohnapfel, and B. Ganapathisubramani.  The instantaneous structure of secondary flows in turbulent boundary layers.  _J. Fluid Mech._ , 862:845–870, 2019.  doi:10.1017/jfm.2018.955.  URL http://doi.org/10.1017/jfm.2018.955.
  * Vanoni (1946) V. A. Vanoni.  Transportation of sediment in suspension.  _Trans. Am. Soc. Civil Eng_ , 101:67–133, 1946.
  * Vowinckel et al. (2017) B. Vowinckel, V. I. Nikora, T. Kempe, and J. Fröhlich.  Spatially-averaged momentum fluxes and stresses in flows over mobile granular beds: a DNS-based study.  _J. Hydraul. Res._ , 55(2):208–223, 2017.  URL https://dx.doi.org/10.1080/00221686.2016.1260658.
  * Wallace et al. (1972) J. M. Wallace, H. Eckelmann, and R. S. Brodkey.  The wall region in turbulent shear flow.  _J. Fluid Mech._ , 54(1):39–48, 1972.  doi:10.1017/S0022112072000515.  URL https://doi.org/10.1017/S0022112072000515.
  * Wang and Cheng (2005) Z. Wang and N. Cheng.  Secondary flows over artificial bed strips.  _Adv. Water Resour._ , 28(5):441–450, 2005.
  * Wang and Cheng (2006) Z. Wang and N. Cheng.  Time-mean structure of secondary flows in open channel with longitudinal bedforms.  _Adv. Water Resour._ , 29(11):1634–1649, 2006\.
  * Wangsawijaya et al. (2020) D. D. Wangsawijaya, R. Baidya, D. Chung, I. Marusic, and N. Hutchins.  The effect of spanwise wavelength of surface heterogeneity on turbulent secondary flows.  _J. Fluid Mech._ , 894(A7), 2020.  doi:10.1017/jfm.2020.262.  URL https://doi.org/10.1017/jfm.2020.262.
  * Wolman and Brush (1961) M. G. Wolman and L. M. Brush.  _Factors controlling the size and shape of stream channels in coarse noncohesive sands_.  US Government Printing Office, 1961.
  * Wong and Parker (2006) M. Wong and G. Parker.  Reanalysis and correction of bed-load relation of meyer-peter and müller using their own database.  _J. Hydr. Eng._ , 132(11):1159–1168, 2006.  doi:10.1061/(ASCE)0733-9429(2006)132:11(1159).  URL https://doi.org/10.1061/(ASCE)0733-9429(2006)132:11(1159).
  * Yang and Anderson (2018) J. Yang and W. Anderson.  Numerical study of turbulent channel flow over surfaces with variable spanwise heterogeneities: topographically-driven secondary flows affect outer-layer similarity of turbulent length scales.  _Flow, Turbulence and Combustion_ , 100(1):1–17, 2018.
  * Zampiron et al. (2020) A. Zampiron, S. Cameron, and V. Nikora.  Secondary currents and very-large-scale motions in open-channel flow over streamwise ridges.  _J. Fluid Mech._ , 887, 2020.  doi:10.1017/jfm.2020.8.  URL https://doi.org/10.1017/jfm.2020.8.