11institutetext: Know-Center GmbH, Graz, Austria
11email<EMAIL_ADDRESS>22institutetext: Graz
University of Technology, Graz, Austria
22email<EMAIL_ADDRESS>33institutetext: University of Graz,
Graz, Austria
33email<EMAIL_ADDRESS>
# Reproducibility in Machine Learning-based Research: Overview, Barriers and
Drivers

Harald Semmelrock 22 Tony Ross-Hellauer 1122 Simone Kopeinik 11 Dieter Theiler
11 Armin Haberl 33 Stefan Thalmann 33 Dominik Kowald${}^{\textrm{(\Letter)}}$
1122

###### Abstract

Research in various fields is currently experiencing challenges regarding the
reproducibility of results. This problem is also prevalent in machine learning
(ML) research. The issue arises, for example, due to unpublished data and/or
source code and the sensitivity of ML training conditions. Although different
solutions have been proposed to address this issue, such as using ML
platforms, the level of reproducibility in ML-driven research remains
unsatisfactory. Therefore, in this article, we discuss the reproducibility of
ML-driven research with three main aims: (i) identifying the barriers to
reproducibility when applying ML in research as well as categorize the
barriers to different types of reproducibility (description, code, data, and
experiment reproducibility), (ii) discussing potential drivers such as tools,
practices, and interventions that support ML reproducibility, as well as
distinguish between technology-driven drivers, procedural drivers, and drivers
related to awareness and education, and (iii) mapping the drivers to the
barriers. With this work, we hope to provide insights and to contribute to the
decision-making process regarding the adoption of different solutions to
support ML reproducibility.

###### Keywords:

Machine Learning Artificial Intelligence Reproducibility Irreproducibility

##  1 Introduction

As in many scientific fields [9, 68], e.g., medicine and behavior science,
research in artificial intelligence (AI) in general, and machine learning (ML)
in particular, faces crucial doubts over the reproducibility of research [36].
This has raised concerns about the reliability and validity of many scientific
findings, risking diminishing confidence in the overall body of scientific
knowledge. Misleading or irreproducible results can lead to wasted resources,
hinder scientific progress, reduce trust in science, and impact decision-
making in various fields [51, 29]. The “reproducibility crisis” has been a
topic of discussion since the mid-2010s, particularly in the medical and
behavioral science fields [9, 55]. Recently, this issue has also been
associated with research involving ML methods [36]. Apart from the common
challenges faced by many disciplines, such as limited sharing of data and
source code, the use of ML introduces unique challenges for reproducibility,
including sensitivity to ML training conditions, sources of randomness [65],
and the increasing use of AutoML tools [42].

While the literature on these issues is expanding [46, 30, 33, 40, 26, 4, 70,
27], there is still no comprehensive overview of the associated barriers and
drivers. For example, in [27], the authors identify and categorize sources of
irreproducibility in ML and how these sources affect conclusions drawn from ML
experiments. However, this study does not investigate drivers to address these
sources of irreproducibility. Thus, our paper provides a contextual
categorization of the barriers and drivers to the four types of ML
reproducibility proposed by (description, code, data, and experiment) [26],
with specific reference to research in both Computer Science and Biomedical
fields. We also propose a Drivers-Barriers-Matrix to summarize and visualize
the results of the discussion. Such an analysis stands to clarify the current
state regarding ML reproducibility, provide guidance for those new to the
subject, and highlight potential gaps, solutions, and avenues for future
exploration.

The paper is structured as follows: in Section 2, we clarify terms and working
definitions. We then analyze the barriers to increased reproducibility of ML-
driven research (Section 3), and next the drivers that support ML
reproducibility, including different tools, practices, and interventions
(Section 4). Finally, we map the barriers to the drivers to help determine the
feasibility of various options for enhancing ML reproducibility (Section 5).
We close the paper with a conclusion and an outlook into our future research
in Section 6.

##  2 Defining Reproducibility

The concept of reproducibility can have different interpretations across
various research fields and even within the same field [23]. To address this
confusion, reproducibility is broadly defined and then further categorized
into various types or degrees. A key distinction is made by Goodman et al.
[25] who specify a fundamental division between whether we (i) mean
reproducible in principle (termed “methods” reproducibility) due to sufficient
description/sharing of methodologies, materials, etc., or (ii) whether
results/conclusions actually prove to be reproducible when experiments or
analyses are re-done. In the second category, they distinguish “results” and
“inferential” reproducibility, depending on whether it is the analyses or
inferences to broader conclusions that are reproduced.

Within ML research, widely accepted definitions that build further on these
key distinctions have been proposed by [30, 26]. We follow and build upon
these latter definitions, and hence here outline them in some length.

Gundersen et al. [28] define reproducibility in general as “the ability of
independent investigators to draw the same conclusions from an experiment by
following the documentation shared by the original investigators”.

Relating to point (ii) of Goodman et al.’s schema, Gundersen and colleagues
[30, 26] further distinguish the targets of reproducibility, i.e., how closely
an experiment can be reproduced:

  * •

Outcome reproducibility requires the reproduced experiment to have the same or
adequately similar outcome as the original experiment. Due to this, the same
analysis and interpretation follow, and the hypothesis is either supported or
rejected by both experiments.

  * •

Analysis reproducibility does not require the reproduced experiment to have
the same/similar outcome; however, if the same/similar analysis and, therefore
also, interpretation can be made, an experiment is analysis reproducible.

  * •

Interpretation reproducibility does not require the reproduced experiment to
have the same/similar outcome nor analysis but requires the interpretation to
be the same as the original one. In such a case, an experiment is
interpretation reproducible.

Figure 1: Types of reproducibility. Adapted from Gundersen [26].

This categorization aims to overcome the problem of ambiguity when making
specific claims about the reproducibility of an experiment. Often in
literature, authors write about reproducing the same “results” of an
experiment. It is not apparent, however, in which cases they mean to achieve
the same computational outcome, i.e., outputs of the algorithms, or whether
they mean to reach the same analysis or interpretation. Therefore, achieving
interpretation reproducibility is a more general and often less stringent
requirement than achieving outcome reproducibility. This categorization is not
specific to ML, but generally applicable to any research field that conducts
computational experiments.

In addition, relating to point (i) of the Goodman et al. schema (“methods”
reproducibility), [26] specifies reproducibility types to which methods can be
made transparent through description or sharing. These four types are defined
as R1 Description, R2 Code, R3 Data, and R4 Experiment. The lower the level of
reproducibility, the less shared information there is, making the experiment
more difficult to reproduce. Giving an example: In general, all published
research experiments are accompanied by a textual description of the
experiment. If this textual description is the only information shared by the
authors, the research is categorized as R1 Description, which, according to
this scheme, is the lowest type of reproducibility. In contrast, if all three
building blocks, i.e., text, code, and data, are shared, the experiment can be
categorized as R4 Experiment, which is the highest type of reproducibility.
Furthermore, the distinction between R2 Code and R3 Data is defined by the
textual description of the original experiment that is accompanied by either
code or data, respectively.

Also, the different types of ML reproducibility exhibit an interplay between
generalizability and reproducibility. R1 Description leads to the highest
generalizability, but to the weakest reproducibility, while R4 Experiment
leads to the highest reproducibility and lowest generalizability.

This relationship is illustrated in Figure 1. Furthermore, Figure 1 visualizes
the building blocks associated with the different types of reproducibility. In
what follows, unless stated otherwise, we use the definitions proposed by
Gundersen and colleagues [26].

##  3 Barriers in ML Reproducibility

In the following, we discuss nine barriers to ML reproducibility, categorized
into the four types of reproducibility mentioned in the previous section.
Where applicable, we give examples within the research fields of Biomedical
Science, and Computer Science.

###  3.1 R1 Description

####  3.1.1 Completeness and quality of reporting.

Research often lacks reproducibility due to missing or vague methodological
details. Mainly, there are three issues in this regard, which often hinder the
reproduction of study results [59]:

  1. 1.

The ML model or training procedure is either incorrectly specified or under-
specified. Reports should give clear details on all steps of the procedure,
even if data and code are not shared. This includes details about which ML
models are used, as well as details on the training data and data
preprocessing.

  2. 2.

The evaluation metrics used to report results are not properly specified.
There are many metrics, which can be used to evaluate ML models, e.g.,
accuracy, receiver-operator-curve (ROC), or mean-squared-error (MSE). It is
important to define these metrics and also explain why they were used.

  3. 3.

Often results are selectively reported, e.g., researchers may only provide
results for the best performance out of many test runs, instead of properly
assessing and reporting the performance over multiple runs. For example, in
research on Natural Language Processing (NLP), often, reproductions turn out
to perform worse than stated in the original publications [11].

Generally, it is important for studies to use a robust methodology and provide
detailed reports, for other researchers to verify results and understand how
analyses were conducted. While ML models have proven highly effective in
Biomedical fields, studies often fall short in providing comprehensive and
high-quality reporting. For example, in studies on predicting cardiometabolic
risk from dietary patterns [56] or supporting the clinical management of
diabetes [39], ML models were observed to be very promising. While this
further enhances the promise of applying ML models for various clinical
prediction tasks, there is a clear need for thorough reporting and validation
of these models to allow for their integration into routine clinical care
[39]. This is also true for the application of ML models for cancer imaging,
where ML models often surpass radiologists in performance, but publications on
these models lack the documentation details needed to reproduce the results
[61].

####  3.1.2 Spin practices and publication bias.

Another flaw, which has been observed in ML-based research and negatively
affects reproducibility, is “spin”. It refers to the misuse of language to
“intentionally or unintentionally affect the interpretation of study
findings”. Accordingly, it is also understood as an inconsistency between the
study results and the conclusions, in the sense that results are over-
generalized or the claimed conclusions are not supported by the scientific
method. This has been shown to impact both the interpretations and decision-
making by readers [5]. In ML-based Biomedical research, the most common
practice of spin includes recommending models for various applications without
providing external validation in the same study. More concretely, the
recommendation to use a model either in a clinical setting or for a different
population is only validated in approximately 15% of the cases. Other observed
instances of spin are invalid comparisons of results to previous studies, and
the use of leading words and strong statements to make the results sound more
prolific [5]. The use of spin in numerous publications might be partly
attributed to the reward system in academia, which often values spin practices
in research, as more prolific results are more likely to lead to accepted
publications. Publication bias in general is a big problem in a variety of
different regards. It seems that research from high-profile institutions is
published more easily and subject to less scrutiny in review, and there also
seems to be a tendency to prefer more complex solutions over simpler ones
addressing the same problem [75].

###  3.2 R2 Code

####  3.2.1 Limited access to code.

Published ML research is often not accompanied by available data and code.
Only one-third of researchers share data, and even fewer share their source
code [37]. This can be for many reasons, such as the increasing pressure on
researchers to publish quickly, which in turn does not allow them to polish
the code and decreases the willingness to release the code, or simply due to
privacy constraints with respect to sharing code. Sharing ML code correctly
requires publishing seven pieces of information: hypothesis, prediction
method, source code, hardware specifications, software dependencies,
experiment setup, and experiment source code. Only if the ML code is
accompanied by all these artifacts, it is specified sufficiently to allow
reproducing an experiment [29]. Unfortunately, current research often does not
meet this requirement, leading to reproducibility issues due to different
software versions, hyperparameter settings, or hardware differences [34, 11].
For example, in Computer Science, the lack of shared code and data
significantly contributes to a lack of reproducibility [19]. Even when code
and data are shared, they are often poorly documented, or the code is only
provided as a skeleton instead of a fully executable code, which does not
ensure reproducibility. Thus, shared code should also include the code and
documentation for data preprocessing, hyperparameter tuning, and managing
random seeds, as well as code for comparisons against baseline models.

###  3.3 R3 Data

####  3.3.1 Limited access to data.

The main barrier concerning R3 Data is that data is often not shared. This
could be due to privacy concerns or a lack of incentives and motivation.
Moreover, many benchmark and training datasets encounter challenges related to
copyright, licensing, and longevity. These datasets may also raise ethical
concerns, such as the unintentional inclusion of privacy-sensitive or harmful
content, making it difficult to share the data for ML model training [57].
Similar to the sharing of source code, sharing only the datasets is
insufficient. For proper use, it is also important to share specific splits,
i.e., the training dataset, validation dataset, and test dataset [29].
Furthermore, data sharing needs to be accompanied by documentation specifying
details about the provenance and preprocessing of data. This is crucial to
investigate methodological errors such as data leakage and bias, which are
described in the following.

####  3.3.2 Data leakage.

In practice, methodological issues such as data leakage often hinder the
reproducibility of ML-based research [40]. This is due to the growing number
of non-experts employing machine learning across different research fields
[24], which is fuelled by the ease of application of ML libraries and no-code
off-the-shelf AI tools. In essence, data leakage happens when data, on which
the ML model should not be trained, leaks into the training process. Data
leakage can be categorized into 3 subcategories [40]:

  1. 1.

No clean train/test split. Here, four variants are possible: (1) training data
and test data are not split at all, (2) test data is also used to select the
best features from the training data (feature selection), (3) test data is
also used for imputation of missing data during preprocessing, and (4)
duplicates occur in the training and test data.

  2. 2.

Use of non-legitimate data. For example, when the use of antihypertensive
drugs is used to predict hypertension. This data is non-legitimate, since it
would not be available in a real-world scenario and cannot realistically be
used to predict hypertension for a new patient.

  3. 3.

Test set is not drawn from the distribution of scientific interest. There are
three possible variants: (i) temporal leakage, which is problematic for ML
models that attempt to predict future outcomes, i.e., when some training
samples have a later timestamp than samples available in the test set, (ii)
the training and test data are not independent of each other, e.g., there
should not be samples in the training and test data that are drawn from the
same person, and (iii) the test set is not chosen selectively; for instance,
if the model is solely evaluated on data, on which it performs well.

####  3.3.3 Bias.

Bias in ML refers to the error introduced by approximating a real-world
problem, which may be complex, by a simplified model, often leading to
systematic deviations from the real world. Furthermore, biases can arise when
the model contains imbalances or reflects existing societal biases [47]. ML
models, which are subject to bias, are prone to generalization issues, and are
therefore potentially problematic for ML reproducibility. There are eight bias
variants that can arise during the data handling phase of ML development [67]:
(i) selection bias - using data not being representative of the target group,
(ii) exclusion bias - excluding particular data samples based on the belief
that they are unimportant, (iii) measurement bias - favoring certain
measurement results, (iv) recall bias - labeling similar data samples
differently, (v) survey bias - introducing data issues stemming from data
collection surveys, (vi) confirmation bias - favoring information, which
confirms previous beliefs, (vii) prejudice bias - including human-related
prejudices in training data, and (viii) algorithmic bias - replicating or
amplifying biases by the inner workings of the algorithms. Bias, such as
selection bias, often leads to the issue of validity shrinkage in Biomedical
science research [38]. For example, in obesity and nutritional research, ML is
used to predict obesity, heart rate, or the risk of a heart attack, based on
data of an individual. Here, validity shrinkage refers to the issue that a
predictive model trained on a subset of data will most probably not perform
well on new samples. The difference between predictive performance on known
data and new data, however, is most often not accounted for in nutritional
science, and therefore also leads to performance claims that cannot be
reproduced [38].

###  3.4 R4 Experiment

####  3.4.1 Inherent nondeterminism.

Inherent nondeterminism in ML models means that results can vary between test
runs, even with identical code, data, and hyperparameters. This variation
occurs from unanticipated influences during training, such as random parameter
initialization, stochastic optimization, and random data subsampling (e.g., in
k-fold cross-validation) [65, 44]. Neural networks are especially known for
their inherent nondeterminism, leading to varied computational outcomes in
multiple reruns due to numerous sources of randomness during training [1]. In
some cases, inherent non-determinism can cause such large variations that
reruns not only yield slightly different outcomes, but also lead to
significant fluctuations in the performance of an ML model [1]. This issue is
exacerbated when other sources of variation, such as different
hyperparameters, are introduced to the ML model. In such cases, the impact can
be magnified, and it is often observed that minor changes in hyperparameters
can result in significant performance loss [11]. For example, a review of the
status quo of reproducibility in NLP research highlights some core issues with
non-determinism that are exemplary for ML research [11]. Simply rerunning the
original code during a reproduction can lead to different computational
outcomes compared to the originally reported ones. Reinforcement learning, a
subfield of ML, is particularly susceptible to these reproducibility issues,
partially because of additional sources of non-determinism in the learning
process, such as the reinforcement learning environment or policy [52].

####  3.4.2 Environmental differences.

Various studies have demonstrated that hardware differences, such as different
GPUs or CPUs, and compiler settings can lead to different computational
outcomes [34]. Additionally, a comparison between the same ML algorithm with
fixed random seeds executed using PyTorch111https://pytorch.org/ and
TensorFlow222https://www.tensorflow.org/ resulted in different performances
[60]. A comparison of the results of an ML experiment between different ML
platforms also shows that out-of-the-box reproducibility is not provided. In
general, it is questionable to what extent reproducibility can be ensured out-
of-the-box [30]. Another important factor is the use of GPUs, which can
increase randomness compared to the use of CPUs. This is due to parallel
optimization and the use of optimizers in ML frameworks, such as PyTorch and
TensorFlow. As a result, some researchers have resorted to solely using CPUs
for executing their experiments. However, this comes at the expense of
runtime-efficiency [3].

####  3.4.3 Limited access to computational resources.

Another barrier to ML reproducibility is the limited access to computational
resources, which most recently has become evident in the case of transformer-
based large language models (LLMs) [10, 82]. These transformer architectures
need a vast amount of data and computational resources, to which most
researchers have limited access. Estimates have calculated the costs to
reproduce one model to be around $1 million to $3.2 million, which largely
increases the difficulty of reproducing results [10]. Another study found that
the needed computational resources are one of the most significant factors
impacting reproducibility [64]. Especially ML models, which require
computational clusters for training and optimization, are notably hard to
reproduce.

##  4 Drivers for ML Reproducibility

In this section, we discuss drivers for ML reproducibility, which we subdivide
into (i) technology-based drivers, (ii) procedural drivers, and (iii) drivers
related to awareness and education.

###  4.1 Technology-based Drivers

####  4.1.1 Hosting services.

Utilizing hosting services offers an efficient way to share code, data, and ML
model parameter settings, thus supporting the reproducibility of ML-driven
research [74]. Examples of hosting services include the runtime environments
of ML platforms. If the original author runs the ML experiment in such a
runtime environment, e.g., Kaggle Notebooks333https://www.kaggle.com/code,
Google Colab444https://colab.google/, or CodaLab555https://codalab.org/,
researchers attempting to reproduce the results should be able to execute the
experiment within the same environment.

The main advantage of using a hosting service is that the provider takes care
of the logistics of code hosting and distribution. However, the main drawbacks
are the limits on data size and computational resources. Since these hosting
services are run in the cloud, there are restrictions on how many resources a
single user can utilize. The limit on resources varies between different
hosting services and is sometimes tied to subscription levels. Because of
these limits, hosting services may not be suitable for all research purposes,
especially considering the compute-intensive nature of state-of-the-art ML
models, such as LLMs. Furthermore, research has indicated that out-of-the-box
reproducibility across multiple ML platforms is not yet guaranteed [30].

####  4.1.2 Virtualization.

Reproducing the environment and setup of any ML experiment requires the
consideration of existing dependencies and software versions, and is usually a
complex task itself. Virtualization can simplify this process by bundling the
essential components of ML models and experiments, such as the environment and
code, into a single package for being shared with other researchers. Thus, if
the authors of a paper build the experiment in a virtual environment, issues
associated with setup reproduction can be greatly reduced. However, the
adoption of virtualization by researchers depends on its user-friendliness and
the effort of integration into their current workflows [12]. Concerns about
virtualization include its limitations in allowing researchers to build upon
them in a scalable manner. Traditional virtual machines (VMs) emulate an
entire operating system for setting up and running experiments. The use of
containerization software like Docker666https://www.docker.com/ has just
recently become popular. Containers are more lightweight and flexible than
VMs, making it easier to adapt environments for follow-up studies [12]. There
are also designated platforms for computational research, such as Code
Ocean777https://codeocean.com/, that offer virtualization via so-called
reproducible capsules. Their focus in particular is to simplify the
virtualization process and allow researchers to focus on the research
questions instead of the standardization of environments [16]. Additionally,
there are many other tools, such as e.g., ReproZip [15], which is one of the
recommended tools by the SIGMOD
Reproducibility888https://reproducibility.sigmod.org/ to streamline
reproducibility, and DetTrace [53], which aims to ensure completely
deterministic computations.

####  4.1.3 Managing sources of randomness.

Many different sources of randomness during ML training lead to
irreproducibility of ML research. Managing these sources of randomness, e.g.,
via random number seeds, deterministic algorithms, or other methods, could
therefore greatly increase reproducibility. Fixed andom number seeds should be
used and published to make ML experiments more reproducible and control a
number of sources of the inherent nondeterminism. A seed is a first value used
to initialize the pseudo-random number generator. When the same seed is used,
the sequence of pseudo-random numbers generated is deterministic, meaning it
will be the same every time the code is run. Additionally, ML models should be
benchmarked and evaluated with multiple random number seeds, such that the
variance can be reported and inform about the true performance of an ML model
[65]. However, fixing the random number seeds can only eliminate algorithmic
sources of randomness. Furthermore, hardware nondeterminism is also avoidable
by using patching, i.e., replacing non-deterministic operations from hardware
with deterministic ones, however, this will lead to a significant computing
overhead [14]. To counteract inherent nondeterminism in reinforcement learning
and achieve reproducible evaluations, there are frameworks, e.g. Gym-Ignition
[22, 13], rl_reach [8], or MinAtar [81], which act as standardized
benchmarking environments. Within them, different algorithms designed for the
same task can be evaluated and compared against each other in a common
environment. The frameworks counteract the effects of inherent nondeterminism
by automatically controlling random seeds and evaluating algorithms over a
number of runs. Additionally, it is an ongoing field of research to implement
fully deterministic reinforcement learning algorithms [52] and make use of
them within such frameworks [73].

####  4.1.4 Privacy-preserving technologies.

Privacy-preserving technologies support reproducibility, as they enable the
collaborative training of ML models without sharing private or sensitive data.
The main benefit of this is, that ML models can make use of larger, and more
diverse data, thus helping to decrease bias and leading to more reproducible
ML models. The main aim of Privacy-Preserving Machine Learning (PPML) is to
facilitate the use of privacy-sensitive data to create better ML models, and,
to allow data owners to collaboratively train ML models on private data. In
that regard, PPML has several requirements. First, protecting the
confidentiality of the training data. Second, preventing the leakage of
sensitive information from ML model parameters and outputs, i.e., to hinder
the re-identification of individuals. Third, achieving the listed security and
privacy aspects while still preserving the utility of the ML model [79]. To
achieve this, a number of different techniques are being used and developed,
such as differential privacy, fully homomorphic encryption, secure multi-party
computation, and federated learning [20]. These techniques are implemented in
software libraries such as TensorFlow Privacy, PySyft, ML Privacy Meter,
CryptFlow, or Crypten [7]. Furthermore, data can be made anonymous by removing
identifiable personal information. However, if too much data is removed, the
ML models may perform poorly. If not enough data is removed, it may still be
possible to re-identify individuals by combining many different non-unique
features [79]. An alternative approach to PPML is to generate synthetic data
that captures the same information as the original data. A robust technique
for creating such datasets can produce readily available datasets of nearly
any size, as demonstrated in Biomedical fields. This approach has led to the
development of Synthea, a software package designed to generate synthetic
patient data and electronic health care records [76].

####  4.1.5 Tools and platforms.

There are many tools and platforms that assist in the implementation and
management of ML models and ML-based applications. A recent study has
evaluated 19 ML tools to gain insights into their concepts constituting
reproducibility support [63]. As a result, five main pillars of ML
reproducibility in tools and platforms were identified: (i) code versioning,
(ii) data access, (iii) data versioning, (iv) experiment logging, and (v)
pipeline creation. Most of these pillars are associated with managing and
keeping track of different artifacts created during phases of the ML lifecycle
(i.e., design, development, and deployment) as for instance, datasets, labels,
code, logs, environment dependencies, random number seeds, or hyperparameters
[69]. Each of these artifacts influences the final results of the ML model.
Consequently, most tools aim to collect, store, and manage these artifacts,
ensuring researchers can access and use them during reproduction attempts.
Notable are also various tools and platforms for experiment tracking [69],
such as:

  * •

DVC999https://dvc.org/ is a version control system for ML projects with a
command-line interface similar to Git101010https://git-scm.com/. It integrates
with Git, supports cloud storage, and handles large versioning of datasets.
DVC ensures full code and data provenance by enabling experiment tracking.

  * •

MLflow111111https://mlflow.org/ is an open-source tool for supporting ML
experiment tracking, ML model deployment, and centralized model storage.
Additionally, it provides an easy-to-use Web dashboard.

  * •

RO-Crate is a specification, implemented by a number of tools, aimed at
aggregating and describing research data and metadata [71]. Although not
specifically designed for ML, RO-Crate can aggregate and represent any
resource, making it applicable for managing ML artifacts as well.

  * •

dToolAI [32] collects and packages ML models together with supplemental
information, such as hyperparameter settings, appropriate metadata, and
persistent URIs for model training data. In contrast to the other tools,
dToolAI is specifically tailored for deep learning models.

AutoML platforms, such as H2O Driverless AI121212https://h2o.ai, Google Cloud
AutoML131313https://cloud.google.com/automl,
DataRobot141414https://www.datarobot.com/platform) are a novel subcategory of
ML tools that aim to aid with every aspect of the ML lifecycle, from data
aggregation to model deployment. Thus, AutoML tools could facilitate more
standardized ML models and also take care of tasks like hyperparameter
optimization. It is however questionable how practical these tools are for
reproducible ML research, since they often hide ML model optimization
procedures. Sometimes, e.g., in the case of H2O Driverless AI, the features go
a step further and promise to be able to prevent problems such as overfitting
or target leakage151515https://h2o.ai/wiki/target-leakage//. In the case of
target leakage, this is done by checking for a strong correlation between a
feature and the target and then taking action, e.g., warning the user or
automatically handling it. This is, however, a very simple solution to the
problem and does not address the more complex cases of data leakage that are
often present in research, e.g., temporal leakage.

###  4.2 Procedural Drivers

####  4.2.1 Standardized datasets and evaluation.

Due to a lack of shared datasets, currently, many researchers in ML-driven
research - most notably in Biomedical fields - use individually acquired data
[46]. The collection of such data is a time-consuming task and bears a
significant risk of causing reproducibility issues, e.g., bias or data
leakage. Often, the number of participants within the datasets is not very
large and, thus, might not translate well to new contexts. Creating shared and
standardized datasets can, therefore, (i) save researchers time in acquiring
new data, (ii) facilitate the collaborative and independent maintenance and
verification of data to minimize methodological errors, and (iii) support
transferability and generalizability through the use of multi-institutional
data [46]. In addition to the standardization of datasets, data cards [62]
provide a consistent and comparable framework for reporting essential aspects
of ML datasets. This includes information, e.g., about access restrictions,
risks and limitations associated with the usage of the dataset, or any
preprocessing steps, amongst many other contents, which are needed for
reproducible ML development.

Another issue is the lack of standardized evaluation methods, which leads to
reported performances of ML models often being overly optimistic [19]. To
ensure the statistical significance of ML model evaluations, it is crucial to
report performance as an aggregate of results obtained from multiple random
runs, and with different random number seeds [17]. Furthermore, ML models
should, if possible, be tested and evaluated on multiple different datasets
[21]. This underscores the need for standardized evaluation methods, which can
be supported by checklists or tools to prevent errors in this critical aspect
of ML research. For this, similarly to data cards, model cards [48] are aimed
to standardize the evaluation and reporting of the performance of ML models
for a variety of use cases. Model cards should inform users about the possible
applications of the ML model and its limitations. For the creation of the
model cards, a toolkit is also being developed by
Google161616https://research.google/blog/introducing-the-model-card-toolkit-
for-easier-model-transparency-reporting/. In reinforcement learning, the
creation of standardized evaluation pipelines is continually being researched
to enable reproducible benchmarking of different reinforcement learning
algorithms [41].

####  4.2.2 Guidelines and checklists.

There are many guidelines and checklists that outline best practices for
increasing the reproducibility of ML. The guidelines are often aimed at
specific parts of the ML workflow. For example, the FAIR guiding
principles171717https://www.go-fair.org/fair-principles/ aim to improve the
management and stewardship of scientific data by making scientific data (i)
findable, (ii) accessible, (iii) interoperable, and (iv) reusable. Other
guidelines promote the transparency and openness of scientific reporting in
general, such as the TOP guidelines181818https://www.cos.io/initiatives/top-
guidelines, which target journals. Similarly, checklists provide a simple
framework for ensuring certain criteria are met. Checklists have been applied
effectively in the past, e.g., in safety-critical systems, where they were
used as early as in 1935 to complete pre-flight checks in Boeing airplanes. A
promising example is the ML checklist proposed in [59], which has been
suggested as best practice by researchers of different fields, e.g., in
chemistry [6]. The checklist requests information about (i) the models and
algorithms being used, (ii) theoretical claims in the research article, (iii)
data, (iv) code, and (v) the ML experiment(s). However, one drawback of
reproducibility checklists when used for academic conferences and journals is
the additional workload they impose on reviewers. To mitigate this workload,
one option would be to leverage LLMs to assist the review process [45].
Finally, numerous guidelines and checklists for ML reproducibility have been
recommended in various research fields [6]. Especially in Biomedical fields,
we have seen a considerable adoption of guidelines and checklists, such as the
TRIPOD statement [18], the CLAIM checklist [49], the ROBUST-ML checklist [2],
or PROBAST [78].

####  4.2.3 Model info sheets and model cards.

Model cards [48] are documentation sheets that provide information about ML
models, including their intended use, potential limitations, and ethical
considerations. They aim to enhance transparency in AI, by detailing aspects
such as data used for training, performance metrics, evaluation methodologies,
and possible biases. Model cards help users to understand and evaluate ML
models more comprehensively, such that they are not being deployed in contexts
other than the intended use, and thus to increase reproducibility. Similarly,
model info sheets also provide documentation about ML models, but are
specifically designed for the detection and prevention of data leakage in ML
models [40]. The model info sheets are published alongside research to enable
other researchers to quickly verify the validity of the data used to train ML
models. Model info sheets require authors to answer detailed questions about
the data and corresponding train/test splits, targeting various types of data
leakage [40]. Despite that, they have two main drawbacks: first, verifying the
correctness of info sheets only works after reproducing the results; second,
completing these sheets requires a certain level of expertise in ML. In
general, model cards and model info sheets represent a promising, low-effort
driver for ML reproducibility. They are especially useful in handling some of
the methodological issues associated with ML models that could arise, e.g.,
when being developed by non-experts [40].

###  4.3 Awareness and Education

Awareness of reproducibility issues and available training/education to
support reproducibility can be a powerful driver for ML reproducibility [77]).

####  4.3.1 Publication policies and initiatives.

To enhance awareness and establish a minimum of reproducibility standards, the
policies of scientific journals are considered an influence factor. A number
of journals already mandate data and/or code availability for publication [59,
58, 31]. However, to address issues such as result manipulation, more
extensive journal participation is needed to for instance introduce
preregistration, where researchers register their research intentions for
future publication. This approach ensures credibility by separating the
research plan from experimental outcomes [72, 54], thereby reducing spin
practices, HARKing, and p-hacking [30]. The ACM TORS (Transactions on
Recommender Systems) journal exemplifies this by allowing preregistration and
publishing “reproducibility papers” dedicated to reproduction studies and
enhancing reproducibility tools. Apart from that, various initiatives have
been launched to raise awareness of reproducibility issues. A few examples are
the following:

  * •

The ReScience journal publishes peer-reviewed papers discussing attempts to
reproduce original publications. These reproductions are published on
GitHub191919https://github.com/, being available to other researchers [66].

  * •

PapersWithCode.com202020https://paperswithcode.com/ is a resource for (i) ML
papers, accompanied by the code, (ii) datasets, and (iii) ML methods. The ML
papers include a link to a repository, which features the code and other
artifacts for reproducing the results.

  * •

Reproducibility challenges, where several researchers try to reproduce many
recent publications in parallel, are being held frequently. These challenges
allow for an analysis of the success rate of reproduction and can be used to
evaluate progress over multiple years [59]. Additionally, conferences such as
the European Conference on Information Retrieval (ECIR), provides special
reproducibility tracks, in which researchers are encouraged to reproduce
existing papers and build upon their results (e.g., [43, 50]).

  * •

The ACM has put together a new emerging interest group on
reproducibility212121https://reproducibility.acm.org/. The main goals are to
(i) contribute to the development of reproducibility standards, practices and
policies, (ii) promote the development and evaluation of tools, techniques,
and methodologies, and (iii) encourage best practices.

  * •

ReproducedPapers.org is another online repository fostering reproductions. It
further focuses on education by incorporating a reproduction project into a
Master’s level ML course at TU Delft [80].

##  5 Mapping Drivers to Barriers

In this section, we map the drivers of reproducibility to the barriers in the
form of a Drivers-Barriers Matrix. This will be based on the definition and
categorization of reproducibility as a foundation (Section 2), and the
identification of the major barriers (Section 3) and drivers (Section 4) of ML
reproducibility. The resulting Drivers-Barriers-Matrix is depicted in Figure 2
and categorizes the barriers into the four different types of ML
reproducibility, i.e., R1 Description, R2 Code, R3 Data, R4 Experiment [26],
and drivers into technology-driven drivers, procedural drivers, and drivers
related to awareness and education.

Our Drivers-Barriers-Matrix shows that there are often multiple drivers for
the same barrier. Consequently, there are also several possible solutions for
a barrier or different aspects of a barrier. The mapping allows to quickly
assess which drivers address the different barriers and which barriers have a
higher or lower number of drivers associated with them. It underlines the need
for context-dependent approaches instead of "one-size-fits-all" solutions, as
the proper selection of a suitable driver depends on the specific conditions
and existing barriers relevant to any ML application. Intersections between
drivers and barriers will be described in more detail below.

####  5.0.1 R1 Description.

Completeness and quality of reporting, as well as spin practices and
publication bias, present the major barriers associated with R1 Description.
These are characterized as missing information in reports and overinflated
results that hinder reproducibility.

The major drivers for completeness and report quality are guidelines and
checklists. Guidelines provide best practices to adopt in order to achieve
reproducible ML research. Furthermore, many checklists exist that
comprehensively state the different pitfalls and provide information on how
they can be avoided. Researchers can use them to ensure their research meets
the desired standards. Furthermore, some of these checklists and guidelines
are enforced by journals, such that research will only be published if certain
criteria are met. In comparison, spin practices are not as easily
identifiable. In this case, the discussion within the research community
centers around removing the incentives for inflating research results. A
particularly effective driver for this is preregistration as an example for
publication policies and initiatives, where researchers submit research
proposals before conducting an experiment. If accepted, the research will be
published, regardless of the outcome, thereby minimizing spin practices.

Figure 2: Drivers-Barriers-Matrix. We map the 9 drivers to the 9 barriers
identified in this paper. The colored boxes show that a specific driver is
applicable to a specific barrier.

####  5.0.2 R2 Code.

Code sharing is essential to reproducibility, which makes limited access to
code a significant barrier in the field. However, it is often neglected as the
process is not trivial. To make shared code useful to the scientific
community, it is necessary to share, in addition to the source code, the
information about the entire software setup and dependencies, including
software versions and hardware configurations. To assist with this,
researchers may consider running their code in hosting services or
virtualization environments, which we identified as drivers for code sharing.
Both have similar advantages, i.e., they can easily be shared and made public
for other researchers to use. As a consequence, it will give reproducers
immediate access to code, including the complete configuration setup, such as
dependencies and versions. Hosting services are the quicker and easier way of
achieving this; however, they may be subject to different resource limits.
Virtualization (e.g., VMs or containers) is more difficult to set up but
offers more flexibility and is not externally (e.g., by a provider) restricted
in capabilities and resources. Furthermore, tools and platforms can be drivers
for reproducibility. A lot of ML tools provide capabilities for code
versioning or other features, which are key to reproducibility. One example is
dToolAI [32], which automatically logs the supplemental information of the
code, i.e., metadata, hyperparameters, and more, which are essential for ML
reproducibility.

####  5.0.3 R3 Data.

Data-related barriers are a severe obstacle to ML reproducibility due to the
research fields’ data-driven nature, where limited access to data forms a
major challenge. Privacy concerns are among the crucial arguments that cause
hesitation in sharing data. The need for data privacy is evident, especially
in the Biomedical fields, which deal with patients’ electronic health records.
Nevertheless, it increases reproducibility issues in ML-based science and,
thus, delays technological progress within these domains. However, there are
several approaches that aim to meet the requirements of sharing sensitive
data: Privacy-preserving technologies allow reproducers to train ML models on
private data without actually possessing the data. This way, reproduction
becomes possible without violating potential privacy regulations. Other than
that, the use of standardized datasets and evaluation can support issues in
regard to dataset meta-information, including the specification of train-test
splits and data provenance. Once again, tools and platforms can assist with
data versioning, and numerous guidelines and checklists have been proposed to
address the provenance of data. These guidelines and checklists are designed
to help researchers to avoid common pitfalls. Current initiatives are
supported by journals that more frequently require data to be shared as part
of a publication.

Concerning methodological errors associated with the data, data leakage is a
major issue, which can, for instance, be mitigated using standardized datasets
and evaluation. Other drivers to solve data leakage are model info sheets and
model cards, which are provided as supplemental information to a published
dataset. Even though there are some limitations to model info sheets, they are
capable of detecting all types of data leakage.

Bias is another methodological error, leading to irreproducible results. This
is because the biased data usually does not generalize well to problems
outside the experimental setup of a specific ML study. Bias has been an
important source of concern, e.g., in Biomedical fields. Effects thereof can
again be minimized using standardized datasets and evaluation or specific
guidelines and checklists, e.g., ROBUST-ML [2].

####  5.0.4 R4 Experiment.

If an ML experiment is shared entirely and code and data are available, i.e.,
reproducibility type R4 Experiment, there are still three barriers, which can
lead to irreproducible results. Inherent non-determinism arises from the
different sources of randomness in ML, and makes it difficult to achieve
repeatable results, even on the same machine. There are, however, methods to
manage the sources of randomness, such as fixed random seeds and deterministic
implementations, while comprehensively mitigating all sources of randomness is
still a very challenging endeavor.

Another barrier is described as environmental differences, which has two main
issues associated with it, i.e., software differences and hardware
differences. Both types of differences can be avoided by using either hosting
services or virtualization; constraints can be assumed to be similar to the
barrier of limited access to code. Limited access to computational resources
constitutes another barrier to ML reproducibility identified in this work. The
issue is particularly problematic for research using LLMs because of their
need for extensive computational resources in training and reproduction.
Hosting services offer a solution, providing access to pre-trained models and
allowing researchers to directly access and run respective models on-site.

##  6 Conclusion

In this paper, we examined the barriers and drivers associated with the four
types of ML reproducibility as outlined by Gundersen et al. (description,
data, code, and experiment) [26], specifically in the contexts of Computer
Science and Biomedical research. We then synthesized this into a Drivers-
Barriers-Matrix to summarize and illustrate which drivers are feasible
solutions for the various barriers. We observe that the barriers to ML
reproducibility can be addressed through three primary drivers: technology-
driven solutions, procedural improvements, and enhanced awareness and
education. It is important to highlight that, in theory, awareness and
education can complement the other drivers and serve as a foundational step
toward overcoming any reproducibility-related challenge. Further, the rise of
AutoML tools for ML development and ML tasks performed by domain experts,
(potentially) not having in-depth computer or data science knowledge, could
become another barrier for reproducibility. However, we are confident that our
findings will also help in this regard and that the topic likely requires
collaborative efforts over the coming years.

We hope that our paper provides practical guidance and orientation for
researchers employing ML, and clarifies the current state of play. Of course,
in such a dynamic and fast-paced research area, this only opens a series of
further questions and avenues for exploration. We recommend further
investigation of the various issues and potential solutions laid out here. We
would also encourage further investigation into the potential role of
platforms [30] or foundation models [35] in further exacerbating or
alleviating these issues.

####  6.0.1 Acknowledgements.

This research is supported by the Horizon Europe project TIER2 (GA:
101094817), and the FFG COMET program.

## References

  * [1] Ahmed, H., Lofstead, J.: Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems. pp. 15–20. ACM, Minneapolis MN USA (2022)
  * [2] Al-Zaiti, S.S., Alghwiri, A.A., Hu, X., Clermont, G., Peace, A., Macfarlane, P., Bond, R.: A clinician’s guide to understanding and critically appraising machine learning studies: a checklist for Ruling Out Bias Using Standard Tools in Machine Learning (ROBUST-ML). European Heart Journal - Digital Health 3(2), 125–140 (2022)
  * [3] Alahmari, S.S., Goldgof, D.B., Mouton, P.R., Hall, L.O.: Challenges for the Repeatability of Deep Learning Models. IEEE Access 8, 211860–211868 (2020), conference Name: IEEE Access
  * [4] Albertoni, R., Colantonio, S., Skrzypczyński, P., Stefanowski, J.: Reproducibility of machine learning: Terminology, recommendations and open issues. arXiv preprint arXiv:2302.12691 (2023)
  * [5] Andaur Navarro, C.L., Damen, J.A., Takada, T., Nijman, S.W., Dhiman, P., Ma, J., Collins, G.S., Bajpai, R., Riley, R.D., Moons, K.G., Hooft, L.: Systematic review finds “spin” practices and poor reporting standards in studies on machine learning-based prediction models. Journal of Clinical Epidemiology 158, 99–110 (2023)
  * [6] Artrith, N., Butler, K.T., Coudert, F.X., Han, S., Isayev, O., Jain, A., Walsh, A.: Best practices in machine learning for chemistry. Nature Chemistry 13(6), 505–508 (2021)
  * [7] Aslanyan, Z., Vasilikos, P.: Privacy-Preserving Machine Learning (2020)
  * [8] Aumjaud, P., McAuliffe, D., Lera, F.J.R., Cardiff, P.: rl_reach: Reproducible reinforcement learning experiments for robotic reaching tasks. Software Impacts 8, 100061 (2021)
  * [9] Baker, M.: 1,500 scientists lift the lid on reproducibility. Nature 533(7604), 452–454 (2016)
  * [10] Beam, A.L., Manrai, A.K., Ghassemi, M.: Challenges to the Reproducibility of Machine Learning Models in Health Care. JAMA 323(4), 305–306 (2020)
  * [11] Belz, A., Agarwal, S., Shimorina, A., Reiter, E.: A systematic review of reproducibility research in natural language processing. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. pp. 381–393 (01 2021)
  * [12] Boettiger, C.: An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review 49(1), 71–79 (2015)
  * [13] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)
  * [14] Chen, B., Wen, M., Shi, Y., Lin, D., Rajbahadur, G.K., Jiang, Z.M.J.: Towards training reproducible deep learning models. In: Proceedings of the 44th International Conference on Software Engineering. p. 2202–2214. ICSE ’22, ACM, New York, NY, USA (2022)
  * [15] Chirigati, F., Rampin, R., Shasha, D., Freire, J.: ReproZip: Computational Reproducibility With Ease. In: Proceedings of the 2016 International Conference on Management of Data. pp. 2085–2088. SIGMOD ’16, ACM, New York, NY, USA (2016)
  * [16] Clyburne-Sherin, A., Fei, X., Green, S.A.: Computational Reproducibility via Containers in Psychology. Meta-Psychology 3 (2019)
  * [17] Colas, C., Sigaud, O., Oudeyer, P.Y.: How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint arXiv:1806.08295 (2018)
  * [18] Collins, G.S., Reitsma, J.B., Altman, D.G., Moons, K.G.: Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD). Circulation 131(2), 211–219 (2015), publisher: American Heart Association
  * [19] Cremonesi, P., Jannach, D.: Progress in Recommender Systems Research: Crisis? What Crisis? AI Magazine 42(3), 43–54 (2021)
  * [20] De Cristofaro, E.: A critical overview of privacy in machine learning. IEEE Security & Privacy 19(4), 19–27 (2021)
  * [21] Dror, R., Baumer, G., Bogomolov, M., Reichart, R.: Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics 5, 471–486 (2017)
  * [22] Ferigo, D., Traversaro, S., Metta, G., Pucci, D.: Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning. In: 2020 IEEE/SICE International Symposium on System Integration (SII). pp. 885–890 (2020)
  * [23] Fidler, F., Wilcox, J.: Reproducibility of Scientific Results. In: Zalta, E.N. (ed.) The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, summer 2021 edn. (2021)
  * [24] Gibney, E.: Could machine learning fuel a reproducibility crisis in science? Nature 608(7922), 250–251 (2022), bandiera_abtest: a Cg_type: News Number: 7922 Publisher: Nature Publishing Group Subject_term: Machine learning, Publishing, Mathematics and computing
  * [25] Goodman, S.N., Fanelli, D., Ioannidis, J.P.: What does research reproducibility mean? Science translational medicine 8(341), 341ps12–341ps12 (2016)
  * [26] Gundersen, O.E.: The fundamental principles of reproducibility. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 379(2197), 20200210 (2021), publisher: Royal Society
  * [27] Gundersen, O.E., Coakley, K., Kirkpatrick, C., Gil, Y.: Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610 (2022)
  * [28] Gundersen, O.E., Coakley, K., Kirkpatrick, C., Gil, Y.: Sources of irreproducibility in machine learning: A review. arXiv preprint arXiv:2204.07610 (2023)
  * [29] Gundersen, O.E., Kjensmo, S.: State of the Art: Reproducibility in Artificial Intelligence. Proceedings of the AAAI Conference on Artificial Intelligence 32(1) (2018)
  * [30] Gundersen, O.E., Shamsaliei, S., Isdahl, R.J.: Do machine learning platforms provide out-of-the-box reproducibility? Future Generation Computer Systems 126, 34–47 (2022)
  * [31] Hardwicke, T.E., Mathur, M.B., MacDonald, K., Nilsonne, G., Banks, G.C., Kidwell, M.C., Hofelich Mohr, A., Clayton, E., Yoon, E.J., Henry Tessler, M., Lenne, R.L., Altman, S., Long, B., Frank, M.C.: Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science 5(8), 180448 (2018), publisher: Royal Society
  * [32] Hartley, M., Olsson, T.S.G.: dtoolAI: Reproducibility for Deep Learning. Patterns 1(5), 100073 (2020)
  * [33] Heil, B.J., Hoffman, M.M., Markowetz, F., Lee, S.I., Greene, C.S., Hicks, S.C.: Reproducibility standards for machine learning in the life sciences. Nature Methods 18(10), 1132–1135 (2021)
  * [34] Hong, S.Y., Koo, M.S., Jang, J., Kim, J.E.E., Park, H., Joh, M.S., Kang, J.H., Oh, T.J.: An Evaluation of the Software System Dependency of a Global Atmospheric Model. Monthly Weather Review 141(11), 4165–4172 (2013), publisher: American Meteorological Society Section: Monthly Weather Review
  * [35] Hosseini, M., Horbach, S.P.J.M., Holmes, K.L., Ross-Hellauer, T.: Open Science at the Generative AI Turn: An Exploratory Analysis of Challenges and Opportunities (2024)
  * [36] Hutson, M.: Artificial intelligence faces reproducibility crisis Unpublished code and sensitivity to training conditions make many claims hard to verify. Science 359(6377), 725–726 (2018)
  * [37] Hutson, M.: Missing data hinder replication of artificial intelligence studies. Science (2018)
  * [38] Ivanescu, A.E., Li, P., George, B., Brown, A.W., Keith, S.W., Raju, D., Allison, D.B.: The Importance of Prediction Model Validation and Assessment in Obesity and Nutrition Research. International journal of obesity (2005) 40(6), 887–894 (2016)
  * [39] Kamel Rahimi, A., Canfell, O.J., Chan, W., Sly, B., Pole, J.D., Sullivan, C., Shrapnel, S.: Machine learning models for diabetes management in acute care using electronic medical records: A systematic review. International Journal of Medical Informatics 162, 104758 (2022)
  * [40] Kapoor, S., Narayanan, A.: Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4(9), 100804 (2023)
  * [41] Khetarpal, K., Ahmed, Z., Cianflone, A., Islam, R., Pineau, J.: RE-EVALUATE: Reproducibility in Evaluating Reinforcement Learning Algorithms. In: 2nd Reproducibility in Machine Learning Workshop at ICML (2018)
  * [42] Koenigstorfer, F., Haberl, A., Kowald, D., Ross-Hellauer, T., Thalmann, S.: Black Box or Open Science? Assessing Reproducibility-Related Documentation in AI Research (2024)
  * [43] Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: 42nd European Conference on IR Research, ECIR 2020. pp. 35–42. Springer (2020)
  * [44] Leventi-Peetz, A.M., Östreich, T.: Deep learning reproducibility and explainable ai (xai). arXiv preprint arXiv:2202.11452 (2022)
  * [45] Liu, R., Shah, N.B.: Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622 (2023)
  * [46] McDermott, M.B.A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., Ghassemi, M.: Reproducibility in machine learning for health research: Still a ways to go. Science Translational Medicine 13(586), eabb1655 (2021)
  * [47] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM computing surveys (CSUR) 54(6), 1–35 (2021)
  * [48] Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I.D., Gebru, T.: Model Cards for Model Reporting. In: Proceedings of the Conference on Fairness, Accountability, and Transparency. pp. 220–229. FAT* ’19, ACM, New York, NY, USA (2019)
  * [49] Mongan, J., Moy, L., Kahn, C.E.: Checklist for Artificial Intelligence in Medical Imaging (CLAIM): A Guide for Authors and Reviewers. Radiology: Artificial Intelligence 2(2), e200029 (2020), publisher: Radiological Society of North America
  * [50] Muellner, P., Kowald, D., Lex, E.: Robustness of meta matrix factorization against strict privacy constraints. In: 43rd European Conference on IR Research, ECIR 2021. pp. 107–119. Springer (2021)
  * [51] Munafò, M.R., Nosek, B.A., Bishop, D.V.M., Button, K.S., Chambers, C.D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.J., Ware, J.J., Ioannidis, J.P.A.: A manifesto for reproducible science. Nature Human Behaviour 1(1), 1–9 (2017), publisher: Nature Publishing Group
  * [52] Nagarajan, P., Warnell, G., Stone, P.: Deterministic implementations for reproducibility in deep reinforcement learning. arXiv preprint arXiv:1809.05676 (2018)
  * [53] Navarro Leija, O.S., Shiptoski, K., Scott, R.G., Wang, B., Renner, N., Newton, R.R., Devietti, J.: Reproducible Containers. In: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems. pp. 167–182. ACM, Lausanne Switzerland (2020)
  * [54] Nosek, B.A., Beck, E.D., Campbell, L., Flake, J.K., Hardwicke, T.E., Mellor, D.T., van ’t Veer, A.E., Vazire, S.: Preregistration Is Hard, And Worthwhile. Trends in Cognitive Sciences 23(10), 815–818 (2019)
  * [55] Open Science Collaboration: Estimating the reproducibility of psychological science. Science 349(6251), aac4716 (2015), publisher: American Association for the Advancement of Science
  * [56] Panaretos, D., Koloverou, E., Dimopoulos, A.C., Kouli, G.M., Vamvakari, M., Tzavelas, G., Pitsavos, C., Panagiotakos, D.B.: A comparison of statistical and machine-learning techniques in evaluating the association between dietary patterns and 10-year cardiometabolic risk (2002–2012): the ATTICA study. British Journal of Nutrition 120(3), 326–334 (2018), publisher: Cambridge University Press
  * [57] Paullada, A., Raji, I.D., Bender, E.M., Denton, E., Hanna, A.: Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns 2(11), 100336 (2021)
  * [58] Peng, R.: The reproducibility crisis in science: A statistical counterattack. Significance 12(3), 30–32 (2015)
  * [59] Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., Larochelle, H.: Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). The Journal of Machine Learning Research 22(1), 164:7459–164:7478 (2021)
  * [60] Pouchard, L., Lin, Y., Van Dam, H.: Replicating Machine Learning Experiments in Materials Science. In: Parallel Computing: Technology Trends, pp. 743–755. IOS Press (2020)
  * [61] Provenzano, D., Rao, Y.J., Goyal, S., Haji-Momenian, S., Lichtenberger, J., Loew, M.: Radiologist vs Machine Learning: A Comparison of Performance in Cancer Imaging. In: 2021 IEEE Applied Imagery Pattern Recognition Workshop (AIPR). pp. 1–10 (2021), iSSN: 2332-5615
  * [62] Pushkarna, M., Zaldivar, A., Kjartansson, O.: Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI. In: Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. pp. 1776–1826. FAccT ’22, ACM, New York, NY, USA (2022)
  * [63] Quaranta, L., Calefato, F., Lanubile, F.: A Taxonomy of Tools for Reproducible Machine Learning Experiments. CEUR Workshop Proceedings (2022)
  * [64] Raff, E.: A Step Toward Quantifying Independently Reproducible Machine Learning Research. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019)
  * [65] Raste, S., Singh, R., Vaughan, J., Nair, V.N.: Quantifying Inherent Randomness in Machine Learning Algorithms. SSRN Electronic Journal (2022)
  * [66] Rougier, N.P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L.A., Benureau, F.C.Y., Brown, C.T., Buyl, P.d., Caglayan, O., Davison, A.P., Delsuc, M.A., Detorakis, G., Diem, A.K., Drix, D., Enel, P., Girard, B., Guest, O., Hall, M.G., Henriques, R.N., Hinaut, X., Jaron, K.S., Khamassi, M., Klein, A., Manninen, T., Marchesi, P., McGlinn, D., Metzner, C., Petchey, O., Plesser, H.E., Poisot, T., Ram, K., Ram, Y., Roesch, E., Rossant, C., Rostami, V., Shifman, A., Stachelek, J., Stimberg, M., Stollmeier, F., Vaggi, F., Viejo, G., Vitay, J., Vostinar, A.E., Yurchak, R., Zito, T.: Sustainable computational science: the ReScience initiative. PeerJ Computer Science 3, e142 (2017), publisher: PeerJ Inc.
  * [67] Rouzrokh, P., Khosravi, B., Faghani, S., Moassefi, M., Vera Garcia, D.V., Singh, Y., Zhang, K., Conte, G.M., Erickson, B.J.: Mitigating Bias in Radiology Machine Learning: 1. Data Handling. Radiology: Artificial Intelligence 4(5), e210290 (2022)
  * [68] Sayre, F., Riegelman, A.: The Reproducibility Crisis and Academic Libraries | Sayre | College & Research Libraries. College and Research Libraries (2018)
  * [69] Schlegel, M., Sattler, K.U.: Management of Machine Learning Lifecycle Artifacts: A Survey. ACM SIGMOD Record 51(4), 18–35 (2023)
  * [70] Semmelrock, H., Kopeinik, S., Theiler, D., Ross-Hellauer, T., Kowald, D.: Reproducibility in machine learning-driven research. arXiv preprint arXiv:2307.10320 (2023)
  * [71] Soiland-Reyes, S., Sefton, P., Crosas, M., Castro, L.J., Coppens, F., Fernández, J.M., Garijo, D., Grüning, B., La Rosa, M., Leo, S., Ó Carragáin, E., Portier, M., Trisovic, A., Community, R.C., Groth, P., Goble, C.: Packaging research artefacts with RO-Crate. Data Science 5(2), 97–138 (2022), publisher: IOS Press
  * [72] Strømland, E.: Preregistration and reproducibility. Journal of Economic Psychology 75, 102143 (2019)
  * [73] Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., Casas, D.d.L., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., et al.: Deepmind control suite. arXiv preprint arXiv:1801.00690 (2018)
  * [74] Tatman, R., VanderPlas, J., Dane, S.: A Practical Taxonomy of Reproducibility for Machine Learning Research. CEUR Workshop Proceedings (2018)
  * [75] Trosten, D.J.: Questionable practices in methodological deep learning research. In: Proceedings of the Northern Lights Deep Learning Workshop. vol. 4 (2023)
  * [76] Walonoski, J., Kramer, M., Nichols, J., Quina, A., Moesel, C., Hall, D., Duffett, C., Dube, K., Gallagher, T., McLachlan, S.: Synthea: An approach, method, and software mechanism for generating synthetic patients and the synthetic electronic health care record. Journal of the American Medical Informatics Association: JAMIA 25(3), 230–238 (2018)
  * [77] Wiggins, B.J., Christopherson, C.D.: The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology 39, 202–217 (2019)
  * [78] Wolff, R.F., Moons, K.G., Riley, R.D., Whiting, P.F., Westwood, M., Collins, G.S., Reitsma, J.B., Kleijnen, J., Mallett, S.: PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies. Annals of Internal Medicine 170(1), 51–58 (2019), publisher: American College of Physicians
  * [79] Xu, R., Baracaldo, N., Joshi, J.: Privacy-preserving machine learning: Methods, challenges and directions. arXiv preprint arXiv:2108.04417 (2021)
  * [80] Yildiz, B., Hung, H., Krijthe, J., Liem, C., Loog, M., Migut, G., Oliehoek, F., Panichella, A., Pawełczak, P., Picek, S., de Weerdt, M., van Gemert, J.: ReproducedPapers.org: Openly Teaching and Structuring Machine Learning Reproducibility. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) pp. 3–11 (2021)
  * [81] Young, K., Tian, T.: Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1903.03176 (2019)
  * [82] Zhang, D., Song, J., Dharmarajan, S., Jung, T.H., Lee, H., Ma, Y., Zhang, R., Levenson, M.: The Use of Machine Learning in Regulatory Drug Safety Evaluation. Statistics in Biopharmaceutical Research pp. 1–5 (2022)


# A Comparison of HDF5, Zarr, and netCDF4
in Performing Common I/O Operations

Sriniket Ambatipudi  Dulles High School
Houston, TX, USA
Email<EMAIL_ADDRESS>Suren Byna  Lawrence Berkeley National
Laboratory
Berkeley, CA, USA
Email<EMAIL_ADDRESS>
###### Abstract

Scientific data is often stored in files because of the simplicity they
provide in managing, transferring, and sharing data. These files are typically
structured in a specific arrangement and contain metadata to understand the
structure the data is stored in. There are numerous file formats in use in
various scientific domains that provide abstractions for storing and
retrieving data. With the abundance of file formats aiming to store large
amounts of scientific data quickly and easily, a question that arises is,
“Which scientific file format is best suited for a general use case?” In this
study, we compiled a set of benchmarks for common file operations, i.e.,
create, open, read, write, and close, and used the results of these benchmarks
to compare three popular formats: HDF5, netCDF4, and Zarr.

_Note: This paper is currently a work in progress, and our results are
representative for a general-purpose use case in which datasets are small in
size and are not using optimizations such as HDF5 or netCDF4 chunking,
asynchronous I/O, sub-filing. We welcome any comments or suggestions regarding
the benchmark located athttps://github.com/asriniket/File-Format-Testing_.

## General Terms / Keywords

Scientific File Formats, HDF5, netCDF4, Zarr

##  I Introduction

With the rapid advancement of experiments, observations, and simulations in
recent years, various domains of science are producing enormous amounts of
data. For example, the Large Hadron Collider (LHC) experiments at CERN produce
90 petabytes of data per year [1]. NASA’s Climate Data Services (CDS) simulate
our planet’s weather and climate models from hours to millennia and produce
datasets up to petabytes in volume [2].

Much of the data produced in scientific experiments, observations, and
simulations are stored in files with various formats. Scientific file formats
offer a medium to store scientific data for long-term processing, which is of
great importance to researchers. Each file format has specific characteristics
that make it suited for a particular use case, and this makes choosing the
appropriate file format an important task. The typical content of scientific
files includes data with a structure and metadata describing the structure and
the data within. Data in these files are often structured as arrays [3]. The
metadata that describes the data often contains the origins of the data,
configurations used in generating or collecting the data, and the location of
the data in the file format for easy access. As such, scientific file formats
will often provide functionality for the storing and retrieval of data and
metadata in files.

There are numerous file formats in existence, such as HDF5 [4], netCDF4 [5],
ROOT [6], Zarr [7], and many more [8]. Each of these file formats exhibits
different performance characteristics and was designed to accomplish a
specific task. For example, the High Energy Physics (HEP) community developed
the ROOT framework to meet the high-performance requirements for multithreaded
read and write operations and support object-oriented programming. As a result
of these design specifications, the ROOT framework is used by CERN in its
research with the Large Hadron Collider [9]. On the other hand, file formats
such as netCDF4 or HDF5 are often used in more general use case scenarios
because of their self-describing capabilities, which allow the storing of
metadata to describe the data within a file. Such self-describing capabilities
allow these file formats to be used in a multitude of applications, like how
HDF5 is used in astronomy, medicine, physics, and many more fields [4].

Because there are a multitude of file formats that are available to store
scientific data, the question of which file format is best suited for a
general use case arises. Previous research [10, 11] has mainly focused on
testing the performance characteristics of individual file formats, like how
HDF5 was tested for its performance in reading a subset of a large array [10]
or how netCDF4’s performance characteristics were analyzed [11]. The first
test revealed that when working with an HDF5 file in Python, the fastest way
to read data is to memory map the file with NumPy, bypassing the HDF5 Python
API (h5py). Memory mapping involves mapping a file’s contents into memory, and
this means that data within a file can be accessed if the location of the data
in terms of an offset is known. The hierarchical structure of file formats
such as HDF5 means that accessing the data within is relatively simple, as the
file’s metadata stores the location of individual datasets. This test was
useful in analyzing the shortcomings of HDF5 in a particular use case,
allowing potential users to reconsider whether the HDF5 file format would be
best suited for their use case. In the second experiment, conducted by the HDF
Group, the netCDF4 file format was tested for its performance characteristics
and then compared to its predecessor, netCDF3. The results of this experiment
showed that netCDF4 generally had slower write speeds than netCDF3, but it had
faster read speeds due to netCDF4’s use of the HDF5 library internally.

This type of testing is useful for analyzing the performance characteristics
of one file format and its shortcomings in specific use cases, but when the
performance characteristics of one file format must be compared to the
performance characteristics of other file formats, a benchmark offers itself
as a viable option because it allows for the objective measurement of the
speed at which each file format is able to perform a specific task. Such a
benchmark has the potential to be a valuable asset to researchers, as it
allows them to choose the file format that is not only suited for their
particular use case but also performs the best in comparison to its
alternatives. This allows the researchers to have ease of access when storing
and modifying data at fast speeds, allowing more time and effort to be put
elsewhere in their project. In this work, we developed a benchmark to compare
the read and write speeds of three multipurpose scientific file formats (HDF5,
netCDF4, and Zarr). This benchmark writes randomized data to a specified
number of datasets within a file and measures the time taken to write the data
to each dataset and the time taken to read the contents of each dataset,
allowing objective comparisons to be drawn between the three file formats’
performance in different operations.

In the remainder of the paper, we first provide a brief background in §II to
the three file formats we used in our evaluation in this study. In §III, we
describe the read and write benchmarks we used in the evaluation. In §IV, we
provide details of the system we used for comparing and evaluating the
performance of the three file formats under different workloads.

##  II Background

###  II-A HDF5

HDF5, or the Hierarchical Data Format 5, is a file format designed to store a
large amount of data in an organized manner. Typically characterized by a
.hdf5 or .h5 file extension, this file format stores data in a manner very
similar to that of a file system. This file format’s primary data models are
groups and datasets. Groups are the overarching structure, and they can hold
other groups or datasets. Datasets store raw data values of a specified data
type and are usually stored within groups [12]. A feature of HDF5 is that it
is able to store data consisting of different data types within the same file
[13]. As mentioned earlier, this file format is self-describing, meaning that
all the groups and datasets within the file format contain metadata describing
their contents. This allows for the data within the file to be mapped in
memory, provided the API supports it. Generally speaking, users use the HDF5
API to issue commands to a lower-level driver, which is in charge of accessing
the file and performing the requested operations [14]. Because the file format
is open source, there has been widespread API support across most modern
languages (Python, C++, and Java).

###  II-B netCDF4

netCDF4 is a file format that is designed to store array-oriented data and is
characterized by a .netc file extension. It stores data in a manner similar to
HDF5, with groups serving as the overarching data structure. Within a group,
there can be other groups or variables. Variables are akin to HDF5 datasets.
Unlike HDF5 datasets, netCDF4 variables cannot be resized once they are
created [15]. To circumvent this, variables can be declared with an unlimited
size in a specified dimension. Similar to HDF5, netCDF4 is a self-describing
file format, and this means that groups and variables both contain metadata
describing their contents. Unlike its predecessor, netCDF3, netCDF4 uses HDF5
as its backend, allowing it to achieve faster read times [11].

###  II-C Zarr

Zarr is a file format that is designed to store large arrays of data and is
characterized by a .zarr file extension. Because it is based on NumPy, it is
geared mainly towards Python users. Similar to both HDF5 and netCDF4, Zarr is
also a hierarchical, self-describing file format that has groups as the
overarching file structure. Each group contains datasets, which are
representative of multidimensional arrays of a homogeneous data type.
Furthermore, the API for this file format was designed to be similar to h5py
(HDF5’s Python API), and as a result, it includes functions based on h5py’s
functions, namely the group creation function [16]. One advantage to using
Zarr is that it provides multiple options to store data by allowing a user to
store a file in memory, in the file system, or in other storage systems with a
similar interface to the first two options [16].

##  III Benchmarks

As a benchmark is being used to compare the performance of the file formats,
the benchmark must only test features of the file format that are supported by
all the file formats being tested. To accomplish this task, we programmed our
benchmark in Python, and this means we will rely on the Python APIs for each
file format being tested to perform the requested operations. Our benchmark
compares the time taken to create a dataset, write data to a dataset, and
finally open that dataset at a later time and read its contents. This can be
categorized into two main types of operations—the writing operation and the
reading operation. Both are very important features to test in a file format,
as the end goal of a file format is to store data for long-term processing.
The faster write and read times are not only indicative of better performance
characteristics but also have a tangible effect on the improvement of an end-
user’s workflow, as less time would be spent performing operations that are
not directly relevant to the task at hand.

To allow for greater flexibility when benchmarking the file formats, we added
a configuration system in which the user is able to specify the testing
parameters such as the number of datasets to create within the file and the
dimensions of the array that will be written to each dataset by editing a
.yaml configuration file. After the benchmark is done, the program then stores
the times taken across multiple trials in a .csv file and plots the data in
the .csv file with matplotlib.pyplot to allow a user to make a definitive
comparison between the file formats being tested. Below, the main operations,
the write operation and the read operation, will be discussed in-depth.

###  III-A Write Benchmark

The write operation is the first operation to be tested in the benchmark. It
creates files with the filename as specified in the configuration file and
extensions .hdf5 for HDF5 files, .netc for netCDF4 files, and .zarr for Zarr
files. The file is placed inside a folder named Files/, to help reduce clutter
in the working directory. Taking information from the configuration file, a
sample data array is generated with dimensions and length as specified. This
sample data array consists of randomly-generated 32-bit floats. Then, the
program creates a dataset within the file and writes the sample data array to
the dataset. This process of generating a sample data array, creating a
dataset, and populating it with the values from the sample data array is
repeated until the benchmark has created the number of datasets as specified
by the configuration file. After the file is populated with data, the
benchmark copies the file to a directory named Files Read/ and renames the
file to avoid any caching effects that may interfere with the read times.
There are numerous ways to mitigate such caching effects, such as waiting for
an extended period of time, but simply moving the file to another directory
and renaming the file is the quickest and easiest way to mitigate the effects
of caching in interfering with the times taken to read from the file. The time
taken to create all the datasets and populate them with data is divided by the
number of datasets to find the average time taken to create and populate one
dataset. Both of these times are then returned to the main program, where they
are written to the .csv output file.

###  III-B Read Benchmark

The benchmark now opens the copied file in the Files Read/ directory and
begins testing the read operations of the three file formats. This operation
consists of opening each dataset within the file and printing its contents to
the standard output. The time taken to open all the datasets and the time
taken to read from all the datasets are once again divided by the number of
datasets within the file to find the average time taken to open and read one
dataset. Both of these times are then returned to the main program, where they
are also written to the .csv output file.

This process of running the write operation benchmark and the read operation
benchmark is then repeated multiple times in order to ensure the consistency
of the data gathered. To avoid filling up the disk with generated test files,
the Files/ and Files Read/ directories are deleted between trials. Finally,
the data from the .csv file are averaged out with pandas and plotted with
matplotlib.pyplot to allow for visualizing a comparison between the tested
file formats in a given operation.

##  IV Performance Evaluation

###  IV-A Experimental setup

The three file formats were tested on a computer running Ubuntu 18.04.5 with
an Intel(R) Xeon(R) Silver 4215R CPU, 196 Gigabytes of RAM, and 960 Gigabytes
of solid-state storage provided by a Micron 5200 Series SSD. The version of
h5py used to test the HDF5 file format was 3.6.0. The version of netCDF4 used
to test the netCDF4 file format was 1.5.8. The version of zarr used to test
the Zarr file format was 2.11.0.

The benchmark parameters that were used in each run of the test can be found
in the tables to the right. Note that the Test Name parameter is automatically
generated by the benchmark and is used to create the generated plot’s title.

Test Name |  2048 Datasets of
---|---
|  [128] Elements
File Name |  2048-Vector
Number Datasets |  2048
Number Elements |  [128]

Test Name |  2048 Datasets of
---|---
|  [128, 128] Elements
File Name |  2048-Matrix
Number Datasets |  2048
Number Elements |  [128, 128]

Test Name |  2048 Datasets of
---|---
|  [128, 128, 128] Elements
File Name |  2048-Tensor
Number Datasets |  2048
Number Elements |  [128, 128, 128]

Test Name |  2048 Datasets of
---|---
|  [256] Elements
File Name |  2048-Datasets
Number Datasets |  2048
Number Elements |  [256]

Test Name |  4096 Datasets of
---|---
|  [256] Elements
File Name |  4096-Datasets
Number Datasets |  4096
Number Elements |  [256]

Test Name |  8192 Datasets of
---|---
|  [256] Elements
File Name |  8192-Datasets
Number Datasets |  8192
Number Elements |  [256]

###  IV-B Data

(a) Create / Open Times

(b) Read / Write Times

Figure 1: 2048 Datasets of [128] Elements

(a) Create / Open Times

(b) Read / Write Times

Figure 2: 2048 Datasets of [128, 128] Elements

(a) Create / Open Times

(b) Read / Write Times

Figure 3: 2048 Datasets of [128, 128, 128] Elements

(a) Create / Open Times

(b) Read / Write Times

Figure 4: 2048 Datasets of [256] Elements

(a) Create / Open Times

(b) Read / Write Times

Figure 5: 4096 Datasets of [256] Elements

(a) Create / Open Times

(b) Read / Write Times

Figure 6: 8192 Datasets of [256] Elements

###  IV-C Discussion

Figure 1 shows the results when 2,048 datasets are created and populated with
a one-dimensional array containing 128 32-bit floats. This graph shows that
the time taken to create and open a dataset in netCDF4 is much faster than
that of HDF5’s or Zarr’s. In comparison to Zarr, HDF5 takes less time to
create a dataset, but it takes slightly more time to open a dataset. When it
comes to writing to datasets or reading from datasets, HDF5 and Zarr share
very similar times in both operations, with netCDF4 trailing by a large
margin.

Figure 2 shows the results when 2,048 datasets are created and populated with
a two-dimensional array containing 128 elements in each dimension. The results
from this test are almost identical to the results from the previous test,
both in terms of the trend and the time taken to complete each operation.

Figure 3 shows the results when 2,048 datasets are created and populated with
a three-dimensional array containing 128 elements in each dimension. The
results from this test follow the same trend as the past two tests, but the
times taken to complete each operation are almost double the times taken in
the past two tests.

These past three bar graphs show the tests in which the number of datasets is
held constant while increasing the number of dimensions in the data array, but
the next three bar graphs will involve increasing the number of datasets while
holding the size of the data array constant in order to measure the effect of
increasing the number of datasets on file-format performance.

Figure 4 shows the results when 2,048 datasets are created and populated with
a one-dimensional array containing 256 elements. The results from this test
mirror those from Figure 1, and this is to be expected as the number of
datasets in both tests is the same, with the size of each dataset varying
slightly.

Figure 5 shows the results when 4,096 datasets are created and populated with
a one-dimensional array containing 256 elements, and Figure 6 shows the
results when 8,192 datasets are created and populated with a one-dimensional
array containing 256 elements. Both graphs are almost identical to Figure 4,
meaning that the number of datasets most likely has no impact on the average
time taken to perform the operations requested.

###  IV-D Write Benchmark Discussion

The results of this benchmark show that a general trend is that when creating
a dataset, netCDF4 takes the least time and is followed by HDF5, which is
followed by Zarr.

When actually writing data to a file, HDF5 takes the least time to write data
to a dataset and is followed by Zarr, which is followed by netCDF4—taking on
average more than double the time of HDF5.

###  IV-E Read Benchmark Discussion

The read benchmarks show results similar to those from the write benchmark.
netCDF4 takes the least time to open a dataset and is followed by Zarr, which
is followed by HDF5.

When reading the data by printing the dataset values to the standard output,
HDF5 takes the least time to read a dataset and is followed by Zarr, which is
followed by netCDF4.

##  V Conclusions

In this paper, we demonstrated a method in which the performance of a file
format can be compared to that of another file format through the running of a
benchmark that tests performance in operations like create, open, read, write,
and close. This paper focused specifically on benchmarking three file formats:
HDF5, netCDF4, and Zarr, as these three file formats are considered to be
general-purpose scientific file formats due to their storing of various types
of data in a hierarchical manner, similar to a file system.

The benchmark was conducted in Python due to the language’s widespread use in
numerous scientific applications, and as such, the Python API for each file
format was tested. To determine the performance of a file format, the time
taken to create a dataset, write data to the dataset, open the dataset once
the file is closed, and read data from the dataset was measured and plotted in
a bar graph. The results of the benchmark show that HDF5 is fastest in reading
or writing to a dataset, netCDF4 is fastest in creating or opening a dataset,
and Zarr generally trails right behind HDF5 in performance.

Future work for this benchmark would include: expanding support to other
programming languages, as this would reveal any potential bottlenecks within
the language-specific API for a file format; testing more file formats in
order to better determine which file format is the fastest; and testing more
aspects of a file format, which may include testing performance in specific
scenarios (i.e., reading a small subset of a dataset, overwriting a dataset).
The code for the benchmark can be found here:
https://github.com/asriniket/File-Format-Testing. We are evaluating further
the overheads of using the Python API on the observed performance as well as
the impact of caching. Considering the small size of the data, the observed
results may have been impacted by caching. This caching effect will typically
be reduced when the data sizes are in gigabytes (GB). We also note that many
applications work with smaller amounts of data that we used in this study. We
encourage readers to try out the benchmarks provided in the GitHub repository
and contribute any optimizations.

##  VI Acknowledgments

This effort was supported in part by the U.S. Department of Energy (DOE),
Office of Science, Office of Advanced Scientific Computing Research (ASCR)
under contract number DE-AC02-05CH11231 with LBNL.

## References

  * [1] CERN  “CERN - Storage”  URL: https://home.cern/science/computing/storage
  * [2] NASA  “NASA’s Climate Data Services (CDS)”  URL: https://www.nccs.nasa.gov/services/climate-data-services
  * [3] Arie Shoshani and Doron Rotem  “Scientific data management. Challenges, technology, and development”  In _Scientific Data Management: Challenges, Technology, and Deployment_ , 2009  DOI: 10.1201/9781420069815
  * [4] HDFGroup  “The HDF5® Library & File Format - The HDF Group”  URL: https://www.hdfgroup.org/solutions/hdf5/
  * [5] UCAR/Unidata  “Unidata — NetCDF”  URL: https://www.unidata.ucar.edu/software/netcdf/
  * [6] CERN  “ROOT: analyzing petabytes of data, scientifically.”  URL: https://root.cern/
  * [7] Zarr Developers  “Zarr”  URL: https://zarr.readthedocs.io/en/stable/
  * [8] Wikipedia  “List of file formats”  URL: https://en.wikipedia.org/wiki/List_of_file_formats#Scientific_data_(data_exchange)
  * [9] Barbara Warmbein  “Big data takes ROOT”  URL: https://home.cern/news/news/computing/big-data-takes-root
  * [10] Cyrille Rossant  “Moving away from HDF5”  URL: https://cyrille.rossant.net/moving-away-hdf5/
  * [11] Choonghwan Lee, MuQun Yang and Ruth Aydt  “NetCDF-4 Performance Report”  URL: https://support.hdfgroup.org/pubs/papers/2008-06_netcdf4_perf_report.pdf
  * [12] Leah A. Wasser  “Hierarchical Data Formats - What is HDF5?”  URL: https://www.neonscience.org/resources/learning-hub/tutorials/about-hdf5
  * [13] HDFGroup  “Introduction to HDF5”  URL: https://portal.hdfgroup.org/display/HDF5/Introduction+to+HDF5
  * [14] HDFGroup  “Chapter 3: The HDF5 File”  URL: https://support.hdfgroup.org/HDF5/doc/UG/FmSource/08_TheFile_favicon_test.html
  * [15] UCAR/Unidata  “NetCDF4 API Documentation”  URL: https://unidata.github.io/netcdf4-python/
  * [16] Zarr Developers  “Zarr”  URL: https://zarr.readthedocs.io/en/stable/tutorial.html#groups/