11institutetext: Know-Center GmbH, Graz, Austria
11email<EMAIL_ADDRESS>22institutetext: Graz
University of Technology, Graz, Austria
22email<EMAIL_ADDRESS>
# Reproducibility in Machine Learning-Driven Research

Harald Semmelrock 22 Simone Kopeinik 11 Dieter Theiler 11 Tony Ross-Hellauer
1122 Dominik Kowald${}^{\textrm{(\Letter)}}$ 1122

###### Abstract

Research is facing a reproducibility crisis, in which the results and findings
of many studies are difficult or even impossible to reproduce. This is also
the case in machine learning (ML) and artificial intelligence (AI) research.
Often, this is the case due to unpublished data and/or source-code, and due to
sensitivity to ML training conditions. Although different solutions to address
this issue are discussed in the research community such as using ML platforms,
the level of reproducibility in ML-driven research is not increasing
substantially. Therefore, in this mini survey, we review the literature on
reproducibility in ML-driven research with three main aims: (i) reflect on the
current situation of ML reproducibility in various research fields, (ii)
identify reproducibility issues and barriers that exist in these research
fields applying ML, and (iii) identify potential drivers such as tools,
practices, and interventions that support ML reproducibility. With this, we
hope to contribute to decisions on the viability of different solutions for
supporting ML reproducibility.

###### Keywords:

Machine Learning Artificial Intelligence Reproducibility Replicability

##  1 Introduction

Similar to other scientific fields [4] [41], research in artificial
intelligence (AI) in general, and machine learning (ML) in particular, is
facing a reproducibility crisis [24]. Here, especially unpublished source-code
and sensitivity to ML training conditions make it nearly impossible to
reproduce existing ML publications, which also makes it very hard to verify
the claims and findings stated in the publications.

One potential solution for enhancing reproducibility in ML is the use of ML
platforms such as OpenML, Google Cloud ML, Microsoft Azure ML or Kaggle.
However, in a recent study [21] found that the same experiment executed on
different platforms leads to different results. This suggests that still a lot
of research is needed until out-of-the-box reproducibility can be provided.
However, a systematic overview of the literature on ML reproducibility is
still missing, especially with respect to the barriers and drivers of
reproducibility that can be found in the literature. An example of a driver
could be code sharing or hosting reproducibility tracks/challenges at
scientific conferences [16]. One example for this is the reproducibility track
at the European Conference on Information Retrieval (ECIR) [28, 31]
With respect to potential barriers, it is still not clear to what extent the
use of ML could even fuel reproducibility issues [17], e.g., via bad ML
practices such as data leakage [26].

This work aims to provide an overview of the situation and identify the
different drivers and barriers present. This should allow for a better
understanding of the following three aspects:

  * •

The situation of ML reproducibility in different research fields (see Section
2).

  * •

Reproducibility issues that exist in research fields applying ML, and the
barriers that cause these issues (see Section 3).

  * •

The drivers that support ML reproducibility, including different tools,
practices, and interventions (see Section 4).

###  1.1 Degrees of Reproducibility

According to [20], there are three degrees of reproducibility in ML, which can
be seen in Table 1.

Table 1: Different degrees of reproducibility according to [20] Type |  Requirement
---|---
(R1) Experiment Reproducibility |  The same implementation (including same software versions, hyperparameters, etc.) of the ML method (i.e., the algorithm) must produce (exactly) the same results when using the same training and test data. If one produces different results, then only the hardware where the ML experiment was reproduced could be the reason for this difference.
(R2) Data Reproducibility |  An alternative implementation of the ML method must produce (almost) the same results when executed using the same data. If there are differences in the results, then probably differences in the concrete implementation (e.g., different versions of a library) are the reasons for this.
(R3) Method Reproducibility |  An alternative implementation of the ML method executed on different data must produce the same results (or at least findings).

We see that R1 is concerned with the exact reproduction of results, i.e.,
output, over multiple runs of the same ML method implementation and data. This
is also often referred to as computational reproducibility. R2, however, is a
bit more general, in a way that the ML method is implemented differently, but
should still be able to produce almost the same results given the same data.
This “Data Reproducibility” requirement makes sure that the information drawn
from the data is consistent and not too heavily dependent on minor
implementation differences. R3 generalizes this even more and is only
concerned with the findings. Relying only on the use of the same ML method,
findings should always be consistent, no matter the exact implementation or
exact data. Accordingly, R3 leads to the highest form of generalizability but
also to the weakest form of reproducibility. This is conversely true for R1,
which leads to the highest form of reproducibility and lowest form of
generalizability. This interplay between generalizability and reproducibility
of the different degrees can be seen in Figure 1.

Figure 1: Degrees of reproducibility. Adapted from [20]

Furthermore, the structure of Figure 1 gives information about what building
blocks the different degrees of reproducibility are concerned with. Thus, R3
is only concerned with the ML method, R2 is concerned with the ML method and
data and R1 requires all three building blocks: the ML method, the data, and
the experiment. This is where the names of the degrees come from.

###  1.2 Reproducibility versus Replicability

The two terms Reproducibility and Replicability are often used interchangeably
in science, even though the difference is often crucial.
For computer science and ML, the Association for Computing Machinery (ACM)
distinguishes the terms in the following way:

  * •

Reproducibility - the results can be obtained by a different team with the
same experimental setup

  * •

Replicability - the results can be obtained by a different team with a
different experimental setup

These definitions harmonize with the definitions seen in the literature [6].
Comparing the definitions to the different degrees of reproducibility coined
by [20], we can see that degree R1 is somewhat similar to what is here
referred to as Reproducibility, and both R2 and R3 can be seen as similar to
what is referred to as Replicability. For the purpose of this work, the
definitions concerning the three degrees of reproducibility [20] will be used.

##  2 ML Reproducibility in Different Research Fields

We mainly focus on two research fields, in which ML reproducibility is
discussed: (i) Computer Science, and (ii) health / life science. Additionally,
we mention a few other research fields, which have seen similar trends in
connection with the reproducibility crisis.

###  2.1 Computer Science

In Computer Science, reproducibility is mainly discussed in two specific sub-
fields of ML i.e., Deep learning and Reinforcement learning, and in two
application fields of ML i.e., Natural language processing and Recommender
systems.

####  2.1.1 Deep learning

Neural networks are widely used in ML and can be applied to both supervised
and unsupervised learning tasks. Neural networks, however, are known to be
inherently non-deterministic and produce different results on multiple reruns,
due to the many sources of randomness during training [2]. Not being able to
get the same results using the exact same code and data is a big challenge to
reproducibility, specifically w.r.t. degree R1. These different outcomes,
however, may not necessarily be statistically different, such that the same
conclusions can still be drawn. This is a distinction, researchers in the area
of using ML for imaging have come to [37]. Their research showed that while
the performance of Convolutional Neural Networks (CNN) on image segmentation
shows slightly different results on multiple reruns, the obtained results are
not statistically different, and the findings are robust.
In order to counteract the effect of different outputs on multiple reruns in
general, the idea of controlling the sources of randomness has been proposed
[2].
To add to this, [42] found that the concrete versions of deep learning
frameworks, such as PyTorch or TensorFlow, can have a big impact on
performance, in a way where upgrading the version of a model being used can
drastically increase or decrease the performance.

####  2.1.2 Reinforcement learning

Reinforcement Learning (RL) is the subfield of ML, where reproducibility
issues are seemingly discussed the most. RL is especially susceptible to
reproducibility issues, partially because of additional sources of non-
determinism in the learning process, which other areas are not subjected to
[32]. To overcome this obstacle, frameworks, which provide reproducible
environments, are commonly used in reinforcement learning research [10] [15]
[46]. These frameworks act as testbeds where different RL algorithms can be
evaluated in a common environment. Additionally, different papers have
outlined the implementation of deterministic reinforcement learning
algorithms, by controlling all the sources of non-determinism [33] [1].
Furthermore, [27] propose to standardize an evaluation pipeline, which can be
used for reproducible benchmarking of different reinforcement algorithms,
similar to competitions like Pommerman and the Learning to Run Challenge.

####  2.1.3 Natural language processing

A review of the status quo in terms of reproducibility in Natural language
processing (NLP) by [8] highlights some core issues, which overlap with the
ones mentioned for neural networks. Even simple rerunning of code can yield
different results, again undermining reproducibility of degree R1.
When trying to reproduce NLP research, it was evident that even minor changes
to parameter settings, such as rare-word thresholds, can drastically change
performance. One striking finding, among others, was that whenever the results
of a reproduction deviated from the original reported results, the reproduced
version performed worse than the original.
A different paper by [14] tried to tackle these performance benchmarking
issues and proposed a statistically sound replicability analysis framework,
where NLP algorithms are benchmarked against each other using multiple
different datasets. This is done in a way that not a specific dataset is
picked for evaluation, on which the algorithm performs well, but multiple
datasets are used to measure performance.

####  2.1.4 Recommender systems

Recommender systems research has also been affected by the reproducibility
crisis. The research area has observed that a stagnation in progress could
happen due to the difficulty of benchmarking new solutions against existing
solutions. This has created a form of “phantom progress” where it is not
evident whether proposed new methods actually perform better than traditional
ones [13]. A study by [7] has evaluated the reasons for this and proposed
solutions to the problem, which heavily overlap with the proposals of other
research areas, e.g., modern publication practices that foster the use of
reproducibility frameworks, and the establishment of best-practice guidelines.

###  2.2 Health and Life Science

There is a debate that the widespread adoption and use of ML has fueled
reproducibility issues in health and life science [17]. Accordingly, domain-
specific research has been done to address and handle these reproducibility
problems. While health and life science may have specific problems and
solutions for reproducibility, a review of the state-of-the-art is important
to come to a conclusion about reproducibility in ML as a whole.

Areas of health / life science making use of ML (often coined ML4H) have had a
lot of trouble with missing data when trying to reproduce a research result.
Data in the medical area is often private and cannot be publicized nor shared
[6] [30].
Importantly, however, health and medical science are fields where
reproducibility is of critical importance, since the verification of results
is important before ML results can be used clinically [6]. To tackle the
problem and overcome the reproducibility crisis in health / life science, a
solution has been proposed by [30]. This solution addresses three main
stakeholders: the data providers, the journals / conferences and the ML4H
research community. The idea is that the three stakeholders should interact in
rigorous and open collaboration. This solution, however, relies on the
collaborative sharing of data, which still has privacy issues connected to it.
Possible solutions to this have also been mentioned by [30], such as (i)
Privacy-Preserving Analysis techniques or specific data collection regimes,
e.g., the Verily Project Baseline.
In general, this proposal follows the ideas of the FAIR data principles
(findability, accessibility, interoperability, and reusability) and appeals
that the scientific data of the area should adhere to these FAIR principles
[12].

In addition to this, many domain-specific tools, such as NiLearn, Clinica or
OSF have been proposed to help with making the research more reproducible
[12].

###  2.3 Other Research Fields

Apart from computer science and health / life science, many different research
fields have benefitted tremendously from ML, but, however, have recently also
faced issues regarding reproducibility. Most of these reproducibility issues
stem either from data leakage [5] or the lack of computational
reproducibility. Across different research fields applying ML, the need for
reproducibility is discussed in literature, a few of which are highlighted
below:

####  2.3.1 Chemistry

ML is used very commonly to interpret patterns in data for chemistry. To keep
reproducibility issues at a minimum, best practices have been proposed by [3],
which introduce guidelines to follow and inform about best practices,
concerning, e.g., code and data sharing, and data leakage.

####  2.3.2 Materials science

ML models are used for the informed design of new materials. Recently,
however, these workflows have increasingly faced reproducibility issues,
especially effects of randomness during training and lack of reproducibility
across platforms and versions [39].

####  2.3.3 Genomics

A systematic review by [5] has found that reported results are often inflated
in recent research and are hard to evaluate, since data leakage is a prevalent
issue in the research field. Galaxy is a biomedical research platform, which
is used in genomics and biomedical research for running analyses and aims to
improve reproducibility [18].

####  2.3.4 Satellite imaging

All kinds of ML techniques, e.g., unsupervised learning techniques, supervised
learning techniques and deep learning are used within image segmentation for
satellite imaging. The Monte-Carlo cross-validation is often used to verify
new methods, however has been found to be very susceptible to data leakage
[34], which negatively impacts reproducibility.

##  3 Barriers in ML Reproducibility

When it comes to the different barriers associated with reproducibility in ML,
it is important to make a distinction between the different degrees of
reproducibility. Some barriers only affect certain degrees of reproducibility.
Computational problems, as an example, which prevent researchers from
reproducing the exact same results using the same code and data is only a
concern for reproducibility of degree R1. On the contrary, degrees R2 and R3
are more concerned with the findings and conclusions than with the exact
outputs.

###  3.1 Computational Problems

Recent studies have shown, that sharing of code and data alone is not enough
to ensure reproducibility even of degree R1. In some cases, not even the
assistance of the original author was enough to reproduce the original results
[22]. This barrier of computational reproducibility can be attributed to a few
factors, which are discussed in the following.

####  3.1.1 Inherent nondeterminism

The inherent nondeterminism in ML is a major reason why reproducibility of
degree R1 cannot be easily achieved. Even if both the data and code used in an
experiment are known, the pseudo-random numbers generated throughout the
training of the ML model can heavily alter its results [2]. Because of this,
fixed random number seeds can be vital to the reproducibility of ML.

####  3.1.2 Environment differences

Studies have shown that both hardware differences, such as different GPUs or
CPUs, as well as different compiler settings can result in different
computation outcomes [23]. Furthermore, a comparison between the same ML
algorithm with fixed random seeds that was executed using PyTorch and
TensorFlow also results in different results [39].
Additionally, a survey, which was conducted by [21] to investigate whether
different ML platforms, such as OpenML or Kaggle, provide out-of-the-box
reproducibility, also uncovered reproducibility issues.

###  3.2 Missing Data and Code

A study by [25] found that published ML research is often not accompanied by
available data and code. Only one third of researchers share the data, and
even fewer share the source code. This can have a lot of reasons, such as
private data or code that itself is based on unpublished code. Furthermore,
the problem may also be attributed to the increasing pressure for researchers
to publish quickly, which in turn does not allow them to polish the code and
decreases the willingness to release the code.

###  3.3 Methodological Problems

When computational problems are handled and code and data are available, then
the reproducibility of degree R1 of concrete experiments should, in theory, be
possible. However, in reality, the results are often based on methodological
errors made throughout the experiment. One well-known methodological error is
data leakage, which can come in different forms [26]. Data leakage has become
a widespread issue of research applying ML recently [26]. This can be
attributed to the increasing amount of non-experts using ML for various
research fields [17], which is fueled by the ease of application of auto-ML
libraries and no-code off-the-shelf AI tools. In essence, data leakage happens
when some form of data, which the ML model should not be trained on, leaks
into the training process. Data leakage can be categorized into 3
subcategories.

####  3.3.1 L1 - No clean train/test split

The subcategory L1 summarizes the most obvious cases of data leakage and is
further split into 4 variants: (1) the training data and test data are not
split at all, (2) the test data is also used for feature selection, (3) the
test data is also used for imputation during preprocessing, and (4) there are
duplicates in the dataset, which occur in both the test and training data.

####  3.3.2 L2 - Use of non-legitimate data

Subcategory L2 is concerned with training data, which contains non-legitimate
features. For example, when the use of anti-hypertensive drugs is used to
predict hypertension [26]. This data is non-legitimate, since it would not be
available in a real world scenario and cannot realistically be used to predict
hypertension for a new patient. Features, which result in data leakage of
subcategory L2 are often synonymous with the target variable. Generally,
deciding whether a feature is non-legitimate for a specific task requires a
lot of domain-knowledge.

####  3.3.3 L3 - Test set is not drawn from distribution of scientific
interest

L3 consists of 3 different types of data leakage. Firstly, temporal leakage is
concerned with ML models, which try to predict future outcomes. In that case,
temporal leakage happens when some training samples have a later timestamp
than samples available in the test set. This would mean that the ML model is
being trained on information coming “from the future”. Accordingly, the ML
model makes use of information, which it cannot have in a realistic scenario,
and therefore performs better than it would in reality. Secondly, the train
and test data need to be independent of each other. This means that there
cannot be samples in both the train and test data which are, e.g, from the
same person. Thirdly, the test set should not be drawn selectively. An example
of this is when the model is only evaluated on specific data, which it
performs well on, and then these results are generalized to all use cases.
Doing this would result in a selection bias.

###  3.4 Structural Problems

####  3.4.1 Privacy concerns

Availability of data is key for the reproducibility of any supervised or
unsupervised ML algorithm. Even for reproducibility of degree R3, where
different data should yield similar conclusions. In this case, it is important
to have some representative dataset, which holds comparable information.
However, the public release of data is not always an option, since the data
can be private, such as health-related patient data or user data collected by
companies.
To counteract this, privacy preserving technologies have been proposed [30].
There are a few different ways how privacy preserving techniques could work,
e.g., to allow the training of ML models without sharing the data, and to work
with encrypted or simulated data.

####  3.4.2 Competitive advantage

Comparing the status-quo in reproducibility of ML in academic research versus
industrial applications gives interesting insights [19]. While both academia
and industry still have a lot of work to do in terms of supporting
reproducibility, there are different main reasons why reproducibility is not
ensured in these two areas. Academic research may just lack enough incentive
or rewards for the increased effort of making sure the results are
reproducible. In contrast to this, in industrial applications it is often a
problem that providing reproducibility may come with a decreased competitive
advantage since.

##  4 Drivers for ML Reproducibility

After review of the status-quo in the different research fields, combined with
the identified barriers, we now present an outline of the main proposals /
drivers to address reproducibility. These drivers could be of critical help in
overcoming the reproducibility crisis.

###  4.1 Standardized Environments

Container software, such as e.g., Docker, could be used to standardize the ML
environment and allow for simple sharing of the container, i.e., environment
and code, with other researchers [9]. Furthermore, Code Ocean is a
computational research platform, which has been specifically developed and
tailored towards the needs of researchers. Because of this, it allows
researchers to focus on the research questions instead of the standardization
of environments [11].

###  4.2 Checklists and Guidelines

The idea of checklists and guidelines has been applied effectively in the
past, e.g., for safety-critical systems. To build on top of that, [38]
proposed a ML reproducibility checklist, which should ensure the inclusion of
necessary information for reproducibility. This checklist has been found
promising during the NeurIPS 2019 reproducibility challenge, and has been
suggested as best practice by researchers of different fields [3]. In Addition
to the use of checklists to make sure specific requirements are met by
different papers (such as proper scientific method or achievement of certain
reproducibility standards etc.), [29] propose to use Large Language Models
(LLMs) to verify these checklists and use them to review papers in general. By
doing this, both the efficiency, and the accuracy of verifying checklists can
be improved, thus further enhancing the viability of checklists. [29] found
that GPT-4 is the LLM, which stood out amongst others like Bard, Alpaca or
LLaMa and is especially capable of doing this task and was able to achieve an
86.6% accuracy of verifying the fulfillment of different checklist questions.

###  4.3 Model Info Sheets

Similar to checklists, model info sheets are questionnaires specifically
tailored towards handling data leakage, i.e., detection and prevention. [26]
propose these so-called model info sheets, which should be filled and
published with the work, in order to allow other researchers to quickly verify
the data used to train the ML model. To do this, the model info sheets require
the authors to answer detailed questions about the data and respective
train/test splits, which are aimed towards each type of data leakage[26].
Model info sheets seem like a promising solution for a lot of types of data
leakage, given that research of non-experts in ML often falls prey to
different types of data leakage [26]. Additionally, the model info sheets are
a low-effort solution for the problem they are trying to solve.

###  4.4 Awareness

Awareness of the reproducibility crisis and informing about ways to support
reproducibility can be a very strong factor in overcoming the crisis, as also
shown by the reproducibility crisis in Psychology [44].
Many different initiatives have been started to increase awareness on the
reproducibility crisis, a few of which are highlighted below:

  * •

The ReScience journal publishes peer-reviewed papers, which are specifically
about trying to reproduce results of original publications. These
reproductions are then made available on GitHub for other researchers [40].

  * •

ReproducedPapers.org is a Webs ite with a similar idea but focuses on
teaching. It is an online repository for papers, which are reproducible. The
ReproducedPapers initiative also aims to include the reproduction of at least
one ML paper in the curriculum of ML students [45].

  * •

Reproducibility challenges, where several researchers try to reproduce many
recent publications in parallel, are being held yearly. These challenges allow
for an analysis of the success rate of reproduction and can be used to
evaluate progress over multiple years [38].

###  4.5 Journals

To increase the awareness and enforce a minimum of reproducibility standards,
journals could adjust their requirements for publication. Recently, it has
already become a standard procedure in many journals to require data and/or
code availability for publication [38] [36] [22].
Furthermore, to address the problem of researchers deliberately tweaking
results for a higher likelihood of being published, journals could introduce
the possibility of preregistration. Preregistration allows researchers to
submit their research intention ahead of time to become registered for the
publication in the future. The journal decides whether to publish the paper
based on the research plan, and researchers then do not have to worry about
the outcome of the experiments, which should improve the credibility of the
findings [43] [35].
The ACM TORS journal (Transactions on Recommender Systems), is a good example
of promoting reproducibility in ML. Firstly, it allows for preregistration,
and secondly, it also allows for publication of “reproducibility papers”,
which are specifically concerned with reproduction studies and tools for
enhancing reproducibility.

##  5 Conclusion and Future Work

Almost all areas of ML, as well as research fields using ML (e.g., health and
life science), have seen signs of a reproducibility crisis. Appropriately,
this reproducibility crisis has received increasing attention recently and is
being discussed in research. It is time to tackle this reproducibility crisis,
which at first needs a common understanding of the definitions and different
degrees of reproducibility.

This work gave an overview over the current state-of-the-art in different
research areas. Additionally, it gave an insight into the different barriers
and drivers for ML reproducibility. For the continuation of this work, we will
dive deeper into the concepts outlined so far and will compare them across
different research fields. With this, we hope that a decision on the viability
of different solutions can be supported.

####  5.0.1 Acknowledgements.

This research is supported by the Horizon Europe project TIER2 under grant
agreement No 101094817.

## References

  * [1] Abel, D.: simple rl: Reproducible Reinforcement Learning in Python. ICLR (2019)
  * [2] Ahmed, H., Lofstead, J.: Managing Randomness to Enable Reproducible Machine Learning. In: Proceedings of the 5th International Workshop on Practical Reproducible Evaluation of Computer Systems. pp. 15–20. ACM, Minneapolis MN USA (2022). https://doi.org/10.1145/3526062.3536353, https://dl.acm.org/doi/10.1145/3526062.3536353
  * [3] Artrith, N., Butler, K.T., Coudert, F.X., Han, S., Isayev, O., Jain, A., Walsh, A.: Best practices in machine learning for chemistry. Nature Chemistry 13(6), 505–508 (2021). https://doi.org/10.1038/s41557-021-00716-z, https://www.nature.com/articles/s41557-021-00716-z, number: 6 Publisher: Nature Publishing Group
  * [4] Baker, M.: 1,500 scientists lift the lid on reproducibility. Nature 533(7604), 452–454 (2016). https://doi.org/10.1038/533452a, https://www.nature.com/articles/533452a, number: 7604 Publisher: Nature Publishing Group
  * [5] Barnett, E., Onete, D., Salekin, A., Faraone, S.V.: Genomic Machine Learning Meta-regression: Insights on Associations of Study Features with Reported Model Performance (2022). https://doi.org/10.1101/2022.01.10.22268751, https://www.medrxiv.org/content/10.1101/2022.01.10.22268751v2, pages: 2022.01.10.22268751
  * [6] Beam, A.L., Manrai, A.K., Ghassemi, M.: Challenges to the Reproducibility of Machine Learning Models in Health Care. JAMA 323(4), 305–306 (2020). https://doi.org/10.1001/jama.2019.20866, https://doi.org/10.1001/jama.2019.20866
  * [7] Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., Gipp, B.: Towards reproducibility in recommender-systems research. User Modeling and User-Adapted Interaction 26(1), 69–101 (2016). https://doi.org/10.1007/s11257-016-9174-x, http://link.springer.com/10.1007/s11257-016-9174-x
  * [8] Belz, A., Agarwal, S., Shimorina, A., Reiter, E.: A Systematic Review of Reproducibility Research in Natural Language Processing (2021). https://doi.org/10.48550/arXiv.2103.07929, http://arxiv.org/abs/2103.07929, arXiv:2103.07929 [cs]
  * [9] Boettiger, C.: An introduction to Docker for reproducible research. ACM SIGOPS Operating Systems Review 49(1), 71–79 (2015). https://doi.org/10.1145/2723872.2723882, https://dl.acm.org/doi/10.1145/2723872.2723882
  * [10] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: OpenAI Gym (2016). https://doi.org/10.48550/arXiv.1606.01540, http://arxiv.org/abs/1606.01540, arXiv:1606.01540 [cs]
  * [11] Clyburne-Sherin, A., Fei, X., Green, S.A.: Computational Reproducibility via Containers in Psychology. Meta-Psychology 3 (2019). https://doi.org/10.15626/MP.2018.892, https://open.lnu.se/index.php/metapsychology/article/view/892
  * [12] Colliot, O., Thibeau-Sutre, E., Burgos, N.: Reproducibility in machine learning for medical imaging (2022). https://doi.org/10.48550/arXiv.2209.05097, http://arxiv.org/abs/2209.05097, arXiv:2209.05097 [cs, q-bio, stat]
  * [13] Cremonesi, P., Jannach, D.: Progress in Recommender Systems Research: Crisis? What Crisis? AI Magazine 42(3), 43–54 (2021). https://doi.org/10.1609/aimag.v42i3.18145, http://ojs.aaai.org/index.php/aimagazine/article/view/18145
  * [14] Dror, R., Baumer, G., Bogomolov, M., Reichart, R.: Replicability Analysis for Natural Language Processing: Testing Significance with Multiple Datasets. Transactions of the Association for Computational Linguistics 5, 471–486 (2017)
  * [15] Ferigo, D., Traversaro, S., Metta, G., Pucci, D.: Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning. In: 2020 IEEE/SICE International Symposium on System Integration (SII). pp. 885–890 (2020). https://doi.org/10.1109/SII46433.2020.9025951, iSSN: 2474-2325
  * [16] Gibney, E.: This AI researcher is trying to ward off a reproducibility crisis. Nature 577(7788), 14–14 (2019). https://doi.org/10.1038/d41586-019-03895-5, https://www.nature.com/articles/d41586-019-03895-5, bandiera_abtest: a Cg_type: News Q&a Number: 7788 Publisher: Nature Publishing Group Subject_term: Machine learning
  * [17] Gibney, E.: Could machine learning fuel a reproducibility crisis in science? Nature 608(7922), 250–251 (2022). https://doi.org/10.1038/d41586-022-02035-w, https://www.nature.com/articles/d41586-022-02035-w, bandiera_abtest: a Cg_type: News Number: 7922 Publisher: Nature Publishing Group Subject_term: Machine learning, Publishing, Mathematics and computing
  * [18] Goecks, J., Nekrutenko, A., Taylor, J., The Galaxy Team: Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biology 11(8), R86 (2010). https://doi.org/10.1186/gb-2010-11-8-r86, https://doi.org/10.1186/gb-2010-11-8-r86
  * [19] Gundersen, O.E.: Standing on the Feet of Giants — Reproducibility in AI. AI Magazine 40(4), 9–23 (2019). https://doi.org/10.1609/aimag.v40i4.5185, https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/5185, number: 4
  * [20] Gundersen, O.E., Kjensmo, S.: State of the Art: Reproducibility in Artificial Intelligence. Proceedings of the AAAI Conference on Artificial Intelligence 32(1) (2018). https://doi.org/10.1609/aaai.v32i1.11503, https://ojs.aaai.org/index.php/AAAI/article/view/11503
  * [21] Gundersen, O.E., Shamsaliei, S., Isdahl, R.J.: Do machine learning platforms provide out-of-the-box reproducibility? Future Generation Computer Systems 126, 34–47 (2022). https://doi.org/10.1016/j.future.2021.06.014, https://www.sciencedirect.com/science/article/pii/S0167739X21002090
  * [22] Hardwicke, T.E., Mathur, M.B., MacDonald, K., Nilsonne, G., Banks, G.C., Kidwell, M.C., Hofelich Mohr, A., Clayton, E., Yoon, E.J., Henry Tessler, M., Lenne, R.L., Altman, S., Long, B., Frank, M.C.: Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science 5(8), 180448 (2018). https://doi.org/10.1098/rsos.180448, https://royalsocietypublishing.org/doi/full/10.1098/rsos.180448, publisher: Royal Society
  * [23] Hong, S.Y., Koo, M.S., Jang, J., Kim, J.E.E., Park, H., Joh, M.S., Kang, J.H., Oh, T.J.: An Evaluation of the Software System Dependency of a Global Atmospheric Model. Monthly Weather Review 141(11), 4165–4172 (2013). https://doi.org/10.1175/MWR-D-12-00352.1, https://journals.ametsoc.org/view/journals/mwre/141/11/mwr-d-12-00352.1.xml, publisher: American Meteorological Society Section: Monthly Weather Review
  * [24] Hutson, M.: Artificial intelligence faces reproducibility crisis Unpublished code and sensitivity to training conditions make many claims hard to verify. Science 359(6377), 725–726 (2018). https://doi.org/10.1126/science.359.6377.725
  * [25] Hutson, M.: Missing data hinder replication of artificial intelligence studies. Science (2018). https://doi.org/10.1126/science.aat3298
  * [26] Kapoor, S., Narayanan, A.: Leakage and the Reproducibility Crisis in ML-based Science (2022), http://arxiv.org/abs/2207.07048, arXiv:2207.07048 [cs, stat]
  * [27] Khetarpal, K., Ahmed, Z., Cianflone, A., Islam, R., Pineau, J.: RE-EVALUATE: Reproducibility in Evaluating Reinforcement Learning Algorithms. ICML (2018), https://openreview.net/forum?id=HJgAmITcgm
  * [28] Kowald, D., Schedl, M., Lex, E.: The unfairness of popularity bias in music recommendation: A reproducibility study. In: Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42. pp. 35–42. Springer (2020)
  * [29] Liu, R., Shah, N.B.: ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing (2023). https://doi.org/10.48550/arXiv.2306.00622, http://arxiv.org/abs/2306.00622, arXiv:2306.00622 [cs]
  * [30] McDermott, M.B.A., Wang, S., Marinsek, N., Ranganath, R., Ghassemi, M., Foschini, L.: Reproducibility in Machine Learning for Health (2019). https://doi.org/10.48550/arXiv.1907.01463, http://arxiv.org/abs/1907.01463, arXiv:1907.01463 [cs, stat]
  * [31] Muellner, P., Kowald, D., Lex, E.: Robustness of meta matrix factorization against strict privacy constraints. In: Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43. pp. 107–119. Springer (2021)
  * [32] Nagarajan, P., Warnell, G., Stone, P.: The Impact of Nondeterminism on Reproducibility in Deep Reinforcement Learning. ICML (2018), https://openreview.net/forum?id=S1e-OsZ4e7
  * [33] Nagarajan, P., Warnell, G., Stone, P.: Deterministic Implementations for Reproducibility in Deep Reinforcement Learning (2019). https://doi.org/10.48550/arXiv.1809.05676, http://arxiv.org/abs/1809.05676, arXiv:1809.05676 [cs]
  * [34] Nalepa, J., Myller, M., Kawulok, M.: Validating Hyperspectral Image Segmentation. IEEE Geoscience and Remote Sensing Letters 16(8), 1264–1268 (2019). https://doi.org/10.1109/LGRS.2019.2895697, http://arxiv.org/abs/1811.03707, arXiv:1811.03707 [cs]
  * [35] Nosek, B.A., Beck, E.D., Campbell, L., Flake, J.K., Hardwicke, T.E., Mellor, D.T., van ’t Veer, A.E., Vazire, S.: Preregistration Is Hard, And Worthwhile. Trends in Cognitive Sciences 23(10), 815–818 (2019). https://doi.org/10.1016/j.tics.2019.07.009, https://www.sciencedirect.com/science/article/pii/S1364661319301846
  * [36] Peng, R.: The reproducibility crisis in science: A statistical counterattack. Significance 12(3), 30–32 (2015). https://doi.org/10.1111/j.1740-9713.2015.00827.x, https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1740-9713.2015.00827.x, _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1740-9713.2015.00827.x
  * [37] Piantadosi, G., Marrone, S., Sansone, C.: On Reproducibility of Deep Convolutional Neural Networks Approaches. In: Kerautret, B., Colom, M., Lopresti, D., Monasse, P., Talbot, H. (eds.) Reproducible Research in Pattern Recognition. pp. 104–109. Lecture Notes in Computer Science, Springer International Publishing, Cham (2019). https://doi.org/10.1007/978-3-030-23987-9_10
  * [38] Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., Larochelle, H.: Improving reproducibility in machine learning research (a report from the NeurIPS 2019 reproducibility program). The Journal of Machine Learning Research 22(1), 164:7459–164:7478 (2021)
  * [39] Pouchard, L., Lin, Y., Van Dam, H.: Replicating Machine Learning Experiments in Materials Science. In: Parallel Computing: Technology Trends, pp. 743–755. IOS Press (2020). https://doi.org/10.3233/APC200105, https://ebooks.iospress.nl/doi/10.3233/APC200105
  * [40] Rougier, N.P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L.A., Benureau, F.C.Y., Brown, C.T., Buyl, P.d., Caglayan, O., Davison, A.P., Delsuc, M.A., Detorakis, G., Diem, A.K., Drix, D., Enel, P., Girard, B., Guest, O., Hall, M.G., Henriques, R.N., Hinaut, X., Jaron, K.S., Khamassi, M., Klein, A., Manninen, T., Marchesi, P., McGlinn, D., Metzner, C., Petchey, O., Plesser, H.E., Poisot, T., Ram, K., Ram, Y., Roesch, E., Rossant, C., Rostami, V., Shifman, A., Stachelek, J., Stimberg, M., Stollmeier, F., Vaggi, F., Viejo, G., Vitay, J., Vostinar, A.E., Yurchak, R., Zito, T.: Sustainable computational science: the ReScience initiative. PeerJ Computer Science 3, e142 (2017). https://doi.org/10.7717/peerj-cs.142, https://peerj.com/articles/cs-142, publisher: PeerJ Inc.
  * [41] Sayre, F., Riegelman, A.: The Reproducibility Crisis and Academic Libraries | Sayre | College & Research Libraries. College & Research Libraries (2018). https://doi.org/https://doi.org/10.5860/crl.79.1.2, https://journals.acrl.org/index.php/crl/article/view/16846
  * [42] Shahriari, M., Ramler, R., Fischer, L.: How Do Deep-Learning Framework Versions Affect the Reproducibility of Neural Network Models? Machine Learning and Knowledge Extraction 4(4), 888–911 (2022). https://doi.org/10.3390/make4040045, https://www.mdpi.com/2504-4990/4/4/45, number: 4 Publisher: Multidisciplinary Digital Publishing Institute
  * [43] Strømland, E.: Preregistration and reproducibility. Journal of Economic Psychology 75, 102143 (2019). https://doi.org/10.1016/j.joep.2019.01.006, https://www.sciencedirect.com/science/article/pii/S0167487018302903
  * [44] Wiggins, B.J., Christopherson, C.D.: The replication crisis in psychology: An overview for theoretical and philosophical psychology. Journal of Theoretical and Philosophical Psychology 39, 202–217 (2019). https://doi.org/10.1037/teo0000137, place: US Publisher: Educational Publishing Foundation
  * [45] Yildiz, B., Hung, H., Krijthe, J., Liem, C., Loog, M., Migut, G., Oliehoek, F., Panichella, A., Pawełczak, P., Picek, S., de Weerdt, M., van Gemert, J.: ReproducedPapers.org: Openly Teaching and Structuring Machine Learning Reproducibility. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12636 LNCS, 3–11 (2021). https://doi.org/10.1007/978-3-030-76423-4_1, iSBN: 9783030764227
  * [46] Young, K., Tian, T.: MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments (2019). https://doi.org/10.48550/arXiv.1903.03176, http://arxiv.org/abs/1903.03176, arXiv:1903.03176 [cs]