# Run, Forest, Run? On Randomization and Reproducibility in Predictive
Software Engineering

Cynthia C. S. Liem  Multimedia Computing Group
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>Annibale Panichella  Software Engineering Research
Group
Delft University of Technology
Delft, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML) has been widely used in the literature to automate
software engineering tasks. However, ML outcomes may be sensitive to
randomization in data sampling mechanisms and learning procedures. To
understand whether and how researchers in SE address these threats, we
surveyed 45 recent papers related to three predictive tasks: defect prediction
(DP), predictive mutation testing (PMT), and code smell detection (CSD). We
found that less than 50% of the surveyed papers address the threats related to
randomized data sampling (via multiple repetitions); only 8% of the papers
address the random nature of ML; and parameter values are rarely reported
(only 18% of the papers). To assess the severity of these threats, we
conducted an empirical study using 26 real-world datasets commonly considered
for the three predictive tasks of interest, considering eight common
supervised ML classifiers. We show that different data resamplings for 10-fold
cross-validation lead to extreme variability in observed performance results.
Furthermore, randomized ML methods also show non-negligible variability for
different choices of random seeds. More worryingly, performance and
variability are inconsistent for different implementations of the conceptually
same ML method in different libraries, as also shown through multi-dataset
pairwise comparison. To cope with these critical threats, we provide practical
guidelines on how to validate, assess, and report the results of predictive
methods.

###### Index Terms:

machine learning, predictive software engineering, statistical analysis,
reproducibility, randomization

##  I Introduction

The application of machine learning (ML) techniques to software engineering
(SE) has attracted increasing interest from researchers and practitioners in
recent years. This has resulted in various successful ML applications, such as
building predictive models for code quality assessment [1], software testing
[2, 3], effort estimation, and software security. At the same time, despite
increasing optimism about the effectiveness of ML techniques for SE,
researchers have started questioning the reproducibility of their results [4,
5]. For example, within the SE community, attention has been raised for best
practices to deal with outcomes of randomized algorithms, and to better
substantiate whether observed performance differences are true differences
through the application of statistical tests [6]. When applying these, it also
is important to be explicitly aware of underlying assumptions made by the test
[6, 7].

Outside of the SE domain, other fields have demonstrated how sensitive
analyses and drawn conclusions may be to _researcher degrees of freedom_ that
are not commonly reported. In Psychology, warnings have been raised that
undisclosed (and non-malicious) choices in data collection, analysis, and
reporting can easily lead to false-positive results and conclusions [8]. In
Music Information Retrieval research, it was shown that implementation design
choices for evaluation metrics can yield highly different outcomes [9], and
that common descriptions and commonly reported parameters of music signal
processing procedures are insufficiently specific to warrant unambiguous,
stable reimplementations [10].

Despite great and broad optimism about ML’s promises, criticism has risen
within the ML field itself on the true merit, reasons behind, and
reproducibility of reported advances. The tendency to appreciate more complex
approaches that outperform alternatives, together with short, fast-paced
publication cycles, has been critiqued to lead to flawed works that lack
practical use and empirical rigor [11, 12]. It has been hinted that the
broader artificial intelligence faces a reproducibility crisis [13], which may
e.g. be due to unreleased source code and unreported critical configuration
details. In deep reinforcement learning, experiments varying random seeds and
hyperparameter values caused major performance differences, and alternative
implementations of the same baseline task were shown to give radically
different results, leading to calls for more thorough reporting practice [14].

While many ML applications in SE use older, more established supervised ML
techniques than those under particular scrutiny today, it still is imaginable
that similar issues may be at play, impacting the trustability of observed
outcomes. This paper therefore seeks to gain more insight into how rigorously
ML techniques are assessed in SE domains. More precisely, we investigate three
important factors that can potentially impact ML result variability,
threatening future reproducibility: (1) the sensitivity of ML methods to
randomized resamplings from datasets for training and validation; (2) the
randomized nature of specific ML learning procedures; (3) whether different
implementations of the conceptually same ML method in different libraries are
actually consistent with one another.

Focusing on three actively researched predictive SE tasks (defect prediction,
predictive mutation testing, and code smell detection), we first perform a
systematic literature review, to understand the extent to which researchers
addressed the factors of our interest in the current literature. Alarmingly,
after rigorously analysing 45 recent papers in the three selected SE tasks, we
did not find a single case in which all three potential treats have been
adequately and explicitly addressed.

Secondly, we present an empirical study that aims to investigate to what
extent predictive SE outcomes may be affected by the factors of our interest.
Our study involves 26 commonly-used datasets in SE from the three predictive
SE tasks of interest, eight ML classifier models, and three alternative
commonly-used ML libraries. Our results show striking inconsistencies and
outcome variabilities, that need to be more carefully and explicitly addressed
by researchers than seemingly is done today. Therefore, we conclude our work
by presenting a set of practical guidelines on how to address these threats
adequately, and how to validate, assess, and report the results in predictive
software engineering.

##  II Systematic Review

Several current ML criticisms on threats to validity and reproducibility may
also hold for common ML practices in SE. First of all, for training and
evaluation, samples or data partitions are randomly drawn from a dataset,
which may lead to particularly (un)lucky samplings. Next to this, models may
include randomization as part of their training procedure (e.g. in a random
forest), implying that different random seeds may give different results.
Finally, hyperparameters have a major impact on outcomes, and thus it would be
good practice to report and discuss them. To assess to what extent these
aspects are acknowledged in current SE practice, we consider three
representative predictive SE tasks that currently heavily build on ML
approaches: _defect prediction (DP)_ , _predictive mutation testing (PMT)_ ,
and _code smell detection (CSD)_.

####  II-1 DP

In this task, ML methods are used to predict the fault-proneness of software
modules (e.g., class or methods) in a given software project with the goal of
prioritizing QA activities. ML for DP has been extensively studied in the last
two decades, as reflected in existing surveys (e.g., [1]), reflection papers
(e.g., [15, 16]), and dedicated tools (e.g., [17]). Defect prediction can be
formulated as a binary classification problem, whose goal is to predict
whether each specific software module (e.g., a class or method) is defective
or not [1, 15, 16]. Multiple features have been used for building prediction
models [15], such as code (e.g., [18]), process (e.g, [19]), and
organizational metrics (e.g, [20]).

####  II-2 PMT

Mutation testing aims to assess the effectiveness of a test suite. Given a
program $P$, mutation testing tools generate program variants (i.e., mutants)
based on a given set of transformation rules (i.e., mutation operators). An
effective test case should pass when executed against the original program $P$
but fails against a mutant $P^{\prime}$. In other words, the test should
differentiate between the original program and its mutated variant. In this
scenario, the mutant $P^{\prime}$ is said to be killed. Instead, the mutant
$P^{\prime}$ survives if the test cases pass on both $P$ and $P^{\prime}$. The
test suite’s effectiveness is then measured as the ratio between the number of
killed mutants and the total number of non-equivalent mutants being generated
[21]. While mutation testing is widely recognized as a high-end test adequacy
criterion [22], it does incur a considerable overhead. Indeed, performing
mutation testing requires to rerun the entire test suite against each
generated mutant. For this reason, researchers have proposed various
strategies that aim to reduce the cost of mutation testing [23].

The purpose of PMT is to produce the mutation testing results without
executing the actual mutants [2, 3]. PMT has been formulated as a binary
classification problem, whose goal is to predict whether each mutant
$P^{\prime}$—generated by a mutation testing tool—is killed or not by a given
test suite $T$ [2, 3]. Widely applied features for PMT include test execution
data (e.g., the number executions of a mutated statement), infection features
(e.g., the type of mutation operator), and propagation features (e.g., the
number of assertions that cover a mutant).

####  II-3 CSD

Code smells are symptoms of poor software design choices that negatively
impact code comprehensibility and future maintenance activities [24]. The
original catalog has also been lately extended to test code [25], and
databases [26]. Over the last decades, researchers have proposed various
static analysis methods and tools to detect code smells automatically [27]. A
complete overview of code smell detection approaches is available in [24].

More recent approaches to detect code smell rely on ML [28, 29, 30]. The
detection problem is reformulated as a binary classification task using
supervised ML methods. More precisely, the goal is to predict/detect whether a
software module (class or method) is affected by a given code smell. These
approaches typically use various static software metrics [24], such as lines
of code, McCabe cyclomatic complexity, the number of input parameters of a
method, and so on.

While not aiming to provide an exhaustive literature review for these three
tasks, we still would like a representative snapshot of current common
practices, obtained in a rigorous and unbiased fashion. Therefore, we conduct
a systematic literature review in line with common practice in the SE domain
[31, 32], as described below.

###  II-A Methodology

We used digital libraries and search engines (Google Scholar, Scopus, and
DBLP) to manually retrieve papers related to the predictive SE tasks of
interest. For querying, we considered the following search keywords:

  * •

DP: (“defect prediction”) OR (“fault prediction”) OR (“bug prediction”) OR
(“defect” AND “deep learning”) OR (“fault” AND “deep learning”) OR (“bug” AND
“deep learning”) OR (“supervised defect prediction”).

  * •

PMT: (“predictive mutation testing”) OR (“supervised mutation testing”) OR
(“predictive mutation analysis”) OR (“machine learning” AND “mutation
testing”) OR (“deep learning” AND “mutation testing”).

  * •

CSD: (“predictive code smell”) OR (“code smell” AND “machine learning”) OR
(“code smell” AND “deep learning”) OR (“design flaw” AND “machine learning”)
OR (“design flaw” AND “deep learning”) OR (“supervised code smells”).

With these keywords initially yielding thousands of hits, we used computer
science and software engineering as additional filters to explicitly restrict
to papers relevant to the SE domain. We then restricted the selection further
to works that appeared at international SE top venues: IEEE Transactions on
Software Engineering (TSE), IEEE Transactions on Reliability, Empirical
Software Engineering (EMSE), Information and Software Engineering (IST),
Journal and Software and System (JSS), Software Testing, Verification and
Reliability (STVR), IEEE Access, Software Quality Journal (SQJ), ACM/IEEE
International Conference on Software Engineering (ICSE), Mining Software
Repositories (MSR), IEEE International Conference on Software Testing,
Verification and Validation (ICST), Automated Software Engineering (ASE),
Symposium on the Foundations of Software Engineering (FSE), ACM International
Symposium on Software Testing and Analysis (ISSTA), IEEE International
Conference on Software Analysis, Evolution and Re-engineering (SANER), and
International Symposium on Software Reliability Engineering (ISSRE). To focus
on original work presentations at sufficient levels of detail, we excluded
keynotes, extended abstracts, tutorials, and posters. The inclusion and
exclusion criteria are detailed in the following paragraphs and for each
predictive tasks separately.

####  II-A1 DP

given the large body of DP research and our focus on a snapshot of current
common practices, we focus our search on papers published in the selected top
venues in 2019 and 2020. We included papers that discuss the application of ML
methods or improved ML procedures (e.g. data balancing), framing DP as a
binary classification task. Hence, we excluded papers that focus on fault
localization, or predicting the severity (or the number of) defects in
software modules.

####  II-A2 PMT

given that the first paper was published in 2016 [2], the body of research in
PMT is still small. For that reason, we included all papers published in the
selected venues between the years 2016 and 2020. We included all papers that
applied ML approaches to mutation testing and its surrounding challenges (e.g.
data balancing, transfer learning), while explicitly focusing on the framing
of PMT as a binary classification task at the mutant level (as this makes
results comparable with those of classical mutation testing). We excluded
papers that use ML-based techniques that do not focus on predictions at this
level.

####  II-A3 CSD

with ML-based CSD also being a very recent research topic, we consider all
papers published in the selected top venues. Beyond this, we include any
further works mentioned in a recent systematic literature review [33], and
apply snowballing as defined by Wohlin [34], i.e., including papers cited by
or that cite the papers retrieved in the previous steps. We included all
papers discussing the application of ML methods or improved ML pipeline
procedures (e.g. data balancing), framing CSD as a classification task. We
excluded papers only focusing on cross-version prediction (as this inherently
implies that training and testing data has strong overlaps, and thus may cause
overfitting). In case conference and extended journal variants exist of the
same work, we only consider the extended (and more recent) journal variants.

###  II-B Review Results

Based on the process described above, we selected 45 papers: 25 papers for DP,
four for PMT, and 16 for CSD, as presented in Table I. We observe the there is
a large variability of strategies used to evaluate ML methods. The most common
validation mechanism is n-fold cross-validation (18 papers111Two papers use a
stratified variant of the n-fold cross validation), followed by cross-project
(12 papers), out-of-sample bootstrap (5 papers), and cross-version (3 papers)
validation. Two articles did not specify what validation procedure was used.
Four papers (i.e., [35, 3, 36, 37]) apply more than one validation mechanism
to assess the investigated ML methods.

Only 22 out of 45 selected papers validated ML methods by performing multiple
data-splitting repetitions in training and test sets. This, even though a
prior DP study did indicate that the random nature of sampling used for the
validation can introduce critical biases [38], and different re-samplings can
indeed lead to different results. For the remaining 23 papers, the authors did
not mention any sampling repetitions, or acknowledge the absence of this in
the threats to validity.

The results for the random seed are even more alarming. Only four out of 45
papers (8%) rerun ML models with different random seeds. The only few papers
that did so are related to DP, while no paper for the other two tasks rerun ML
methods with multiple random seeds. For example, the heavily-used random
forest, which even has been recommended as  most effective classifier in terms
of performance [39], will by definition be sensitive to different random
seeds. While systematic and statistically sound methodologies to assessed
randomized algorithms are well-established in other SE sub-domains [6],
statistical tests on observed performance outcomes in predictive SE are only
very rarely being mentioned.

The most widely used ML library is Weka [40] (20 papers), followed by R [41]
(8 papers), Scikit-learn [42] (5 papers), and MATLAB [43] (3 papers). Keras
and Tensorflow are the leading libraries for approaches based on Deep
Learning. Finally, six papers did not report the actual ML library or
framework used in the empirical evaluation. This raises some concerns about
future replicability.

As for hyperparameter tuning, reporting the parameter values is not a common
practice in predictive SE. Out of 45 papers, 29 (64%) did not report the
parameter values. 11 papers mention to have used “default parameters”. 9
papers performed parameter tuning, but often did not reported the final
(tuned) parameter values. Only one paper reported the tuned values [39], and
other two papers did so but partially [28, 44]. 8 papers give indications of
concrete parameter values being used; 8 other papers explicitly mention that a
selection of parameter values are reported, or that reports are only provided
for selected ML methods.

Defect Prediction | Evaluation Type | # Re-samplings | # Random Seeds | Library | Parameter values | Notes
---|---|---|---|---|---|---
Kondo et al., EMSE, 2019 [45] | Out-of-sample bootstrap | 100 | Not Reported | R + Weka | Partially Tuned, Not Reported | Tuned only supervised methods
Fan et al., TSE, 2019 [46] | Out-of-sample bootstrap | 1000 | Not Reported | R | Not Reported | —
Chen et al., IEEE Access, 2019 [47] | Cross-project (29%-71% split) | Not Reported | 10 | Keras | Partially Reported | Keras with default parameters
Zhou et al., IST, 2019 [48] | 2-fold cross-validation | 100 | Not Reported | Scikit-learn | Partially Reported | Baselines with default parameters
Liu et al., IST, 2019 [49] | Cross-project | 10 | 10 | LIBLINEAR + Weka | Not reported | SMOreg with default parameters
Xu et al, IST, 2019 [50] | Stratified sampling | 30 | Not Reported | MATLAB | Reported | —
Huang et al., EMSE, 2019 [51] | 10-fold cross-validation | 10 | Not Reported | Weka + R | Not Reported | Default parameters
Ni et al., JSS, 2019 [52] | Stratified sampling | 10 | Not Reported | Weka | Reported | —
Shippey et al., IST, 2019 [53] | Stratified cross-validation | 100 | Not Reported | Weka | Not Reported | Default Parameters
Hoang et al, MSR, 2019 [54] | Cross-Validation | Not Reported | Not Reported |  | Reported | —
Yu et al., IEEE Access, 2019 [55] | Cross-project | Not Reported | Not Reported | Weka | Reported | —
Bennin et al., EMSE, 2019 [56] | 10-fold cross-validation | 10 | Not Reported | MATLAB + R | Not Reported | —
Zhang et al., IEEE Access, 2019 [57] | Cross-version | Not Reported | Not Reported | Weka | Not Reported | Default Parameters
Gong et al., IEEE TR, 2020 [58] | Cross-project | Not Reported | 20 | unknown | Not Reported | —
Cabral et al., ICSE, 2019 [59] | Cross-version | 30 | Not Reported | unknown | Tuned, Not Reported | —
Mori et al., EMSE, 2019 [60] | 5-fold cross-validation | 5 | Not Reported | Weka + R | Reported | Reported seeds for data splitting
Chen et al., IST, 2019 [61] | 5-fold cross validation | 25 | Not Reported | scikit-learn | Tuned, Not Reported | —
Jiarpakdee et al., TSE, 2019 [62] | Out-of-sample bootstrap | 100 | Not Reported | R | Partially Reported | Default values
Jiarpakdee et al., TSE, 2019 [63] | Out-of-sample bootstrap | 100 | Not Reported | R | Partially Reported | Default values
Chen et al., STVR, 2020 [64] | Cross-project | Not Reported | Not Reported | Weka | Not Repored | Default values and ack. the diff. across libraries
Chen et al., TSE, 2020 [65] | Cross-project | 20 | Not Reported | MATLAB | Not Reported | Friedman test + Nemenyi Test
Amasaki, EMSE, 2020 [35] | Cross-project and version | Not Reported | Not Reported | CrossPare [17] | Not Reported | —
Afric et al., JSS, 2020 [66] | Random Splits | 30 | 30 | Scikit-learn | Not Reported | Default Values
Yuan et al., IEEE Access, 2020 [67] | 2-fold cross-validation | 10 | Not Reported | Scikit-learn + CrossPare [17] | Not Reported | Default Values
Alqadi & Maletic, SANER, 2020 [68] | Out-of-sample bootstrap | 1000 | Not Reported | R | Not Reported | —
Predictive Mutation Testing | Evaluation Type | # Re-samplings | # Random Seeds | Library | Parameter values | Notes
Zhang et al., ISSTA, 2016 [2] | Cross-project | NA | Not Reported | Weka | Not Reported | —
Zhang et al., TSE, 2018 [3] | Cross-version, cross-project | NA | Not Reported | Weka | Not Reported | —
Mao et al., ICST, 2019 [39] | Cross-project 5-Folds | Not Reported | Not Reported | Scikit-learn | Tuned, Reported | Multiple reps. only for feature selection
Naeem et al., IEEE Access, 2019 [69] | Cross-project | NA | Not Reported | Keras | Tuned, Not Reported | —
Code Smell Detection | Evaluation Type | # Re-samplings | # Random Seeds | Library | Parameter values | Notes
Kreimer, ENTCS, 2005 [70] | Leave-one-out | NA | Not Reported | Weka | Not Reported | Small-scale prototype
Vaucher et al., WCRE, 2009 [71] | 10-fold cross-validation | Not Reported | Not Reported | Weka | Not Reported | Focus on rule extraction
Hassaine et al., QUATIC, 2010 [72] | 3-fold cross-validation | Not Reported | Not Reported | Not Reported | Not Reported | No Variability Indicators
Maneerat & Muenchaisri, JCSSE, 2011 [73] | 10-fold cross-validation | “at least 10 times” | Not Reported | Weka 3.6 | Not Reported | Default parameters, no Variability Indicators
Khomh et al., JSS, 2011 [36] | 3-fold cross-validation / cross-project | Not Reported | Not Reported | Weka | Not Reported | No Variability Indicators
Maiga et al., WCRE, 2012 [37] | 10-fold cross-validation / cross-project | Not Reported | Not Reported | Weka | Not Reported | No Variability Indicators
Maiga et al., ASE, 2012 [74] | Not Reported | Not Reported | Not Reported | Weka | Not Reported | No Variability Indicators
Amorim et al., ISSRE, 2015 [75] | 10-fold cross-validation | Not Reported | Not Reported | Not Reported | Not Reported | No Variability Indicators
Arcelli Fontana et al., EMSE, 2016 [28] | 10-fold cross-validation | 10 | Not Reported | Weka | Tuned, Partially Reported | Standard deviation reported
Arcelli Fontana & Zanoni, KBS, 2017 [29] | Stratified 10-fold cross-validation | 10 | Not Reported | Weka, KNIME | Taken From Previous Work | Standard deviation reported
Fakhoury et al., SANER, 2018 [76] | Leave-one-out | NA | Not Reported | Weka, Tensorflow | Tuned, Reported | —
Di Nucci et al., SANER, 2018 [30] | Same as [28] | Same as [28] | Same as [28] | Same as [28] | Same as [28] | No Variability Indicators
Barbez et al., JSS, 2020 [44] | Cross-project | 1 | Not Reported | Tensorflow | Tuned, Partially Reported | —
Guggulothu & Moiz [77], SQJ, 2020 | 10-fold cross-validation | 10 | Not Reported | Not Reported | Not Reported | No Variability Indicators
Pecorelli et al., JSS, 2020 [78] | 10-fold cross-validation | Not Reported | Not Reported | Not Reported | Tuned, not reported | Classifier library not explicitly mentioned.
Liu et al., TSE, 2020 [79] | Cross-project | 1 | Not Reported | Keras | Not Reported | —

TABLE I: Papers in the related literature that focus on defect prediction,
predictive mutation testing, and code smell detection.

##  III Empirical Study

As evidenced in our literature review, potential performance variability in
predictive SE models is not frequently acknowledged. However, beyond [38],
focusing on DP, no clear insight exists yet into the extent to which these
factors have impact across predictive SE tasks. For example, the commonly used
$n$-fold cross-validation scheme internally averages over $n$ train-test
cycles, which is assumed to increase robustness and generalizability. Hence,
it is unclear to what extent it practically matters that the initial
partitioning into $n$ folds is randomized. Therefore, we conduct an empirical
study, focusing on three research questions:

  * •

RQ1: To what extent does the performance of cross-validated ML models vary
when considering random fold repartitionings of the same dataset?

  * •

RQ2: To what extent does the performance of randomized ML models vary when
using different random seeds?

  * •

RQ3: To what extent are implementations of the same conceptual ML model in
different libraries consistent?

###  III-A Datasets

For our three predictive SE tasks of interest, we consider commonly used,
openly available datasets222All data resources and repositories come from
external, independent research. The exact input data files as used in our
analyses are released in anonymized form as part of our supplementary
material, available at https://figshare.com/s/998dd6c07173f78bd64c.

####  III-A1 Defect Prediction (DP)

We use data from two repositories333From the combined repository at
https://github.com/klainfo/DefectData.git.: PROMISE [80], and the repository
built by D’Ambros et al. [81], which have been widely used in prior studies in
defect prediction (e.g., [45, 35, 58, 81]). From the PROMISE repository [80],
we considered six Java projects that are among the largest in terms of data
points, that consider open-source projects (where applicable, according to
latest versions), and that have representative data characteristics (i.e.
defects not being the majority class): Apache Ant v1.7, Camel v1.6, Ivy v2.0,
jEdit v4.3, Lucene v2.4, and Apache Tomcat. From the repository built by
D’Ambros et al. [81], we considered five available projects: Eclipse JDT Core
3.4, Lucene v2.4.0, Mylyn v3.1, Eclipse PDE UI v3.4.1, Equinox v3.4. The two
repositories differ on the features (metrics) being used; further details are
available in the original papers [18, 81]. The dataset sizes range from 341
(Lucene from PROMISE) to 2196 (Mylyn from D’Ambros et al.) data points
(average dataset size: 877).

####  III-A2 Predictive Mutation Testing (PMT)

We use the data of the latest versions of the 9 base Git projects used in
[3]444https://tinyurl.com/y6e4yyrr: Java APNS, AssertJ, Joda-Time, Linear
Algebra for Java, Apache Commons Lang, Message Pack for Java, UAAA, Vraptor,
and Wire Mobile Protocol Buffers. For feature details, we refer to the
original paper [3]. Dataset sizes range from 115 (Java APNS) to 32,930 (Joda-
Time) data points (average dataset size: 10,278).

####  III-A3 Code Smell Detection (CSD)

We used the six datasets used in
[28]555https://essere.disco.unimib.it/machine-learning-for-code-smell-
detection/. Each dataset corresponds to one specific type of code smell (god
class, data class, feature envy, long method, long parameter list, and switch
statements). For feature details, we refer to the original paper [28]. From
each dataset, we remove columns with string values that are irrelevant to
predictions (e.g., library/method names). Furthermore, we remove rows which
contain missing values (indicated by a ‘?’ in the original files), as the
question of how to treat these missing values is not trivially defined.
Compared to the previous two tasks, the datasets for this task are smaller and
within a closer range; dataset sizes range from 360 (long parameter list) to
393 (data class) data points.

###  III-B Study Design

In our experiments, we frame our Predictive SE tasks of interest as binary
classification tasks, and consider them under a 10-fold cross-validation setup
(the most popular evaluation setup from Section II-B), which will be rerun 100
times. To answer RQ1, for each dataset, 100 random 10-fold repartitionings are
made; to answer RQ2, for each explicitly randomized ML model, 100 different,
explicit random seeds are used. As for the choice of ML models, we consider
models that conceptually occur in three commonly used ML libraries (RQ3):
Weka666https://www.cs.waikato.ac.nz/ml/weka/, scikit-learn (sklearn) in
Python, and the classifiers governed by the mlr3777https://cran.r-project.org
framework [82] governing well-established ML packages in R. More specifically,
we will consider Weka v3.8.4 in Java 11, sklearn v0.22.1 in Python 3.6, and
mlr3 v0.5.0 in R v3.6.2.

Following this, for RQ1, we consider eight common supervised ML models: (1)
Naive Bayes classifier (NB), (2) Linear Discriminant Analysis (LDA), (3)
$k$-Nearest Neighbor classification (kNN), (4) Logistic Regression (LR), (5)
Random Forest (RF), (6) Support Vector Machine (SVM), (7) Decision Tree (DT)
and (8) Quadratic Discriminant Analysis (QDA). For RQ2, we investigated for
which subset of these models explicit random seeds could be set in the
libraries. Both in Weka and sklearn, these were the RF, SVM and DT models888In
R, random state is implicit, and needs to be set as a global variable before
calling the training procedure., and we therefore will focus on these three
models in RQ3.

Library |  Classifier Name |  Default Parameters | References
---|---|---|---
Weka |  NaiveBayes |  batchSize=100, numDecimalPlaces=2, useKernelEstimator=false | [83]
sklearn |  GaussianNB |  priors∗, var_smoothing=1e-09 | —
R |  naiveBayes |  threshold=0.001, eps=0 | [84]
Weka |  LDA |  R=1e-6 | [83]
sklearn |  LinearDiscriminant-Analysis |  solver = ‘svd’, shrinkage∗, priors∗, n_components = min(n_classes-1, n_features), store_covariance=False, tol=0.0001 | [85, 86]
R |  lda |  prior=proportions, tol=0.0001, method=‘moment’, CV=F |  | [87, 88]
---
Weka |  IBk |  batchSize=100, numDecimalPlaces=2, -k 1, -W 0, -A ‘weka.core.neighbour-search.LinearNNSearch’, -A ‘weka.core.EuclideanDistance’, -R first-last |  | [83, 89]
---
sklearn |  KNeighborsClassifier |  n_neighbors=5, weights=‘uniform’, algorithm=‘auto’, leaf_size=30, p=2, metric=‘minkowski’, metric_params∗, n_jobs∗ | —
R |  kknn |  k=7, distance=2, kernel=‘optimal’, ykernel∗, scale=TRUE | [90, 91]
Weka |  SimpleLogistic |  batchSize=100, numDecimalPlaces=2, -I 0 -M 500 -H 50 -W 0.0 | [83, 92, 93]
sklearn |  Logistic-Regression |  penalty=‘l2’, dual=False, tol=0.0001, C=1.0, fit_intercept=True, intercept_scaling=1, class_weight∗, random_state∗, solver=‘lbfgs’, max_iter=100, multi_class =‘auto’, verbose=0, warm_start=False, n_jobs∗, l1_ratio∗ | [94, 95]
R |  glm |  weights=rep.int(1, nobs), start∗, etastart∗, mustart∗, offset=rep.int(0, nobs), family=gaussian(), intercept=T, singular.ok=T | [41]
Weka |  RandomForest |  batchSize=100, numDecimalPlaces=2, -P 100, -I 100, -num-slots 1 -K 0, -M 1.0, -V 0.001, -S 1 | [83, 96]
sklearn |  RandomForestClassifier |  n_estimators = 100, criterion = ‘gini’, max_depth∗, min_samples_split = 2, min_samples_leaf = 1, min_weight_fraction_leaf = 0.0, max_features = sqrt(n_features), max_leaf_nodes∗, min_impurity_decrease = 0.0, min_impurity_split∗, bootstrap = True, oob_score = False, n_jobs∗, random_state∗, verbose = 0, warm_start = False, class_weight∗, ccp_alpha = 0.0, max_samples∗ | [96]
R |  ranger |  num.trees=500, mtry∗, importance=”none”, write.forest =T, probability = F, min.node.size∗, max.depth∗, replace = T, sample.fraction = ifelse(replace, 1, 0.632), case.weights∗, class.weights∗, splitrule∗, num.random.splits = 1, alpha = 0.5, minprop = 0.1, split.select.weights∗, always.split.variables∗, respect.unordered.factors∗, scale.permutation.importance = F, local.importance = F, regularization.factor = 1, regularization.usedepth = F, keep.inbag = F, inbag∗, holdout = F, quantreg = F, oob.error = T, num.threads∗, save.memory = F, verbose = T, dependent.variable.name∗, status.variable.name∗, classification∗, | [97, 96]
Weka |  SMO |  batchSize = 100, numDecimalPlaces=2, -C 1.0, -L 0.001, -P 1.0e12, -N 0, -V -1, -K ‘weka.classifier.functions.supportVector.PolyKernel’, -E 1.0, -C 250007, -calibrator ‘weka.classifiers.function.Logistic’, -R 1.0E-8, -M -1 -num-decimal-places 4 | [83, 98, 99, 100]
sklearn |  SVC |  C=1.0, kernel=‘rbf’, degree=3, gamma=‘scale’, coef0=0.0, shrinking=True, probability=True, tol=0.001, cache_size=200, class_weight∗, verbose=False, max_iter=-1, decision_function_shape=‘ovr’, break_ties=False, random_state∗ | [101, 102]
R |  svm |  scale=T, type∗, kernel=”radial”, degree=3, gamma=if (is.vector(x)) 1 else 1 / ncol(x), coef0=0, cost=1, nu=0.5, class.weights∗, cachesize=40, tolerance=0.001, epsilon=0.1, shrinking=T, cross=0, probability=F, fitted=T | [84, 101]
Weka |  J48 |  batchSize=100, numDecimalPlaces=2, -C 0.25, -M 2 | [83, 103]
sklearn |  DecisionTreeClassifier |  criterion = ‘gini’, splitter = ‘best’, max_depth∗, min_samples_split = 2, min_samples_leaf = 1, min_weight_fraction_leaf = 0.0, max_features∗, random_state∗, max_leaf_nodes∗, min_impurity_decrease = 0.0, min_impurity_split∗, class_weight∗, presort=‘deprecated’, ccp_alpha=0.0 | [85, 104]
R |  rpart |  method=‘class’, minsplit=20, minbucket=round(minsplit/3), cp=0.01, maxcompete=4, maxsurrogate=5, usesurrogate=2, xval=10, surrogatestyle=0, maxdepth=30 parms=‘gini’ | [105, 104]
weka |  QDA |  R=1e-6 | [83]
sklearn |  QuadraticDiscriminant-Analysis |  priors∗, reg_param=0.0,  store_covariance=False, tol=0.0001 |  | [85, 86]
---
R |  qda |  prior=proportions, tol=0.0001, method=‘moment’, CV=F | [87, 88]
Parameters marked with * are set equal to NULL (in R) or None (in Python)
TABLE II: Specifications and used (default) settings of the considered
classifiers for the three libraries of interest. In case library documentation
explicitly References work related to implementation and default parameter
configurations, we list those here.

To highlight potential differences between libraries, we fix and store data
partitionings and random seed choices for the 100 runs once999Exact
configurations are shared as part of the supplementary material at
https://figshare.com/s/998dd6c07173f78bd64c, and reuse these for all
libraries. We will run the classifiers with their default parameters, unless
our experiments explicitly demand a specific parameter setting101010For the
RQ2 experiments, we override default random seed parameters explicitly;
furthermore, in order to be able to evaluate AUC scores, we explicitly set SVM
classifiers in sklearn to output probability estimates.. In doing this, we
purposefully mimic a scenario of naively running the ML models in their out-
of-the-box form; as our review in Section II-B showed, this is not uncommon in
literature. Details of exact classifier names and default parameter values in
all libraries of interest are given in Table II.

To minimize the researcher degrees of freedom and maximize reproducibility, we
also minimize preprocessing of the data in the datasets, and purposefully do
not apply any rebalancing or normalization steps, nor any hyperparameter
tuning. We emphasize that in none of our approaches, we seek to optimize for
performance explicitly; instead, we seek to quantify natural performance
variability.

For RQ1, we will run 26 (datasets) $\times$ 8 (ML models) = 208 experiments
per ML library, performing 100 10-fold cross-validations for each experiment.
To answer RQ3, we repeat this procedure for Weka, sklearn, and R. As we reuse
the same pre-generated 100 10-fold cross-validation partitionings, the $i$-th
experimental run for any ML library will have been based on the exact same
randomized partitioning for the 10-fold cross-validation. For the 208 $\times$
3 (libraries) experiments, we calculate the performance median and range,
according to 3 metrics, resulting in 1872 median-range pairs.

For RQ2, for each dataset, we take the first 10-fold partitioning considered
as part of RQ1, and then run the cross-validation 100 times, now considering
different, explicitly set random seeds in each run. In total, for this RQ, we
run 26 $\times$ 3 = 78 experiments, each encompassing 100 runs, wherein the
$i$-th experimental run, we use $i$ as an explicit random seed to be passed to
the three randomized ML models of interest111111In Weka, this also meant that
we had to set the ReducedError-Pruning parameter to true, to allow J48 using
different seeds.. Repeating this for all three ML libraries of interest, we
obtain 78 $\times$ 3 $\times$ 3 = 702 median-range pairs.

###  III-C Evaluation metrics

For all three RQs, we consider three evaluation metrics: the commonly applied
(1) F-measure and (2) Area Under the Curve (AUC), plus the (3) Matthews
Correlation Coefficient [5]. F-measure requires to set a cut-off (usually set
to 0.5) on the confidence levels produced by ML methods to decide whether a
given data point should be marked as a true case (e.g., defective class in
defect prediction) or not. Instead, the AUC metric does not require a cut-off
and, therefore, it is often preferred to other metrics [15]. Finally, the
Matthews Correlation Coefficient (unlike the F-measure and AUC) uses all four
quadrants of the confusion matrix, and has been advocated as a more stable
metric in the recent DP literature [5]. To compute these metrics, we make use
of the implementations offered by the various ML libraries.

For our analysis, we first consider the median performance (i.e., F-measure,
AUC, and Matthews Correlation Coefficient) achieved by each classifier on each
dataset and across 100 independent runs (100 data resamplings for RQ1 and 100
random seeds for RQ2). To analyze the variability in the results across
different runs, we compute the range. This is a standard dispersion metric in
statistics, reflecting the difference between the largest and smallest values
within a given distribution. In our case, this concerns the maximum
performance difference of a given classifier on a given dataset, observed over
the 100 independent runs for the RQ of interest; in other words, the
difference between best-case and worst-case performance in particularly
‘(un)lucky’ runs in our experiments. Due to space considerations, we will
present observed median results and the range values through swarm
plots121212Complete outcomes in tabular form are offered as supplementary
material at https://figshare.com/s/998dd6c07173f78bd64c: scatter plots in
which points corresponding to the same value are plotted in close proximity to
one another, rather than on top of each other. This way, both the local
density and overall spread in a distribution can be seen. Per RQ, we will plot
medians and ranges for each choice of the dataset, ML model, metric and ML
library. Results for datasets within the same predictive SE task will be
included in the same plot.

To answer RQ3 from a statistical point of view, we will not use the Scott-
Knott Effect Size Difference proposed in [38]; instead, following the
commentary in [7], we will use the Friedman test [106] with the Nemenyi post-
hoc test [107]. The Friedman test is the non-parametric equivalent to ANOVA
with unreplicated block design [106]. Due to its non-parametric nature, it
does not make any assumption about the type of distribution (i.e., the data
does not need to follow a normal distribution), and allows us to assess the
significance of differences across multiple test attempts (in our case,
datasets) when using different treatments (in our case, ML methods and their
implementations). We use the confidence level $\alpha=0.95$ [106].

While the Friedman test reveals whether results statistically differ among
different treatments, it does not tell for which combination of treatments the
significance holds. To further understand this aspect, we use the Nemenyi
post-hoc test [107]. The Nemenyi test computes the average rank achieved by
each treatment across multiple datasets. Treatments with the best ranking are
preferable over the others. Two treatments (e.g., RF in Weka vs. RF in R)
significantly different if their corresponding average ranks differ by at
least the given critical distance (CD) [107].

Figure 1: Swarm plots of median performance observed over 100 runs per
dataset, using different random 10-fold cross-validation partitions. Figure 2:
Swarm plots of median performance observed over 100 runs per dataset, using
different random 10-fold cross-validation partitions. Figure 3: Swarm plots of
performance ranges observed over 100 runs per dataset, using different random
10-fold cross-validation partitions. Figure 4: Swarm plots of performance
ranges observed over 100 runs per dataset, using different random seeds for
the ML training.

##  IV Empirical Results

###  IV-A Results

_Median performances._ Median performance observations indicate what
(absolute) performance outcomes are observed ‘in the middle’ of the
performance range, when considering multiple runs. Swarm plots for RQ1 and
RQ2, both considering all three ML libraries (RQ3), are presented in Figure 1
and 2, respectively. Comparing results for RQ1 and RQ2, we notice similar
distributional patterns and ranges, and also note that performance for an ML
model may greatly differ, depending on the considered dataset.

Considering Weka to be the most frequently used library in the field, even
while the current models were not explicitly tuned and no domain-specific data
pre-processing was performed, observed median performance numbers appear in
alignment with performance reports in the existing literature (e.g. from the
meta-analysis in [33]). However, when not only looking at results from Weka,
remarkably, different ML libraries appear to give inconsistent results. For
example, observed performances for sklearn’s kNN, LR and SVM classifiers
appear to perform worse than the other two ML libraries for various datasets
in the CSD task. Also, in some cases, for the same dataset and the same
conceptual ML model, classifiers may crash or raise warnings in one library,
but not the other. As one example, in R, no LDA and QDA outcomes can be
obtained for any dataset in the CSD task, due to constant values being
observed within groups for LDA, and rank deficiencies being observed for QDA.
In sklearn, LDA completes without any warnings on the same inputs; indeed,
collinearity warnings are given for QDA, but the experiments complete without
crashing. In Weka, no crashes are observed, and no warnings are given.

Figure 5: Results of the statistical comparison based on the Friedman test and
the Nemenyi post-hoc test.

_Performance ranges._ To highlight differences between particularly good and
bad runs, we calculate and analyze performance ranges. The performance ranges
for RQ1 and RQ2 are shown in Figures 3 and 4, respectively, again representing
outcomes for all three ML libraries of interest (RQ3).

We observe large ranges, confirming that the choice of data partition may
cause considerable performance differences. Considering that the F-measure and
AUC can range between $[0.0,1.0]$ (while the Matthews Correlation Coefficient
can range between $[-1.0,1.0]$), observed performance ranges for RQ1 (Figure
3) are particularly alarming for these two metrics: e.g., for the DP task,
considering the AUC, in some cases, performance ranges higher than 0.4 are
observed. The dataset for which the largest differences were observed was
jEdit v4.3; upon closer inspection, we found ths dataset to be highly skewed
(containing only 11 defective classes over a total of 492 classes in the
project); as a consequence, considering 10-fold partitioning, there may have
been folds without any true positives.

Comparing range results for RQ1 and RQ2 (Figure 4), two interesting
observations can be drawn for RQ2. First of all, in the absolute sense,
performance ranges for runs based on different random seeds are smaller than
those for runs based on different data repartitionings. Furthermore, as
datasets grow larger in size, in line with statistical expectations,
variability will decrease; this can e.g. be seen by comparing ranges observed
for PMT (for which datasets were considerably larger in size) with those
observed in the two other tasks.

Considering RQ3, it is striking that certain libraries yield
(near-)deterministic results for different random seed choices, while others
do not. For example, DT outcomes for Weka and R are very consistent, while for
sklearn, they differ for different random seeds.

_Multi-dataset Statistical Analysis._ While we observed that median
performance outcomes are in alignment with currently reported performance in
literature, can any ML model be claimed to be superior over all other models?
For this, as discussed before, we conduct a Friedman test and a Nemenyi post-
hoc test, comparing model ranks, based on median AUC performance. To
immediately consider this question in the context of RQ3, we consider _each
combination of ML model and ML library_ as a treatment. As different
predictive SE tasks consider different problems, potentially with different
data characteristics, we conduct separate statistical analyses per task.

Our results are shown in Figure 5; several striking aspects stand out. First
of all, while RF models consistently are high ranked for all three ML
libraries, concerning the critical distance, they cannot be claimed as
statistically superior to all other ML models. Next to this, while different
ML libraries reimplement the same conceptual ML model, the default
configurations of these models do not give equivalent results. This is
evidenced by different libraries relating to the same model generally not
grouping together in rankings; in some cases, e.g. for LDA and kNN, ranks may
greatly differ.

###  IV-B Discussion

While in terms of median performance, results of our studied ML models appear
to be in realistic ballparks, very large differences may occur for different
randomized runs, especially when considering different data repartitionings.
Furthermore, considering parallel implementations of ML models in various
common libraries, we observe major inconsistencies in performance outcomes
between libraries.

Several explanations can be found for these inconsistencies. First of all,
while referring to the conceptually same ML model, underlying design choices
may not actually implement the same model. For example, as for DT, the J48
decision tree implemented in Weka is a C4.5 tree [103], while sklearn and R
implement CART trees [104]. Careful examination of Table II also reveals major
differences in default parameter settings for the different models. For
example, for kNN, the default $k$ is 1 for Weka, 5 for sklearn and 7 for R;
for SVM, the default kernel in Weka is polynomial, while it is a radial basis
function in sklearn and R.

It should however be emphasized that no optimal default parameter setting
exists. When looking at Figure 5, considering alternative model
implementations, no consistent order is found between our three ML libraries.
That the RF models, even despite many different parameters in different
libraries, still form the most consistent high-performing block in the
rankings, can be attributed to them being an ensemble method, performing
bootstrap aggregating on decision trees. As a consequence, while the
underlying methods and parameters may differ, if many estimators are combined,
results may converge and become more robust.

##  V Threats to Validity

Internal Validity. Several threats to internal validity may be identified.
Firstly, despite our best effort applying procedural best practices [31, 32],
our systematic review may still miss relevant works. However, we sought to
provide a representative and recent snapshot, rather than an exhaustive
overview. Furthermore, patterns with regard to (the absence of) detail
reporting were so strong, that even the addition of further works will be
unlikely to alter these patterns.

Secondly, as we very explicitly used default ML model parameter values
(without hyperparameter tuning), and did not apply any domain-specific data
pre-processing, one can challenge to what extent our experimental runs
considered realistic approaches to the predictive SE tasks of interest.
However, as followed from our review in Section II-B, the use of default
values is not uncommon, and when hyperparameters are tuned, there is no
obvious optimal parameter configuration to replicate, transfer and/or ‘plug
in’ trivially. Furthermore, our goal was not to solve each predictive SE task
in the best way possible, but rather to give insights into what threats to
reproducibility may already occur in minimal-intervention, out-of-the-box
setups. Therefore, we feel the choice not to do domain-specific data pre-
processing is justified, as doing this would lead to additional, non-trivially
reported researcher degrees of freedom to be added to the procedure.

External Validity. Questions may be raised regarding the generalizability of
our findings. Our chosen datasets all come from well-established, openly
available sources that are more broadly used in the field. In choosing the
datasets, we also were careful to prioritize newer versions, larger datasets,
and realistic projects. Still, it can be questioned to what extent our
datasets representatively and comprehensively represent the predictive SE
tasks of interest, also considering that all project-related datasets consider
Java projects. However, our interest in randomization-induced performance
variability and the comparison of parallel implementations of the same ML
model actually is not dependent on the input data—in fact, we could have
applied a similar procedure to any other binary classification task in any
other field. However, with the importance of ML for predictive SE, we did
choose to restrict to data from this field, and our setup can easily be
expanded to further datasets and further tasks.

##  VI Practical Guidelines for Predictive SE

We have shown that commonly underreported aspects of ML procedures can lead to
large performance differences and unexpected outcome inconsistencies. From our
findings, we can distill several practical guidelines that can help improve
the reproducibility and statistical soundness of studies.

Guideline 1. Rerun ML models on many different resamplings of the same
dataset.

As we demonstrated, different random partitionings for 10-fold cross-
validation may lead to strongly differing performance outcomes. Our review in
Section II-B showed that multiple fold repartitionings are rarely considered
in cross-validation setups; if they are, they are rarely repeated more than 10
times. However, typically considered ML models and datasets in predictive SE,
such as the ones we used in this paper, are not so large or complex that they
demand extensive computational resources. Thus, it was inexpensive to perform
100 runs in this work, and we feel explicit justification is needed when
researchers choose not to perform many reruns.

While our study focused on within-project cross-validation, repeated
resampling is also possible for cross-project [65], and cross-version [59]
validation. Beyond taking full projects or versions as alternatives to folds
in a cross-validation setup, combinations of versions and/or projects may be
randomly drawn to increase an ML model’s robustness. In such cases, we
recommend using many repeated resamplings too.

Guideline 2. Rerun ML models considering multiple, explicitly controlled
random seeds.

While observed performance ranges were smaller for RQ2 than for RQ1, different
random seeds choices were still shown to cause non-negligible performance
variability, which was also inconsistent for implementations of the same model
in different libraries. While this is currently hardly acknowledged in the
literature, we recommend to consciously rerun models on a given data split
configuration with multiple, different, and explicitly controlled random
seeds. For our study, performing 100 runs was practically feasible.

Guideline 3. Be aware that seemingly non-randomized ML procedures may converge
differently.

While explicit randomization only occurs for a few ML models, iterative
numerical methods may be applied during the training phase. This can again
lead to performance variability, depending on (1) the solver being used
(randomized or not), (2) the number of iterations (if used as stopping
criterion), and (3) the tolerance threshold. In our study, the default
settings for NB, LDA, KNN, LR, and QDA (see Table II) use deterministic
solvers with very small tolerance values, such that consistent results were
obtained for multiple reruns under the same data and parameter configuration.
Researchers should explicitly be aware of this and check the configurations of
the solvers.

In addition, similar considerations come into play when hyperparameters are
tuned. While hyperparameter search procedures were sometimes reported in our
review, explicit final outcomes rarely are. If paper assets would not allow
such reports, we call for both the search procedure and final parameter
choices to be explicitly included in replication packages.

Guideline 4. Explicitly report parameter values, library/function names, and
library versions.

As our results show, running default settings of the (conceptually) same ML
model in different libraries yields very different results. Judging from these
outcomes, generic statements like “we trained an SVM on the openly available
dataset X” are irreproducible. While design choices and default settings
differ between different ML libraries, as shown in Table II and discussed in
Section IV, we also emphasize that they may change across multiple versions of
the same library. For example, for RF in sklearn, the default number of trees
in the forest (the n_estimators parameter)
changed131313https://tinyurl.com/yyd4kqbu from 10 (versions $<$ 0.22) to 100
(version $\geq$ 0.22).

Guideline 5. Consider which outcomes can truly be considered as statistically
significant.

Only very few works in our survey made use of statistical tests to assess what
differences can truly be deemed significant. Interest in statistically sound
performance assessment is emerging in the SE domain [6, 38, 7], and for our
current experiments, the Friedman test complemented with a post-hoc Nemenyi
test (as recommended in [107] and [7]) was appropriate. For our work, a major
realization following this procedure has been that _while RF models
consistently rank high in performance, from a statistical perspective, they
cannot be considered fully superior to other ML models_. We recommend that the
SE community more consciously take these types of tests into account when
reporting and assessing performance outcomes.

##  VII Conclusions and Future Work

In this work, we provided a systematic review of common reporting practices
regarding ML procedures in Predictive SE, indicating that randomization
aspects are very rarely explicitly acknowledged in current work. We ran
experiments with multiple randomized runs (in terms of fold partitioning and
random seed choice) on representative datasets in three Predictive SE tasks,
for eight common ML models implemented by three common ML libraries, and
observed considerable performance differences. Outcomes from different library
implementations of the conceptually same ML model were inconsistent, likely
due to greatly differing default parameter choices.

From our findings, it became clear that much more conscious awareness of these
issues is needed, and that much more transparency in reporting is needed to
make outcomes reproducible. To this end, we proposed several practical
guidelines. Given the continued interest in ML for SE, and the increasing
number of papers on ML observed in SE top venues, we hope these guidelines
will help in improving best practices and strengthening the field.

## References

  * [1] S. Hosseini, B. Turhan, and D. Gunarathna, “A systematic literature review and meta-analysis on cross project defect prediction,” _IEEE Trans. Softw. Eng._ , vol. 45, no. 2, pp. 111–147, 2019.
  * [2] J. Zhang, Z. Wang, L. Zhang, D. Hao, L. Zang, S. Cheng, and L. Zhang, “Predictive mutation testing,” in _Proc. 25th ISSTA_. ACM, 2016, pp. 342–353.
  * [3] J. Zhang, L. Zhang, M. Harman, D. Hao, Y. Jia, and L. Zhang, “Predictive mutation testing,” _IEEE Trans. Softw. Eng._ , vol. 45, no. 9, pp. 898–918, 2018.
  * [4] Z. Mahmood, D. Bowes, T. Hall, P. C. Lane, and J. Petrić, “Reproducibility and replicability of software defect prediction studies,” _Information and Software Technology_ , vol. 99, pp. 148–163, 2018.
  * [5] M. Shepperd, D. Bowes, and T. Hall, “Researcher bias: The use of machine learning in software defect prediction,” _IEEE Trans. Softw. Eng._ , vol. 40, no. 6, pp. 603–616, 2014.
  * [6] A. Arcuri and L. Briand, “A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering,” _Softw. Test. Verif. Reliab._ , vol. 24, no. 3, pp. 219–250, 2014.
  * [7] Herbold, Steffen, “Comments on ScottKnottESD in response to“ An empirical comparison of model validation techniques for defect prediction models”,” _IEEE TSE_ , 2017.
  * [8] J. P. Simmons, L. D. Nelson, and U. Simonsohn, “False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant,” _Psychol. Sci._ , vol. 22, no. 11, pp. 1359–1366, 2011.
  * [9] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. P. W. Ellis, “mir_eval: A transparent implementation of common MIR metrics,” in _Proc. 15th ISMIR Conf._ , 2014.
  * [10] B. McFee, J. W. Kim, M. Cartwright, J. Salamon, R. M. Bittner, and J. P. Bello, “Open-Source Practices for Music Signal Processing Research: Recommendations for Transparent, Sustainable, and Reproducible Audio Research,” _IEEE Signal Process. Mag._ , vol. 36, 2019.
  * [11] Z. C. Lipton and J. Steinhardt, “Research for practice: Troubling trends in machine-learning scholarship,” _Commun. ACM_ , vol. 62, no. 6, pp. 45–53, May 2019.
  * [12] D. Sculley, J. Snoek, A. Wiltschko, and A. Rahimi, “Winner’s Curse? On Pace, Progress, and Empirical Rigor,” in _Proc. 6th ICLR, Workshop Track_ , 2018\. [Online]. Available: https://openreview.net/forum?id=rJWF0Fywf
  * [13] M. Hutson, “Artificial intelligence faces reproducibility crisis,” _Science_ , vol. 359, no. 6377, pp. 725–726, 2018. [Online]. Available: https://science.sciencemag.org/content/359/6377/725
  * [14] P. Henderson, R. Islam, P. Bachman, J. Pineau, D. Precup, and D. Meger, “Deep Reinforcement Learning That Matters,” in _Proc. AAAI-18_. AAAI, 2018.
  * [15] Y. Kamei and E. Shihab, “Defect prediction: Accomplishments and future challenges,” in _23rd Int. Conf. SANER_ , vol. 5, 2016, pp. 33–45.
  * [16] Z. Wan, X. Xia, A. E. Hassan, D. Lo, J. Yin, and X. Yang, “Perceptions, expectations, and challenges in defect prediction,” pp. 1–1, 2018\.
  * [17] S. Herbold, “Crosspare: A tool for benchmarking cross-project defect predictions,” in _30th ASE Workshop (ASEW)_ , 2015, pp. 90–96.
  * [18] T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” _IEEE Trans. Softw. Eng._ , vol. 33, no. 1, pp. 2–13, 2006.
  * [19] N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in _Proc. 27th ICSE_ , 2005, pp. 284–292.
  * [20] M. Cataldo, A. Mockus, J. A. Roberts, and J. D. Herbsleb, “Software dependencies, work dependencies, and their impact on failures,” _IEEE Trans. Softw. Eng._ , vol. 35, no. 6, pp. 864–878, 2009.
  * [21] Y. Jia and M. Harman, “An analysis and survey of the development of mutation testing,” _IEEE Trans. Softw. Eng._ , vol. 37, no. 5, pp. 649–678, 2011.
  * [22] P. Ammann and J. Offutt, _Introduction to software testing_. Cambridge University Press, 2016.
  * [23] A. J. Offutt and R. H. Untch, “Mutation 2000: Uniting the orthogonal,” in _Mutation testing for the new century_. Springer, 2001, pp. 34–44.
  * [24] T. Sharma and D. Spinellis, “A survey on software smells,” _J. Syst. Softw._ , vol. 138, pp. 158–173, 2018.
  * [25] A. Van Deursen, L. Moonen, A. Van Den Bergh, and G. Kok, “Refactoring test code,” in _Proc. 2nd Int. Conf. XP_ , 2001, pp. 92–95.
  * [26] T. Sharma, M. Fragkoulis, and D. Spinellis, “Does your configuration code smell?” in _Proc. 13th MSR_. IEEE, 2016, pp. 189–200.
  * [27] E. Fernandes, J. Oliveira, G. Vale, T. Paiva, and E. Figueiredo, “A review-based comparative study of bad smell detection tools,” in _Proc. 20th Int. Conf. EASE_ , 2016, pp. 1–12.
  * [28] F. Arcelli Fontana, M. V. Mäntylä, M. Zanoni, and A. Marino, “Comparing and experimenting machine learning techniques for code smell detection,” _Empir. Softw. Eng._ , vol. 21, pp. 1143––1191, 2016.
  * [29] F. Arcelli Fontana and M. Zanoni, “Code smell severity classification using machine learning techniques,” _Knowl. Based Sys._ , vol. 128, pp. 43–58, 2017.
  * [30] D. Di Nucci, F. Palomba, D. A. Tamburri, A. Serebrenik, and A. De Lucia, “Detecting Code Smells using Machine Learning Techniques: Are We There Yet?” in _Proc. 25th SANER_. IEEE, 2018.
  * [31] B. Kitchenham, “Procedures for Undertaking Systematic Reviews,” Computer Science Department, Keele University (TR/SE-0401) and National ICT Australia Ltd. (0400011T.1), Joint Technical Report, 2004.
  * [32] B. Kitchenham, O. P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, “Systematic literature reviews in software engineering–a systematic literature review,” _Inf. Softw. Technol._ , vol. 51, no. 1, pp. 7–15, 2009\.
  * [33] M. I. Azeem, F. Palomba, L. Shi, and Q. Wang, “Machine learning techniques for code smell detection: A systematic literature review and meta-analysis,” _Inf. Softw. Technol._ , vol. 108, pp. 115–138, 2019.
  * [34] C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in _Proc. 18th Int. Conf. EASE_ , 2014, pp. 1–10.
  * [35] S. Amasaki, “Cross-version defect prediction: use historical data, cross-project data, or both?” _Empir. Softw. Eng._ , pp. 1–23, 2020.
  * [36] F. Khomh, S. Vaucher, Y.-G. Guéhéneuc, and H. Sahraoui, “BDTEX: A GQM-based Bayesian approach for the detection of antipatterns,” _J. Syst. Softw._ , vol. 84, pp. 559–572, 2011.
  * [37] A. Maiga, N. Ali, N. Bhattacharya, A. Sabané, Y.-G. Guéhéneuc, and E. Aïmeur, “SMURF: A SVM-based Incremental Anti-pattern Detection Approach,” in _Proc. 19th WCRE_. IEEE, 2012.
  * [38] C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “An empirical comparison of model validation techniques for defect prediction models,” _IEEE Trans. Softw. Eng._ , vol. 43, no. 1, pp. 1–18, 2017.
  * [39] D. Mao, L. Chen, and L. Zhang, “An extensive study on cross-project predictive mutation testing,” in _Proc. 12th IEEE Conf. ICST_. IEEE, 2019, pp. 160–171.
  * [40] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, “The WEKA data mining software: an update,” _SIGKDD Explorations_ , vol. 11, no. 1, pp. 10–18, 2009.
  * [41] R Core Team, _R: A Language and Environment for Statistical Computing_ , R Foundation for Statistical Computing, Vienna, Austria, 2012, ISBN 3-900051-07-0. [Online]. Available: http://www.R-project.org/
  * [42] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” _J. Mach. Learn. Res._ , vol. 12, pp. 2825–2830, 2011.
  * [43] MATLAB, _version 7.10.0 (R2010a)_. Natick, Massachusetts: The MathWorks Inc., 2010.
  * [44] A. Barbez, F. Khomh, and Y.-G. Guéhéneuc, “A machine-learning based ensemble method for anti-patterns detection,” _J. Syst. Softw._ , vol. 161, p. 110486, 2020.
  * [45] M. Kondo, C.-P. Bezemer, Y. Kamei, A. E. Hassan, and O. Mizuno, “The impact of feature reduction techniques on defect prediction models,” _Empir. Softw. Eng._ , vol. 24, no. 4, pp. 1925–1963, 2019.
  * [46] Y. Fan, X. Xia, D. A. da Costa, D. Lo, A. E. Hassan, and S. Li, “The impact of changes mislabeled by szz on just-in-time defect prediction,” _IEEE Trans. Softw. Eng._ , 2019.
  * [47] D. Chen, X. Chen, H. Li, J. Xie, and Y. Mu, “Deepcpdp: Deep learning based cross-project defect prediction,” _IEEE Access_ , vol. 7, pp. 184 832–184 848, 2019.
  * [48] T. Zhou, X. Sun, X. Xia, B. Li, and X. Chen, “Improving defect prediction with deep forest,” _Inf. Softw. Technol._ , vol. 114, pp. 204–216, 2019.
  * [49] C. Liu, D. Yang, X. Xia, M. Yan, and X. Zhang, “A two-phase transfer learning model for cross-project defect prediction,” _Inf. Softw. Technol._ , vol. 107, pp. 125–136, 2019.
  * [50] Z. Xu, J. Liu, X. Luo, Z. Yang, Y. Zhang, P. Yuan, Y. Tang, and T. Zhang, “Software defect prediction based on kernel PCA and weighted extreme learning machine,” _Inf. Softw. Technol._ , vol. 106, pp. 182–200, 2019\.
  * [51] Q. Huang, X. Xia, and D. Lo, “Revisiting supervised and unsupervised models for effort-aware just-in-time defect prediction,” _Empir. Softw. Eng._ , vol. 24, no. 5, pp. 2823–2862, 2019.
  * [52] C. Ni, X. Chen, F. Wu, Y. Shen, and Q. Gu, “An empirical study on pareto based multi-objective feature selection for software defect prediction,” _J. Syst. Softw._ , vol. 152, pp. 215–238, 2019.
  * [53] T. Shippey, D. Bowes, and T. Hall, “Automatically identifying code features for software defect prediction: Using ast n-grams,” _Inf. Softw. Technol._ , vol. 106, pp. 142 – 160, 2019. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0950584918302052
  * [54] T. Hoang, H. Khanh Dam, Y. Kamei, D. Lo, and N. Ubayashi, “Deepjit: An end-to-end deep learning framework for just-in-time defect prediction,” in _Proc. 16th MSR_ , 2019, pp. 34–45.
  * [55] Q. Yu, J. Qian, S. Jiang, Z. Wu, and G. Zhang, “An empirical study on the effectiveness of feature selection for cross-project defect prediction,” _IEEE Access_ , vol. 7, pp. 35 710–35 718, 2019.
  * [56] K. E. Bennin, J. W. Keung, and A. Monden, “On the relative value of data resampling approaches for software defect prediction,” _Empir. Softw. Eng._ , vol. 24, no. 2, pp. 602–636, 2019.
  * [57] J. Zhang, J. Wu, C. Chen, Z. Zheng, and M. R. Lyu, “Cds: A cross–version software defect prediction model with data selection,” _IEEE Access_ , vol. 8, pp. 110 059–110 072, 2020.
  * [58] L. Gong, S. Jiang, L. Bo, L. Jiang, and J. Qian, “A novel class-imbalance learning approach for both within-project and cross-project defect prediction,” _IEEE Trans. Reliab._ , vol. 69, no. 1, pp. 40–54, 2020\.
  * [59] G. G. Cabral, L. L. Minku, E. Shihab, and S. Mujahid, “Class imbalance evolution and verification latency in just-in-time software defect prediction,” in _Proc. 41st ICSE_ , 2019, pp. 666–676.
  * [60] T. Mori and N. Uchihira, “Balancing the trade-off between accuracy and interpretability in software defect prediction,” _Empir. Softw. Eng._ , vol. 24, no. 2, pp. 779–825, 2019.
  * [61] X. Chen, D. Zhang, Y. Zhao, Z. Cui, and C. Ni, “Software defect number prediction: Unsupervised vs supervised methods,” _Inf. Softw. Technol._ , vol. 106, pp. 161 – 181, 2019. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0950584918302076
  * [62] J. Jiarpakdee, C. Tantithamthavorn, and A. E. Hassan, “The impact of correlated metrics on the interpretation of defect models,” _IEEE Trans. Softw. Eng._ , pp. 1–1, 2019.
  * [63] J. Jiarpakdee, C. Tantithamthavorn, H. K. Dam, and J. Grundy, “An empirical study of model-agnostic techniques for defect prediction models,” _IEEE Trans. Softw. Eng._ , pp. 1–1, 2020.
  * [64] X. Chen, Y. Mu, Y. Qu, C. Ni, M. Liu, T. He, and S. Liu, “Do different cross-project defect prediction methods identify the same defective modules?” _J. Softw.: Evol. Process_ , vol. 32, no. 5, p. e2234, 2020.
  * [65] H. Chen, X. Jing, Z. Li, D. Wu, Y. Peng, and Z. Huang, “An empirical study on heterogeneous defect prediction approaches,” _IEEE Trans. Softw. Eng._ , pp. 1–1, 2020.
  * [66] P. Afric, L. Sikic, A. S. Kurdija, and M. Silic, “REPD: Source code defect prediction as anomaly detection,” _J. Syst. Softw._ , p. 110641, 2020.
  * [67] Z. Yuan, X. Chen, Z. Cui, and Y. Mu, “ALTRA: Cross-Project Software Defect Prediction via Active Learning and Tradaboost,” _IEEE Access_ , vol. 8, pp. 30 037–30 049, 2020.
  * [68] B. S. Alqadi and J. I. Maletic, “Slice-based cognitive complexity metrics for defect prediction,” in _2020 IEEE 27th Int. Conf. SANER_ , 2020, pp. 411–422.
  * [69] M. R. Naeem, T. Lin, H. Naeem, F. Ullah, and S. Saeed, “Scalable mutation testing using predictive analysis of deep learning model,” _IEEE Access_ , vol. 7, pp. 158 264–158 283, 2019.
  * [70] J. Kreimer, “Adaptive Detection of Design Flaws,” _Electronic Notes in Theoretical Computer Science_ , vol. 141, pp. 117–136, 2005.
  * [71] S. Vaucher, F. Khomh, N. Moha, and Y.-G. Guéhéneuc, “Tracking Design Smells:Lessons from a Study of God Classes,” in _Proc. 16th WCRE_. IEEE, 2009.
  * [72] S. Hassaine, F. Khomh, Y.-G. Guéhéneuc, and S. Hamel, “IDS: An Immune-inspired Approach forthe Detection of Software Design Smells,” in _Proc. 7th Int. Conf. QUATIC_. IEEE, 2010.
  * [73] N. Maneerat and P. Muenchaisri, “Bad-smell Prediction from Software Design Model Using Machine Learning Techniques,” in _Proc. 8th Int. JCSSE_. IEEE, 2011.
  * [74] A. Maiga, N. Ali, N. Bhattacharya, A. Sabané, Y.-G. Guéhéneuc, G. Antoniol, and E. Aïmeur, “Support Vector Machines for Anti-pattern Detection,” in _Proc. 27th IEEE/ACM Int. Conf. ASE_. IEEE, 2012.
  * [75] L. Amorim, E. Costa, N. Antunes, B. Fonseca, and M. Ribeiro, “Experience Report: Evaluating the Effectiveness ofDecision Trees for Detecting Code Smells,” in _Proc. 26th ISSRE_. IEEE, 2015.
  * [76] S. Fakhoury, V. Arnaoudova, C. Noiseux, F. Khomh, and G. Antoniol, “Keep It Simple: Is Deep Learning Good for Linguistic Smell Detection?” in _Proc. IEEE 25th Int. Conf. SANER_. IEEE, 2018.
  * [77] T. Guggulothu and S. A. Moiz, “Code smell detection using multi-label classification approach,” _Softw. Qual. J._ , 2020.
  * [78] F. Pecorelli, D. Di Nucci, C. De Roover, and A. De Lucia, “A large empirical assessment of the role of data balancing in machine-learning-based code smell detection,” _J. Syst. Softw._ , vol. 169, p. 110693, 2020.
  * [79] H. Liu, J. Jin, Z. Xu, Y. Zou, Y. Bu, and L. Zhang, “Deep Learning Based Code Smell Detection,” _IEEE Trans. Softw. Eng._ , 2020.
  * [80] G. Boetticher, “The PROMISE repository of empirical software engineering data,” 2007.
  * [81] M. D’Ambros, M. Lanza, and R. Robbes, “An extensive comparison of bug prediction approaches,” in _Proc. 7th MSR_. IEEE, 2010, pp. 31–41.
  * [82] M. Lang, M. Binder, J. Richter, P. Schratz, F. Pfisterer, S. Coors, Q. Au, G. Casalicchio, L. Kotthoff, and B. Bischl, “mlr3: A modern object-oriented machine learning framework in R,” _J. Open Source Softw._ , dec 2019. [Online]. Available: https://joss.theoj.org/papers/10.21105/joss.01903
  * [83] I. H. Witten and E. Frank, “Data mining: practical machine learning tools and techniques with java implementations,” _ACM SIGMOD Rec._ , vol. 31, no. 1, pp. 76–77, 2002.
  * [84] D. Meyer, E. Dimitriadou, K. Hornik, A. Weingessel, and F. Leisch, _e1071: Misc Functions of the Department of Statistics, Probability Theory Group (Formerly: E1071), TU Wien_ , 2019, r package version 1.7-3. [Online]. Available: https://CRAN.R-project.org/package=e1071
  * [85] T. Hastie, R. Tibshirani, and J. Friedman, _The Elements of Statistical Learning_ , 2nd ed. Springer, 2008.
  * [86] R. O. Duda, P. E. Hart, and D. G. Stork, _Pattern Classification_ , 2nd ed. Wiley, 2000.
  * [87] W. N. Venables and B. D. Ripley, _Modern Applied Statistics with S_ , 4th ed. New York: Springer, 2002, iSBN 0-387-95457-0. [Online]. Available: http://www.stats.ox.ac.uk/pub/MASS4
  * [88] B. D. Ripley, _Pattern Recognition and Neural Networks_. Cambridge University Press, 1996.
  * [89] D. Aha and D. Kibler, “Instance-based learning algorithms,” _Mach. Learn._ , vol. 6, pp. 37–66, 1991.
  * [90] K. Schliep and K. Hechenbichler, _kknn: Weighted k-Nearest Neighbors_ , 2016, r package version 1.3.1. [Online]. Available: https://CRAN.R-project.org/package=kknn
  * [91] K. Hechenbichler and K. P. Schliep, “Weighted k-Nearest-Neighbor Techniques and Ordinal Classification,” Ludwig-Maximilians University Munich, Discussion Paper 399, SFB 386, 2004.
  * [92] N. Landwehr, M. Hall, and E. Frank, “Logistic model trees,” _Mach. Learn._ , vol. 95, no. 1-2, pp. 161–205, 2005.
  * [93] M. Sumner, E. Frank, and M. Hall, “Speeding up logistic model tree induction,” in _9th Eur. Conf. Princ. Prac. KDD_. Springer, 2005, pp. 675–683.
  * [94] Wikipedia contributors, “Broyden–Fletcher–Goldfarb–Shanno algorithm — Wikipedia, The Free Encyclopedia”,” 2020, [Online; accessed August 25, 2020]. [Online]. Available: https://en.wikipedia.org/wiki/Broyden%E2%80%93Fletcher%E2%80
%93Goldfarb%E2%80%93Shanno_algorithm

  * [95] Z. H. Fu, “Comparison of Gradient Descent, Stochastic Gradient Descent and L-BFGS,” 2016, [Online; accessed August 25, 2020]. [Online]. Available: http://www.fuzihao.org/blog/2016/01/16/Comparison-of-Gradient-Descent-Stochastic-Gradient-Descent-and-L
-BFGS/
  * [96] L. Breiman, “Random Forests,” _Mach. Learn._ , 2001.
  * [97] M. N. Wright and A. Ziegler, “ranger: A fast implementation of random forests for high dimensional data in C++ and R,” _J. Stat. Softw._ , vol. 77, no. 1, pp. 1–17, 2017.
  * [98] J. Platt, “Fast training of support vector machines using sequential minimal optimization,” in _Advances in Kernel Methods - Support Vector Learning_ , B. Schoelkopf, C. Burges, and A. Smola, Eds. MIT Press, 1998. [Online]. Available: http://research.microsoft.com/j̃platt/smo.html
  * [99] S. Keerthi, S. Shevade, C. Bhattacharyya, and K. Murthy, “Improvements to Platt’s SMO Algorithm for SVM Classifier Design,” _Neur. Comp._ , vol. 13, no. 3, pp. 637–649, 2001.
  * [100] T. Hastie and R. Tibshirani, “Classification by pairwise coupling,” in _Adv. Neural Inf. Proc. Sys._ , M. I. Jordan, M. J. Kearns, and S. A. Solla, Eds., vol. 10. MIT Press, 1998.
  * [101] C.-C. Chang and C.-J. Lin, “LIBSVM: A Library for Support Vector Machines,” _ACM Trans. Intell. Syst. Technol._ , vol. 2, no. 3, 2011.
  * [102] J. C. Platt, “Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods,” in _Advances in Large Margin Classifiers_. MIT Press, 1999.
  * [103] R. Quinlan, _C4.5: Programs for Machine Learning_. San Mateo, CA: Morgan Kaufmann Publishers, 1993.
  * [104] L. Breiman, J. Friedman, C. J. Stone, and R. A. Olshen, _Classification and Regression Trees_. Wadsworth, 1984\.
  * [105] T. Therneau and B. Atkinson, _rpart: Recursive Partitioning and Regression Trees_ , 2019, r package version 4.1-15. [Online]. Available: https://CRAN.R-project.org/package=rpart
  * [106] M. Friedman, “A comparison of alternative tests of significance for the problem of m rankings,” _Ann. Math. Stat._ , vol. 11, no. 1, pp. 86–92, 1940\.
  * [107] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” _J. Mach. Learn. Res._ , vol. 7, no. Jan, pp. 1–30, 2006.