# Over-optimism in benchmark studies and the multiplicity of design and
analysis options when interpreting their results

Christina Nießl 111Corresponding author, e-mail<EMAIL_ADDRESS>muenchen.de, Institute for Medical Information Processing, Biometry and
Epidemiology, Ludwig Maximilians University Munich, Marchioninistr. 15,
D-81377, Munich, Germany. Institute for Medical Information Processing,
Biometry and Epidemiology, Ludwig Maximilians University Munich (Germany)
0000-0003-2425-7858 Moritz Herrmann  Department of Statistics, Ludwig
Maximilians University Munich (Germany)  0000-0002-4893-5812 Chiara Wiedemann
Institute for Medical Information Processing, Biometry and Epidemiology,
Ludwig Maximilians University Munich (Germany)  Giuseppe Casalicchio
Department of Statistics, Ludwig Maximilians University Munich (Germany)
0000-0001-5324-5966 Anne-Laure Boulesteix  Institute for Medical Information
Processing, Biometry and Epidemiology, Ludwig Maximilians University Munich
(Germany)  0000-0002-2729-0947

(August 28, 2024)

###### Abstract

In recent years, the need for neutral benchmark studies that focus on the
comparison of methods from computational sciences has been increasingly
recognised by the scientific community. While general advice on the design and
analysis of neutral benchmark studies can be found in recent literature,
certain amounts of flexibility always exist. This includes the choice of data
sets and performance measures, the handling of missing performance values and
the way the performance values are aggregated over the data sets. As a
consequence of this flexibility, researchers may be concerned about how their
choices affect the results or, in the worst case, may be tempted to engage in
questionable research practices (e.g. the selective reporting of results or
the post-hoc modification of design or analysis components) to fit their
expectations or hopes. To raise awareness for this issue, we use an example
benchmark study to illustrate how variable benchmark results can be when all
possible combinations of a range of design and analysis options are
considered. We then demonstrate how the impact of each choice on the results
can be assessed using multidimensional unfolding. In conclusion, based on
previous literature and on our illustrative example, we claim that the
multiplicity of design and analysis options combined with questionable
research practices lead to biased interpretations of benchmark results and to
over-optimistic conclusions. This issue should be considered by computational
researchers when designing and analysing their benchmark studies and by the
scientific community in general in an effort towards more reliable benchmark
results.

##  1 Introduction and related work

With the constant development of new methods in computational sciences (e.g.
machine learning and bioinformatics), it is becoming increasingly difficult
for data analysts to keep pace with scientific progress and to select the most
appropriate method for their data and research question out of the many
existing approaches. This problem is addressed by benchmark studies, which
systematically analyse and compare the performance of several methods in
different conditions using simulated or real data sets.
In many cases, benchmark studies are performed as part of a paper introducing
a new method, usually with the intention to demonstrate the superiority of the
new method over existing ones. Accordingly, they can be considered as biased
in favour of the newly proposed method and should be seen as an informal
method comparison rather than a real benchmark study (Norel et al.,, 2011;
Boulesteix et al.,, 2013; Buchka et al.,, 2021). In contrast, so called
neutral benchmark studies are defined as benchmark studies that focus on the
comparison itself and are ideally performed by reasonably neutral authors,
i.e. authors who (1) are equally experienced with all considered methods and
(2) design and analyse the study in a rational way (Boulesteix et al.,, 2017).
These characteristics make neutral benchmark studies essentially unbiased.
Therefore, recommendations resulting from such studies are especially relevant
both for method users and developers (Boulesteix et al.,, 2018).
Regarding the appropriate design and analysis of benchmark studies, the
available literature ranges from general guidelines (Weber et al.,, 2019;
Boulesteix,, 2015) and statistical frameworks (Demšar,, 2006; Hothorn et al.,,
2005; Eugster et al.,, 2012; Boulesteix et al.,, 2015, all with focus on
supervised learning), to recommendations for context-specific benchmarks (e.g.
Mangul et al.,, 2019; Bokulich et al.,, 2020; Zimmermann,, 2020; Kreutz,,
2019). However, for many issues relevant in practice (e.g. the selection of
data sets and performance measures), no concrete guidance or methodology can
be found. This means that researchers are usually faced with a high amount of
flexibility when conducting their benchmark study.
As a consequence, researchers who are aware of these issues, although making
well-considered design and analysis choices prior to conducting the benchmark
study, might be concerned about how their choices affect the results. On the
other hand, the high amount of flexibility could tempt less aware researchers
to engage in questionable research practices (see John et al.,, 2012, in the
context of applied research) when conducting their benchmark study. This
includes the selective reporting of results (e.g. reporting the results of
only one performance measure although performance was originally assessed by
two measures) and the modification of specific design and/or analysis
components of the benchmark study after seeing the results (e.g. using
performance measures other than those originally selected). Of course, these
practices are not questionable on their own. For example, it is fine to use an
alternative performance measure if the current one does not produce meaningful
results as long as the change of performance measure is adequately justified
and documented. However, practices such as the selective reporting of results
or the post-hoc modification of benchmark components do become questionable if
they are applied to fit the researchers’ expectations or hopes. For example,
researchers might seek an “exciting” result (e.g. a clear-cut result
suggesting a univocal winner as opposed to vague tendencies) or have a
specific presumption in mind that they want to be confirmed by the results
(e.g. the superiority of a certain method or class of methods that they are
more familiar with or that has performed well in previous benchmark studies).
The problem with such research practices is that they are likely to produce
over-optimistic results, i.e. results with an optimistic bias towards the
researchers’ expectations and hopes. While we are convinced that very few
researchers have the actual intention to cheat (Ioannidis et al.,, 2014), it
should not be understated that “even an honest person is a master of self-
deception” (Nuzzo,, 2015), meaning that every researcher is at risk of
engaging in questionable research practices. Moreover, the non-neutrality that
leads to such practices in the first place is difficult to avoid completely
and is likely to arise in a subconscious manner even in studies intended as
neutral. Note also that the actual neutrality of neutral benchmark studies can
only be checked to a certain extent. For example, one may review the authors’
publication lists to identify the methods they are most familiar with, but
this gives only a partial picture of someone’s (non-)neutrality.
In application fields of statistics (e.g. medicine and psychology), the
multiplicity of analysis strategies and the associated risk of over-optimistic
results are well-known issues (Ioannidis,, 2005; Simmons et al.,, 2011;
Hoffmann et al.,, 2021) and terms such as “p-hacking” or “fishing expeditions”
have been discussed by many (Head et al.,, 2015; Wagenmakers et al.,, 2012).
However, in methodological research including benchmark studies, this topic is
covered rather sparsely. Existing literature on the risk and prevention of
over-optimism in benchmark studies is either limited to general considerations
in benchmarking guidelines (Weber et al.,, 2019; Boulesteix et al.,, 2017) or
to benchmark studies that are performed as part of a paper introducing a new
method (Norel et al.,, 2011; Boulesteix,, 2015), which can be transferred to
neutral benchmark studies only to a limited extent. Similarly, the scarce
literature that empirically investigates the effects of over-optimism in
benchmark studies in a quantitative manner is either also devoted to the bias
affecting evaluations of a newly proposed method to other existing methods
(Jelizarow et al.,, 2010; Buchka et al.,, 2021), or focusing on the selection
of data sets (MacIà et al.,, 2013; Yousefi et al.,, 2010).
In this paper, we illustrate and discuss the multiplicity of options regarding
the design and analysis of neutral benchmark studies based on real data sets
and examine its effect on the results. Note that although we focus on neutral
benchmark studies based on real data, our results are also relevant to
benchmarks comparing new to existing methods and, to some extent, benchmarks
based on simulated data. We will empirically address the multiplicity of
options and its effects in a twofold approach. In the first step, in order to
raise awareness of the multiplicity of possible results and the over-optimism
that may arise from questionable research practices, we use the results of a
recently published benchmark study to illustrate how variable the resulting
method rankings are when different options for design and analysis are
considered. In the second step, we propose a framework based on
multidimensional unfolding (Borg and Groenen,, 2005) that enables researchers
to assess the impact of each choice on the method rankings. More precisely,
the framework allows to analyse when and how using alternative options for a
specific choice affects the results and can thus be an effective strategy to
prevent biased interpretations and over-optimistic conclusions.
The exemplary study we will use throughout the paper to illustrate our
proposed framework and the multiplicity of possible options and results is a
benchmark experiment by Herrmann et al., (2020) comparing the performance of
13 survival prediction methods based on 18 real so-called “multi-omics” data
sets. Note that our paper does not intend to question the results of this
study. Instead, it should be seen as extended analysis of the benchmark study,
which by assessing the multiplicity of results and examining the impact of
each choice, makes the results of Herrmann et al., (2020) even more reliable
and meaningful.
The remainder of this paper is structured as follows: we review and discuss a
selection of design and analysis choices in the context of benchmark studies
in Section 2, and describe the design of the study as well as the principle of
multidimensional unfolding in Section 3. The results are presented in Section
4, which is followed by a discussion in Section 5 and concluding remarks in
Section 6.

##  2 Examples of design and analysis choices in benchmark studies

###  2.1 Setting

In this section, we discuss some of the choices that researchers are faced
with when conducting a benchmark study based on real data sets. In general,
most choices that have to be made to conduct a benchmark study relate to (1)
the general aim of the study, (2) the design of the study, or (3) the analysis
of the performance results; see the left part of Figure 1. Choices that belong
to the first category are, for instance, the choice of methods to be compared
or the type of outcome variable to be considered. However, in this paper, we
focus on choices regarding the design of the study (i.e. how the aim of the
study is addressed) and the analysis of performance results (i.e. how the
$L\times M$ matrix of results generated by each considered performance measure
is analysed, where $L$ and $M$ are the numbers of data sets and methods,
respectively). It is important to note that these choices should ideally be
made prior to conducting the benchmark study. However, we conjecture that they
are in practice often made post-hoc, i.e. after seeing the results—which can
amount to questionable research practices. When reading a benchmark study,
there is no way to check when the choices were made.
For each choice, we will give concrete examples of possible options that will
later be analysed with regard to their effect on the results; see the right
part of Figure 1. For this purpose, we consider the benchmark study by
Herrmann et al., (2020) mentioned above. The authors compare the performance
of $M=13$ survival prediction methods (here denoted as BlockForest, Clinical
Only, CoxBoost, CoxBoost Favoring, Glmboost, Grridge, Ipflasso, Kaplan-Meier,
Lasso, Prioritylasso, Prioritylasso Favoring, Ranger and Rfsrc) on $L=18$ real
multi-omics data sets. See the original paper (Herrmann et al.,, 2020) for
details on the methods, the benchmark experiment and the results.

Figure 1: Examples of choices that researchers are usually faced with when
conducting a benchmark study including options used in the example benchmark
study by Herrmann et al., (2020) (second column) and alternative options
(third column). Options that are considered in our illustration are coloured
in purple.

###  2.2 Design choices

####  2.2.1 Data sets

The selection of data sets is an important design choice in every benchmark
study, as the performances are usually highly variable across data sets (Weber
et al.,, 2019; Novianti et al.,, 2015). To make meaningful statements and
prevent the study from being underpowered, it is recommended to consider an
adequate number of data sets (Boulesteix et al.,, 2017). Although there are
suggestions on how to calculate the minimum required number (Boulesteix et
al.,, 2015), it seems that the number of included data sets is usually based
on practical criteria (such as availability or computational cost) rather than
statistical considerations (MacIà et al.,, 2013). Moreover, if the benchmark
study aims at external validation, the number of data sets that can be
included in the benchmark study is usually limited, as for many data sets
there is often no comparable data set available that could be used for
external validation.
Concerning the type of data sets, researchers should include data sets that
are representative for the domain of interest and diverse enough to make sure
the methods can be evaluated under a wide range of conditions (Gatto et al.,,
2016; Weber et al.,, 2019). Corresponding inclusion criteria for the data sets
should be defined before conducting the benchmark study (Boulesteix et al.,,
2017). However, the decision on how the inclusion criteria are defined lies
with the researcher. In many benchmark studies, the exact search strategy or
inclusion criteria are not reported transparently, suggesting that in these
cases, there might be no clearly defined inclusion criteria at all.
In the benchmark study by Herrmann et al., (2020), the authors selected all
cancer data sets with five different multi-omics groups and more than 100
samples from the TCGA research network (http://cancergenome.nih.gov).
Additionally, they excluded data sets that did not have observations for every
data type or less than 5% effective cases (i.e. patients with event),
resulting in a total of $L=18$ data sets. However, depending on their research
interest, Herrmann et al., (2020) could have set additional constraints. For
example, if the authors had been interested in the performance of the methods
on data sets with a small number of effective cases, they could have adjusted
the inclusion criteria accordingly (e.g. set $n_{e}<30$). The other way
around, one may decide to ignore data sets with a small number of events (e.g.
set $n_{e}\geq 30$) because it is questionable if it makes sense to fit models
in this case at all.
In this paper, we will address the multiplicity of possible options regarding
the selection of data sets and its impact on the results by considering
subgroups of the original $L=18$ data sets defined based on some of the data
sets’ characteristics. The considered characteristics are the number of
clinical variables ($clin$), the number of observations ($n$), the number of
effective observations ($n_{e}$), and the number of variables ($p$). For each
data set characteristic, we will only consider data sets that are smaller
($<$) or greater or equal ($\geq$) than the median value of the respective
data set characteristic over the 18 considered data sets. This results in
eight groups with 8 to 10 data sets.

####  2.2.2 Quantitative performance measure

Another important aspect of benchmarking is the choice of evaluation criteria,
which usually includes both quantitative performance measures and other
measures such as runtime or qualitative features such as user-friendliness.
Although all these evaluation criteria are important, we will focus on
quantitative performance measures in this paper.
The choice of performance measure is usually context-specific, i.e. it depends
on the type of methods and data addressed in the benchmark study, as well as
on the aspects of performance that are considered the most important by the
researcher (Weber et al.,, 2019; Morris et al.,, 2019). It is also often a
non-trivial choice. For some tasks such as classification, researchers are
spoilt for choice considering the variety of measures they can choose from
(e.g. accuracy, sensitivity/specificity, area under the curve or F1-Score),
which makes decisions difficult (Mangul et al.,, 2019; Robinson and Vitek,,
2019). In contrast, for more complex situations they might have to design
their own performance measures, which can also be challenging (Weber et al.,,
2019). To provide a more complete picture of the methods’ behaviour and avoid
over-optimism, it can be useful to consider more than one performance measure
(Norel et al.,, 2011). However, there is no way to objectively determine the
adequate number of performance measures as this is highly context dependent.
In the benchmark study by Herrmann et al., (2020), the primary performance
measure is the integrated Brier score (Graf et al.,, 1999; denoted as ibrier).
Additionally, they consider Uno’s C-index (Uno et al.,, 2011; denoted as
cindex). The authors justify their decision to use the ibrier as primary
measure by the fact that cindex only measures the discriminatory power and is
not a strictly proper scoring rule (Blanche et al.,, 2019), while the ibrier
additionally measures calibration. However, they argue that if the main
interest lies in ranking patients according to their risk, then the cindex
would also be a valid measure. Furthermore, they reason that it makes sense to
include the cindex for the purpose of comparability with other studies, since
it is a widely used performance measure. Accordingly, depending on which
aspect of performance they would have considered more important, Herrmann et
al., (2020) could have also used the cindex as primary performance measure or
only selected one of the two performance measures. In this paper, we will thus
compare the results of ibrier and cindex.

###  2.3 Analysis choices

####  2.3.1 Handling of missing performance values

Because of non-convergence or other computational issues, methods sometimes
fail to output a result for a specific data set. In the context of resampling
procedures such as cross-validation or bootstrapping, the consequence is that
performance values may be missing for all or part of the resampling iterations
for some data sets. This problem seems to be common especially in benchmarks
of larger scale (Bischl et al.,, 2013). While there is at least some
literature devoted to the selection of data sets and performance measures, the
issue of missing performance values in some combinations of data sets and
methods is almost completely ignored. Many authors of benchmark studies do not
report how they handled missing performance values, and there is to our
knowledge no corresponding guidance available.
Bischl et al., (2013) mention several possible ad-hoc options that could be
applied if the missing values occur only on a subset of resampling iterations,
namely that missing values could be imputed by the worst possible value or by
the mean of the remaining performance values obtained for this combination of
data set and method—although both options are not ideal in their opinion.
Another ad-hoc option they actually use for their benchmark study is a mixed
strategy, where the imputed value is sampled from an estimated normal
distribution of the remaining values if the method fails in less than 20% of
the resampling iterations. If the method fails in more than 20% of the
resampling iterations, the worst possible value is used for imputation.
Herrmann et al., (2020), who use cross-validation as resampling procedure and
also face the problem of failing iterations, use a similar 20%-threshold rule
as Bischl et al., (2013). However, instead of sampling from a normal
distribution, they use the mean performance value of the remaining iterations
and instead of the worst possible value, they assign values of the performance
measures corresponding to random prediction (i.e. 0.25 for ibrier and 0.5 for
cindex).
Since there seems to be no common agreement on how to handle missing values in
this context, other sensible options would also be justifiable. For example,
missing values could be imputed by a formula that weights the mean performance
value and the random performance value used by Herrmann et al., (2020)
according to the proportion of missing values, thus avoiding the choice of an
arbitrary threshold. For the ibrier, where 0 corresponds to the best possible
value and 0.25 to random prediction, the imputed value for the considered
combination of data set and method could be defined as

$\displaystyle
x_{impute}=0.25-(0.25-\frac{\sum_{i\in\mathcal{I}}x_{i}}{|\mathcal{I}|})_{+}\cdot(1-r),$
(1)

where $\mathcal{I}$ is the set of indices of the non-failed iterations,
$x_{i}$ is the ibrier value for iteration $i\in\mathcal{I}$ and $r$ is the
proportion of missing values. For two methods with the same mean value for
non-failed iterations, the method with more missing values obtains a worse
performance value. Moreover, the imputed value is equal to 0.25 if a method
has 100% failures for a data set, or a mean value greater or equal than 0.25
(which makes sense since fluctuations above the value 0.25 corresponding to
random prediction are not relevant). Another advantage of this weighted
imputation procedure is that it reduces to the mean when the proportion of
missing values $r$ tends to 0—as intuitively expected. The corresponding
formula for the cindex can be found in the supplementary material.
In this paper, we will consider four imputation methods that can be used to
handle the issue of missing performance values: the 20%-threshold rule used by
Herrmann et al., (2020), the weighted method in Eq. (1), imputation using
values that correspond to random prediction, and imputation using the average
of the non-failed iterations.

####  2.3.2 Aggregation of performance values across data sets

Although it is common to analyse the methods’ individual performances across
data sets (e.g. using graphical tools), most benchmark studies ultimately
aggregate the performance values over the data sets to generate an overall
method evaluation, for example in the form of a ranking or a list of methods
that show statistically significant differences in performance. While there is
much literature addressing statistical testing procedures in benchmark
experiments based on a single data set (Dietterich,, 1998; Hothorn et al.,,
2005) or several data sets (Demšar,, 2006; Eisinga et al.,, 2017), there seems
to be no consensus on how to generate an overall method ranking from several
data sets, which we will focus on in this section.
For example, the performance values can be aggregated using standard summary
measures such as the mean, median, minimum, maximum or standard deviation
(Mersmann et al.,, 2015). Since the distribution of performance values can be
considerably skewed, some authors advise against using the mean or median as
aggregation method. Instead, they recommend assigning ranks to the methods for
each data set such that the best method in the corresponding data set obtains
rank 1 and the worst method rank $M$, where $M$ is the number of considered
methods (Demšar,, 2006; Hornik and Meyer,, 2007). The resulting ranks are then
usually aggregated using the mean (e.g. Kibekbaev and Duman,, 2016; Verenich
et al.,, 2019) or, less often, the median (e.g. Orzechowski et al.,, 2018).
Other possible aggregation methods include counting the number of times a
method performs best, often divided by the number of data sets to obtain a
value between 0 and 1 (e.g. De Cnudde et al.,, 2020; Fernández-Delgado et
al.,, 2014; Wu et al.,, 2020). Some of these authors suggest to not only
consider the best performing method for each data set but also the set of
methods performing similarly to the best method. Accordingly, Fernández-
Delgado et al., (2014) consider the number of data sets in which a method
achieves 95% or more of the maximum accuracy (i.e. the accuracy achieved by
the best performing method in that data set) divided by the total number of
data sets. In the same vein, Wu et al., (2020) estimate the probability of
achieving good performance as the number of data sets for which the method is
among the top three methods divided by the total number of data sets.
Note that all aggregation methods presented so far are based on point
estimates of the methods’ performances. Although less frequently used in
practice, it is also possible to generate method rankings based on the results
of statistical tests (i.e. pairwise comparisons indicating if method $1$
performs significantly better than method $2$) using consensus rankings
(Hornik and Meyer,, 2007).
If more than one performance measure and/or other evaluation criteria (e.g.
runtime) are considered, researchers also have to decide if rankings arising
from multiple criteria should be combined in some form (e.g. Eugster et al.,,
2012) or should be considered separately, as suggested by Weber et al.,
(2019). Specifically, Weber et al., (2019) recommend to identify a set of
consistently high performing methods based on the individual rankings and then
highlight the different strengths of each method.
Herrmann et al., (2020) aggregate the performance values based on ibrier and
cindex using the mean and consider each ranking separately. To assess the
heterogeneity of performances across data sets, they also calculate the
resulting standard deviations and confidence intervals and perform paired
t-tests. In our illustration, we will consider four aggregation methods that
can be used to generate method rankings: mean (as used by Herrmann et al.,,
2020), median, mean rank and number of times a method performs best. If two
methods obtain the same rank according to the number of times they perform
best, they are additionally ranked by the number of times their performance
lies within the 5% environment of the best performing method. This applies if
$\frac{|\overline{x}_{m}-\overline{x}_{best}|}{\overline{x}_{best}}<0.05$,
where $\overline{x}_{m}$ denotes the performance (cindex or ibrier) of method
$m$ and $\overline{x}_{best}$ the performance of the best performing method in
the corresponding data set. We denote this aggregation method (i.e. counting
the number of times a method performs best and the number of times it lies
within the 5% environment as secondary ranking method) as best0.05. Note that
since we only evaluate the results of one performance measure at a time
(ibrier or cindex), we are not considering different options for combining
rankings that result from more than one performance measure.

##  3 Methods

###  3.1 Design of the study

To illustrate the variability of benchmark results with respect to design and
analysis choices, we use the benchmark results from Herrmann et al., (2020)
and systematically examine different combinations of design and analysis
options. Specifically, we consider all combinations of options regarding the
choice of data sets (9 options), performance measure (2 options), imputation
method (4 options), and aggregation method (4 options) described in Section 2
and Figure 1. This results in $9\times 2\times 4\times 4=288$ combinations. We
then compare the 288 resulting rankings of the 13 survival prediction methods,
where a rank of 1 corresponds to the best performing method and a rank of 13
to the worst performing method (average ranks are assigned in case of ties).

###  3.2 Multidimensional unfolding

The impact of each choice on the method rankings is assessed using
multidimensional unfolding (Coombs,, 1964; Borg and Groenen,, 2005), which we
will briefly introduce in the remainder of this section. Multidimensional
unfolding is a technique that represents preference data as distances in a
low-dimensional space. It locates $K$ ideal points representing the subjects
(in our case, $K=288$ combinations) and $M$ object points representing the
objects (in our case, $M=13$ methods) such that the distances from each ideal
point to the object points correspond to the observed preference values. The
closer an object point lies to a subject’s ideal point, the stronger the
subject’s preference for that object. Accordingly, the ideal point itself
corresponds to maximal preference (Borg et al.,, 2013).
Multidimensional unfolding takes non-negative dissimilarities $\delta_{km}$
($k=1,\dots,K$; $m=1,\dots,M$) as input, which are the preference values
possibly converted in a way that small values correspond to high preferences.
In our case, where the preference values are ranks, this is not necessary
since a small rank already indicates high preference. Moreover, the number of
dimensions $dim$ must be specified, which we set to $dim=2$ as it is done in
most applications of multidimensional unfolding. To find the coordinates for
the points representing the $K$ subjects and $M$ objects, a loss function
(stress) is minimised. It is defined as

$\sigma^{2}(\hat{\boldsymbol{D}},\boldsymbol{Z}_{1},\boldsymbol{Z}_{2})=\sum_{k=1}^{K}\sum_{m=1}^{M}w_{km}(\hat{d}_{km}-d_{km}(\boldsymbol{Z}_{1},\boldsymbol{Z}_{2}))^{2},$
(2)

where $w_{km}$ denotes a non-negative a priori weight (which is set to
$w_{km}=1$ by default), and $\boldsymbol{Z}_{1}\in\mathbb{R}^{K\times dim}$
and $\boldsymbol{Z}_{2}\in\mathbb{R}^{M\times dim}$ are the coordinates for
the points representing the subjects and objects, respectively. Moreover,
$d_{km}(\boldsymbol{Z}_{1},\boldsymbol{Z}_{2})$ denotes the fitted Euclidean
distances

$d_{km}(\boldsymbol{Z}_{1},\boldsymbol{Z}_{2})=\sqrt{\sum_{s=1}^{dim}(z_{1ks}-z_{2ms})^{2}}.$
(3)

The matrix $\hat{\boldsymbol{D}}\in\mathbb{R}_{0}^{+K\times M}$ contains the
disparities $\hat{d}_{km}=f(\delta_{km})$, which are the optimally scaled
dissimilarities. This means that the loss function in Eq. (2) is not only
minimised with respect to $\boldsymbol{Z}_{1}$ and $\boldsymbol{Z}_{2}$ but
also with respect to a function $f(\cdot)$ that transforms the dissimilarities
$\delta_{km}$ into disparities $\hat{d}_{km}$ (the function class depends on
the assumed scale level). If the preference data is available in form of
rankings, $f(\delta_{km})$ reflects a monotone step function that is found
through monotonic regression on the dissimilarities. This type of
multidimensional unfolding is referred to as ordinal or non-metric unfolding.
To avoid degenerate solutions due to equal disparities which occur
particularly often in non-metric unfolding, it is recommended to use a
penalised version of the stress function in (2) that involves the coefficient
of variation $v(\hat{\boldsymbol{D}})$. The penalised stress function is
minimised through numerical optimisation using a strategy called SMACOF
(Stress Majorization of a Complicated Function) and is implemented in an R
package of the same name (de Leeuw and Mair,, 2009). For details on
multidimensional unfolding and its implementation see Mair et al., ming
(forthcoming), Borg and Groenen, (2005) and Busing et al., (2005).

##  4 Results

For full reproducibility, the entire analysis and the results presented in
this section are publicly available in the GitHub repository
https://github.com/NiesslC/overoptimism_benchmark.

###  4.1 Overall variability and step-wise optimisation

As a first step, we compare the method rankings resulting from all 288
combinations of design and analysis options. Figure 2 shows the corresponding
rank distribution for each method. Importantly, it reveals that any method can
achieve almost any rank. On one hand, all methods but one achieve rank 1 (8
methods) or 2 (4 methods) for at least one combination. The exception is
Kaplan-Meier, which does not use any feature information and can achieve ranks
as small as 3. On the other hand, 10 methods are found to be the worst or one
of the two worst methods (i.e. have rank 13 or 12.5, respectively) for at
least one combination. The highest rank obtained by the remaining methods
(Clinical Only, BlockForest, and CoxBoost Favoring) ranges from 10 to 11.5.
Figure 2 also reveals that the ranks are distributed differently for each
method. For example, while Clinical Only obtains rank 1 or 2 in approximately
50% of the combinations, the ranks of Ranger are more evenly distributed.

Figure 2: Rank distribution of 13 methods generated by 288 combinations of
design and analysis options.

While considering all combinations of options provides valuable information on
the overall variability of results, it is not a realistic scenario concerning
over-optimism in the sense that no researcher conducting a benchmark study
would try all possible combinations to obtain a favourable result (unless they
are actively cheating, which we do not assume here). Therefore, we
additionally illustrate how easy it is to modify the method rankings if the
design and analysis options are selected in a step-wise optimisation process,
which might represent a more realistic scenario. In our illustration, the
step-wise optimisation for each method is performed as follows: In each step
(i.e. for each choice), the option that yields the best rank for the
considered method (or the best performance value in case of equal ranks) is
selected. If all options yield the same result, the default option is used. As
default options, we use all 18 data sets, ibrier as primary performance
measure, the 20%-threshold rule as imputation method, and the mean as
aggregation method. This corresponds to the setting of Herrmann et al.,
(2020). Moreover, we assume that a favourable result is a small rank for a
specific method. Note that this may not always be the case, e.g. if one
expects a reference method such as Kaplan-Meier to obtain a high rank or
considers a group of several methods as target.
Figure 3 displays the optimisation process if the ranks are optimised in the
order: (1) imputation method, (2) aggregation method, (3) performance measure,
and (4) data sets. It shows that for 8 of 13 methods, the best rank achieved
by step-wise optimisation corresponds to the smallest possible rank for the
corresponding method (i.e. the smallest rank that can be achieved when all 288
combinations are considered) and for another three methods, the step-wise
optimisation achieves one rank higher than the smallest possible rank. Only
two methods (Prioritylasso and Grridge) show a larger discrepancy between
step-wise optimisation and considering all possible combinations. However,
this is not too surprising considering the few cases and thus very specific
combinations where they achieve small ranks (see Figure 2). If a step is
missing in the optimisation process of a certain method, this indicates that
the corresponding step did not improve the rank of that method. In fact, all
methods except Lasso and Prioritylasso Favoring require no more than two
optimisation steps.
Note that the results of the step-wise optimisation depend on the default
options. For example, when cindex instead of ibrier is used as default option,
the resulting ranks are higher (see Figure S1 in the supplementary material).
Moreover, the results depend on the order in which the ranks are optimised.
The order shown in Figure 3 is realistic in the sense that researchers might
find it more problematic to modify components of the benchmark study that are
generally considered as important (i.e. performance measure or data sets) and
thus only resort to them if the previous optimisation steps (i.e imputation
method or aggregation method) do not yield a favourable result. However, other
orders in which the ranks are optimised would also be conceivable. For
example, the selection of data sets could be optimised first since it offers
many options and can be easily modified by eliminating specific data sets. In
this case, the selection of data sets remains the only optimisation step for
many methods since the subsequent steps do not lead to an improvement (see
Figure S2 in the supplementary material), which already indicates the large
impact of data set selection, discussed in more detail in the next section.

Figure 3: Step-wise optimisation of method ranks by (1) imputation method
(blue), (2) aggregation method (pink), (3) performance measure (green), and
(4) data sets (yellow). The dotted line corresponds to the smallest and
highest possible ranks when all 288 combinations are considered. Missing steps
indicate that they did not lead to an improved rank. Default options
correspond to Herrmann et al., (2020).

###  4.2 Impact of individual design and analysis choices

To gain additional insight concerning the impact of each design and analysis
choice, the method rankings are analysed using multidimensional unfolding.
Figure 4 displays the resulting unfolding solution that represents the
rankings of all 288 combinations regarding the 13 methods. Before looking at
the different colourings of the ideal points in Figure 4(a)-4(d), we can make
some general observations on how the combinations and methods are scaled in
the plot (which is identical for each figure). First, the unfolding solution
clearly shows that the method rankings can differ widely depending on which
combination of design and analysis options is considered, which is consistent
with the results presented in Section 4.1. Second, similar to the rank
distribution in Figure 2, the unfolding solution indicates that some methods
tend to achieve smaller ranks than other methods. This applies specifically to
Clinical Only, CoxBoost Favoring, and BlockForest, which are scaled close to
the origin and thus have a small distance to most ideal points. In contrast,
other methods such as Lasso and Kaplan-Meier can be found in the periphery of
the plot, indicating that they obtain rather high ranks by most combinations.
Of course, the degree to which the presented unfolding solution reflects the
actual rankings depends on its goodness-of-fit (a perfect fit usually requires
as many dimensions as there are methods, i.e. $dim=M=13$). However, following
Mair et al., (2016), the unfolding solution in Figure 4 fits the ranking data
reasonably well (see the supplementary material for diagnostic figures and
measures).
An important feature of the unfolding solution in Figure 4 is that not only
the distances between ideal and object points can be interpreted, but also the
distances within ideal and object points. This means that, in contrast to the
rank distribution in Figure 2, the unfolding solution also provides
information about which methods are ranked similarly and which combinations of
design and analysis options yield similar rankings. We make use of the latter
(i.e. the fact that the unfolding solution indicates which combinations yield
similar rankings) to assess the impact of each design and analysis choice on
the method rankings. For this purpose, the unfolding solution is supplemented
with additional information, which results in Figure 4(a)-4(d): For each
choice, the ideal points are coloured according to the option that was used in
the respective combination, with the default option (i.e. the option used in
Herrmann et al., 2020) coloured in grey. Moreover, we connect each ideal point
representing the default option to the ideal points representing the
alternative options given that the other three choices remain the same.
Although this makes the representation dependent on which option is used as
the default, for reasons of clarity, we refrain from additionally connecting
the alternative options with each other.
The resulting plot for the choice of performance measure is displayed in
Figure 4(a). The grey lines indicate that the distances between most ideal
points corresponding to pairs of ibrier and cindex within one specific setting
(i.e. combinations where the other three choices remain the same) are large.
Accordingly, the choice of performance measure strongly impacts the resulting
method ranking for most settings. Figure 4(a) also reveals that the ideal
points corresponding to ibrier and cindex form two clearly separated clusters.
Accordingly, the variability in the method rankings is reduced if the
performance measure is fixed. This applies in particular to the cindex, whose
corresponding ideal points show considerably less variation than the ideal
points corresponding to the ibrier. With regard to the remaining three choices
(data sets, imputation method, and aggregation method), this means that their
impact is smaller if the cindex is used as performance measure. This finding
might be explained by the fact that the cindex only measures discriminatory
power (see Section 2) and might thus be more robust to changes in the
remaining design and analysis choices than the ibrier.
As can be seen from Figure 4(b), another important choice that accounts for a
large part of the variability in the method rankings is the selection of data
sets, especially if the ibrier is used as performance measure (compare with
Figure 4(a)). Figure 4(b) also reveals that within the two clusters
corresponding to cindex and ibrier, the ideal points are roughly clustered
according to the group of data sets that was used in the respective
combination. This indicates that keeping the data sets fixed in addition to
the performance measure again reduces the variability in the method rankings.
Regarding the type of data sets used in each combination, Figure 4(b) shows
that within both clusters of performance measure, the ideal points
corresponding to small and large values of each data set characteristic lie
approximately opposite to each other while the ideal points representing all
18 data sets are located between them. With regard to the choice of data sets,
the largest discrepancy between two rankings can thus be expected when
comparing the results of two groups that correspond to small and large values
of one of the considered data set characteristics. Using all 18 data sets, on
the other hand, results in a compromise between the two extremes.
As already stated above, the variability in the method rankings is
considerably reduced if performance measure and data sets are fixed, which in
turn means that the variations caused by using different imputation or
aggregation methods are expected to be small. This finding is confirmed by
Figure 4(c) and Figure 4(d). The grey lines indicate that variations in the
method rankings caused by deviations from the default imputation or
aggregation method mainly arise for ibrier as the performance measure and all
groups of data sets except those with many clinical variables or large values
of $n$ or $n_{e}$ (compare with Figure 4(a) and Figure 4(b)). In some of the
other settings, the impact of the choice of imputation and aggregation method
is so small that the ideal points corresponding to different
imputation/aggregation methods have the same coordinates (i.e. yield the same
ranking). This applies in particular to the choice of imputation method, which
generally has less impact on the method rankings than the choice of
aggregation method, as can be seen from comparing Figure 4(c) and Figure 4(d).

(a)

(b)

(c)

(d)

Figure 4: Unfolding solution representing the rankings of 288 combinations of
design and analysis options (ideal points; circles) regarding 13 methods
(triangles). For each choice, the ideal points are coloured according to the
option that was used in the respective combination (default options
corresponding to Herrmann et al.,, 2020 are grey). Each ideal point
representing a default option is connected to the ideal points representing
alternative options, given that the other three choices remain the same.

The distances between ideal points of default and alternative options that are
represented as grey lines in Figure 4(a)-4(d) can also be summarised as
boxplots, which are displayed in Figure 5. This representation provides
information that is technically also included in Figure 4(a)-4(d), but is
presented more clearly in Figure 5. For example, it shows for each choice
which alternative option used instead of the default option tends to yield the
highest variations in the method rankings (e.g. for the choice of imputation
method, it is the option that uses the mean of the non-failed iterations as
imputation value). Moreover, Figure 5 reveals that according to the unfolding
solution, the largest discrepancy between two rankings generated by only
varying one design or analysis option is achieved by using the median instead
of the mean as aggregation method. This is an unexpected finding since it has
already been stated above and can also be seen from Figure 5 that in most
settings (i.e. combinations where the other three choices remain the same),
the choice of aggregation method tends to have a smaller impact on the method
rankings than the choice of performance measure and data sets. A major
drawback of Figure 5 is that in contrast to Figure 4(a)-4(d), it does not
provide any information about how similar the rankings generated by the
alternative options are, nor about how the ranks of the individual methods
change.

Figure 5: Distances between ideal points of combinations that represent
default and alternative options of one specific choice (given that the other
three choices remain the same), derived from the unfolding solution in Figure
4. The larger the distance, the larger the discrepancy between the two
rankings generated by using the alternative option instead of the default
option.

Of course, all findings concerning the impact of the individual design and
analysis choices depend on the number and type of options considered for each
choice. Specifically, for the choice of data sets, we only consider a small
subset of possible options and we focus, in addition to the 18 original data
sets, on groups of approximately equal size (8 to 10 data sets) generated by
specific data set characteristics. We thus complement our analysis by
illustrating the impact of the choice of data sets if more options are
considered, especially with regard to the number of data sets. For this
purpose, we keep performance measure, imputation method, and aggregation
method fixed to their respective default option and randomly draw 50
permutations of the 18 original data sets. For each of these permutations we
store the method rankings generated by only considering the first $l$ data
sets with $l=1,..,17$, and remove duplicate groups of data sets (which mainly
occur for groups with 1, 2, or 17 data sets). This results in 774 rankings
including one ranking generated by the 18 original data sets, which are all
represented in the unfolding solution in Figure 6. The widely distributed
ideal points clearly indicate that the choice of data sets is even more
essential if the number of data sets is not restricted and the groups of data
sets are not defined based on specific data set characteristics (as it was the
case above in Figure 4). As one might have expected, we also observe that the
variability in the method rankings increases if the number of data sets
decreases. Accordingly, the most extreme rankings (i.e. rankings that differ
the most from the ranking generated using all 18 data sets) occur for groups
with only a few data sets. Since Figure 4(a) revealed that the impact of the
choice of data sets strongly depends on the choice of performance measure, we
repeat the analysis using cindex as performance measure (see Figure S3 in the
supplementary material). Similar to Figure 4(b), the impact of the choice of
data sets is considerably reduced. However, as in Figure 6, the variability in
the method rankings increases with decreasing number of data sets.

Figure 6: Unfolding solution representing 774 rankings (circles) of 13 methods
(triangles) generated by randomly sampling different groups of data sets while
performance measure, imputation method, and aggregation method are fixed to
their respective default option.

##  5 Discussion

###  5.1 Summary

In this paper, we addressed the multiplicity of design and analysis options in
the context of benchmark studies and the associated risk of over-optimistic
results. As a preliminary step, we reviewed literature related to the choice
of four design and analysis choices that researchers are usually faced with
when conducting a benchmark study based on real data sets, namely the choice
of data sets, the choice of quantitative performance measure, the choice of
imputation method for missing performance values and the choice of aggregation
method to generate an overall method ranking.
We then used the benchmark study by Herrmann et al., (2020) to illustrate how
variable the resulting method rankings of a benchmark study can be when all
possible combinations of a range of design and analysis options are
considered. In fact, in this example, the results were so variable that any
method could achieve almost any rank, i.e. each method could almost be
presented as best or worst method for at least one combination of design and
analysis options. For the more realistic scenario where the design and
analysis options are not systematically examined for each combination but
selected in a step-wise optimisation process, we observed that the variability
in the method rankings is smaller but still remarkable.
In addition to examining the overall variability in the method rankings, we
also investigated the individual impact of each choice on the results using
multidimensional unfolding. As might be expected, the choice of performance
measure and data sets accounts for a large part of the variability in the
method rankings. The impact of the choice of imputation and aggregation
method, on the other hand, tends to be considerably smaller but still non-
negligible in many settings. In general, the impact of each choice depends on
the options used for the other three choices, with the choice of performance
measure affecting the impact of the remaining choices most strongly. In an
additional analysis, we increased the number of considered options for the
choice of data sets, which clearly showed that the variability in the method
rankings increases if the number of data sets decreases and once again
emphasised the importance of the choice of data sets.

###  5.2 Limitations

Of course, the specific results obtained for the example study by Herrmann et
al., (2020) should only be seen as an illustration that cannot be generalised
to other benchmark studies. Moreover, one possible reason why the method
rankings are so variable is that in our example benchmark study, many
performance differences are small and the performance values differ widely
across data sets, as discussed in the original study by Herrmann et al.,
(2020). The focus of our study was on ranks, which do not reflect the size of
the differences between the methods’ performances or the heterogeneity across
data sets. On the one hand, taking these aspects into account rather than
focusing on ranks may lead to much less variable results, particularly if one
relies on statistical tests. On the other hand, the multiplicity of possible
analysis options is not limited to the analysis of ranks: there are also
plenty of possibly ways to analyse performance differences and the
heterogeneity across data sets, even if statistical tests are performed (e.g.
paired t-test or Wilcoxon signed-rank test with or without correction for
multiple testing, or global tests such as the Friedman test).

###  5.3 Negative consequences and possible solutions

Despite these limitations, our illustration suggests that, as a consequence of
the multiplicity of design and analysis options, the results of benchmark
studies could be much more variable than many researchers realise. Combined
with questionable research practices (e.g. the selective reporting of results
or the targeted modification of specific design and analysis components), this
potentially high variability of benchmark results can lead to biased
interpretations and over-optimistic conclusions regarding the performance of
some of the considered methods. Given the high level of evidence that is
attributed to neutral benchmark studies (Boulesteix et al.,, 2017), a
“neutral” benchmark study that is in fact biased could thus negatively affect
both methodological and applied research by misleading method users and
developers (Weber et al.,, 2019).
Fortunately, there are several strategies to prevent over-optimistic benchmark
results that arise from the multiplicity of design and analysis options, some
of which are already applied by many researchers, including Herrmann et al.,
(2020). For example, strategies inspired from blinding in clinical trials can
help to reduce non-neutrality and/or the potential to exploit the multiplicity
of possible options. Specifically, blinding could be realised by labelling the
methods with non-informative names (e.g. Method A, Method B, etc.) such that
researchers have no information about the performance of each method until the
end of the study (Boulesteix et al.,, 2017). If the benchmark study is based
on simulated data, researchers could also be blinded to the data generation
process, which prohibits the possibility to tune the parameters of selected
methods according to the known ground truth (e.g. Kreutz et al.,, 2020).
The remaining strategies to prevent over-optimistic results can be summarised
using the work of Hoffmann et al., (2021), who formalise the effect of both
random sources of uncertainty (including sampling uncertainty) and epistemic
sources of uncertainty (resulting in a multiplicity of possible analysis
strategies and thus opening the door to questionable research practices) on
the replicability of research findings. They outline six steps researchers
from all empirical research fields can take to make their own research more
replicable and credible. In brief, researchers should (1) be aware of the
multiplicity of possible analysis strategies, (2) reduce uncertainty, (3)
integrate uncertainty, (4) report uncertainty, (5) acknowledge uncertainty,
and (6) publish all research code, data and material. Although Hoffmann et
al., (2021) focus on applied rather than methodological research, we argue
that their recommended steps can also be applied to address the sources of
uncertainty that arise from the design and analysis of benchmark studies.

##### Step 1

In the context of benchmark studies, the first step to reduce the risk of
over-optimistic results is to simply be aware of the multiplicity of possible
design and analysis options and the potential for questionable research
practices. We can only speculate about how much awareness for this issue is
already present in methodological research but hope that this paper
contributes to raising it.

##### Step 2

The second step suggested by Hoffmann et al., (2021) is to reduce sources of
uncertainty. In the context of benchmark studies, this could be realised by
consulting existing benchmarking guidelines found in literature. However, as
discussed in this paper, guidelines for many issues relevant in practice are
still lacking. We claim that more guidance and standardised approaches are
needed in this context. Regarding the choice of data sets, uncertainty could
be reduced if the number of data sets to include in the study would be
consequently based on statistical considerations such as power calculation
(e.g. Boulesteix et al.,, 2015) and if data sets would be selected according
to strict and well-considered inclusion criteria. Both aspects are facilitated
if structured and well-documented databases exist for the type of data to be
studied.

##### Step 3

As a third step, Hoffmann et al., (2021) recommend to integrate remaining
sources of uncertainty that could not be reduced in the second step. Analysis
approaches such as confidence intervals, statistical tests or boxplots that
take the heterogeneity of performance values across data sets into account can
be seen as first steps towards integrating the uncertainty regarding the
choice of data sets. However, they do not provide much information about how
the benchmark results would change if only certain subgroups of data sets
would be considered. A more advanced but less common way to integrate
uncertainty regarding the choice of data sets is to analyse the relationship
between method performance and data set characteristics (e.g. Oreski et al.,,
2017; Kreutz et al.,, 2020; Eugster et al.,, 2014). Concerning the choice of
evaluation criteria (including quantitative performance measures), the
aggregation of method rankings resulting from different criteria into an
overall ranking can be seen as an attempt towards integrating uncertainty.
However, to our knowledge, currently existing approaches such as consensus
rankings (Hornik and Meyer,, 2007) do not provide any measure of uncertainty.

##### Step 4

For all sources that cannot be adequately integrated, Hoffmann et al., (2021)
suggest to systematically report the results of alternative analysis
strategies, which, in the context of benchmark studies, would be alternative
design and analysis options. While reporting the results of alternative
analysis strategies, e.g. in the form of a sensitivity analysis, is a common
procedure in applied research (Hoffmann et al.,, 2021), to our knowledge it is
rarely performed in benchmark studies (especially if they are based on real
data sets). However, considering the lack of ways to reduce and integrate
uncertainty when designing and analysing benchmark studies, adequately
reporting the results of alternative options seems to be all the more
important. One reason for the lack of uncertainty reporting in benchmark
studies could be that, to our knowledge, no suitable framework has been
available so far. This gap could be filled by the framework based on
multidimensional unfolding that we used in this paper. It can be seen as a
systematic version of standard sensitivity analysis that allows to graphically
assess the variability of the method rankings with respect to a large number
of different combinations of design and analysis options. It also provides
information about the individual impact of each choice on the method ranking
and thus enables researchers to analyse when and how using alternative options
for a specific choice affects the results. In this way, the risk of misleading
readers is reduced and the benchmark results become even more reliable and
valuable. Moreover, using the framework allows to identify critical choices
that substantially affect the results and should therefore be particularly
well justified in future benchmark studies and be given more consideration in
benchmarking guidelines.

##### Step 5

The next important step suggested by Hoffmann et al., (2021) is to accept the
inherent uncertainty of scientific findings. In the context of benchmark
studies, this implies that researchers should clearly state that the benchmark
results are conditional on the selected design and analysis options
(Boulesteix et al.,, 2013; Hornik and Meyer,, 2007). In this vein, researchers
should also acknowledge that just as in applied research, generalisations from
a single study are usually not appropriate (Amrhein et al.,, 2019; Hoffmann et
al.,, 2021). This emphasises the need for more high-quality benchmark studies
and for meta-analyses of benchmark studies (e.g. Gardner et al.,, 2019),
which, however, are still rare and unfortunately sometimes not considered as
full-fledged research by the scientific community (Boulesteix et al.,, 2020).
Another aspect also related to the acceptance of uncertainty is to recognise
that statistical inference within exploratory analyses should be treated with
great caution (Amrhein et al.,, 2019; Hoffmann et al.,, 2021). Similar to
applied research, strictly confirmatory benchmark studies could be realised by
pre-registration of design- and analysis plans, as recently implemented in the
context of the so-called pre-registration experiment (see
https://preregister.science) or through the “registered report” publication
format (Chambers,, 2013), which has meanwhile been adopted by several
interdisciplinary journals that also accept computational papers.
It is also important to recall that there is usually no best method for all
scenarios and data sets (the well-known ”no free lunch” theorem; Wolpert,,
2002). Especially for data sets and evaluation criteria, it might thus be
advisable to accept the uncertainty that is associated with their choice by
putting more focus on the analysis of the individual strengths and weaknesses
of each method than on an aggregated overall ranking. This can for example be
realised by individually analysing the rankings generated by each evaluation
criterion and by investigating the relationship between method performance and
data set characteristics (see Step 3).

##### Step 6

As a final step, the publication of codes and (if possible) data sets that
ideally allow the extension to alternative options and additional methods can
reduce the impact of over-optimism since it enables readers to run alternative
analyses and to reveal potentially biased results.

##  6 Conclusion

In conclusion, our illustration suggests that benchmark results can be highly
variable with respect to design and analysis choices, which can lead to biased
interpretations and over-optimistic conclusions. However, there is a wide
range of strategies that can help to avoid these pitfalls. We hope that our
proposed framework makes a useful contribution towards this objective. While a
certain amount of over-optimism can probably never be completely avoided,
addressing this problem will lead to more reliable and valuable benchmark
results.

## Funding Information

This work was supported by the German Federal Ministry of Education and
Research (01IS18036A) and by the German Research Foundation (BO3139/4-3,
BO3139/7-1, BO3139/6-2) to ALB. The authors of this work take full
responsibilities for its content.

## Acknowledgements

The authors thank Anna Jacob for language correction.

## References

  * Amrhein et al., (2019) Amrhein, V., Trafimow, D., and Greenland, S. (2019).  Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication.  The American Statistician, 73:262–270.
  * Bischl et al., (2013) Bischl, B., Schiffner, J., and Weihs, C. (2013).  Benchmarking local classification methods.  Computational Statistics, 28:2599–2619.
  * Blanche et al., (2019) Blanche, P., Kattan, M. W., and Gerds, T. A. (2019).  The c-index is not proper for the evaluation of $t$-year predicted risks.  Biostatistics, 20:347–357.
  * Bokulich et al., (2020) Bokulich, N. A., Ziemski, M., Robeson, M. S., and Kaehler, B. D. (2020).  Measuring the microbiome: Best practices for developing and benchmarking microbiomics methods.  Computational and Structural Biotechnology Journal, 18:4048–4062.
  * Borg and Groenen, (2005) Borg, I. and Groenen, P. J. F. (2005).  Modern multidimensional scaling: Theory and applications.  Springer, New York, 2 edition.
  * Borg et al., (2013) Borg, I., Groenen, P. J. F., and Mair, P. (2013).  Applied multidimensional scaling.  Springer, Berlin Heidelberg.
  * Boulesteix, (2015) Boulesteix, A.-L. (2015).  Ten simple rules for reducing overoptimistic reporting in methodological computational research.  PLoS Computational Biology, 11:e1004191.
  * Boulesteix et al., (2018) Boulesteix, A.-L., Binder, H., Abrahamowicz, M., and Sauerbrei, W. (2018).  On the necessity and design of studies comparing statistical methods.  Biometrical Journal, 60:216–218.
  * Boulesteix et al., (2015) Boulesteix, A.-L., Hable, R., Lauer, S., and Eugster, M. J. A. (2015).  A statistical framework for hypothesis testing in real data comparison studies.  The American Statistician, 69:201–212.
  * Boulesteix et al., (2020) Boulesteix, A.-L., Hoffmann, S., Charlton, A., and Seibold, H. (2020).  A replication crisis in methodological research?  Significance, 17:18–21.
  * Boulesteix et al., (2013) Boulesteix, A.-L., Lauer, S., and Eugster, M. J. A. (2013).  A plea for neutral comparison studies in computational sciences.  PLoS ONE, 8:e61562.
  * Boulesteix et al., (2017) Boulesteix, A.-L., Wilson, R., and Hapfelmeier, A. (2017).  Towards evidence-based computational statistics: Lessons from clinical research on the role and design of real-data benchmark studies.  BMC Medical Research Methodology, 17:138.
  * Buchka et al., (2021) Buchka, S., Hapfelmeier, A., Gardner, P. P., Wilson, R., and Boulesteix, A.-L. (2021).  On the optimistic performance evaluation of newly introduced bioinformatic methods.  Genome Biology, 22:152.
  * Busing et al., (2005) Busing, F. M. T. A., Groenen, P. J. K., and Heiser, W. J. (2005).  Avoiding degeneracy in multidimensional unfolding by penalizing on the coefficient of variation.  Psychometrika, 70:71–98.
  * Chambers, (2013) Chambers, C. D. (2013).  Registered reports: A new publishing initiative at Cortex.  Cortex, 49:609–610.
  * Coombs, (1964) Coombs, C. H. (1964).  A theory of data.  Wiley, New York.
  * De Cnudde et al., (2020) De Cnudde, S., Martens, D., Evgeniou, T., and Provost, F. (2020).  A benchmarking study of classification techniques for behavioral data.  International Journal of Data Science and Analytics, 9:131–173.
  * de Leeuw and Mair, (2009) de Leeuw, J. and Mair, P. (2009).  Multidimensional scaling using majorization: SMACOF in R.  Journal of Statistical Software, 31:1–30.
  * Demšar, (2006) Demšar, J. (2006).  Statistical comparisons of classifiers over multiple data sets.  Journal of Machine Learning Research, 7:1–30.
  * Dietterich, (1998) Dietterich, T. G. (1998).  Approximate statistical tests for comparing supervised classification learning algorithms.  Neural Computation, 10:1895–1923.
  * Eisinga et al., (2017) Eisinga, R., Heskes, T., Pelzer, B., and Grotenhuis, M. (2017).  Exact p-values for pairwise comparison of Friedman rank sums, with application to comparing classifiers.  BMC Bioinformatics, 18:68.
  * Eugster et al., (2012) Eugster, M. J. A., Hothorn, T., and Leisch, F. (2012).  Domain-based benchmark experiments: Exploratory and inferential analysis.  Austrian Journal of Statistics, 41:5–26.
  * Eugster et al., (2014) Eugster, M. J. A., Leisch, F., and Strobl, C. (2014).  (Psycho-)analysis of benchmark experiments: A formal framework for investigating the relationship between data sets and learning algorithms.  Computational Statistics and Data Analysis, 71:986–1000.
  * Fernández-Delgado et al., (2014) Fernández-Delgado, M., Cernadas, E., Barro, S., and Amorim, D. (2014).  Do we need hundreds of classifiers to solve real world classification problems?  Journal of Machine Learning Research, 15:3133–3181.
  * Gardner et al., (2019) Gardner, P. P., Watson, R. J., Morgan, X. C., Draper, J. L., Finn, R. D., Morales, S. E., and Stott, M. B. (2019).  Identifying accurate metagenome and amplicon software via a meta-analysis of sequence to taxonomy benchmarking studies.  PeerJ, 7:e6160.
  * Gatto et al., (2016) Gatto, L., Hansen, K. D., Hoopmann, M. R., Hermjakob, H., Kohlbacher, O., and Beyer, A. (2016).  Testing and validation of computational methods for mass spectrometry.  Journal of Proteome Research, 15:809–814.
  * Graf et al., (1999) Graf, E., Schmoor, C., Sauerbrei, W., and Schumacher, M. (1999).  Assessment and comparison of prognostic classification schemes for survival data.  Statistics in Medicine, 18:2529–2545.
  * Head et al., (2015) Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., and Jennions, M. D. (2015).  The extent and consequences of p-hacking in science.  PLoS Biology, 13:e1002106.
  * Herrmann et al., (2020) Herrmann, M., Probst, P., Hornung, R., Jurinovic, V., and Boulesteix, A.-L. (2020).  Large-scale benchmark study of survival prediction methods using multi-omics data.  Briefings in Bioinformatics.  10.1093/bib/bbaa167.
  * Hoffmann et al., (2021) Hoffmann, S., Schönbrodt, F., Elsas, R., Wilson, R., Strasser, U., and Boulesteix, A.-L. (2021).  The multiplicity of analysis strategies jeopardizes replicability: Lessons learned across disciplines.  Royal Society Open Science, 8:201925.
  * Hornik and Meyer, (2007) Hornik, K. and Meyer, D. (2007).  Deriving consensus rankings from benchmarking experiments.  In Decker, R. and Lenz, H.-J., editors, Advances in data analysis, pages 163–170, Berlin, Heidelberg. Springer.
  * Hothorn et al., (2005) Hothorn, T., Leisch, F., Zeileis, A., and Hornik, K. (2005).  The design and analysis of benchmark experiments.  Journal of Computational and Graphical Statistics, 14:675–699.
  * Ioannidis, (2005) Ioannidis, J. P. A. (2005).  Why most published research findings are false.  PLoS Medicine, 2:e124.
  * Ioannidis et al., (2014) Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., and David, S. P. (2014).  Publication and other reporting biases in cognitive sciences: Detection, prevalence, and prevention.  Trends in Cognitive Sciences, 18:235–241.
  * Jelizarow et al., (2010) Jelizarow, M., Guillemot, V., Tenenhaus, A., Strimmer, K., and Boulesteix, A.-L. (2010).  Over-optimism in bioinformatics: An illustration.  Bioinformatics, 26:1990–1998.
  * John et al., (2012) John, L. K., Loewenstein, G., and Prelec, D. (2012).  Measuring the prevalence of questionable research practices with incentives for truth telling.  Psychological Science, 23:524–532.
  * Kibekbaev and Duman, (2016) Kibekbaev, A. and Duman, E. (2016).  Benchmarking regression algorithms for income prediction modeling.  Information Systems, 61:40–52.
  * Kreutz, (2019) Kreutz, C. (2019).  Guidelines for benchmarking of optimization approaches for fitting mathematical models.  Genome Biology, 20:281.
  * Kreutz et al., (2020) Kreutz, C., Can, N. S., Bruening, R. S., Meyberg, R., Mérai, Z., Fernandez-Pozo, N., and Rensing, S. A. (2020).  A blind and independent benchmark study for detecting differentially methylated regions in plants.  Bioinformatics, 36:3314–3321.
  * MacIà et al., (2013) MacIà, N., Bernadó-Mansilla, E., Orriols-Puig, A., and Kam Ho, T. (2013).  Learner excellence biased by data set selection: A case for data characterisation and artificial data sets.  Pattern Recognition, 46:1054–1066.
  * Mair et al., (2016) Mair, P., Borg, I., and Rusch, T. (2016).  Goodness-of-fit assessment in multidimensional scaling and unfolding.  Multivariate Behavioral Research, 51:772–789.
  * (42) Mair, P., Groenen, P. J. F., and de Leeuw, J. (Forthcoming).  More on multidimensional scaling and unfolding in R: smacof version 2\.  Journal of Statistical Software.
  * Mangul et al., (2019) Mangul, S., Martin, L. S., Hill, B. L., Lam, A. K. M., Distler, M. G., Zelikovsky, A., Eskin, E., and Flint, J. (2019).  Systematic benchmarking of omics computational tools.  Nature Communications, 10:1393.
  * Mersmann et al., (2015) Mersmann, O., Preuss, M., Trautmann, H., Bischl, B., and Weihs, C. (2015).  Analyzing the BBOB results by means of benchmarking concepts.  Evolutionary Computation, 23:161–185.
  * Morris et al., (2019) Morris, T. P., White, I. R., and Crowther, M. J. (2019).  Using simulation studies to evaluate statistical methods.  Statistics in Medicine, 38:2074–2102.
  * Norel et al., (2011) Norel, R., Rice, J. J., and Stolovitzky, G. (2011).  The self-assessment trap: Can we all be better than average?  Molecular Systems Biology, 7:537.
  * Novianti et al., (2015) Novianti, P. W., Jong, V. L., Roes, K. C., and Eijkemans, M. J. (2015).  Factors affecting the accuracy of a class prediction model in gene expression data.  BMC Bioinformatics, 16:199.
  * Nuzzo, (2015) Nuzzo, R. (2015).  How scientists fool themselves — and how they can stop.  Nature, 526:182–185.
  * Oreski et al., (2017) Oreski, D., Oreski, S., and Klicek, B. (2017).  Effects of dataset characteristics on the performance of feature selection techniques.  Applied Soft Computing, 52:109–119.
  * Orzechowski et al., (2018) Orzechowski, P., La Cava, W., and Moore, J. H. (2018).  Where are we now? A large benchmark study of recent symbolic regression methods.  In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’18, page 1183–1190, New York, NY, USA. Association for Computing Machinery.
  * Robinson and Vitek, (2019) Robinson, M. D. and Vitek, O. (2019).  Benchmarking comes of age.  Genome Biology, 20:205.
  * Simmons et al., (2011) Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011).  False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant.  Psychological Science, 22:1359–1366.
  * Uno et al., (2011) Uno, H., Cai, T., Pencina, M. J., D’Agostino, R. B., and Wei, L. J. (2011).  On the C-statistics for evaluating overall adequacy of risk prediction procedures with censored survival data.  Statistics in Medicine, 30:1105–1117.
  * Verenich et al., (2019) Verenich, I., Dumas, M., La Rosa, M., Maggi, F. M., and Teinemaa, I. (2019).  Survey and cross-benchmark comparison of remaining time prediction methods in business process monitoring.  ACM Transactions on Intelligent Systems and Technology, 10:34.
  * Wagenmakers et al., (2012) Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L. J., and Kievit, R. A. (2012).  An agenda for purely confirmatory research.  Perspectives on Psychological Science, 7:632–638.
  * Weber et al., (2019) Weber, L. M., Saelens, W., Cannoodt, R., Soneson, C., Hapfelmeier, A., Gardner, P. P., Boulesteix, A.-L., Saeys, Y., and Robinson, M. D. (2019).  Essential guidelines for computational method benchmarking.  Genome Biology, 20:125.
  * Wolpert, (2002) Wolpert, D. H. (2002).  The supervised learning no-free-lunch theorems.  In Roy, R., Köppen, M., Ovaska, S., Furuhashi, T., and Hoffmann, F., editors, Soft computing and industry: Recent applications, pages 25–42. Springer, London.
  * Wu et al., (2020) Wu, Z., Zhu, M., Kang, Y., Leung, E. L.-h., Lei, T., Shen, C., Jiang, D., Wang, Z., Cao, D., and Hou, T. (2020).  Do we need different machine learning algorithms for QSAR modeling? A comprehensive assessment of 16 machine learning algorithms on 14 QSAR data sets.  Briefings in Bioinformatics.  10.1093/bib/bbaa321.
  * Yousefi et al., (2010) Yousefi, M. R., Hua, J., Sima, C., and Dougherty, E. R. (2010).  Reporting bias when using real data sets to analyze classification performance.  Bioinformatics, 26:68–76.
  * Zimmermann, (2020) Zimmermann, A. (2020).  Method evaluation, parameterization, and result validation in unsupervised data mining: A critical survey.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10:e1330.

## Supplementary material

### Weighted imputation method for missing cindex performance values

For the cindex, where 1 corresponds to the best possible value and 0.5 to
random prediction, the imputed value for the considered combination of data
set and method that corresponds to the proposed “weighted imputation method”
is

$\displaystyle
x_{impute}=0.5+(\frac{\sum_{i\in\mathcal{I}}x_{i}}{|\mathcal{I}|}-0.5)_{+}\cdot(1-r),$
(4)

where $\mathcal{I}$ is the set of indices of the non-failed iterations,
$x_{i}$ is the cindex value for iteration $i\in\mathcal{I}$ and $r$ is the
proportion of missing values.

### Additional figures step-wise optimisation

Figure S1: Step-wise optimisation of method ranks by (1) imputation method
(blue), (2) aggregation method (pink), (3) performance measure (green), and
(4) data sets (yellow). The dotted line corresponds to the smallest and
highest possible ranks when all 288 combinations are considered. Missing steps
indicate that they did not lead to an improved rank. Default options
correspond to Herrmann et al., (2020) except performance measure, which is set
to cindex. Figure S2: Step-wise optimisation of method ranks by (1) data sets
(yellow), (2) performance measure (green), (3) aggregation method (pink), and
(4) imputation method (blue). The dotted line corresponds to the smallest and
highest possible ranks when all 288 combinations are considered. Missing steps
indicate that they did not lead to an improved rank. Default options
correspond to Herrmann et al., (2020).

### Additional figures unfolding

Figure S3: Unfolding solution representing 774 rankings (circles) of 13
methods (triangles) generated by randomly sampling different groups of data
sets while imputation method and aggregation method are fixed to their
respective default option and performance measure is set to cindex.

### Goodness-of-fit unfolding solutions

The assessment of the goodness-of-fit is based on Mair et al., (2016). We
assess the fit of three unfolding models presented in this paper:

  * •

Model 1: Unfolding solution representing the rankings of 288 combinations of
design and analysis options regarding 13 methods

  * •

Model 2: Unfolding solution representing 774 rankings of 13 methods generated
by randomly sampling different groups of data sets while performance measure,
imputation method, and aggregation method are fixed to their respective
default option

  * •

Model 3: Unfolding solution representing 774 rankings of 13 methods generated
by randomly sampling different groups of data sets while imputation method and
aggregation method are fixed to their respective default option and
performance measure is set to cindex

##### Permutation test

We test the null hypothesis that the unfolding solution is obtained from a
random permutation of dissimilarities. Rejecting the null hypothesis provides
some evidence that the unfolding solution captures a structural signal. For
all three unfolding models, the resulting p-value is $<0.001$.

##### Scree plots

We generate scree plots with varying number of dimensions (i.e.
$dim=1,\dots,12$, since $dim=13$ results in a stress value of $0$). Ideally,
we would see an elbow at $dim=2$ (the dimension chosen for the unfolding
models in this paper), which would indicate that additional dimensions
represent only random components of the data (Borg et al.,, 2013). Although no
clear elbow is visible in Figure S4-S6, the scree plots indicate that the
stress is already considerably low for $dim=2$. Note that in Figure S6, the
iteration limit was reached when running the unfolding models for $dim\geq 6$
and the stress is close to 0, which might indicate degenerate solutions.

Figure S4: Scree plot for unfolding model 1. The stress value for $dim=2$,
which was used in our application, is coloured in red. Figure S5: Scree plot
for unfolding model 2. The stress value for $dim=2$, which was used in our
application, is coloured in red. Figure S6: Scree plot for unfolding model 3.
The stress value for $dim=2$, which was used in our application, is coloured
in red.

##### Stress-per-point

To check for influential points that should be subject to special
consideration, we can look at the stress-per-point values (SPP). The SPP
values are assessed separately for subjects (here: combinations of design and
analysis options) and objects (here: methods). As can be seen from Figure
S7-S9, there are no extreme outliers for any of the three unfolding models
presented in this paper. On the method side, all stress proportions are
smaller than 14%, and on the combination side, most stress proportions are
smaller than 1%.

Figure S7: Stress-per-point for methods (left) and combinations of design and
analysis options (right) for unfolding model 1. The greater the stress
proportion, the more the point contributes to the misfit of the unfolding
solution. The dotted line represents the stress proportion if every
method/combination contributed equally to the misfit. Figure S8: Stress-per-
point for methods (left) and combinations of design and analysis options
(right) for unfolding model 2. The greater the stress proportion, the more the
point contributes to the misfit of the unfolding solution. The dotted line
represents the stress proportion if every method/combination contributed
equally to the misfit. Figure S9: Stress-per-point for methods (left) and
combinations of design and analysis options (right) for unfolding model 3. The
greater the stress proportion, the more the point contributes to the misfit of
the unfolding solution. The dotted line represents the stress proportion if
every method/combination contributed equally to the misfit.