# Systematic Overestimation of Machine Learning Performance in Neuroimaging
Studies of Depression

Claas Flint indicates that the authors contributed equally to the work and
should be regarded as first authors.  Faculty of Mathematics and Computer
Science, University of Münster, Germany  Micah Cearns  Department of
Psychiatry, Melbourne Medical School, The University of Melbourne, Parkville,
Australia  Nils Opel  Department of Psychiatry, University of Münster, Germany
Ronny Redlich  Department of Psychiatry, University of Münster, Germany  David
M. A. Mehler  Department of Psychiatry, University of Münster, Germany  Daniel
Emden  Department of Psychiatry, University of Münster, Germany  Nils R.
Winter  Department of Psychiatry, University of Münster, Germany  Ramona
Leenings  Department of Psychiatry, University of Münster, Germany  Simon B.
Eickhoff  Institute of Neuroscience and Medicine (INM-7) Research Center
Jülich  Institute of Systems Neuroscience, Medical Faculty, Heinrich Heine
University Düsseldorf, Düsseldorf, Germany  Tilo Kircher  Department of
Psychiatry and Psychotherapy, University of Marburg, Germany  Axel Krug
Department of Psychiatry and Psychotherapy, University of Marburg, Germany
Igor Nenadic  Department of Psychiatry and Psychotherapy, University of
Marburg, Germany  Volker Arolt  Department of Psychiatry, University of
Münster, Germany  Scott Clark  Discipline of Psychiatry, School of Medicine,
University of Adelaide, Australia  Bernhard T. Baune  Discipline of
Psychiatry, School of Medicine, University of Adelaide, Australia  Department
of Psychiatry, Melbourne Medical School, The University of Melbourne,
Parkville, Australia  The Florey Institute of Neuroscience and Mental Health,
The University of Melbourne, Parkville, Australia  Xiaoyi Jiang  Faculty of
Mathematics and Computer Science, University of Münster, Germany  Udo
Dannlowski indicates that the authors contributed equally to the work and
should be regarded as senior authors.  Tim Hahn

###### Abstract

We currently observe a disconcerting phenomenon in machine learning studies in
psychiatry: While we would expect larger samples to yield better results due
to the availability of more data, larger machine learning studies consistently
show much weaker performance than the numerous small-scale studies. Here, we
systematically investigated this effect focusing on one of the most heavily
studied question in the field, namely the classification of patients suffering
from Major Depressive Disorder (MDD) and healthy controls. Drawing upon a
balanced sample of $N=$1868$$ MDD patients and healthy controls from our
recent international Predictive Analytics Competition (PAC), we first trained
and tested a classification model on the full dataset which yielded an
accuracy of $61\text{\,}\mathrm{\char 37\relax}$. Next, we mimicked the
process by which researchers would draw samples of various sizes ($N=4$ to
$N=150$) from the population and showed a strong risk of overestimation.
Specifically, for small sample sizes ($N=20$), we observe accuracies of up to
$95\text{\,}\mathrm{\char 37\relax}$. For medium sample sizes ($N=100$)
accuracies up to $75\text{\,}\mathrm{\char 37\relax}$ were found. Importantly,
further investigation showed that sufficiently large test sets effectively
protect against performance overestimation whereas larger datasets per se do
not. While these results questions the validity of a substantial part of the
current literature, we outline the relatively low-cost remedy of larger test
sets.

_K_ eywords machine learning $\cdot$ neuroimaging $\cdot$ major depression
$\cdot$ overestimation $\cdot$ small sample size $\cdot$ clinical translation

##  1 Introduction

In psychiatry, we are witnessing an explosion of interest in machine learning
(ML) and artificial intelligence for prediction and biomarker discovery,
paralleling similar developments in personalized medicine[1, 2, 3, 4]. In
contrast to the majority of investigations employing classic group-level
statistical inference, ML approaches aim to build models which allow for
individual (i.e. single subject) predictions, thus enabling direct assessment
of individual differences and clinical utility[5]. While this constitutes a
major advancement for clinical translation, recent results of large-scale
investigations have given rise to a fundamental concern in the field:
Specifically, machine learning studies including larger samples did not yield
stronger performance, but consistently showed weaker results than studies
drawing on small samples, calling into question the validity and
generalisability of a large number of highly published proof-of-concept
studies.

The magnitude of this issue was impressively illustrated by the results of the
Predictive Analytics Competition (PAC 2018; http://photon-ai.com/pac) in which
participants developed machine learning models to classify healthy controls
(HC) and depressive patients (MDD) based on structural MRI data from
$N=$2240$$ participants. Despite the best efforts of $170$ machine learners in
$49$ teams from around the world, performance ranged between
$60\text{\,}\mathrm{\char 37\relax}$ and $65\text{\,}\mathrm{\char 37\relax}$
percent accuracy in a large, independent test set. This is in strong contrast
to the numerous smaller studies showing accuracies of
$80\text{\,}\mathrm{\char 37\relax}$ or more[6, 7, 8]. Further empirical
studies focusing on other disorders support this observed effect of
performance deterioration with increasing sample size: In a large-scale
investigation, Neuhaus & Popescu[9] aggregated original studies across
disorder categories, including schizophrenia (total observation $N=$5563$$),
MDD ($N=$2042$$), and attention deficit hyperactivity disorder (ADHD,
$N=$8084$$), finding an inverse relationship between sample size and balanced
accuracy (schizophrenia, $r=$-0.34$$; MDD, $r=$-0.32$$; and ADHD,
$r=$-0.43$$). Similar results were observed in a recent review of $200$
neuroimaging classification studies of brain disorders, finding a general
trend towards lower reported accuracy scores in larger samples[10]. Given that
model performance would be expected to increase with more data, these results
hint at a fundamental issue hampering current predictive biomarker studies in
psychiatry.

From a methodological point of view, it has been known since the early 90’s
that training samples should be large when there is a high number of features
(i.e. measured variables) and a complex classification rule being fit to a
dataset[11]. Recent works have further reiterated this point[12]. Moreover, it
is possible that these effects have been further amplified by certain cross-
validation schemes. For example, Kambeitz et al.[13] observed higher accuracy
estimates in studies using hold-out cross-validation strategies compared to
10-fold and leave-one-out (LOOCV), whilst Varoquaux et al observed that LOOCV
leads to unstable and biased estimates, concluding that repeated random splits
should be preferred[14].

Although these findings have sparked in-depth conceptual considerations[15],
empirical investigations of this problem have been limited to specific cross-
validation schemes and small test set sizes[16]. Here, we aim to
systematically investigate the effects of both train and test set sample sizes
on machine learning model performance in neuroimaging based MDD
classification. In addition, as it is possible that effects of systematic
overestimation have arisen due to suboptimal pipeline configurations (i.e.,
the disproportionate use of LOOCV and linear Support Vector Machines on
samples containing more predictors than observations), we also test a further
$48$ different pipeline configurations to quantify the influence of these
additional factors. To demonstrate that this effect was not dependent on the
data or any pipeline configurations used in our analyses, we repeat the
process using a dummy classifier that ignores all input variables and attempts
to only predict class (outcome) proportions. To quantify the magnitude of
these effects in each configuration, we drew samples of various sizes from the
PAC dataset – mimicking the process by which researchers would draw samples
from the population of ML studies reported in the literature. The resulting
probability distributions are investigated.

##  2 Materials and methods

To investigate the effects of both train and test set sample sizes on machine
learning model performance in neuroimaging based MDD classification, we
repeatedly drew samples of different sizes from the PAC dataset to imitate the
procedure reported in the literature. Subsequently, the resulting probability
distributions are investigated.

###  2.1 Data description

The PAC dataset comprised anonymized, pre-processed MRI data of $N=$2240$$
individuals obtained from two large, independent, ongoing studies - the
Münster Neuroimaging Cohort[17, 18] ($N=$724$$ MDD; $N=$285$$ HC) and the
FOR2107-study[19] ($N=$582$$ MDD, $N=$649$$ HC). Case/control status was
diagnosed with the SCID-IV[20] interview employed by trained clinical raters
in both studies. In both cohorts, exclusion criteria were any neurological or
severe medical condition, MRI contraindications, substance related disorders,
Benzodiazepine treatment and, head injuries. For healthy controls, any current
or previous psychiatric disorder or use of any psychotropic substances. The
Münster Neuroimaging Cohort was scanned at one single MRI-scanner with the
same sequence, while the FOR2107-study was scanned at two independent
sites[21], yielding $3$ different scanner types/sequences. The structural
T1-weighted magnetic resonance imaging (MRI) scans were pre-processed with the
CAT12 toolbox (http://www.neuro.uni-jena.de/cat, r1184) using default
parameters to obtain modulated, normalized grey matter segments (resolution
$$1.5$\times$1.5$\times$1.5\text{\,}{\mathrm{mm}}^{3}$$) which were used for
the present analysis. Furthermore, age, gender, scanner type, and total
intracranial volume (TIV) were provided.

###  2.2 Machine learning pipeline

To ensure the unbiased approximation of the model’s performance in previously
unseen patients (i.e. model generalization), we trained and tested all models
in a pipeline to prevent information leaking between patients used for
training and validation. First, we used random under sampling to derive a
balanced sample of $934$ MDD cases and $934$ healthy controls (HC) (see
Supplementary appendix A for summary statistics). To reduce the computational
effort, all images have been scaled down to a voxel size of
$$3$\times$3$\times$3\text{\,}{\mathrm{mm}}^{3}$$. Following, every image was
converted to a vector, where every voxel served as a feature. For
standardization, the features were scaled to have zero mean and unit variance.
Finally, a linear support vector machine (SVM) with default parameters
(sklearn[22], v0.20) was trained and model performance was calculated and
analysed in the subsequent analyses. The effect of varying scanner-
distributions across samples was found to be negligible (see Supplementary
appendix C).

###  2.3 Overall sample size effects

To first examine the effects of overall-set size, we randomly sampled the PAC
dataset in steps of $1$ from $N=4$ to $N=150$. For each $N$ value we created
$1000$ balanced samples. This yielded $147\,000$ random samples with equal
numbers of patients and healthy controls. Applying the SVM pipeline described
above to each of these samples independently allowed us to obtain a
distribution of accuracy scores for each $N$ value, respectively. On each
sample one SVM was trained with default parameters using LOOCV. Thus, we
trained a total of $11\,315\,000$ SVMs. Since the computational effort
increases quadratically with increasing sample size, and in addition, the most
recent neuroimaging ML studies using LOOCV rarely exceed $N=150$, we decided
to stop at this value. Following, we evaluated the distribution of accuracy
scores estimated for each sample size ($N=4$ to $150$).

###  2.4 Training set size effects

Second, to examine the effects of training set size, we varied the size of
each training set and then tested performance on a fixed hold-out set.
Specifically, we randomly sampled the PAC dataset in steps of $1$ from $N=4$
to $150$ as was done in the previous analysis. For each $N$ value, we created
$1000$ balanced samples and used each of them to train different models. To
then test the effects of training set size on test set performance, we tested
each trained model on a balanced test set of $N=300$. From the resulting
distributions, we quantified the probability of overestimating accuracy as a
function of training set size.

###  2.5 Test set size effects

To assess the effects of test set size on classification accuracy, we randomly
sampled a balanced group of $300$ subjects from our full sample of $934$ MDD
cases and $934$ HC’s. From this sample, we randomly sub-sampled test sets of
$N=4$ to $150$ in steps of $1$. For each value of $N$, we took $1000$ samples,
resulting in the creation of $147\,000$ random test samples. From the
remaining $784$ MDD cases and $784$ HC’s, we took out another
$20\text{\,}\mathrm{\char 37\relax}$ sample ($\text{MDD}=157$ and
$\text{HC}=157$). The prediction of this sample provides a reliable basis for
an overall performance estimation and allows for a comparison with the results
obtained from smaller samples. We then trained a single SVM on the remaining
sample ($\text{MDD}=627$, $\text{HC}=627$). The trained SVM was then tested on
each test set sample ($N=4$ to $150$). From the resulting test set accuracy
distributions, we derived the probability of obtaining accuracy scores between
$50\text{\,}\mathrm{\char 37\relax}$ and $90\text{\,}\mathrm{\char 37\relax}$
accuracy by chance. See Figure 1 for an overview of all analyses.

Figure 1: Workflow for overall, train, and test set size systematic
overestimation analyses.

###  2.6 Generalizability of statistical effect

####  2.6.1 Alternative pipeline configurations

As we have attempted to hold other components of the modelling process
constant so any observed effects of systematic overestimation can be
attributed to sample size alone, it is also possible that our results may be
dependent on the basic configuration of our machine learning pipeline (e.g.,
the use of a linear SVM, default hyperparameters, and LOOCV). Therefore, we
trained a further $48$ pipeline configurations, including the use of both
linear and radial basis function SVMs and a Random Forest classifier, all of
which have demonstrated their efficacy in neuroimaging classification
studies.[13] Within these configurations, we conducted dimensionality
reduction using principal components analysis as well as f-test based feature
selection. This allowed us to assess whether our findings were being
confounded by the large number of predictors used in the main analysis. For
further information see Supplement Appendix E and for all results see
supplementary results E.1, supplementary figures E.1 – E.48 and supplementary
tables E.2 – E.49. Dummy classifier In order to show that the observed effects
were not specific to the PAC dataset or any of the alternative pipeline
configurations in these analyses, we repeated the procedures described above
with a dummy classifier. Our dummy classifier assumed a prior probability for
MDD vs control classification based on the percentage proportion of each class
in the training data (prevalence). As the dataset was balanced with random
under sampling, the prior and subsequent ground truth of the model was equal
to exactly $50\text{\,}\mathrm{\char 37\relax}$. This approach allowed us to
compare our distribution of dummy performance estimates derived from our
subsample analysis to this ground truth value. Importantly, this approach
allows for the quantification of accuracy overestimation as a function of
sample size completely independent of any unique characteristics that may be
specific to the PAC dataset or our pipeline configurations. Therefore, we can
then be sure that any subsequent changes in classifier performance are
attributable to sample size alone.

##  3 Results

###  3.1 Overall sample size effects

In the overall sample size analysis using LOOCV, we were able to show that the
risk of overestimating the classifier performance increases with decreasing
overall sample size (Figure 2a, Supplements Table B.1a). Specifically,
accuracies of $70\text{\,}\mathrm{\char 37\relax}$ or higher are observed with
a probability of $13\text{\,}\mathrm{\char 37\relax}$ on sample sizes of
$N=20$ whereas this probability is reduced to $2\text{\,}\mathrm{\char
37\relax}$ for sample sizes of $N=100$. In addition, sample size has a
profound impact on the variability of accuracy estimates: For samples of size
$N=20$, accuracies ranged from $10\text{\,}\mathrm{\char 37\relax}$ to
$95\text{\,}\mathrm{\char 37\relax}$ ($\text{standard
deviation}=$15\text{\,}\mathrm{\char 37\relax}$$) while for samples of
$N=100$, accuracies ranged between $35\text{\,}\mathrm{\char 37\relax}$ and
$81\text{\,}\mathrm{\char 37\relax}$ ($\text{standard
deviation}=$6\text{\,}\mathrm{\char 37\relax}$$) (Figure 2b, Supplements Table
B.1b). Note that this effect is symmetrical and also applies to the
underestimation of performance (Figure 2b). Additionally, the results of the
dummy classifier (Figure 2c, d, Supplements Table B.2) show that the observed
overestimation effect is a general effect of sample size as previously pointed
out by Varoquaux[16]. As the regularization of the SVM is sensitive to the
total number of outliers, which may increase in parallel with sample size, we
conducted an additional analysis with adjusted $C$ parameters, with the
observed effect remaining constant across these analyses (see Supplements
Figure D.1 and D.2).

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVMs to yield an accuracy exceeding a certain threshold as a
function of sample size employing LOOCV.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVMs as a function of sample
size employing LOOCV.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (c) Probabilities
for dummy classifiers’s to get an accuracy above a certain chance level
related to the size of the used sample.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (d)
Minimum, maximum and mean results for the dummy classifiers related to the
size of the used sample size for training and testing.

Figure 2: Effects of varying overall sample sizes employing LOOCV.

###  3.2 Training set size effects

When examining the effects of training set size ($N=4$ to $150$) using a large
test set for evaluation ($N=300$), we did not observe any systematic
overestimation (Figure 3a, Supplements Table B.3a). In fact, models trained on
virtually any training set size from $N=4$ to $150$ were sufficient to obtain
maximum model performance. However, increasing training sample size decreased
the probability of obtaining very low performance estimate. For training sets
of size $N=20$, accuracies ranged from $32\text{\,}\mathrm{\char 37\relax}$ to
$69\text{\,}\mathrm{\char 37\relax}$ ($\text{standard
deviation}=$7.1\text{\,}\mathrm{\char 37\relax}$$) while for training sets of
$N=100$, accuracies ranged between $51\text{\,}\mathrm{\char 37\relax}$ and
$70\text{\,}\mathrm{\char 37\relax}$ ($\text{standard
deviation}=$3.0\text{\,}\mathrm{\char 37\relax}$$) (Figure 3b, Supplements
Table B.3b). In accordance with the overall sample size analysis, the results
of the dummy classifier (Figure 3c, d, Supplements Table B.4) showed that this
observed effect was general in nature.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVMs to yield an accuracy exceeding a certain threshold as a
function of training sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVMs as a function of
training sample size.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (c) Probabilities
for the dummy classifier to get an accuracy above a certain chance level
related to the size of the training set size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (d)
Minimum, maximum and mean results for the dummy classifier related to the size
of the used sample size for training.

Figure 3: Results as a function of training set sizes with a fixed test set
size of $N=300$.

###  3.3 Test set size effects

For our analysis varying test set size, we found a similar pattern of
systematic overestimation as that in our first overall sample size analysis
using LOOCV. With a sample size of $N=20$, we obtain results of
$70\text{\,}\mathrm{\char 37\relax}$ accuracy or higher with a probability of
$30\text{\,}\mathrm{\char 37\relax}$, whilst the mean accuracy on the full
dataset ($N=268$) was only $61\text{\,}\mathrm{\char 37\relax}$. This
probability dropped to $13\text{\,}\mathrm{\char 37\relax}$ when the test
sample size was $N=100$. For test sets of size $N=20$, accuracies ranged from
$35\text{\,}\mathrm{\char 37\relax}$ to $95\text{\,}\mathrm{\char 37\relax}$
($\text{standard deviation}=$10\text{\,}\mathrm{\char 37\relax}$$) whilst for
test sets of size $N=100$, accuracies ranged from $51\text{\,}\mathrm{\char
37\relax}$ and $79\text{\,}\mathrm{\char 37\relax}$ ($\text{standard
deviation}=$4\text{\,}\mathrm{\char 37\relax}$$) (Figure 4b, Supplements Table
B.6b). Running the analysis again using our dummy classifier, we were able to
show that the general pattern of systematic overestimation was independent of
the specific dataset used (Figure 4c, d, Supplements Table B.6). To show the
independent and generalizable character of the observed effect, we repeated
the analysis on the $48$ unique pipeline configurations discussed above (see
Supplementary appendix E). Specifically, the results are comparable to the
original used configuration, i.e. an SVM with a linear kernel and no
preprocessing. Finally, our analysis of scanner sites revealed no effects on
model performance (see Supplementary appendix D).

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVMs to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVMs as a function of test
sample size.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (c) Probabilities
for the dummy classifier to get an accuracy above a certain chance level
related to the size of the test set size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (d)
Minimum, maximum and mean results for the dummy classifier related to the size
of the used sample size for testing.

Figure 4: Results as a function of variable test set sizes with and a fixed
classifier.

##  4 Discussion

Sparked by the observation that machine learning studies drawing on larger
neuroimaging samples consistently showed weaker results than studies drawing
on smaller ones, we drew samples of various sizes from the PAC dataset,
thereby mimicking the process by which researchers would draw samples from the
population of ML studies reported in the literature. When applying a linear
SVM with LOOCV, as is the most common approach in the neuroimaging
literature[10], we observed higher risk for artificially high performance
estimates in smaller samples. Importantly, our analyses then revealed that
this is primarily due to small test set size, not training set size.
Generally, this shows that small test sample size alone may explain many of
the highly optimistic results published in recent years. When considering the
well-established effect of publication bias, this finding appears even more
likely.

Our results are the first to disentangle the effects of training and test set
size effects which are typically inseparable in common cross-validation
frameworks such as LOOCV. This delineation of effects enabled two important
insights for biomarker discovery and outcome prediction. First, researchers
need to validate their models on large, independent test sets. Our results
indicate that in the PAC dataset, a test set size of $N=100$ was already
sufficient to lower the probability of obtaining artificially good performance
(i.e. $70\text{\,}\mathrm{\char 37\relax}$ or higher) to
$13\text{\,}\mathrm{\char 37\relax}$. With a median $N$ of less than $100$ in
many published studies[10], this may seem challenging. However, online
infrastructure for independent machine learning model evaluation is available
(e.g. www.photon-ai.com/repo). If researchers open-source their models, anyone
- independent of technical knowledge or machine learning expertise - can
evaluate them. This way, large independent test datasets can be obtained in a
short time without the need for data sharing. This is not restricted to
neuroimaging data, but any machine learning model. In addition, efforts from
consortia will also help to mitigate this problem and should be considered by
machine learning practitioners.

Second, the size of the training set alone cannot serve as an indicator of
later model performance. Larger training sets are more likely to generalize to
new data and broaden model scope (i.e. about which groups within a population
a given model can make reasonable predictions), however, in the current
analysis, the linear rule learned by an SVM on high-dimensional neuroimaging
data could be approximated with only a handful of samples. From a training set
size of $30$ onward, we no longer observed any increase in model performance.
This somewhat counterintuitive effect arises whenever a simple rule is
approximated. For higher complexity models (i.e. models capable of learning
more complex rules), we of course expect performance increases as training
sample size increases. However, considering the results of the PAC competition
(https://photon-ai.com/pac), high complexity models such as Deep Learning
approaches did not yield higher performance when trained with $1000$ samples.
Thus, we conclude that simple models are competitive for sample sizes of up to
$N=$1000$$ for this particular classification problem. Whether more complex
rules can be discovered beyond this point or whether other fundamental issues
hamper biomarker discovery in psychiatry (cf. e.g. biotype identification[19]
and normative modelling approaches[23]) remains an open question.

An intuitive criticism of our main analyses would be that it has merely
replicated methods similar to those of previous low-quality works (for
example, studies using only linear SVMs with default parameters, tested in
LOOCV schemes, on samples with many more predictors than observations). Whilst
this pipeline configuration was used in the current analysis to a) hold
constant properties, that if varied, may have been indistinguishably
responsible for changes in accuracy overestimation, and b) replicate the most
commonly used ML pipeline configuration in our field, it was important to
conduct complementary analyses to rule out these confounds. Therefore, we
tested a further 48 ML pipeline configurations (see Supplementary appendix F)
using both linear (a linear SVM) and non-linear (a radial basis function SVM
and a Random Forest) classifiers. In addition, we conducted PCA based
dimensionality reduction as well as f-test based feature selection within
these classifiers to delineate whether our findings were confounded by the
large size of the predictor space relative to our number of observations.
Importantly, all pipeline configurations demonstrated the same pattern of
systematic overestimation as that in our main analyses. The second potential
criticism of the current work is that these findings may merely be modality
specific, limiting the generalizability of these findings across domains.
However, the use of a dummy classifier that completely ignored the input
predictor space (the voxels), and instead, classified samples based only on
their prevalence in training ($\text{MDD}=$50\text{\,}\mathrm{\char
37\relax}$$, $\text{Control}=$50\text{\,}\mathrm{\char 37\relax}$$), showed
the same pattern of sample size based systematic overestimation across all
pipeline configurations, thus, demonstrating a generalizable statistical
effect regardless of the data modality used.

Given the profound effect of test set size on systematic overestimation, it is
important to consider why this effect may arise. Previous work by Schnack &
Kahn[24] suggest that patient characteristics in smaller samples tend to be
more homogenous. In the case of small $N$, participants may be more likely to
be recruited from the same data collection site and of a similar age (for
example, in the case of a university recruited convenience sample). In
addition, stringent recruitment criterion may be easily met, resulting in a
well-defined phenotype that is not truly representative of the parent
population of interest. Whilst this explanation makes sense for samples
collected in this manner, it fails to explain why we observed this phenomenon
in our random sampling procedure, and more importantly, with our dummy
classifiers that paid no attention to participants, their characteristics, or
the inputted predictor variables. This observation suggests a mechanism for
systematic overestimation that is not just sample/patient specific or
contingent on sample homogeneity, but instead, inherent in the behaviour of
classification models when tested on small samples.

In addition to sample size, other issues such as data leakage[13], are likely
contributing to the systematic overestimation seen in the literature.
Dedicated cross-platform software to help avoid data leakage is freely
available (e.g. PHOTON, www.photon-ai.com or Sci-kit learn[22]). Finally, code
should be made available on request or provided in an online repository (e.g.
GitHub or GitLab) upon submission for review. In addition, a more elaborate
evaluation framework including the analysis of model scope assessment as well
as incremental utility and risk analysis is needed to move the field beyond
proof-of-concept studies. The success of current translational efforts in
neuroimaging and psychiatry will crucially depend on the timely adoption of
guidelines and rules for medical machine learning models (for an in-depth
introduction, see [15]).

In summary, our results indicate that - while many of the most highly
published results might strongly overestimate true performance - evaluation on
large test sets constitutes a straightforward remedy. Given that simple, low-
complexity models such as linear SVMs did not gain from larger training set
size, researchers should not discard their models due to low training $N$ but
seek evaluation on a large test set for any model showing good performance.

## Acknowledgements

### FOR2107

This work was funded by the German Research Foundation (DFG, grant FOR2107
DA1151/5-1 and DA1151/5-2 to UD; SFB-TRR58, Projects C09 and Z02 to UD) and
the Interdisciplinary Center for Clinical Research (IZKF) of the medical
faculty of Münster (grant Dan3/012/17 to UD). TH was supported by the German
Research Foundation (DFG grants HA7070/2-2, HA7070/3, HA7070/4).

This work is part of the German multicenter consortium ”Neurobiology of
Affective Disorders. A translational perspective on brain structure and
function”, funded by the German Research Foundation (Deutsche
Forschungsgemeinschaft DFG; Forschungsgruppe/Research Unit FOR2107).

##### Principal investigators (PIs) with respective areas of responsibility in
the FOR2107 consortium are:

Work Package WP1, FOR2107/MACS cohort and brain imaging: Tilo Kircher (speaker
FOR2107; DFG grant numbers KI 588/14-1, KI 588/14-2), Udo Dannlowski (co-
speaker FOR2107; DA 1151/5-1, DA 1151/5-2), Axel Krug (KR 3822/5-1, KR
3822/7-2), Igor Nenadic (NE 2254/1-2), Carsten Konrad (KO 4291/3-1). WP2,
animal phenotyping: Markus Wöhr (WO 1732/4-1, WO 1732/4-2), Rainer Schwarting
(SCHW 559/14-1, SCHW 559/14-2). WP3, miRNA: Gerhard Schratt (SCHR 1136/3-1,
1136/3-2). WP4, immunology, mitochondriae: Judith Alferink (AL 1145/5-2),
Carsten Culmsee (CU 43/9-1, CU 43/9-2), Holger Garn (GA 545/5-1, GA 545/7-2).
WP5, genetics: Marcella Rietschel (RI 908/11-1, RI 908/11-2), Markus Nöthen
(NO 246/10-1, NO 246/10-2), Stephanie Witt (WI 3439/3-1, WI 3439/3-2). WP6,
multi method data analytics: Andreas Jansen (JA 1890/7-1, JA 1890/7-2), Tim
Hahn (HA 7070/2-2), Bertram Müller-Myhsok (MU1315/8-2), Astrid Dempfle (DE
1614/3-1, DE 1614/3-2). CP1, biobank: Petra Pfefferle (PF 784/1-1, PF
784/1-2), Harald Renz (RE 737/20-1, 737/20-2). CP2, administration. Tilo
Kircher (KI 588/15-1, KI 588/17-1), Udo Dannlowski (DA 1151/6-1), Carsten
Konrad (KO 4291/4-1).

##### Data access and responsibility:

All PIs take responsibility for the integrity of the respective study data and
their components. All authors and coauthors had full access to all study data.

##### Acknowledgements and members by Work Package (WP):

###### WP1:

Henrike Bröhl, Katharina Brosch, Bruno Dietsche, Rozbeh Elahi, Jennifer
Engelen, Sabine Fischer, Jessica Heinen, Svenja Klingel, Felicitas Meier, Tina
Meller, Torsten Sauder, Simon Schmitt, Frederike Stein, Annette Tittmar,
Dilara Yüksel (Dept. of Psychiatry, Marburg University). Mechthild Wallnig,
Rita Werner (Core-Facility Brainimaging, Marburg University). Carmen Schade-
Brittinger, Maik Hahmann (Coordinating Centre for Clinical Trials, Marburg).
Michael Putzke (Psychiatric Hospital, Friedberg). Rolf Speier, Lutz Lenhard
(Psychiatric Hospital, Haina). Birgit Köhnlein (Psychiatric Practice,
Marburg). Peter Wulf, Jürgen Kleebach, Achim Becker (Psychiatric Hospital
Hephata, Schwalmstadt-Treysa). Ruth Bär (Care facility Bischoff, Neunkirchen).
Matthias Müller, Michael Franz, Siegfried Scharmann, Anja Haag, Kristina
Spenner, Ulrich Ohlenschläger (Psychiatric Hospital Vitos, Marburg). Matthias
Müller, Michael Franz, Bernd Kundermann (Psychiatric Hospital Vitos, Gießen).
Christian Bürger, Fanni Dzvonyar, Verena Enneking, Stella Fingas, Janik
Goltermann, Hannah Lemke, Susanne Meinert, Jonathan Repple, Kordula Vorspohl,
Bettina Walden, Dario Zaremba (Dept. of Psychiatry, University of Münster).
Harald Kugel, Jochen Bauer, Walter Heindel, Birgit Vahrenkamp (Dept. of
Clinical Radiology, University of Münster). Gereon Heuft, Gudrun Schneider
(Dept. of Psychosomatics and Psychotherapy, University of Münster). Thomas
Reker (LWL-Hospital Münster). Gisela Bartling (IPP Münster). Ulrike Buhlmann
(Dept. of Clinical Psychology, University of Münster).

###### WP2:

Marco Bartz, Miriam Becker, Christine Blöcher, Annuska Berz, Moria Braun,
Ingmar Conell, Debora dalla Vecchia, Darius Dietrich, Ezgi Esen, Sophia Estel,
Jens Hensen, Ruhkshona Kayumova, Theresa Kisko, Rebekka Obermeier, Anika
Pützer, Nivethini Sangarapillai, Özge Sungur, Clara Raithel, Tobias Redecker,
Vanessa Sandermann, Finnja Schramm, Linda Tempel, Natalie Vermehren, Jakob
Vörckel, Stephan Weingarten, Maria Willadsen, Cüneyt Yildiz (Faculty of
Psychology, Marburg University). WP4: Jana Freff, Silke Jörgens, Kathrin
Schwarte (Dept. of Psychiatry, University of Münster). Susanne Michels,
Goutham Ganjam, Katharina Elsässer (Faculty of Pharmacy, Marburg University).
Felix Ruben Picard, Nicole Löwer, Thomas Ruppersberg (Institute of Laboratory
Medicine and Pathobiochemistry, Marburg University).

###### WP5:

Helene Dukal, Christine Hohmeyer, Lennard Stütz, Viola Schwerdt, Fabian
Streit, Josef Frank, Lea Sirignano (Dept. of Genetic Epidemiology, Central
Institute of Mental Health, Medical Faculty Mannheim, Heidelberg University).
WP6: Anastasia Benedyk, Miriam Bopp, Roman Keßler, Maximilian Lückel, Verena
Schuster, Christoph Vogelbacher (Dept. of Psychiatry, Marburg University).
Jens Sommer, Olaf Steinsträter (Core-Facility Brainimaging, Marburg
University). Thomas W.D. Möbius (Institute of Medical Informatics and
Statistics, Kiel University).

###### CP1:

Julian Glandorf, Fabian Kormann, Arif Alkan, Fatana Wedi, Lea Henning, Alena
Renker, Karina Schneider, Elisabeth Folwarczny, Dana Stenzel, Kai Wenk, Felix
Picard, Alexandra Fischer, Sandra Blumenau, Beate Kleb, Doris Finholdt,
Elisabeth Kinder, Tamara Wüst, Elvira Przypadlo, Corinna Brehm (Comprehensive
Biomaterial Bank Marburg, Marburg University).

The FOR2107 cohort project (WP1) was approved by the Ethics Committees of the
Medical Faculties, University of Marburg (AZ: 07/14) and University of Münster
(AZ: 2014-422-b-S).

## Conflict of Interest

Biomedical financial interests or potential conflicts of interest: Tilo
Kircher received unrestricted educational grants from Servier, Janssen,
Recordati, Aristo, Otsuka, neuraxpharm.

The other authors (Claas Flint, Micah Cearns, Nils Opel, Ronny Redlich, David
M. A. Mehler, Daniel Emden, Nils R. Winter, Ramona Leenings, Simon B.
Eickhoff, Axel Krug, Igor Nenadic, Volker Arolt, Scott Clark, Xiaoyi Jiang,
Udo Dannlowski, Tim Hahn) declare no conflicts of interest.

## References

  * [1] Alison M. Darcy, Alan K. Louie, and Laura Weiss Roberts.  Machine learning and the profession of medicine.  JAMA - Journal of the American Medical Association, 315(6):551–552, 2016.
  * [2] Harris A. Eyre, Ajeet B. Singh, and Charles Reynolds.  Tech giants enter mental health.  World Psychiatry, 15(1):21–22, 2016.
  * [3] John D.E. Gabrieli, Satrajit S. Ghosh, and Susan Whitfield-Gabrieli.  Prediction as a humanitarian and pragmatic contribution from human cognitive neuroscience.  Neuron, 85(1):11–26, 2015.
  * [4] M I Jordan and T M Mitchell.  Machine learning: Trends, perspectives, and prospects.  Science (New York, N.Y.), 349(6245):255–60, 2015.
  * [5] T. Hahn, A. A. Nierenberg, and S. Whitfield-Gabrieli.  Predictive analytics in mental health: Applications, guidelines, challenges and perspectives.  Molecular Psychiatry, 22(1):37–43, 2017.
  * [6] Blair A. Johnston, J. Douglas Steele, Serenella Tolomeo, David Christmas, and Keith Matthews.  Structural MRI-based predictions in patients with treatment-refractory depression (TRD).  PLoS ONE, 10(7):1–16, 2015.
  * [7] Benson Mwangi, Klaus P. Ebmeier, Keith Matthews, and J. Douglas Steele.  Multi-centre diagnostic classification of individual structural neuroimaging scans from patients with major depressive disorder.  Brain, 135(5):1508–1521, 2012.
  * [8] Meenal J. Patel, Carmen Andreescu, Julie C. Price, Kathryn L. Edelman, Charles F. Reynolds, and Howard J. Aizenstein.  Machine learning approaches for integrating clinical and imaging features in late-life depression classification and response prediction.  International Journal of Geriatric Psychiatry, 30(10):1056–1067, 2015.
  * [9] Andres H. Neuhaus and Florin C. Popescu.  Sample Size, Model Robustness, and Classification Accuracy in Diagnostic Multivariate Neuroimaging Analyses.  Biological Psychiatry, 84(11):e81–e82, 2018.
  * [10] Mohammad R Arbabshirani, Sergey Plis, Jing Sui, and Vince D Calhoun.  Single subject prediction of brain disorders in neuroimaging: Promises and pitfalls.  NeuroImage, 145:137–165, 2017.
  * [11] S. Raudys and A. Jain.  Small Sample Size Effects in Statistical Pattern Recognition: Recommendations for Practitioners.  IEEE Transactions on Pattern Analysis & Machine Intelligence, 13:252–264, 1991.
  * [12] Tjeerd van der Ploeg, Peter C Austin, and Ewout W Steyerberg.  Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints.  BMC Medical Research Methodology, 14(1):137, 2014.
  * [13] Joseph Kambeitz, Carlos Cabral, Matthew D. Sacchet, Ian H. Gotlib, Roland Zahn, Mauricio H. Serpa, Martin Walter, Peter Falkai, and Nikolaos Koutsouleris.  Detecting Neuroimaging Biomarkers for Depression: A Meta-analysis of Multivariate Pattern Recognition Studies.  Biological Psychiatry, 82(5):330–338, 2017.
  * [14] Gaël Varoquaux, Pradeep Reddy Raamana, Denis A. Engemann, Andrés Hoyos-Idrobo, Yannick Schwartz, and Bertrand Thirion.  Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines.  NeuroImage, 145(August 2015):166–179, 2017.
  * [15] Tim Hahn, Ulrich Ebner-Priemer, and Andreas Meyer-Lindenberg.  Transparent Artificial Intelligence – A Conceptual Framework for Evaluating AI-based Clinical Decision Support Systems.  OSF Preprints, 2019.
  * [16] Gaël Varoquaux.  Cross-validation failure: Small sample sizes lead to large error bars.  NeuroImage, 180:68–77, oct 2018.
  * [17] Udo Dannlowski, Harald Kugel, Dominik Grotegerd, Ronny Redlich, Janina Suchy, Nils Opel, Thomas Suslow, Carsten Konrad, Patricia Ohrmann, Jochen Bauer, Tilo Kircher, Axel Krug, Andreas Jansen, Bernhard T. Baune, Walter Heindel, Katharina Domschke, Andreas J. Forstner, Markus M. Nöthen, Jens Treutlein, Volker Arolt, Christa Hohoff, Marcella Rietschel, and Stephanie H. Witt.  NCAN Cross-Disorder Risk Variant Is Associated with Limbic Gray Matter Deficits in Healthy Subjects and Major Depression.  Neuropsychopharmacology, 40(11):2510–2516, 2015.
  * [18] U. Dannlowski, H. J. Grabe, K. Wittfeld, J. Klaus, C. Konrad, D. Grotegerd, R. Redlich, T. Suslow, N. Opel, P. Ohrmann, J. Bauer, P. Zwanzger, I. Laeger, C. Hohoff, V. Arolt, W. Heindel, M. Deppe, K. Domschke, K. Hegenscheid, H. Völzke, D. Stacey, H. Meyer Zu Schwabedissen, H. Kugel, and B. T. Baune.  Multimodal imaging of a tescalcin (TESC)-regulating polymorphism (rs7294919)-specific effects on hippocampal gray matter structure.  Molecular Psychiatry, 20(3):398–404, 2015.
  * [19] Tilo Kircher, Markus Wöhr, Igor Nenadic, Rainer Schwarting, Gerhard Schratt, Judith Alferink, Carsten Culmsee, Holger Garn, Tim Hahn, Bertram Müller-Myhsok, Astrid Dempfle, Maik Hahmann, Andreas Jansen, Petra Pfefferle, Harald Renz, Marcella Rietschel, Stephanie H. Witt, Markus Nöthen, Axel Krug, and Udo Dannlowski.  Neurobiology of the major psychoses: a translational perspective on brain structure and function—the FOR2107 consortium.  European Archives of Psychiatry and Clinical Neuroscience, 0(0):0, 2018.
  * [20] H.-U. Wittchen, U. Wunderlich, S. Gruschwitz, and M. Zaudig.  SKID I. Strukturiertes Klinisches Interview für DSM-IV. Achse I: Psychische Störungen. Interviewheft und Beurteilungsheft. Eine deutschsprachige, erweiterte Bearb. d. amerikanischen Originalversion des SKID I.  Hogrefe, Göttingen, 1997.
  * [21] Christoph Vogelbacher, Thomas W.D. Möbius, Jens Sommer, Verena Schuster, Udo Dannlowski, Tilo Kircher, Astrid Dempfle, Andreas Jansen, and Miriam H.A. Bopp.  The Marburg-Münster Affective Disorders Cohort Study (MACS): A quality assurance protocol for MR neuroimaging data.  NeuroImage, 172(December 2017):450–460, 2018.
  * [22] Scikit-learn.  Pipeline: chaining estimators, 2018.
  * [23] Andre F. Marquand, Iead Rezek, Jan Buitelaar, and Christian F. Beckmann.  Understanding Heterogeneity in Clinical Cohorts Using Normative Models: Beyond Case-Control Studies.  Biological Psychiatry, 80(7):552–561, 2016.
  * [24] Hugo G. Schnack and René S. Kahn.  Detecting neuroimaging biomarkers for psychiatric disorders: Sample size matters.  Frontiers in Psychiatry, 7(MAR), 2016.

Supplemental Materials

Systematic Overestimation of Machine Learning Performance

in Neuroimaging Studies of Depression

##  Appendix A Summary statistics for the final study sample

|  | MDD | HC
---|---|---|---
| n | 934 | 934
Sex | female | 556 | 548
male | 378 | 368
Scanner | 1 | 285 | 512
2 | 395 | 236
3 | 254 | 186
(a) Summary of sample size (n), sex and scanner site.

|  | Mean | SD | Min | Max
---|---|---|---|---|---
MDD | Age ($n=934$) | $37.76$ | $12.94$ | $16$ | $65$
TIV ($n=933$) | $1575.94$ | $152.97$ | $1115.00$ | $2189.00$
HC | Age ($n=934$) | $34.15$ | $12.42$ | $17$ | $65$
TIV ($n=933$) | $1575.61$ | $156.38$ | $1146.23$ | $2706.76$
(b) Summage of Age (in years) and TIV (total intracranial volume).

Table A.1: Summary statistics for the final study sample. MDD = Major
Depressive Disorder; HC = Healthy Controls.

##  Appendix B Additional tabluar presentations

This section contains the tabular presentation of the graphical visualized
data in the paper.

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
overall sample size | 10 | $0.54$ | $0.54$ | $0.43$ | $0.33$ | $0.15$ | $0.15$ | $0.05$ | $0.05$ | $0.01$
20 | $0.64$ | $0.52$ | $0.40$ | $0.24$ | $0.13$ | $0.06$ | $0.01$ | $0.01$ | $0.00$
30 | $0.70$ | $0.61$ | $0.39$ | $0.29$ | $0.11$ | $0.06$ | $0.01$ | $0.00$ | $0.00$
40 | $0.78$ | $0.63$ | $0.47$ | $0.23$ | $0.09$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
50 | $0.82$ | $0.70$ | $0.51$ | $0.27$ | $0.08$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
60 | $0.81$ | $0.62$ | $0.41$ | $0.19$ | $0.05$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
70 | $0.88$ | $0.72$ | $0.42$ | $0.20$ | $0.07$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
80 | $0.90$ | $0.73$ | $0.48$ | $0.18$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.89$ | $0.76$ | $0.45$ | $0.20$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.93$ | $0.75$ | $0.47$ | $0.17$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $0.94$ | $0.81$ | $0.48$ | $0.14$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $0.96$ | $0.83$ | $0.45$ | $0.12$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the overall sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
overall sample size | 10 | $0.55$ | $0.60$ | $0.00$ | $1.00$ | $0.20$
20 | $0.56$ | $0.60$ | $0.10$ | $0.95$ | $0.15$
30 | $0.58$ | $0.58$ | $0.20$ | $0.87$ | $0.12$
40 | $0.59$ | $0.60$ | $0.25$ | $0.82$ | $0.10$
50 | $0.59$ | $0.60$ | $0.16$ | $0.82$ | $0.09$
60 | $0.58$ | $0.58$ | $0.32$ | $0.82$ | $0.08$
70 | $0.59$ | $0.60$ | $0.36$ | $0.81$ | $0.07$
80 | $0.59$ | $0.60$ | $0.36$ | $0.76$ | $0.07$
90 | $0.59$ | $0.60$ | $0.34$ | $0.77$ | $0.07$
100 | $0.60$ | $0.60$ | $0.35$ | $0.81$ | $0.06$
125 | $0.59$ | $0.60$ | $0.41$ | $0.74$ | $0.05$
150 | $0.59$ | $0.59$ | $0.41$ | $0.73$ | $0.05$
(b) Summary of the accuracy value distributions for certain overall sample
size.

Table B.1: Effects of varying overall sample sizes for training and testing a
linear SVM employing LOOCV.

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
overall sample size | 10 | $0.25$ | $0.25$ | $0.17$ | $0.10$ | $0.03$ | $0.03$ | $0.01$ | $0.01$ | $0.00$
20 | $0.32$ | $0.18$ | $0.10$ | $0.04$ | $0.02$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
30 | $0.33$ | $0.22$ | $0.06$ | $0.03$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
40 | $0.37$ | $0.18$ | $0.09$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
50 | $0.39$ | $0.21$ | $0.07$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
60 | $0.40$ | $0.16$ | $0.04$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.40$ | $0.16$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.41$ | $0.13$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.41$ | $0.14$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.43$ | $0.11$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $0.46$ | $0.13$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $0.44$ | $0.09$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the overall sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
overall sample size | 10 | $0.44$ | $0.40$ | $0.00$ | $1.00$ | $0.17$
20 | $0.47$ | $0.45$ | $0.15$ | $0.80$ | $0.11$
30 | $0.48$ | $0.47$ | $0.23$ | $0.80$ | $0.09$
40 | $0.49$ | $0.50$ | $0.25$ | $0.70$ | $0.08$
50 | $0.49$ | $0.48$ | $0.28$ | $0.74$ | $0.07$
60 | $0.49$ | $0.50$ | $0.28$ | $0.68$ | $0.06$
70 | $0.49$ | $0.50$ | $0.31$ | $0.69$ | $0.06$
80 | $0.49$ | $0.50$ | $0.30$ | $0.66$ | $0.06$
90 | $0.49$ | $0.49$ | $0.34$ | $0.68$ | $0.05$
100 | $0.50$ | $0.50$ | $0.33$ | $0.64$ | $0.05$
125 | $0.50$ | $0.50$ | $0.33$ | $0.63$ | $0.05$
150 | $0.50$ | $0.49$ | $0.37$ | $0.63$ | $0.04$
(b) Summary of the accuracy value distributions for certain overall sample
size.

Table B.2: Effects of varying overall sample sizes for training and testing a
Dummy Classifier employing LOOCV.

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
train sample size | 10 | $0.80$ | $0.62$ | $0.42$ | $0.16$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
20 | $0.87$ | $0.75$ | $0.55$ | $0.20$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
30 | $0.94$ | $0.83$ | $0.65$ | $0.26$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
40 | $0.97$ | $0.90$ | $0.72$ | $0.29$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
50 | $0.97$ | $0.93$ | $0.76$ | $0.31$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
60 | $0.99$ | $0.95$ | $0.76$ | $0.30$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.99$ | $0.96$ | $0.83$ | $0.35$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.99$ | $0.97$ | $0.83$ | $0.34$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.98$ | $0.89$ | $0.36$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $0.98$ | $0.87$ | $0.35$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $0.99$ | $0.91$ | $0.39$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $0.99$ | $0.94$ | $0.39$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the train sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
train sample size | 10 | $0.57$ | $0.58$ | $0.33$ | $0.69$ | $0.08$
20 | $0.59$ | $0.61$ | $0.32$ | $0.69$ | $0.07$
30 | $0.61$ | $0.62$ | $0.36$ | $0.69$ | $0.06$
40 | $0.62$ | $0.63$ | $0.35$ | $0.70$ | $0.05$
50 | $0.62$ | $0.63$ | $0.38$ | $0.70$ | $0.05$
60 | $0.63$ | $0.64$ | $0.46$ | $0.70$ | $0.04$
70 | $0.63$ | $0.64$ | $0.45$ | $0.71$ | $0.04$
80 | $0.63$ | $0.64$ | $0.45$ | $0.70$ | $0.03$
90 | $0.64$ | $0.64$ | $0.49$ | $0.71$ | $0.03$
100 | $0.64$ | $0.64$ | $0.51$ | $0.70$ | $0.03$
125 | $0.64$ | $0.64$ | $0.51$ | $0.71$ | $0.03$
150 | $0.64$ | $0.65$ | $0.51$ | $0.70$ | $0.03$
(b) Summary of the accuracy value distributions for certain train sample size.

Table B.3: Results of a fixed validation set of $n=300$ and varying train
sample sizes on a linear SVM using default parameter. ($C=1$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
train sample size | 10 | $0.49$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
20 | $0.47$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
30 | $0.47$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
40 | $0.47$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
50 | $0.45$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
60 | $0.46$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.48$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.49$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.47$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.48$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $0.45$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $0.45$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the train sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
train sample size | 10 | $0.50$ | $0.50$ | $0.41$ | $0.59$ | $0.03$
20 | $0.50$ | $0.50$ | $0.39$ | $0.59$ | $0.03$
30 | $0.50$ | $0.50$ | $0.41$ | $0.58$ | $0.03$
40 | $0.50$ | $0.50$ | $0.38$ | $0.60$ | $0.03$
50 | $0.50$ | $0.50$ | $0.42$ | $0.61$ | $0.03$
60 | $0.50$ | $0.50$ | $0.40$ | $0.60$ | $0.03$
70 | $0.50$ | $0.50$ | $0.41$ | $0.58$ | $0.03$
80 | $0.50$ | $0.50$ | $0.42$ | $0.62$ | $0.03$
90 | $0.50$ | $0.50$ | $0.42$ | $0.58$ | $0.03$
100 | $0.50$ | $0.50$ | $0.40$ | $0.60$ | $0.03$
125 | $0.50$ | $0.50$ | $0.40$ | $0.59$ | $0.03$
150 | $0.50$ | $0.50$ | $0.39$ | $0.61$ | $0.03$
(b) Summary of the accuracy value distributions for certain train sample size.

Table B.4: Results of a fixed validation set of $n=300$ and varying train
sample sizes on a Dummy Classifier.

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.78$ | $0.78$ | $0.66$ | $0.54$ | $0.29$ | $0.29$ | $0.10$ | $0.10$ | $0.02$
20 | $0.91$ | $0.81$ | $0.69$ | $0.48$ | $0.30$ | $0.17$ | $0.06$ | $0.06$ | $0.00$
30 | $0.96$ | $0.90$ | $0.73$ | $0.57$ | $0.27$ | $0.15$ | $0.02$ | $0.01$ | $0.00$
40 | $0.98$ | $0.91$ | $0.80$ | $0.49$ | $0.25$ | $0.09$ | $0.02$ | $0.01$ | $0.00$
50 | $0.99$ | $0.97$ | $0.85$ | $0.58$ | $0.22$ | $0.07$ | $0.00$ | $0.00$ | $0.00$
60 | $1.00$ | $0.97$ | $0.84$ | $0.56$ | $0.23$ | $0.05$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.99$ | $0.88$ | $0.61$ | $0.21$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.90$ | $0.56$ | $0.17$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.94$ | $0.63$ | $0.17$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $0.99$ | $0.94$ | $0.58$ | $0.13$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.97$ | $0.63$ | $0.13$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.98$ | $0.66$ | $0.08$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.66$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.67$ | $0.65$ | $0.35$ | $0.95$ | $0.11$
30 | $0.66$ | $0.67$ | $0.40$ | $0.93$ | $0.08$
40 | $0.66$ | $0.65$ | $0.42$ | $0.88$ | $0.07$
50 | $0.66$ | $0.66$ | $0.48$ | $0.88$ | $0.06$
60 | $0.66$ | $0.67$ | $0.47$ | $0.83$ | $0.06$
70 | $0.66$ | $0.67$ | $0.50$ | $0.83$ | $0.05$
80 | $0.66$ | $0.66$ | $0.49$ | $0.80$ | $0.05$
90 | $0.66$ | $0.67$ | $0.53$ | $0.78$ | $0.04$
100 | $0.66$ | $0.66$ | $0.51$ | $0.79$ | $0.04$
125 | $0.66$ | $0.66$ | $0.54$ | $0.78$ | $0.03$
150 | $0.66$ | $0.66$ | $0.56$ | $0.75$ | $0.03$
(b) Summary of the accuracy value distributions for certain test sample size.

Table B.5: Effects of varying test sample sizes on a fixed pretrained linear
SVM.

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.39$ | $0.39$ | $0.29$ | $0.17$ | $0.05$ | $0.05$ | $0.01$ | $0.01$ | $0.00$
20 | $0.42$ | $0.24$ | $0.13$ | $0.05$ | $0.02$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
30 | $0.42$ | $0.28$ | $0.10$ | $0.05$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
40 | $0.44$ | $0.19$ | $0.08$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
50 | $0.44$ | $0.20$ | $0.06$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
60 | $0.45$ | $0.15$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.45$ | $0.17$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.44$ | $0.10$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.43$ | $0.12$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.47$ | $0.10$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $0.53$ | $0.09$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $0.45$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.50$ | $0.50$ | $0.10$ | $1.00$ | $0.16$
20 | $0.50$ | $0.50$ | $0.15$ | $0.85$ | $0.11$
30 | $0.50$ | $0.50$ | $0.23$ | $0.73$ | $0.09$
40 | $0.50$ | $0.50$ | $0.28$ | $0.72$ | $0.07$
50 | $0.50$ | $0.50$ | $0.32$ | $0.70$ | $0.06$
60 | $0.50$ | $0.50$ | $0.30$ | $0.67$ | $0.06$
70 | $0.50$ | $0.50$ | $0.34$ | $0.67$ | $0.05$
80 | $0.50$ | $0.50$ | $0.35$ | $0.64$ | $0.05$
90 | $0.50$ | $0.50$ | $0.36$ | $0.67$ | $0.04$
100 | $0.50$ | $0.50$ | $0.39$ | $0.63$ | $0.04$
125 | $0.50$ | $0.50$ | $0.39$ | $0.62$ | $0.04$
150 | $0.50$ | $0.50$ | $0.40$ | $0.59$ | $0.03$
(b) Summary of the accuracy value distributions for certain test sample size.

Table B.6: Effects of varying test sample sizes on a fixed pretrained Dummy
Classifier.

##  Appendix C Influence of the scanner distribution

Scanner site distributions were not well balanced in the PAC sample. This
imbalance was even stronger in our randomly drawn sub-samples. In order to
determine the influence of the different scanner-sites on model accuracy, we
took the same methodical approach that we used for the “train sample size
effect analysis” and determined the scanner distribution for each set.
Following, we calculated Spearman’s rho between the scanner-distribution and
the accuracy of the hold-out test set ($n=300$). The scanner-distribution for
each set was approximated using Gini’s index. Here, we show that the scanner
distribution has a statistically significant influence on test set accuracy
but explains only 0.79% of the variance.

##  Appendix D Adjustment of SVM regularization based on sample size

The regularization of a linear SVM depends on the absolute number of outliers
in a sample. To exclude the possibility that the use of default
hyperparameters (a constant value of $C=1$) caused the effect that we
observed, we have adjusted the SVM regularization (our $C$ hyperparameter)
based on the size of each analyses sub-sample. In this adjustment, a constant
value $k$ is divided by the sample size. In order to approximate the default
parameter $C=1$ on average, we have set $k=75$. For example, $C$ would be
$C=100/75=$1.33$$ for a sample size of $n=100$. In this case, the adjusted
regularization did not deliver results as good as what we observed with $C=1$
(see Figure D.1, D.2). Therefore, we conclude that the default value of $C=1$
across changing N values did not increase the probability of overestimating
accuracy as sample size decreased.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVMs to yield an accuracy exceeding a certain threshold as a
function of sample size employing LOOCV.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVMs as a function of sample
size employing LOOCV.

Figure D.1: Effects of varying overall sample sizes employing LOOCV with an
adjusted $C$ parameter with $C=\frac{75}{\text{sample size}}$.

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVMs to yield an accuracy exceeding a certain threshold as a
function of training sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVMs as a function of
training sample size.

Figure D.2: Results as a function of training set sizes with a fixed test set
size of $N=300$ and an adjusted $C$ parameter with $C=\frac{75}{\text{sample
size}}$.

##  Appendix E Alternative Machine Configurations

In order to exclude the possibility that the observed effects can only be
traced back to our specific configuration, we have tested other usual
configurations. The configurations consist of a preprocessing and a
classification. In the preprocessing step, we used a method for reduction of
feature space and a method for features selection. The reduction of the
feature space was achieved with a Principal Component Analysis (PCA), where
only a certain number the first components were used. Afterwards, for feature
selection, an ANOVA was calculated and a specific number of feature beginning
with the highest F-value were taken. The actual configuration used for
preprocessing is listed in Table E.1. For the classification, we chose three
specific machines:

  1. 1.

An SVM with a linear kernel and default parameter. ($C=$1.0$$)

  2. 2.

An SVM with a rbf kernel and default parameter ($C=$1.0$$;
$\gamma=1/n_{\text{feature}}$)

  3. 3.

A Random Forrest and default parameter ($n_{\text{estimators}}=$100$$)

In combination with the preprocessing, this results in 48 configurations.
Since the found effect is limited to the test set size, we have only repeated
our analyses for the test set for each of these configurations.

|  | PCA $n_{\text{components}}$
---|---|---
|  | no | all | $500$ | $50$ | $10$
ANOVA $k_{\text{best}}$ | no | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
$10$ | $\times$ | $\times$ | $\times$ | $\times$ |
$100$ | $\times$ | $\times$ | $\times$ |  |
$1000$ | $\times$ | $\times$ |  |  |
$10\,000$ | $\times$ |  |  |  |
$50\,000$ | $\times$ |  |  |  |
Table E.1: The $\times$ marks the used combinations of configurations for the
preprocessing. The _no_ -label means that this preprocessing step were left
out.

###  E.1 Results

The results are comparable to the original used configuration, an SVM with a
linear kernel and no preprocessing (see Figure E.1 to E.48 and Table E.2 to
E.49). The results thus underline a generally valid character of the findings.
An outlier in the results can be found for a specific configuration, a PCA
with 10 components and the SVM with rbf kernel (see Figure E.20 and Table
E.21). This result can be explained due overfitting, the machine constantly
returns one constant class as prediction.

####  E.1.1 Graphical results representation

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.1: Results as a function of variable test set sizes with a fixed
classifier. A Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.2: Results as a function of variable test set sizes with a fixed
classifier. An SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.3: Results as a function of variable test set sizes with a fixed
classifier. An linear SVM was trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.4: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.5: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.6: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, an linear SVM was trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.7: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.8: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.9: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, an linear SVM was trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.10: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.11: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.12: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.
Following, an linear SVM was trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.13: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10\,000$ features were taken based on the ANOVA
F-values. Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.14: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10\,000$ features were taken based on the ANOVA
F-values. Following, an SVM with a rbf kernel was trained with default
parameters. ($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.15: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10\,000$ features were taken based on the ANOVA
F-values. Following, an linear SVM was trained with default parameters.
($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.16: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $50\,000$ features were taken based on the ANOVA
F-values. Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.17: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $50\,000$ features were taken based on the ANOVA
F-values. Following, an SVM with a rbf kernel was trained with default
parameters. ($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.18: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $50\,000$ features were taken based on the ANOVA
F-values. Following, an linear SVM was trained with default parameters.
($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.19: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $10$ components were retained. Following, a Random Forest was
trained with default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.20: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $10$ components were retained. Following, an SVM with a rbf
kernel was trained with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.21: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $10$ components were retained. Following, an linear SVM was
trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.22: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. Following, a Random Forest was
trained with default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.23: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. Following, an SVM with a rbf
kernel was trained with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.24: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. Following, an linear SVM was
trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.25: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, a Random Forest was trained with
default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.26: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, an SVM with a rbf kernel was trained
with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.27: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $50$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.28: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. Following, a Random Forest was
trained with default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.29: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. Following, an SVM with a rbf
kernel was trained with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.30: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. Following, an linear SVM was
trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.31: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, a Random Forest was trained with
default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.32: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, an SVM with a rbf kernel was trained
with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.33: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $10$ features were taken
based on the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.34: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $100$ features were taken
based on the ANOVA F-values. Following, a Random Forest was trained with
default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.35: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $100$ features were taken
based on the ANOVA F-values. Following, an SVM with a rbf kernel was trained
with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.36: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and $500$ components were retained. For feature selection an ANOVA
was computed inside the the pipeline and the top $100$ features were taken
based on the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.37: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. Following, a Random Forest was
trained with default parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.38: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. Following, an SVM with a rbf
kernel was trained with default parameters. ($C=$1.0$$;
$\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.39: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. Following, an linear SVM was
trained with default parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.40: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $10$ features were taken based on
the ANOVA F-values. Following, a Random Forest was trained with default
parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.41: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $10$ features were taken based on
the ANOVA F-values. Following, an SVM with a rbf kernel was trained with
default parameters. ($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.42: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $10$ features were taken based on
the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.43: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $100$ features were taken based
on the ANOVA F-values. Following, a Random Forest was trained with default
parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.44: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $100$ features were taken based
on the ANOVA F-values. Following, an SVM with a rbf kernel was trained with
default parameters. ($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.45: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $100$ features were taken based
on the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for Random Forest to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the Random Forest as a function of test
sample size.

Figure E.46: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $1000$ features were taken based
on the ANOVA F-values. Following, a Random Forest was trained with default
parameters. ($n_{\text{estimators}}=$100$$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for SVM with rbf kernel to yield an accuracy exceeding a certain threshold as
a function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the SVM with rbf kernel as a function of
test sample size.

Figure E.47: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $1000$ features were taken based
on the ANOVA F-values. Following, an SVM with a rbf kernel was trained with
default parameters. ($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

255075100125150sample size0%20%40%60%80%100%probabilityabove 50%above 55%above
60%above 65%above 70%above 75%above 80%above 85%above 90% (a) Probabilities
for linear SVM to yield an accuracy exceeding a certain threshold as a
function of test sample size.

50100150sample size0%20%40%60%80%100%balanced accuracymaximummeanminimum (b)
Minimum, maximum and mean results for the linear SVM as a function of test
sample size.

Figure E.48: Results as a function of variable test set sizes with a fixed
classifier. To reduce the dimensionality of the feature space a PCA was
performed and all components were retained. For feature selection an ANOVA was
computed inside the the pipeline and the top $1000$ features were taken based
on the ANOVA F-values. Following, an linear SVM was trained with default
parameters. ($C=$1.0$$)

####  E.1.2 Tabular results representation

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.79$ | $0.79$ | $0.65$ | $0.54$ | $0.28$ | $0.28$ | $0.08$ | $0.08$ | $0.01$
20 | $0.92$ | $0.81$ | $0.67$ | $0.47$ | $0.29$ | $0.15$ | $0.07$ | $0.06$ | $0.00$
30 | $0.96$ | $0.91$ | $0.74$ | $0.58$ | $0.29$ | $0.15$ | $0.03$ | $0.01$ | $0.00$
40 | $0.98$ | $0.93$ | $0.80$ | $0.50$ | $0.24$ | $0.07$ | $0.01$ | $0.00$ | $0.00$
50 | $0.99$ | $0.96$ | $0.86$ | $0.59$ | $0.23$ | $0.08$ | $0.00$ | $0.00$ | $0.00$
60 | $1.00$ | $0.98$ | $0.87$ | $0.55$ | $0.23$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.99$ | $0.89$ | $0.61$ | $0.21$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.90$ | $0.54$ | $0.16$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.93$ | $0.61$ | $0.14$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.94$ | $0.59$ | $0.14$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.96$ | $0.64$ | $0.12$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.99$ | $0.66$ | $0.07$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.66$ | $0.70$ | $0.20$ | $1.00$ | $0.14$
20 | $0.67$ | $0.65$ | $0.35$ | $0.95$ | $0.10$
30 | $0.67$ | $0.67$ | $0.37$ | $0.90$ | $0.08$
40 | $0.66$ | $0.65$ | $0.42$ | $0.85$ | $0.07$
50 | $0.66$ | $0.66$ | $0.46$ | $0.84$ | $0.06$
60 | $0.67$ | $0.67$ | $0.50$ | $0.80$ | $0.05$
70 | $0.67$ | $0.67$ | $0.51$ | $0.81$ | $0.05$
80 | $0.66$ | $0.66$ | $0.53$ | $0.81$ | $0.05$
90 | $0.66$ | $0.67$ | $0.51$ | $0.78$ | $0.04$
100 | $0.66$ | $0.66$ | $0.55$ | $0.77$ | $0.04$
125 | $0.66$ | $0.66$ | $0.54$ | $0.76$ | $0.03$
150 | $0.66$ | $0.66$ | $0.57$ | $0.75$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.2: Results as a function of variable test set sizes with a fixed
classifier. A Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.79$ | $0.79$ | $0.65$ | $0.55$ | $0.29$ | $0.29$ | $0.10$ | $0.10$ | $0.01$
20 | $0.91$ | $0.80$ | $0.65$ | $0.45$ | $0.25$ | $0.12$ | $0.04$ | $0.03$ | $0.00$
30 | $0.96$ | $0.91$ | $0.71$ | $0.56$ | $0.26$ | $0.14$ | $0.03$ | $0.01$ | $0.00$
40 | $0.98$ | $0.91$ | $0.79$ | $0.48$ | $0.21$ | $0.04$ | $0.01$ | $0.00$ | $0.00$
50 | $0.99$ | $0.96$ | $0.84$ | $0.55$ | $0.20$ | $0.07$ | $0.00$ | $0.00$ | $0.00$
60 | $0.99$ | $0.97$ | $0.83$ | $0.49$ | $0.19$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.99$ | $0.86$ | $0.60$ | $0.16$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.91$ | $0.53$ | $0.15$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.99$ | $0.92$ | $0.62$ | $0.14$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.95$ | $0.57$ | $0.13$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.96$ | $0.63$ | $0.12$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.98$ | $0.64$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.66$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.66$ | $0.65$ | $0.30$ | $0.95$ | $0.10$
30 | $0.66$ | $0.67$ | $0.40$ | $0.90$ | $0.08$
40 | $0.66$ | $0.65$ | $0.45$ | $0.93$ | $0.07$
50 | $0.66$ | $0.66$ | $0.46$ | $0.86$ | $0.06$
60 | $0.66$ | $0.65$ | $0.50$ | $0.82$ | $0.05$
70 | $0.66$ | $0.66$ | $0.50$ | $0.79$ | $0.05$
80 | $0.66$ | $0.66$ | $0.53$ | $0.84$ | $0.04$
90 | $0.66$ | $0.67$ | $0.50$ | $0.80$ | $0.04$
100 | $0.66$ | $0.66$ | $0.54$ | $0.77$ | $0.04$
125 | $0.66$ | $0.66$ | $0.56$ | $0.78$ | $0.03$
150 | $0.66$ | $0.66$ | $0.59$ | $0.75$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.3: Results as a function of variable test set sizes with a fixed
classifier. An SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.77$ | $0.77$ | $0.63$ | $0.53$ | $0.27$ | $0.27$ | $0.08$ | $0.08$ | $0.01$
20 | $0.90$ | $0.78$ | $0.65$ | $0.46$ | $0.27$ | $0.14$ | $0.05$ | $0.04$ | $0.00$
30 | $0.95$ | $0.89$ | $0.69$ | $0.54$ | $0.23$ | $0.12$ | $0.02$ | $0.01$ | $0.00$
40 | $0.98$ | $0.90$ | $0.78$ | $0.45$ | $0.19$ | $0.06$ | $0.01$ | $0.00$ | $0.00$
50 | $1.00$ | $0.97$ | $0.83$ | $0.55$ | $0.18$ | $0.06$ | $0.01$ | $0.00$ | $0.00$
60 | $1.00$ | $0.96$ | $0.82$ | $0.51$ | $0.18$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.98$ | $0.84$ | $0.55$ | $0.16$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.89$ | $0.52$ | $0.16$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.89$ | $0.57$ | $0.12$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.93$ | $0.52$ | $0.10$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.96$ | $0.58$ | $0.10$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.98$ | $0.61$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.65$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.66$ | $0.65$ | $0.30$ | $0.95$ | $0.11$
30 | $0.66$ | $0.67$ | $0.43$ | $1.00$ | $0.08$
40 | $0.65$ | $0.65$ | $0.38$ | $0.90$ | $0.07$
50 | $0.66$ | $0.66$ | $0.46$ | $0.86$ | $0.06$
60 | $0.66$ | $0.67$ | $0.48$ | $0.82$ | $0.05$
70 | $0.66$ | $0.66$ | $0.50$ | $0.80$ | $0.05$
80 | $0.66$ | $0.66$ | $0.50$ | $0.80$ | $0.05$
90 | $0.66$ | $0.66$ | $0.53$ | $0.78$ | $0.04$
100 | $0.66$ | $0.66$ | $0.52$ | $0.79$ | $0.04$
125 | $0.66$ | $0.66$ | $0.54$ | $0.76$ | $0.03$
150 | $0.66$ | $0.66$ | $0.57$ | $0.75$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.4: Results as a function of variable test set sizes with a fixed
classifier. An linear SVM was trained with default parameters. ($C=$1.0$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.67$ | $0.67$ | $0.52$ | $0.41$ | $0.18$ | $0.18$ | $0.06$ | $0.06$ | $0.01$
20 | $0.81$ | $0.64$ | $0.49$ | $0.28$ | $0.13$ | $0.05$ | $0.01$ | $0.01$ | $0.00$
30 | $0.87$ | $0.77$ | $0.49$ | $0.34$ | $0.12$ | $0.05$ | $0.01$ | $0.00$ | $0.00$
40 | $0.93$ | $0.78$ | $0.60$ | $0.25$ | $0.07$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
50 | $0.95$ | $0.84$ | $0.60$ | $0.28$ | $0.06$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
60 | $0.97$ | $0.84$ | $0.51$ | $0.20$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.98$ | $0.88$ | $0.50$ | $0.23$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.99$ | $0.87$ | $0.58$ | $0.17$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.99$ | $0.93$ | $0.62$ | $0.21$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.99$ | $0.93$ | $0.63$ | $0.15$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $0.98$ | $0.64$ | $0.14$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $0.98$ | $0.68$ | $0.09$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.61$ | $0.60$ | $0.10$ | $1.00$ | $0.15$
20 | $0.61$ | $0.60$ | $0.20$ | $0.90$ | $0.10$
30 | $0.61$ | $0.60$ | $0.33$ | $0.87$ | $0.09$
40 | $0.62$ | $0.62$ | $0.40$ | $0.80$ | $0.07$
50 | $0.61$ | $0.62$ | $0.40$ | $0.84$ | $0.06$
60 | $0.61$ | $0.62$ | $0.43$ | $0.78$ | $0.06$
70 | $0.61$ | $0.60$ | $0.43$ | $0.80$ | $0.05$
80 | $0.61$ | $0.61$ | $0.42$ | $0.77$ | $0.05$
90 | $0.61$ | $0.61$ | $0.48$ | $0.76$ | $0.04$
100 | $0.61$ | $0.61$ | $0.49$ | $0.74$ | $0.04$
125 | $0.61$ | $0.62$ | $0.50$ | $0.72$ | $0.03$
150 | $0.61$ | $0.61$ | $0.52$ | $0.70$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.5: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.77$ | $0.77$ | $0.64$ | $0.54$ | $0.28$ | $0.28$ | $0.09$ | $0.09$ | $0.01$
20 | $0.90$ | $0.80$ | $0.66$ | $0.45$ | $0.26$ | $0.12$ | $0.05$ | $0.04$ | $0.00$
30 | $0.94$ | $0.90$ | $0.67$ | $0.53$ | $0.24$ | $0.13$ | $0.03$ | $0.01$ | $0.00$
40 | $0.98$ | $0.91$ | $0.75$ | $0.45$ | $0.21$ | $0.05$ | $0.01$ | $0.00$ | $0.00$
50 | $0.99$ | $0.96$ | $0.85$ | $0.57$ | $0.21$ | $0.08$ | $0.01$ | $0.00$ | $0.00$
60 | $1.00$ | $0.97$ | $0.82$ | $0.48$ | $0.17$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.98$ | $0.83$ | $0.55$ | $0.16$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.98$ | $0.89$ | $0.52$ | $0.13$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.99$ | $0.90$ | $0.57$ | $0.14$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.94$ | $0.51$ | $0.10$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.95$ | $0.55$ | $0.09$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.98$ | $0.59$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.66$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.66$ | $0.65$ | $0.30$ | $0.95$ | $0.10$
30 | $0.66$ | $0.67$ | $0.37$ | $0.90$ | $0.08$
40 | $0.65$ | $0.65$ | $0.42$ | $0.85$ | $0.07$
50 | $0.66$ | $0.66$ | $0.48$ | $0.84$ | $0.06$
60 | $0.66$ | $0.65$ | $0.50$ | $0.82$ | $0.05$
70 | $0.66$ | $0.66$ | $0.49$ | $0.83$ | $0.05$
80 | $0.66$ | $0.66$ | $0.53$ | $0.80$ | $0.05$
90 | $0.66$ | $0.66$ | $0.52$ | $0.78$ | $0.04$
100 | $0.66$ | $0.66$ | $0.53$ | $0.79$ | $0.04$
125 | $0.66$ | $0.66$ | $0.56$ | $0.76$ | $0.03$
150 | $0.66$ | $0.65$ | $0.57$ | $0.73$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.6: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.80$ | $0.80$ | $0.69$ | $0.57$ | $0.30$ | $0.30$ | $0.11$ | $0.11$ | $0.02$
20 | $0.93$ | $0.84$ | $0.72$ | $0.51$ | $0.32$ | $0.15$ | $0.07$ | $0.05$ | $0.01$
30 | $0.97$ | $0.94$ | $0.78$ | $0.63$ | $0.32$ | $0.18$ | $0.04$ | $0.01$ | $0.00$
40 | $0.99$ | $0.94$ | $0.86$ | $0.60$ | $0.30$ | $0.10$ | $0.02$ | $0.00$ | $0.00$
50 | $1.00$ | $0.99$ | $0.92$ | $0.69$ | $0.30$ | $0.10$ | $0.01$ | $0.00$ | $0.00$
60 | $1.00$ | $0.99$ | $0.90$ | $0.61$ | $0.27$ | $0.05$ | $0.01$ | $0.00$ | $0.00$
70 | $1.00$ | $1.00$ | $0.92$ | $0.70$ | $0.28$ | $0.07$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.95$ | $0.67$ | $0.23$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.97$ | $0.78$ | $0.23$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.97$ | $0.70$ | $0.20$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.99$ | $0.77$ | $0.23$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $1.00$ | $0.83$ | $0.15$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.67$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.67$ | $0.70$ | $0.35$ | $1.00$ | $0.10$
30 | $0.68$ | $0.67$ | $0.40$ | $0.93$ | $0.08$
40 | $0.68$ | $0.68$ | $0.42$ | $0.88$ | $0.07$
50 | $0.68$ | $0.68$ | $0.48$ | $0.86$ | $0.06$
60 | $0.68$ | $0.67$ | $0.52$ | $0.83$ | $0.05$
70 | $0.68$ | $0.67$ | $0.54$ | $0.83$ | $0.05$
80 | $0.67$ | $0.68$ | $0.54$ | $0.82$ | $0.05$
90 | $0.68$ | $0.68$ | $0.56$ | $0.80$ | $0.04$
100 | $0.67$ | $0.67$ | $0.56$ | $0.80$ | $0.04$
125 | $0.68$ | $0.67$ | $0.57$ | $0.80$ | $0.03$
150 | $0.68$ | $0.67$ | $0.60$ | $0.76$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.7: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $10$ features were taken based on the ANOVA F-values.
Following, an linear SVM was trained with default parameters. ($C=$1.0$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.73$ | $0.73$ | $0.62$ | $0.49$ | $0.24$ | $0.24$ | $0.09$ | $0.09$ | $0.01$
20 | $0.86$ | $0.73$ | $0.58$ | $0.37$ | $0.19$ | $0.09$ | $0.03$ | $0.03$ | $0.00$
30 | $0.92$ | $0.85$ | $0.63$ | $0.47$ | $0.15$ | $0.08$ | $0.01$ | $0.00$ | $0.00$
40 | $0.96$ | $0.84$ | $0.70$ | $0.37$ | $0.15$ | $0.04$ | $0.01$ | $0.00$ | $0.00$
50 | $0.98$ | $0.94$ | $0.80$ | $0.47$ | $0.15$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
60 | $0.99$ | $0.91$ | $0.72$ | $0.38$ | $0.11$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.96$ | $0.75$ | $0.43$ | $0.09$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.97$ | $0.81$ | $0.39$ | $0.08$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.98$ | $0.81$ | $0.43$ | $0.06$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $0.99$ | $0.86$ | $0.36$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.89$ | $0.38$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.93$ | $0.36$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.64$ | $0.60$ | $0.20$ | $1.00$ | $0.15$
20 | $0.64$ | $0.65$ | $0.25$ | $0.95$ | $0.10$
30 | $0.64$ | $0.63$ | $0.40$ | $0.87$ | $0.08$
40 | $0.64$ | $0.65$ | $0.40$ | $0.88$ | $0.07$
50 | $0.65$ | $0.64$ | $0.44$ | $0.82$ | $0.06$
60 | $0.64$ | $0.63$ | $0.47$ | $0.85$ | $0.06$
70 | $0.64$ | $0.64$ | $0.49$ | $0.80$ | $0.05$
80 | $0.64$ | $0.65$ | $0.50$ | $0.81$ | $0.05$
90 | $0.64$ | $0.64$ | $0.51$ | $0.78$ | $0.04$
100 | $0.64$ | $0.64$ | $0.52$ | $0.81$ | $0.04$
125 | $0.64$ | $0.64$ | $0.53$ | $0.75$ | $0.03$
150 | $0.64$ | $0.64$ | $0.55$ | $0.72$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.8: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.73$ | $0.73$ | $0.60$ | $0.47$ | $0.20$ | $0.20$ | $0.05$ | $0.05$ | $0.01$
20 | $0.86$ | $0.71$ | $0.56$ | $0.34$ | $0.19$ | $0.07$ | $0.02$ | $0.01$ | $0.00$
30 | $0.92$ | $0.85$ | $0.63$ | $0.44$ | $0.17$ | $0.09$ | $0.02$ | $0.01$ | $0.00$
40 | $0.96$ | $0.85$ | $0.67$ | $0.35$ | $0.15$ | $0.04$ | $0.01$ | $0.00$ | $0.00$
50 | $0.97$ | $0.91$ | $0.73$ | $0.40$ | $0.12$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
60 | $0.99$ | $0.92$ | $0.71$ | $0.34$ | $0.09$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
70 | $1.00$ | $0.96$ | $0.72$ | $0.39$ | $0.09$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
80 | $1.00$ | $0.95$ | $0.74$ | $0.32$ | $0.06$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.99$ | $0.78$ | $0.37$ | $0.06$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $0.99$ | $0.83$ | $0.32$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.88$ | $0.30$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.90$ | $0.33$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.63$ | $0.60$ | $0.10$ | $1.00$ | $0.14$
20 | $0.63$ | $0.65$ | $0.35$ | $0.95$ | $0.10$
30 | $0.64$ | $0.63$ | $0.37$ | $0.90$ | $0.09$
40 | $0.64$ | $0.62$ | $0.42$ | $0.88$ | $0.07$
50 | $0.63$ | $0.64$ | $0.42$ | $0.84$ | $0.06$
60 | $0.64$ | $0.63$ | $0.48$ | $0.80$ | $0.05$
70 | $0.64$ | $0.64$ | $0.47$ | $0.79$ | $0.05$
80 | $0.63$ | $0.64$ | $0.50$ | $0.76$ | $0.05$
90 | $0.64$ | $0.63$ | $0.51$ | $0.77$ | $0.04$
100 | $0.64$ | $0.64$ | $0.53$ | $0.75$ | $0.04$
125 | $0.64$ | $0.64$ | $0.54$ | $0.73$ | $0.03$
150 | $0.64$ | $0.64$ | $0.57$ | $0.71$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.9: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.69$ | $0.69$ | $0.56$ | $0.44$ | $0.19$ | $0.19$ | $0.05$ | $0.05$ | $0.00$
20 | $0.81$ | $0.69$ | $0.54$ | $0.32$ | $0.16$ | $0.06$ | $0.02$ | $0.02$ | $0.00$
30 | $0.89$ | $0.82$ | $0.53$ | $0.34$ | $0.10$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
40 | $0.93$ | $0.80$ | $0.59$ | $0.27$ | $0.10$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
50 | $0.96$ | $0.87$ | $0.61$ | $0.31$ | $0.07$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
60 | $0.97$ | $0.85$ | $0.54$ | $0.19$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.99$ | $0.92$ | $0.58$ | $0.26$ | $0.04$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.99$ | $0.92$ | $0.67$ | $0.21$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $0.95$ | $0.65$ | $0.22$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $0.96$ | $0.68$ | $0.18$ | $0.02$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $0.99$ | $0.74$ | $0.17$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $0.76$ | $0.12$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.62$ | $0.60$ | $0.10$ | $1.00$ | $0.15$
20 | $0.62$ | $0.60$ | $0.30$ | $0.95$ | $0.10$
30 | $0.62$ | $0.63$ | $0.33$ | $0.87$ | $0.08$
40 | $0.62$ | $0.62$ | $0.38$ | $0.82$ | $0.07$
50 | $0.62$ | $0.62$ | $0.40$ | $0.84$ | $0.06$
60 | $0.61$ | $0.62$ | $0.45$ | $0.80$ | $0.05$
70 | $0.62$ | $0.61$ | $0.46$ | $0.77$ | $0.05$
80 | $0.62$ | $0.62$ | $0.49$ | $0.77$ | $0.05$
90 | $0.62$ | $0.62$ | $0.49$ | $0.77$ | $0.04$
100 | $0.62$ | $0.62$ | $0.47$ | $0.75$ | $0.04$
125 | $0.62$ | $0.62$ | $0.52$ | $0.72$ | $0.03$
150 | $0.62$ | $0.62$ | $0.54$ | $0.71$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.10: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $100$ features were taken based on the ANOVA F-values.
Following, an linear SVM was trained with default parameters. ($C=$1.0$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.81$ | $0.81$ | $0.71$ | $0.60$ | $0.35$ | $0.35$ | $0.11$ | $0.11$ | $0.02$
20 | $0.94$ | $0.84$ | $0.72$ | $0.52$ | $0.33$ | $0.16$ | $0.06$ | $0.05$ | $0.01$
30 | $0.97$ | $0.95$ | $0.78$ | $0.65$ | $0.33$ | $0.20$ | $0.04$ | $0.01$ | $0.00$
40 | $0.99$ | $0.95$ | $0.86$ | $0.57$ | $0.30$ | $0.09$ | $0.02$ | $0.00$ | $0.00$
50 | $0.99$ | $0.98$ | $0.89$ | $0.67$ | $0.28$ | $0.10$ | $0.01$ | $0.00$ | $0.00$
60 | $1.00$ | $0.99$ | $0.88$ | $0.64$ | $0.26$ | $0.05$ | $0.01$ | $0.00$ | $0.00$
70 | $1.00$ | $0.99$ | $0.92$ | $0.71$ | $0.27$ | $0.06$ | $0.01$ | $0.00$ | $0.00$
80 | $1.00$ | $0.99$ | $0.95$ | $0.65$ | $0.24$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.97$ | $0.74$ | $0.24$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.98$ | $0.74$ | $0.25$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.99$ | $0.81$ | $0.24$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $1.00$ | $0.82$ | $0.15$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.68$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.68$ | $0.70$ | $0.35$ | $0.95$ | $0.10$
30 | $0.68$ | $0.67$ | $0.43$ | $0.93$ | $0.08$
40 | $0.67$ | $0.68$ | $0.45$ | $0.85$ | $0.07$
50 | $0.67$ | $0.68$ | $0.48$ | $0.86$ | $0.06$
60 | $0.68$ | $0.68$ | $0.50$ | $0.85$ | $0.05$
70 | $0.68$ | $0.67$ | $0.50$ | $0.84$ | $0.05$
80 | $0.67$ | $0.68$ | $0.50$ | $0.80$ | $0.05$
90 | $0.68$ | $0.68$ | $0.53$ | $0.81$ | $0.04$
100 | $0.68$ | $0.68$ | $0.56$ | $0.79$ | $0.04$
125 | $0.68$ | $0.68$ | $0.58$ | $0.78$ | $0.03$
150 | $0.68$ | $0.68$ | $0.59$ | $0.76$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.11: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.
Following, a Random Forest was trained with default parameters.
($n_{\text{estimators}}=$100$$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.82$ | $0.82$ | $0.69$ | $0.62$ | $0.35$ | $0.35$ | $0.13$ | $0.13$ | $0.03$
20 | $0.93$ | $0.84$ | $0.72$ | $0.53$ | $0.33$ | $0.17$ | $0.06$ | $0.06$ | $0.00$
30 | $0.98$ | $0.95$ | $0.78$ | $0.64$ | $0.34$ | $0.20$ | $0.06$ | $0.01$ | $0.00$
40 | $0.99$ | $0.95$ | $0.87$ | $0.56$ | $0.29$ | $0.12$ | $0.03$ | $0.01$ | $0.00$
50 | $1.00$ | $0.98$ | $0.92$ | $0.71$ | $0.30$ | $0.10$ | $0.01$ | $0.00$ | $0.00$
60 | $1.00$ | $0.98$ | $0.91$ | $0.66$ | $0.29$ | $0.06$ | $0.01$ | $0.00$ | $0.00$
70 | $1.00$ | $1.00$ | $0.94$ | $0.73$ | $0.30$ | $0.08$ | $0.01$ | $0.00$ | $0.00$
80 | $1.00$ | $1.00$ | $0.97$ | $0.71$ | $0.28$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
90 | $1.00$ | $1.00$ | $0.98$ | $0.81$ | $0.30$ | $0.04$ | $0.00$ | $0.00$ | $0.00$
100 | $1.00$ | $1.00$ | $0.97$ | $0.73$ | $0.23$ | $0.03$ | $0.00$ | $0.00$ | $0.00$
125 | $1.00$ | $1.00$ | $0.99$ | $0.81$ | $0.26$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $1.00$ | $1.00$ | $0.88$ | $0.20$ | $0.01$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.69$ | $0.70$ | $0.20$ | $1.00$ | $0.15$
20 | $0.68$ | $0.70$ | $0.40$ | $0.95$ | $0.10$
30 | $0.68$ | $0.67$ | $0.33$ | $0.93$ | $0.08$
40 | $0.68$ | $0.68$ | $0.45$ | $0.93$ | $0.07$
50 | $0.68$ | $0.68$ | $0.44$ | $0.84$ | $0.06$
60 | $0.68$ | $0.68$ | $0.52$ | $0.83$ | $0.05$
70 | $0.68$ | $0.69$ | $0.54$ | $0.84$ | $0.05$
80 | $0.68$ | $0.68$ | $0.55$ | $0.82$ | $0.04$
90 | $0.68$ | $0.68$ | $0.54$ | $0.80$ | $0.04$
100 | $0.68$ | $0.68$ | $0.56$ | $0.79$ | $0.04$
125 | $0.68$ | $0.68$ | $0.58$ | $0.78$ | $0.03$
150 | $0.68$ | $0.68$ | $0.60$ | $0.77$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.12: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.
Following, an SVM with a rbf kernel was trained with default parameters.
($C=$1.0$$; $\gamma=\nicefrac{{1}}{{n_{\text{feature}}}}$)

|  |  $\geq$ accuracy (%)
---|---|---
| $n$ | 50 | 55 | 60 | 65 | 70 | 75 | 80 | 85 | 90
test sample size | 10 | $0.61$ | $0.61$ | $0.44$ | $0.33$ | $0.13$ | $0.13$ | $0.04$ | $0.04$ | $0.00$
20 | $0.68$ | $0.51$ | $0.35$ | $0.17$ | $0.07$ | $0.03$ | $0.01$ | $0.01$ | $0.00$
30 | $0.77$ | $0.62$ | $0.32$ | $0.20$ | $0.06$ | $0.02$ | $0.00$ | $0.00$ | $0.00$
40 | $0.81$ | $0.61$ | $0.40$ | $0.13$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
50 | $0.86$ | $0.68$ | $0.41$ | $0.15$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
60 | $0.90$ | $0.66$ | $0.33$ | $0.09$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
70 | $0.93$ | $0.72$ | $0.32$ | $0.10$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
80 | $0.95$ | $0.71$ | $0.33$ | $0.06$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
90 | $0.96$ | $0.74$ | $0.30$ | $0.05$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
100 | $0.97$ | $0.73$ | $0.31$ | $0.03$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
125 | $0.99$ | $0.83$ | $0.27$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
150 | $1.00$ | $0.86$ | $0.23$ | $0.01$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
(a) Probabilities to yield an accuracy exceeding a certain threshold related
to the test sample size.

| $n$ | mean | median | min | max | std
---|---|---|---|---|---|---
test sample size | 10 | $0.58$ | $0.60$ | $0.10$ | $1.00$ | $0.15$
20 | $0.57$ | $0.60$ | $0.20$ | $0.90$ | $0.11$
30 | $0.58$ | $0.57$ | $0.30$ | $0.80$ | $0.08$
40 | $0.58$ | $0.57$ | $0.38$ | $0.80$ | $0.07$
50 | $0.58$ | $0.58$ | $0.38$ | $0.78$ | $0.07$
60 | $0.58$ | $0.58$ | $0.42$ | $0.75$ | $0.06$
70 | $0.58$ | $0.59$ | $0.43$ | $0.76$ | $0.05$
80 | $0.58$ | $0.57$ | $0.44$ | $0.71$ | $0.05$
90 | $0.58$ | $0.58$ | $0.42$ | $0.77$ | $0.04$
100 | $0.58$ | $0.58$ | $0.46$ | $0.71$ | $0.04$
125 | $0.58$ | $0.58$ | $0.47$ | $0.68$ | $0.03$
150 | $0.58$ | $0.58$ | $0.49$ | $0.68$ | $0.03$
(b) Summary of the accuracy value distributions for certain sample sizes.

Table E.13: Results as a function of variable test set sizes with a fixed
classifier. For feature selection an ANOVA was computed inside the the
pipeline and the top $1000$ features were taken based on the ANOVA F-values.