# Is K-fold cross validation the best model selection method for Machine
Learning?

J.M. Gorriz, F. Segovia, J Ramirez
Department of Signal Theory and Communications
University of Granada
Granada, Spain
&A. Ortiz
Department of Communications Engineering
University of Malaga
Malaga, Spain
<EMAIL_ADDRESS>
&John Suckling
Department of Psychiatry
University of Cambridge
Cambridge, UK
<EMAIL_ADDRESS>
&International Initiatives
for the Alzheimer’s Disease Neuroimaging Initiative (ADNI)
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

As a technique that can compactly represent complex patterns, machine learning
has significant potential for predictive inference. K-fold cross-validation
(CV) is the most common approach to ascertaining the likelihood that a machine
learning outcome is generated by chance and frequently outperforms
conventional hypothesis testing. This improvement uses measures directly
obtained from machine learning classifications, such as accuracy, that do not
have a parametric description. To approach a frequentist analysis within
machine learning pipelines, a permutation test or simple statistics from data
partitions (i.e. folds) can be added to estimate confidence intervals.
Unfortunately, neither parametric nor non-parametric tests solve the inherent
problems around partitioning small sample-size datasets and learning from
heterogeneous data sources. The fact that machine learning strongly depends on
the learning parameters and the distribution of data across folds
recapitulates familiar difficulties around excess false positives and
replication. The origins of this problem are demonstrated by simulating common
experimental circumstances, including small sample sizes, low numbers of
predictors, and heterogeneous data sources. A novel statistical test based on
K-fold CV and the Upper Bound of the actual error (K-fold CUBV) is composed,
where uncertain predictions of machine learning with CV are bounded by the
_worst case_ through the evaluation of concentration inequalities. Probably
Approximately Correct-Bayesian upper bounds for linear classifiers in
combination with K-fold CV is used to estimate the empirical error. The
performance with neuroimaging datasets suggests this is a robust criterion for
detecting effects, validating accuracy values obtained from machine learning
whilst avoiding excess false positives.

_K_ eywords K-fold cross-validation $\cdot$ linear support vector machines
$\cdot$ statistical learning theory $\cdot$ permutation tests $\cdot$ Magnetic
Resonance Imaging $\cdot$ Upper Bounding.

##  1 Introduction

Machine learning (ML) systems [1] are designed to solve tasks without
knowledge of the underlying distributions of the data. In other words, ML
systems are agnostic models [2] that have clear potential for processing
complex imaging datasets. The unknown probability density function (pdf) in
high-dimensional spaces from which data samples are drawn is modeled by fixing
the ML system complexity, typically the number of parameters of the neural
network architectures [3], and optimising the parameters through learning on a
training set. At the decision stage of ML systems, high dimensional features
are commonly projected onto low-dimensional sub-spaces [4] easing the learning
task and permitting inference. System performance is frequently assessed by
cross-validation (CV) techniques that divide the overall data into partitions
(i.e. folds) and then systematically train and test the ML system on each fold
averaging the resulting outcome measures. CV with limited sample sizes can
lead to large standard errors [5, 6]. No closed solution to this common
problem in data analysis has been found in the literature, although there are
several results within statistical learning theory (SLT) that could relieve
the issue [7].

Multivariate, data-driven approaches and ML systems offer an attractive
alternative with high detection rates available in several research fields,
e.g. neuroscience [8, 9, 10, 11]. Between-group analyses in neuroimaging are
typically conducted with the General Linear Model (GLM) and classical
statistics [12]. Although straightforward to interpret [13], the assumptions
of the univariate GLM are frequently violated [14] making the test more
conservative and encouraging less than optimal analytic practices. Inflated
type I error rates, p-hacking, bad experimental designs have all become
problematic affecting reproducibility and replicability across science and
engineering [15], including NI [16, 17].

Migration to ML [18, 19] does not preclude false positives and misleading
results. For example, outputs from some ML systems with multivariate features
are strongly affected by high-noise levels in the data [20, 21]. Here,
although questionable research practices such as p-hacking111the deliberate
selection of the level of significance in the seek of the desired effect are
not problematic for ML, there is a strong dependence of performance on the
construction of the learning and validation datasets [6].

There have been several commentaries in the literature [5, 16, 20, 22] about
the high variability of performance found across CV folds in several
neuroimaging data analyses with clear implications for predictive inference.
However, no quantitative solution has been given to this issue beyond
increasing sample size and warning of errors across CV folds. In the context
of small sample-sizes and heterogeneous data, CV strongly underestimates the
actual error, which can be interpreted as the violation of the assumption of
_ergodicity_ [23]; that is, the average behavior (performance) of any system
can be described from a collection of random samples (a set of random
performances in each CV loop).

What would happen if our process was not ergodic? We can make use of the
concept of a stable inducer [24] to describe the consequential effect. An
inducer is the algorithm that creates a classfier from a training set. It is
stable for a given dataset if it creates classifiers that make the same
predictions when given perturbed datasets. If the sample is strongly
perturbed, e.g. due to small-sized and heterogeneous samples drawn from
multivariate distributions, the process of learning from specific training
folds cannot be efficiently extrapolated, in statistical terms, to test folds.

Permutation analyses [25] can augment ML systems to test for statistical
significance by the computation of p-values from the distribution of a
statistic under the null hypothesis. Assuming the data is i.i.d., this null
distribution is obtained by permuting the class labels a large number of times
and training and testing the ML system each time. However, the estimation of
the statistics could be biased with heterogeneous datasets due to the use of
only a single instance of the K-folds in the CV. A high degree of confidence
in the results cannot be obtained if the essential element of these analyses,
the predictive inference based on an averaged accuracy from specific folds, is
flawed.

In this paper we propose the use of K-fold CV in combination with a
statistical test based on the analysis of the _worst case_ (upper bounding the
actual error) to estimate the probabilities of uncertainty from a sample drawn
from a population. This method, named K-fold Cross Upper Bounding Validation
(CUBV) allows control over the performance of the system. Given the empirical
K-fold CV-error we calculate the upper-bound of the actual error to assess the
degree of significance of the classifier’s performance. Upper-bounding the
actual error is a statistical learning technique based on concentration
inequalities (CI) that constructs efficient algorithms for prediction,
estimation, clustering, and feature learning. CI deal with deviations of
functions of independent random variables from their expectations; here, the
deviation of empirical errors in each training fold from the real error
achieved by the classifier given the complete dataset. This solution for
evaluating the error variability is similar to the multiple comparisons
p-value corrections, such as Bonferroni or Random Field Theory, within
hypothesis-driven methods for statistical inference[26, 27].

##  2 K-fold Cross Validation: theory and practice

Statistical inference using ML systems with neuroimaging data is usually
performed by means of predictive inference. Datasets are split into CV folds
(training and test sets) and the ability of statistical classifiers to
extrapolate from the samples in the training sets is assessed on the test set.
Performance is measured in terms of the classifier error that is then used as
a statistic to test for between-group differences; e.g. by estimating p values
from permutation testing. The degree of reliability of this approach will
depend on the sample size and the data complexity with both affecting the
quality of the learning procedure.

###  2.1 Predictive Inference and Machine Learning

Given the input and output spaces in a binary classification problem,
$\mathcal{X}$ and $\mathcal{Y}$ respectively, we observe N i.i.d. pairs
$(\mathbf{x}_{i},y_{i})\in(\mathbb{R}^{n},\\{0,1\\})$ sampled according to an
unknown $P$. We try to predict $y$ from $x$ by means of a function $f\in
F:\mathcal{X}\rightarrow\mathcal{Y}$ that is chosen minimizing the expected
loss $R(f)=\mathbb{E}(\mathcal{L}(y,f(x))=P(f(x)\neq y)$. By definition, the
K-fold CV estimated error is given by:

$R_{N}(f)=\frac{1}{N}\sum_{(x_{i},y_{i})\in D}1-\delta(f(D\backslash
D_{i}),y_{i})$ (1)

where $\delta(.)$ is the Kronecker´s delta. Typically, $K=10$ and represents a
trade-off between variance and bias of the estimation of the error. Another
common choice is $K=N$, known as the leave-one-out (LOO) CV. $D$ is the whole
dataset, and $D_{i}$, for $i=1,\ldots,K$, are the mutually exclusive folds.

As previously pointed out [24], a better estimation of this error would
include all the possible combinations $\binom{N}{\frac{N}{K}}$, although it is
usually too expensive computationally. Indeed, the variance of the error
strongly depends on $K$ if the classifier $f$ is unstable under perturbations.
This happens when deleting instances of the training folds provokes a
substantial change in the outcome, mainly due to data heterogeneity.

A major drawback in using a predictive inference approach is that the CV is
sub-optimal when performing a hypothesis test for a given _sample size_ and
threshold [28] 222Proof: the Neyman-Pearson lemma..

###  2.2 Statistical Inference with permutation analyses

A p-value estimation by means of a permutation test [29] can be conducted on
the classification results from K-fold CV of a ML system [30]. In this way,
the error of the trained ML system is the statistic used to test for between-
group differences. In a pattern classification problem we evaluate how likely
is the performance of the system using the labeled data and empirical
distribution compared to the distribution from a number, $M$, of label
permutations that simulate “no effect’, i.e. samples are i.i.d (the null
distribution) [31]. In other words, we assess the probability of the observed
value of the error under the null hypothesis by permutation $\pi$, as the
following:

$p_{value}=\frac{\\#(\mathcal{R_{\pi}}\leq\mathcal{R})}{M+1}$ (2)

where $\\#(.)$ is the number of times the error in the permutation is less or
equal to the error obtained with the observed effect. If this value is less
than our level of significance, e.g. $\alpha=0.05$, then the distribution of
data significantly differs between groups [25].

To sum up, the state-of-the-art in this field often employs only a single-
instance estimation from a particular set of CV folds and, at most, a
subsequent permutation analysis to simulate the null distribution and test for
statistical significance. However, the whole procedure, i.e. the assessment of
the true effect and the null distribution from incomplete datasets, is
substantially _biased_ if the samples are small and heterogeneous.

##  3 Performance of K-fold Cross Validation in Common Experimental Designs

The following examples illustrate the key problems found in the literature
that affect the reproducibility of ML results in neuroimaging and other
related areas. In summary, contemporary results are based on a single instance
of the pdf from the training folds as well as the selection of a particular CV
fold as the test set with which predictive inference is assessed. Thus, even
with the same datasets, the performance of one study informed by another can
differ substantially with potentially contradictory interpretations if the
data folds are not identical.

###  3.1 The null experiment – control of type I errors

Figure 1: Null distribution of accuracy values using K-fold CV (in green font)
obtained from sampling the pdf (on the left) and permuting the fold
distribution (on the right). In blue we show the proposed K-fold CUBV method
to control FP in this null experiment. Note that in this example dimension
$n=2$ and Cohen’s $d=0$.

Assume that two laboratories A and B acquire experimental data from two
populations and seek an effect in a group comparison that is actually
trivially small, i.e. dependent on the sample realization (figure 1). Both
laboratories believe that an effect exists although the patterns belonging to
both classes are drawn from the same, unknown, Gaussian distribution:
$\mathcal{N}(0,1)$. Given their sample realizations, they apply their ML
algorithms for pattern classification using a properly established CV-scheme
and obtain accuracy values. The results obtained by the two laboratories are,
of course, not identical (a different sample realization in each lab) and
_normally_ distributed around $0.5$. This expected theoretical result is
achieved if we follow the same scheme with an increasing number of
laboratories and average the set of accuracy values.

How and when does a specific laboratory reject the null hypothesis that there
is an effect? In order to test statistical significance a laboratory compares
its observed accuracy with the set of values obtained by permuting conditions
on its particular sample realization with the aim of modelling the null
distribution. Is this sufficiently accurate? What if the sample realization is
randomly providing accuracies from an overfitted solution? Even though the
null distribution is properly modelled, the effect is being tested by
_assuming the alternative hypothesis_ in a classification task, thus
overfitting could affect the outcome measure. This is the main problem of
using ML algorithms in group comparisons; one can never be assured of the
statistical significance of the averaged accuracy values obtained from folds
using empirical measures only.

What if the null distribution is not properly modelled by a particular
realization, e.g. by an insufficient number of permutations, $M$? This
situation compounds overfitting and non-reliable conclusions could be drawn
from the experiment.

What if the patterns are more complex than those drawn from one-single-mode
Gaussian pdf? In this case ML could be providing good classification results,
distinguishing sub-patterns within the same condition in imbalanced datasets.
This question is related to the assumptions, e.g. the Gaussianity and
homogeneity assumptions, that are frequently violated in random effect
analyses where the mixing proportion of the effect should be considered
instead [32].

As an illustration, K-fold CV was evaluated in a classification task
consisting of synthetic i.i.d. data drawn from two completely overlapped
Gaussian distributions, representing two groups to be compared, in $n=2$
dimensions; that is, with Cohen’s $d=0$ between the distribution centroids. We
assessed the control of False Positives (FP) in this experiment where any
effects detected are type I errors as a function of sample size.

As shown in figure 1, the experiment on the left shows a symmetric
distribution of the statistic when we sample the Gaussian pdf and directly
compare the results. On the right of 1 is the _estimated_ null-distribution of
classifier error obtained when permuting labels $F=100$ times using a single-
point realization of the sample. It is non-symmetrically distributed and
biased around $50\%$. In short, the statistical tests performed in a
permutation analysis result in excess FP (or false negatives depending on the
sign of the bias) above or below random chance.

###  3.2 Classification variability across independent (multi-sample)
experiments

Figure 2: Bottom left: example of a classification problem using linear
decision functions and samples drawn from two Gaussian pdfs with Cohen’s $d=2$
similar to the problem described in section 3.1. Bottom right: averaged
accuracy and its standard deviation versus sample size are displayed in green
font for the standard K-fold CV, $K=10$. The theoretical error achieved by
linear classifiers and the whole dataset ($2*10^{4}$ samples) in this problem
is displayed by the black line.

Assume two laboratories A and B are exploring a specific neurological
condition versus a control pattern by a classification task. Datasets obtained
from different experimental setups and signal processing pipelines may be
modelled by a fixed, but unknown pdf. Each laboratory enrols a different
cohort of participants and after data acquisition and processing they obtain
noisy i.i.d samples (theoretically) drawn from different samples of the same
pdf. We may simulate this scenario with synthetic data by randomly generating
two samples of sample size $N$ from 2-dimensional normal distributions
$\mathcal{N}(0,1)$ and $\mathcal{N}(d,1)$, where $d$ is the Cohen’s distance
between the centroids. Unfortunately, even following the same protocols,
laboratories A and B obtain different performance accuracies averaged across
CV folds.

In general, classification results obtained from K-fold CV reflects the
variability in the sample at different sample sizes, $N$, even when the effect
is large ($d=2$) as shown in figure 2. Note in this figure how the accuracy is
symmetrically distributed around the real effect (black line) and the
decreasing variability with increasing $N$, as expected. However, with a
realization of $20$ samples we readily see that the effect could be
overestimated in half of the laboratories that undertake this experiment
(values above $95\%$) or underestimated on the other half (values around
$85\%$) when the true effect is around $90\%$. It would be desirable to
provide an additional test to evaluate if under the specific experimental
conditions (sample size, number of predictors, classifier complexity, etc.)
laboratories A or B give performances that could be extrapolated elsewhere and
are close to the real effect.

###  3.3 Classification variability across CV-folds in single sample
experiments

An increasing number of international challenges and initiatives as well as
open source databases foster the common situation whereby laboratory A shares
data with B to test the same hypothesis [33, 34, 35]. Although both
laboratories have the same single realization of the sample, neither obtained
the same classification results or estimated the same effect level in data,
_if any_. This is because the selected fold distribution in the CV was
different, thus predictive accuracy in the test data and its average may be
slightly or substantially dissimilar. This scenario can be modelled in the
same manner as in the preceding examples by randomly permuting the training
and test folds $F$ times.

As an illustration, we simulated two data distributions with $d=1.57$
following the procedure shown in [36] (figure 3). We observed a “non-
symmetric” distribution of accuracy values around the real effect level with
increasing sample size. The results obtained by many laboratories are similar
(less variability than the previous example), but _biased_ mainly due to small
sample sizes and the single realization of the sample.

Figure 3: Bottom left: Example of a classification problem with a single
sample generated following the procedure described in [36] and, bottom right:
a $F=100$-folds distribution for $K=10$. See also figure 2.

Finally, data may be non-Gaussian distributed, e.g. see an example in $n=2$
dimensions in figure 3, generating imbalanced modes. In the previous examples,
we assumed that samples were drawn from one dominant mode and an almost
symmetric Gaussian pdf. Complex data following imbalanced multi-modal pdfs
[36] further increases the variability of the performance obtained at each
lab. Moreover, the size of effect strongly influences the variability of the
accuracy results, with small samples increasing variability, as shown in the
following sections; see figure 9.

##  4 The K-fold Cross Upper Bound Validation test

###  4.1 Upper Bounding the actual error under the worst-case scenario

The main goal of SLT is to provide a framework for addressing the problem of
statistical inference [37, 38]. One of its most notable achievements is to
establish simple and powerful confidence intervals for bounding the actual
risk of misclassification, $\mathcal{R}(f)$ [7]. In particular, we are
interested in the estimation of the risk from an empirical quantity with
probability at least $1-\eta$ as:

$\mathcal{R}(f_{N})\leq\mathcal{R}_{N}(f_{N})+\Delta(N,\mathcal{F})$ (3)

where $f_{N}$ is estimated to prevent overfitting 333A classifier that
perfectly predicts the labels of the training data but often fails to predict
them on the test set., e.g. restricting the class of functions with
$f_{N}=\underset{f\in\mathcal{F}}{\arg\min}\mathcal{R}_{N}(f)$, and
$\Delta(N,\mathcal{F})$ is an upper bound of the actual risk. In the worst
case, the inequality turns into an equality. This deviation or inequality can
be interpreted from several perspectives of classical probability theory [37]
in order to assess how close the sum of independent random variables
(empirical risk) are to their expectations (actual risk).

###  4.2 A Probably Approximately Correct Bayesian bound

Here, we employ one of the major advances in this field based on Probably
Approximately Correct (PAC)-Bayesian theory [39]. In particular, we evaluate a
dropout bound inspired by the recent success of dropout training in deep
neural networks. The bound depicted in equation 3 is expressed in terms of the
underlying distribution $Q$ that draws the function $f$ from the set of
“rules”, $\mathcal{F}$.

For any constant, $\lambda>1/2$, and class of linear classifiers,
$f=\omega\in\mathbb{R}^{n}$ that are selected according to the distribution
$Q$, we have that with probability at least $1-\eta$ over the draw of the
sample, the following CI hold for all the distributions $Q$ with dropout rate
$\delta$:

$\mathcal{R}(\boldsymbol{\upomega})\leq\mathcal{R}_{N}(\boldsymbol{\upomega})+\min_{1\leq
i\leq
k}\frac{1}{2\lambda_{i}-1}\left(\mathcal{R}_{N}(\boldsymbol{\upomega})+\frac{2\lambda_{i}^{2}}{N}\left(\mathcal{D}(Q,Q_{u}))+\ln\frac{k}{\eta}\right)\right)$
(4)

where $\mathcal{D}(Q,Q_{u})=\frac{1-\delta}{2}||\boldsymbol{\upomega}||^{2}$
is the Kullback-Leibler divergence from $Q$ to the uniform distribution
$Q_{u}=\mathcal{N}(0,1)^{n}$ and $\lambda\in(1/2,10)$ can take $k$ different
values [39].

###  4.3 A statistical test based on K-fold CV and upper bounding

The statistical test proposed in this section, the K-fold CUBV test,
formalizes the selection of the accuracy threshold (usually $50\%$) as the
threshold for detection of an effect in a between-group analysis. Given a
classifier with fixed complexity and number of predictors (dimensions of the
input pattern) in a small sample-size dataset, if the effect is small we
cannot guarantee statistical significance with just an empirical error.
Instead, we propose to reject the null-hypothesis if the analysis of the worst
case with at least a probability $1-\eta$ provides an upper bound that
satisfies:

$\mathcal{R}(f_{N})=\mathcal{R}_{N}(f_{N})+\Delta(N,\mathcal{F})\leq\eta$ (5)

where we use CV to estimate the empirical risk $\mathcal{R}_{N}(f_{N})$. Note
that this probability $\eta$ refers to the analysis of the worst case and is
not related to the level of significance $\alpha$ of a classical statistical
test. With an $\alpha<0.05$ only $1$ out of $20$ random effects would be
considered as a real effect by rejecting the null hypothesis. In the worst
case analysis we could soften this condition to $\eta=0.5$ as it refers to the
supremum of the deviation between actual and empirical errors [40], a value
that an efficient SLT algorithm, such as a linear Support Vector Machine (SVM)
[41], could not achieve given the sample set.

In figure 4 we collect all the ideas presented in previous illustrations of
different experimental scenarios. We show the operation of the proposed CUBV
test for detecting effects with good performance using the regular K-fold CV
and a single-point realization of the sample. Here, performance is described
in terms of variability of the error and its deviation from the actual risk.
As shown in figure 4, theoretically sampling the unknown pdf (in green font)
provides a biased estimation of the theoretical error. Moreover, using a
single realization (dash-dotted lines) does not improve the situation, but
quite the opposite especially at small sample sizes. Finally, when the
deviation obtained by the CUBV technique is in the majority above $\eta=0.5$
(blue-shaded area on the right) one can be assured that performance of a
single-point realization based K-fold CV in “laboratory 1” can be
“extrapolated” to other laboratories. Note that the size of effect (real
classification rates of around $75\%$) influences this decision.

Figure 4: Performance of K-fold CV in common experimental designs. Typical
large biobanks include data across modalities including neuroimaging,
biosensors, genetic, clinical, omics, etc. data. A set of synthetic and real
MRI samples, obtained from $N_{c}$ sources and expressed as $n$ dimensional
features, are analysed. The theoretical error achieved by a (linear)
classifier can be assessed by resubstitution on the infinite population (pdf).
Then, the K-fold CV error is estimated by (theoretically) sampling this pdf
($M$ times) or by (realistically) permuting the learning folds using a single
realization of the sample ($F$ times). The proposed CUBV test rejects the
null-hypothesis in the blue-shaded area.

##  5 Materials and Methods

###  5.1 Synthetic datasets to measure heterogeneous and multi-cluster
datasets

Current biomedical data provide multi-modal/dimensional sources of information
which result in complex and heterogeneous datasets [42, 43] (see figure 4).
Real data in between-group analyses are usually characterized by a multimodal
pdf that draws samples from several sources of variability (covariates) per
group such as sex, age, socioecomic status, genetic profiles, and so on [44]
that all influence the effect under observation [22, 46]. Assuming data from
each group is defined by a balanced set of Gaussian clusters, each cluster
representing a data source, we generate realistic data following the procedure
described in [36], or simply by generating samples given each Gaussian cluster
in a $n$-dimensional space.

SLT provides additional statistical measures to control: i) the degree of
reality of the pattern generated by these procedures (typically used in the
extant literature to evaluate models in a controlled, experimental fashion),
or simply; ii) the ability of the selected classifier complexity to perform
well in a classification task.

Heterogeneity in data is not always proportional to the number of (unknown)
sources generating samples when, for example, they “correlate” within a class.
The concept of data-(in)dependent measures of complexity of sets of functions
derived by SLT can be evaluated for this purpose [7]. In general, capacity
measures, e.g. the Rademacher average, establish a connection between
empirical errors (derived from the training set) and the unobserved
generalization error; i.e. the actual error. One of these measures is the
well-known VC dimension [37] that for the class of functions $\mathcal{F}$ of
linear classifiers is easy to compute: $h=n+1$. In a nutshell, the VC
dimension of the class of linear classifiers is the size of the largest set
that it can shatter. For example, in two dimensions we can shatter up to three
samples using linear classifiers (see figure 5).

Figure 5: Data complexity and VC dimension in $n=2$. In two dimensions the
number of non-intersecting convex hulls is, in general, $2^{h}$ for a set of
points or distant clusters with cardinality less than the Radon number ($n+2$)
[45]. Assuming balanced sources and $N_{c}>n+2$ we have only $Nc$ separable
simulations whilst the number of non-separable simulations grow with order
$\sim\frac{2^{N_{c}+1}}{\sqrt{2\pi N_{c}}}$

Assume a number, $N_{c}$, of $n$-dimensional Gaussian pdfs (clusters) from
which samples are drawn with centroids randomly distributed “far away” from
one another (non-overlapping) in the input space. The VC dimension for linear
classifiers establishes that out of $2^{N_{c}}$ different binary combinations,
linear classifiers can shatter only $2^{n+1}$ simulations. If the clusters are
grouped into balanced binary classes $\\{0,1\\}$ we have only
$\binom{N_{c}}{N_{c}/2,N_{c}/2}$ 444The multinomial coefficient or the number
of ways to put $m$ clusters into $k$ classes,
$\binom{m}{m_{1},m_{2},\ldots,m_{k}}:=\frac{m!}{m_{1}!m_{2}!\ldots m_{k}!}$,
where $m_{j}$ is the number of clusters in class $j$. different classification
simulations instead of $2^{N_{c}}$. For example, generating $6$ clusters ($3$
per class) in $n=2$ dimensions, we have $20$ different classification
simulations out of $64$. Removing the inverse simulations (inverting labels)
only $10$ remain and among them $7$ represent a problem that a linear
classifier cannot shatter (those described by multimodal pdfs or subsets with
intersecting convex hulls). This example is depicted in figures 5 and 6
(bottom). This procedure allows us to select which simulated data samples
represent a realistic data pool for subsequent simulations. In general, any
data-dependent capacity measure detects if the sample is likely to be drawn
from a multimodal pdf. In this case, we can visually/analytically identify the
number of simulations following this realistic pattern.

Figure 6: We generate realistic datasets [36] including several modes by
selecting a different number of clusters or sources generating the data
sample, $N_{c}=\\{2,4,6\\}$, and setting two balanced classes in all possible
configurations:
$\\#\text{simulations}=\frac{1}{2}\binom{N_{c}}{N_{c}/2}=\\{1,3,10\\}$.
Bottom: three separable linear simulations 1,4,10; seven non separable linear
simulations: 2, 3, 5-9.

###  5.2 Simulation of single cluster Gaussian PDFs

We generated synthetic data to model different scenarios in neuroimaging data
analysis whereby observation vectors $\mathbf{x}$ were simulated for each
group from Gaussian distributions (similar to bottom left in figure 2) with
different effect sizes ($d:0-2$), sample sizes ($N:20-500$), and dimensions
($n:1-6$). To complement the results presented in the preliminary example in
section 3, we computed the power of the k-fold CV-based permutation test and
the upper-bounding test (considered as an independent test) in these commonly
encountered experiments. We drew samples from $n=2$ Gaussian pdfs with only
two clusters (one per group) and increasing $d$. Note that the aim of this
paper is the combination of both; that is, using the CUBV technique to
validate the K-fold CV test within a permutation analysis.

###  5.3 Simulation of complex multi-cluster Gaussian PDFs

We repeated the simulations increasing the number of clusters, $N_{c}$, per
group (see figure 6) and constructing pdfs for each group from an imbalanced
number of samples drawn from each cluster. This procedure increases the
complexity of the simulation affecting the generalisation of performance of
K-Fold CV. The idea behind this simulation is quite simple: by chance the
fitted classifiers at the training stage cannot perform well on the test set
because patterns in both groups (training and test) are drawn from different
pdf modes at different ratios.

###  5.4 Neuroimaging dataset

Datasets provided by the International challenge for automated prediction of
_mild cognitive impairment_ (MCI) from MRI data
(https://inclass.kaggle.com/c/mci-prediction) were considered for the
evaluation of the proposed method in a neuroimaging context.

MRI scans were selected from the Alzheimer’s disease Neuroimaging Initiative
(ADNI, http://www.adni-info.org) and preprocessed by Freesurfer (v5.3) [33].
The dataset consisted of 429 demographic, clinical, and cortical and
subcortical MRI features for each participant. Participants were in four
groups according to their diagnostic status: healthy control (HC), AD
patients, MCI individuals whose diagnosis did not change in the follow-up, and
converter MCI (cMCI) individuals that progressed from MCI to AD in the follow-
up period. The training dataset contained $400$ ADNI individuals ($100$ HC,
$100$ MCI, $100$ cMCI and $100$ AD). Up to $20$-dimensional partial least
squares (PLS) based features were extracted from the MRI-study groups
according to [36] and then combined to provide different classification
problems with heterogeneous sources ($N_{c}=4$) by combining the resulting
groups.

###  5.5 Monte Carlo Performance Evaluation and Power Calculations

Monte Carlo (MC) simulation was employed to determine the number of trials of
a random variable, or statistic $T,$ required to achieve detection with a
given a threshold [47], e.g. $P(T>\gamma)>0.5$.

The probability that a $T$-statistic, consisting of an average of normally
distributed random variables, is greater than a threshold is equal to
$P(T>\gamma)=Q\left(\frac{\gamma}{\sqrt{\sigma^{2}/N})}\right)$, where $Q$ is
the right-tail probability and $\sigma$ is the standard deviation of the
random variable. Then, by estimating this probability using a number of $M$
trials we can evaluate the minimum value to achieve detection at a given
significance level $\alpha$ as:

$M\geq\frac{[Q^{-1}(\alpha/2)]^{2}(1-P)}{\epsilon^{2}P}$ (6)

where $\epsilon$ is the maximum deviation allowed between $P$ and its
estimation $\hat{P}$. $\hat{P}$ is computed by counting the number of times
$T$ is greater than the selected threshold in $M$ trials.

MC simulations were performed in controlled experiments to undertake power
calculations. We estimated the power of the test ($1-\beta$) across
permutations as the number of times the p-value (equation 2) was less than the
significance level, 0.05, divided by the total number of permutations. This
was done for different sample sizes under ideal conditions, i.e. mono-modal
Gaussian pdf per class, and with imbalanced and multimodal Gaussian data (see
the experimental section 6).

###  5.6 Experimental Designs

We performed an analysis of the scheme shown in figure 4 under the
experimental conditions described in the illustrated scenarios, 3. For each
experimental design, three classification methods were compared: K-fold CV and
K-fold CUBV, all with $K=10$. The experimental designs were chosen to: i)
evaluate the power of the tests; ii) assess the theoretical (ensemble) and
real (single realization) performance of the regular K-fold CV test; and iii)
carry out experiments on a MRI multiclass dataset [33].

Power calculations for the null experiment, where $d=0$ (3.1), were undertaken
to understand nominal FP control. These values were included within power
calculations at different effect sizes $d=\\{0\ldots 4\\}$ in $n=2$
dimensions, data complexity $N_{c}=\\{2,4\\}$, samples sizes $N=\\{20\ldots
500\\}$, as shown in the experimental part. To assess the probability of
detection we evaluated the MC performance of the regular K-fold CV (number of
trials required to detect the effect) and the detection ability of the test
based on CUBV.

All the experiments were carried out in the “ideal case” based on sampling the
theoretical pdf $M=\\{100,1000\\}$ times or label permuting $F=\\{100,1000\\}$
times a single realization of the sample (real case). In addition, in both
cases we generated data from single mode pdf and multimode pdf, thus
increasing the complexity of the problem. Finally, real MRI data was analysed
using this methodology to validate the findings achieved on a statistical
significant level.

##  6 Results

###  6.1 Results on the null experiment

Results for the null experiment, $d=0$, are displayed in figures 11 and 12.
Power of the K-fold CV-based permutation test was above the significance level
confirming a FP rate in some experimental setups beyond an admissible level,
figures 17 and 18. This effect is partly controlled by increasing the sample
size with a low data complexity. The K-fold CUBV, used as a statistical test,
always provided power below the significance level and thus can be considered
conservative within ML-based inference approaches.

####  6.1.1 Control of type I errors across independent (multi-sample)
experiments

In figure 7 we display the analysis of the FP rate under the null-hypothesis
with increasing dimension $n$ and sample size $N$. Performance is plotted for
all the validation methods with $M=100$ samples obtained from theoretical
Gaussian pdfs and $N_{c}=2$, the MC performance evaluation and the worst case
detection analysis. CUBV controlled type I errors in all the analysed
simulations. Note that both detection methods (MC and CUBV) yields similar
results of no effect, but the FP rate in K-fold CV is always above the nominal
value.

Figure 7: Examples of performance, FP rates and MC performance evaluation
across independent (multi-sample) experiments

####  6.1.2 Control of type I errors in single sample experiments

In figure 8 we display the standalone analysis of the FP rate under the null-
hypothesis with increasing dimension $n$ and sample size $N$. We plotted the
performance of all the validation methods with $F=100$ samples obtained from
the fold permutations of one single realization and $N_{c}=2$, the MC
performance evaluation and the worst case detection analysis. This again
highlights the ability of the K-fold CUBV to control type I errors unlike
K-fold CV whose optimistic performance is confirmed by the MC performance
evaluation (detection is achieved under the null hypothesis when colored lines
go below the blue line: 8, bottom).

Figure 8: Examples of performance, FP rates and MC performance evaluation in
single sample experiments.

###  6.2 Classification variability across independent (multi-sample)
experiments

####  6.2.1 K-fold variability vs complexity

In figure 9 we show the classification values obtained averaging $M=1000$
experiments and $\\{1,3,10\\}$ simulations using K-fold CV as a function of
the number of clusters (complexity) that generate the data, and sample size
for an up to $6$ dimensional problem. The variability of the performance
increases with complexity and sample size, although the dimensionality of the
features can partly relieve this problem (see figure scale). Unfortunately,
most of the methods usually employed in neuroimaging are univariate, typically
voxelwise, and the conditions most frequently encountered are those reflected
in the left side of figure 9. The curves in this figure are compared to the
estimated null-distribution to make an inference about the data, as shown in
figure 11.

Figure 9: The accuracy values (average and standard deviation) obtained in
K-fold CV versus complexity ($N_{c}$) and sample size $N$ with $M=1000$ and a
large effect size, in a $n=1$ (top) and $n=6$ (bottom) binary classification
task. Figure 10: The accuracy values (average and standard deviation) obtained
in CUBV versus complexity ($N_{c}$) and sample size $N$ with $M=1000$ and a
large effect size, in a $n=1$ (top) and $n=6$ (bottom) binary classification
task. Note the dashed line in blue corresponding to the random threshold.

####  6.2.2 Power and detection analysis in single mode pdf

In figure 11 we show the behaviour of both approaches, K-fold CV and CUBV,
with changes to Cohen’s d and effect size in a $n=2$ classification task.
Statistical power is computed against $d$ by evaluating the probability that
$\text{p value}<0.05$ given the set of trials, $M$.

When using the CUBV technique, the power of the test is computed by
thresholding at the level of random chance (equation 5). However, if we
compute the number of required MC trials to achieve detection with a degree of
confidence $\alpha=0.05$ we obtain an unexpected result, as shown in figure 11
on the right. The number of required MC trials to achieve detection for small
effect sizes is about $7$ times the sample size. Thus, for example, if $N=500$
then $M\sim 3700$, even under the controlled experimental conditions of this
simulation (samples drawn from a Gaussian pdf). Conversely, the CUBV technique
achieves a significant detection with just a few samples. It’s clear that
based on the information on the right of figure 11, we can validate the
results obtained with the classical ML-based permutation test using this
technique, unlike the MC evaluation as it theoretically requires many trials
to achieve detection.

####  6.2.3 Power and detection analysis in multi-mode pdf

We repeated the power analysis of the previous section ($n=2$ and $N_{c}=4$)
in the non-separable case predicted by the capacity measure (see figure 12)
assuming a balanced sample per cluster and per group (i.e. sources have the
same effect on the observed variable), and with a ratio $r=1/3$ per cluster in
both classes (imbalanced case).

Here, we readily see that the detection ability of the test using K-fold CV is
less powerful than in the “ideal” case and greater than the level of
significance with no effect ($d=0$), thus providing a FP rate greater than the
nominal value.

The MC performance is worse than in the previous case, e.g. the curve does not
decrease with sample size at values between 2 and 3 dB in the worst scenario
(balanced sample), thus the number of trials needed to achieve detection at
the level of confidence is between $7-20$ times the sample size. The CUBV
technique is expected to achieve detection with fewer samples ($(N,d)=(200,4)$
and $(300,3)$, respectively). In the imbalanced case, the benefits of the CUBV
approach are even clearer as it preserves detection ability whilst controlling
FP.

Finally, figure 13 illustrates the operation of the CUBV approach and the
regular K-fold CV against $d$ for $n=2$. Substantial control of FP is achieved
by the combination of both approaches; figure 13 top left. Optimistic or
conservative estimations of the “real effect” (small, medium and large
effects) are predicted by the CUBV method with errors below $0.5$ at small
sample sizes.

###  6.3 Classification variability across CV-folds in single sample
experiments

####  6.3.1 Performance analysis including complexity

This simulation is close to empirical studies commonly found in the
contemporary literature. Given one, and only one dataset with an increasing
number of clusters or data complexity we ran a permutation test by randomly
selecting training and testing folds repeatedly ($F=100$) under varying sample
size ($N:20-500$), dimension ($n:1-6$) and complexity ($N_{c}:2-6$) using a
prior procedure [36]. The effects considered were medium to large as shown in
figure 6.

As previously, the behaviour of the estimator of the error and its average
were not as symmetric as the ideal case, with consequences for small sample-
sizes and low dimensions, as predicted by the capacity measures detailed in
section 5.1. As an example, in figure 14 for $n=2$ and $N_{c}=4$ there was $1$
out of $3$ situations (number of cases out of different simulations that
cannot be shattered) where the estimation was biased towards underestimating
the real effect. In this case (2 out of 3), the test developed in section 4.3
did not reject the null-hypothesis (no effect) for any value of $N$. In cases
1 and 3, significant rejection is achieved above $N=50$ samples using K-fold
CUBV.

In figures 15 and 16 we repeated the same analysis carried out in section
6.2.1 with a single realisation and averaging over the set of simulations with
increasing complexity.

####  6.3.2 Power and detection analysis in single mode pdf

With the same parameter configuration at the previous section 6.2.2, we
computed the same measures using only a single realization of the sample. In
figure 17 we show the expected behaviour of both approaches, K-fold CV and
CUBV, varying Cohen’s d and sample size in a $n=2$ classification task.
Statistical power was computed by evaluating the probability that $\text{p
}<0.05$ for a set of trials, $F=100$. Surprisingly, the number of FPs for a
number of parameter configurations of the K-fold CV method was clearly
admissible even at large sample sizes.

####  6.3.3 Power and detection analysis in multi-mode pdf

With the same parameter configuration than previous section 6.2.3 we computed
the same measures using only a single realization of the sample. The power and
detection analysis with increasing complexity is shown in figure 18. The
results are in line with those shown in the previous section, where we
highlighted the ability of the proposed CUBV method to adequately control FPs.

###  6.4 Results on real MRI data

MRI datasets, described in section 5.4, were evaluated with the proposed CV
methods. Multiclass classification problems are usually divided into multiple,
separate binary classifications. In this section we analyse three binary
classification problems P1: HC+MCI vs AD+MCIc; P2: HC+MCIc vs MCI+AD; P3:
HC+AD vs MCI+MCIc, that arise by combining the conditions studied in onset and
progression of Alzheimer’s Disease (HC, AD, MCI and MCIc). Cohen’s distance
$d$ for the set of problems can be seen in figure 19. We ran $F=1000$ CV
experiments and averaged the accuracy values obtained from K-fold, LOO and the
proposed CV methods.

####  6.4.1 Real data analysis

In figure 19 we plotted the Cohen’s distance for the analysed groups. Note
that the null distribution is not properly modelled although $M=1000$
permutations were used to simulate the condition of no effect. Indeed there is
a small effect hidden in data of about $0.1$. This can be explained by the
presence of multiple clusters in each group and the similarities among them.

Data obtained after feature extraction is rather similar to that analysed in
the previous simulated examples as shown in figure 20. The presence of non-
Gaussian distributions with multiple modes represents the typical scenario
where standard CV methods are usually employed.

####  6.4.2 Mean classification variability and MC performance

In figure 21 we show the variability in the mean accuracy values obtained by
LOO and K-fold CV ($F=1000$) that are compared with a threshold of $0.5$. The
CUBV method shows an almost monotonic behaviour converging to the theoretical
error value with increasing sample size and dimension. On the other hand, in
figure 22 the MC analysis is shown for both approaches where it can readily be
seen that a larger number of trials is needed to achieve detection using the
K-fold CV method (limited to 40 dB for visualization purposes and above $M=N$
at all times) and the increased probability of detection using the proposed
approach with increasing dimension and sample size.

####  6.4.3 Power Analysis

Finally, we plotted in figures 23 and 24 the normalized cumulative sum of beta
values with increasing dimension and sample size for all the analyzed problems
(P1-3), and the null experiment. Any deviation from the slope equal to 1 line
in the K-fold CV means a detection with power less than 1, unlike CUBV that
detects an effect when accuracy exceeds $0.5$. Observe in the null experiment
that K-fold CV is mainly operating above the significance level
($\alpha=0.05$) with increasing sample size and dimension. In the latter
experiment, CUBV only performs weakly at specific values of small sample sizes
and high dimensions. Note that the null distribution is incorrectly modelled
across sample sizes.

##  7 Discussion

From the set of experiments undertaken here, we have demonstrated that the
proposed CUBV method is effective for assessing the variability of accuracy
values obtained by K-fold CV. Moreover, the evidence derived from these
experiments suggests that CUBV is a robust approach to statistical inference.

Whenever accuracy values were associated with a large deviation between actual
and empirical risks, the CUBV method did not generate significant results over
that expected by random chance; that is, the null hypothesis was not
erroneously rejected. Under the same conditions, K-fold CV provided accuracies
above and below the threshold for significant across all effects sizes where
CUBV indicated that the unseen CV accuracy values, drawn from an unknown pdf,
were expected to have alarming uncertainty. In this sense, CUBV is a trade-off
between the proper control of false positives and the power to detect true
effects of the K-fold CV test.

Classical methods to perform power calculations and evaluate the required
number of trials to achieve detection (the MC evaluation method) revealed that
K-fold CV usually operates far below its theoretical performance. In other
words, the MC method can be considered an over-conservative method and novel
complex AI approaches that leverage these methods to a great extent cannot not
be rigorously validated within these frameworks.

The theoretical findings are also applicable to the MRI-based samples from AD
patients. The models devised in this paper together with the simulation of
realistic datasets create suitable exemplars for characterising performance in
neuroimaging applications. A simple comparison between the datasets from
figure 20 with those in figures 12 and 6 reveals the similarity in the results
obtained. Nevertheless, the scatter plots and data distributions projected in
the dimensions are clear examples demonstrating that the conditions to provide
stable inducers are not met, and thus exploration of alternative validation
methods is a priority.

At the final stage of any (image) ML-based processing system classifiers learn
from folds of the limited amount of complex and multimodal samples. The
conservative nature of the CUBV method results in robust detection that shows
a monotonic behavior with sample size and feature dimension; see figure 22.
The detection ability of K-fold CV methods in the search for real effects is
arguable in the light of false positives from the null-experiment. As shown in
figure 23 their normalized cumulative sum of power shows a linear dependence
on sample size and feature dimension. Thus, increasing sample size does not
control of the risk of false positive findings, in fact quite the opposite.
One of the reasons for the poor performance is the difficulty in modelling the
null distribution, as shown in figure 19. Even with $F=1000$ label
permutations and given the “single-point” $N=400$ sample dataset, there is an
effect of about $0.1$ hidden in data that provokes a flawed statistical
analysis. It is worth mentioning that, with increasing complexity of simulated
data or when the problem to be solved is challenging, e.g. P2 and P3 using the
MRI dataset, the CUBV method is even more useful for controlling the FP rate
as shown in the experimental part.

Finally, we emphasize the relevance of highlighting negative results in order
to improve science555Please see the column in Nature about this issue
https://www.nature.com/articles/d41586-019-02960-3. Positive results are often
the main goal of any research paper and we seldom evaluate our algorithms on
putative task designs with no effect. This is the case described in section
3.1. This analysis is important because it is usually employed to
approximately model the null-distribution of the test-statistic in permutation
analyses, e.g. performance or accuracy in a classification task using ML
techniques. In permutation analysis the performance obtained from the paired
data and labels is compared to that obtained by randomly permuting the group
labels a large number of times, and should be distributed around $50\%$. If
the distribution of the performance is non-symmetric around random chance, and
is then biased, the result derived from the test data is likely to be flawed.
This would mean that the distribution of data differs between groups under the
null hypothesis, which violates the i.i.d. assumption and the estimation of
p-values could lead to incorrect conclusions at the family-wise level [25].

##  8 Conclusions

Standard CV methods in combination with ML were evaluated to ascertain whether
statistical inferences made by data-driven approaches are sufficiently
consistent. They are frequently claimed to outperform conventional statistical
approaches such as hypothesis testing. However, this improvement is based on
measures (e.g. accuracy) derived from ML classification tasks that do not have
a parametric description and depend on the experimental setup; i.e. derived
from classification/prediction folds of the dataset to establish confidence
intervals.

As shown in this paper, small sample-size datasets and learning from
heterogeneous data sources strongly influences their performance and results
in poor replication. A novel statistical test based on K-fold CV and the Upper
Bound of the actual error (K-fold CUBV) was proposed to tackle the uncertain
predictions of ML and CV methods. The analysis of the _worst case_ obtained by
a (PAC)-Bayesian upper bound for linear classifiers in combination with the
K-fold CV estimation is a robust criterion to detect effects validating
accuracy values obtained from ML models and avoiding false positives,
complementing the regular K-fold method for CV.

## Acknowledgments

This work was supported by the MCIN/ AEI/10.13039/501100011033/ and FEDER “Una
manera de hacer Europa” under the RTI2018-098913-B100 project, by the
Consejería de Economía, Innovación, Ciencia y Empleo (Junta de Andalucía) and
FEDER under CV20-45250, A-TIC-080-UGR18, B-TIC-586-UGR20 and P20-00525
projects.

## References

  * [1] Y. LeCun et al. Deep learning. Nature 521, 436–444 (2015).
  * [2] MT Ribeiro, et al. Model-agnostic interpretability of machine learning arXiv preprint arXiv:1606.05386. 2016.
  * [3] P Grohs, et al. Mathematical Aspects of Deep Learning. Cambridge University Press. ISBN 9781009025096. https://doi.org/10.1017/9781009025096.
  * [4] L.van der Maaten et al. Visualizing Data using t-SNE. Journal of Machine Learning Research 2008 vol 9, num 86, 2579–2605.
  * [5] G. Varoquaux. Cross-validation failure: Small sample sizes lead to large error bars. NeuroImage 180 (2018) 68-77.
  * [6] Gorgen, K., et al. The same analysis approach: Practical protection against the pitfalls of novel neuroimaging analysis methods. NeuroImage, 180, 19-30. 2018.
  * [7] S. Boucheron et al. Concentration Inequalities: A Nonasymptotic Theory of Independence ISBN: 9780199535255 Oxford University Press
  * [8] J.Mouro-Miranda, et al. Classifying brain states and determining the discriminating activation patterns: Support vector machine on functional MRI data. NeuroImage, 28, 980-995. (2005).
  * [9] Y. Zhang et al. Multivariate lesion-symptom mapping using support vector regression. Hum Brain Mapp. 2014 Dec;35(12):5861-76.
  * [10] JM Gorriz, et al. A connection between pattern classification by machine learning and statistical inference with the General Linear Model. IEEE Journal of Biomedical and Health Informatics 2021.
  * [11] JM Gorriz, et al. A hypothesis-driven method based on machine learning for neuroimaging data analysis. Neurocomputing Volume 510, 21 October 2022, Pages 159-171
  * [12] K.J.Friston, et al. Statistical Parametric Maps in functional imaging: A general linear approach Hum. Brain Mapp. 2:189-210 (1995)
  * [13] K.J.Friston, et al. Classical and Bayesian inference in neuroimaging: theory NeuroImage, 16 (2) (2002), pp. 465-483
  * [14] J.D. Rosenblatt, et al. Revisiting multi-subject random effects in fMRI: Advocating prevalence estimation. NeuroImage 84 (2014): 113-121.
  * [15] National Academies of Sciences, Engineering, and Medicine. (2019). Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303.
  * [16] A.Eklund, et al. Cluster failure: Inflated false positives for fMRI. Proceedings of the National Academy of Sciences Jul 2016, 113 (28) 7900-7905.
  * [17] S. Noble, et al. Cluster failure or power failure? Evaluating sensitivity in cluster-level inference. NeuroImage, 209, 116468,2020.
  * [18] Z Wang, et al. Support vector machine learning-based fMRI data group analysis. NeuroImage 36 (4), 1139-1151. 2007
  * [19] Z Wang. A hybrid SVM–GLM approach for fMRI data analysis. Neuroimage 46 (3), 608-615. 2009.
  * [20] Jollans L,et al. Quantifying performance of machine learning methods for neuroimaging data. Neuroimage. 2019 Oct 1;199:351-365.
  * [21] M.J. McKeown et. al. Independent component analysis of functional MRI: what is signal and what is noise? Curr Opin Neurobiol. 2003 Oct; 13(5): 620–629.
  * [22] J.M.Górriz, et al. A Machine Learning Approach to Reveal the NeuroPhenotypes of Autisms. International journal of neural systems, 1850058. 2019.
  * [23] G. Gallavotti. Ergodicity, ensembles, irreversibility in Boltzmann and beyond Springer March 1995 Journal of Statistical Physics 78(5):1571-1589
  * [24] R. Kohavi. A study of cross-validation and bootstrap for accuracy estimation and model selection. International Joint Conference on Artificial Intelligence (IJCAI), pp 1–7, 1995.
  * [25] B. Phipson et al. Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology: Vol. 9: Iss. 1, Article 39. (2010)
  * [26] R.S.J. Frackowiak, et al. Human Brain Function (Second Edition). Chap. 44. Introduction to Random Field Theory. ISBN 978-0-12-264841-0 Academic Press. 867-879, 2004.
  * [27] T.E.Nichols. Multiple testing corrections, nonparametric methods, and random field theory. NeuroImage 62 (2012) 811-815
  * [28] K.J. Friston. Sample size and the fallacies of classical inference. NeuroImage 81 (2013) 503–504
  * [29] E T Bullmore et al. Global, voxel, and cluster tests, by theory and permutation, for a difference between two groups of structural MR images of the brain IEEE Trans Med Imaging (1999) Jan;18(1):32-42.
  * [30] P.T. Reiss, et al. Cross-validation and hypothesis testing in neuroimaging: an irenic comment on the exchange between Friston and Lindquist et al. Neuroimage. 2015 August 1; 116: 248-254
  * [31] C. Jimenez-Mesa et al. A non-parametric statistical inference framework for Deep Learning in current neuroimaging. Information Fusion Volume 91, March 2023, Pages 598-611.
  * [32] J.D. Rosenblatt, et al. Better-than-chance classification for signal detection. Biostatistics (2016).
  * [33] A. Sarica, et al. A machine learning neuroimaging challenge for automated diagnosis of Alzheimer’s disease. Editorial on special issue: Machine learning on MCI, vol 302, Journal of Neuroscience Methods. 2018.
  * [34] C.C.Jack,Jr. ,et al. NIA-AA Research Framework: Toward a biological definition of Alzheimer’s disease. Alzheimers Dement. 2018 Apr; 14(4): 535?562.
  * [35] J.M.Gorriz, et al. Artificial intelligence within the interplay between natural and artificial computation: Advances in data science, trends and applications. Neurocomputing Volume 410, 14 October 237-270 2020.
  * [36] J.M.Górriz, et al. On the computation of distribution-free performance bounds: Application to small sample sizes in neuroimaging. Pattern Recognition 93, 1-13, 2019.
  * [37] V. Vapnik. Estimation dependencies based on Empirical Data. Springer-Verlach. 1982 ISBN 0-387-90733-5
  * [38] D. Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation Volume 100, Issue 1, September 1992, Pages 78-150
  * [39] D. McAllester, A PAC-Bayesian Tutorial with A Dropout Bound. arXiv 10.48550/ARXIV.1307.2118,2013
  * [40] J.M.Gorriz, et al. Statistical Agnostic Mapping: A framework in neuroimaging based on concentration inequalities. Information Fusion Volume 66, February 2021, Pages 198-212
  * [41] C.J.C Burges. A tutorial on support vector machines for pattern recognition Data Mining and Knowledge Discovery, 2 (2) (1998), pp. 121-167
  * [42] Zhang YD, et al. Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation. Inf Fusion. 2020 Dec;64:149-187.
  * [43] J.N. Acosta et al. Multimodal biomedical AI. Nat Med 28, 1773–1784 (2022).
  * [44] C.S. Hyatt et al. The quandary of covarying: A brief review and empirical examination of covariate use in structural neuroimaging studies on psychological variables. Neuroimage 205, 116225
  * [45] H. Tverberg, A Generalization of Radon’s Theorem, Journal of the London Mathematical Society, Volume s1-41, Issue 1, 1966, Pages 123-128.
  * [46] M. Leming, et al. Ensemble Deep Learning on Large, Mixed-Site fMRI Datasets in Autism and Other Tasks. M Leming, International Journal of Neural Systems. Vol. 30, No. 07, 2050012. 2020.
  * [47] S.M. Kay. Fundamentals of Statistical Signal Processing: Detection theory. Prentice-Hall PTR, 1998 013504135X, 9780135041352.

Figure 11: Analysis of the ideal case. Top-up: statistical power of K-fold and
CUBV (top-down) CV permutation tests; bottom-up: MC performance evaluation and
CUBV detection (bottom-down). Figure 12: Analysis of the non-ideal case
($N_{c}=4$ and $n=2$). Left column: samples and $d$ values -top-; statistical
power of CV permutation tests - middle up K-fold CV, middle-down CUBV; MC
performance of K-fold CV -bottom up- and CUBV detection -bottom down- using a
balanced dataset. Right column: the same measures using an imbalanced sample
with $r=1/3$ per cluster in each group. Figure 13: Distribution of accuracy
values ($M=100$) vs. sample size and $d$ for the non-ideal case. We show a
$n=2$ classification problem sampling from $N_{c}=4$ Gaussian pdfs ($2$ per
cluster) using an imbalanced dataset ($r=1/3$) and $d=\\{0,1,2,4\\}$. Note the
biased regions within the green area.

Figure 14: Examples of classical permutation tests based on regular CV and
CUBV decisions depending on sample variability and classifier complexity. Note
that whenever there is a clear deviation between the green and black lines or
a high variability in K-fold CV is obtained, the CUBV decision is below $0.5$
and thus the test does not reject the null hypothesis. This decision is
warning that the green curve values obtained cannot be extrapolated to other
experiments thus the analysis by another laboratory could be providing
substantially different outcomes, specially in problem 2 out of 3. Figure 15:
The analysis depicted in figure 9 is replicated here using a single
realization. Observe the theoretical accuracy highlighted by the blue line
Figure 16: The same analysis as in figure 10 using a single realisation Figure
17: Examples, power and detection analysis in single-mode pdf using a single
sample realization Figure 18: Examples, power and detection analysis in multi-
mode pdf using a single sample realization Figure 19: Cohen’s distance
obtained from binary classes (whole datasets) versus dimension (PLS features).
Note that Problem 1: HC+MCI vs AD+MCIc; Problem 2: HC+MCIc vs MCI+AD; Problem
3: HC+AD vs MCI+MCIc. The null distribution is also plotted and was obtained
with $1000$ label permutations. Note that with this sample even the null
distribution has a small (incorrectly modelled) effect. Figure 20: Examples of
data analyzed in this section with several dimensions and problems. Each
classification problem corresponds with a different Cohen’s $d$, whilst for
dimension $n>2$ only the first two vector components are displayed. Note the
similarities of the resulting samples, after feature extraction, with
simulated data. Figure 21: Accuracy values for selected CV methods as a
function of number of dimensions. Note the CUBV technique alerts on the
variability of the results obtained by LOO and K-fold CV, especially in small
effect and sample sizes, and low dimensions. With increasing sample size
estimations, classical methods converge to the theoretical error rate
(obtained by resubstitution of the whole dataset in the same manner as with
simulated data). Figure 22: Top: MC evaluation of K-fold CV in real datasets
by averaging the results of Problems 1, 2 and 3. Note that values are limited
to 40 db for visualization purposes. Bottom: detection analysis of the CUBV
method versus sample size ($N$) and feature dimension ($n$). Figure 23:
Normalized cumulative sum of ($1-\beta$) values
($P_{c}:=\frac{\sum_{j=1}^{(N,n)}(1-\beta_{j})}{\\#\text{experiments}}$) for
$N$ and $n$ versus dimension/sample size, respectively. Top: null experiment;
bottom: Problem 1 Figure 24: Normalized cumulative sum of ($1-\beta$) values
for $N$ and $n$ versus dimension/sample size, respectively. Top: Problem 2;
bottom: Problem 3.