#  Adapting Machine Learning Diagnostic Models to New Populations Using a
Small Amount of Data: Results from Clinical Neuroscience

Rongguang Wang  Department of Electrical and Systems Engineering, University
of Pennsylvania Center for AI and Data Science for Integrated Diagnostics,
University of Pennsylvania  Center for Biomedical Image Computing and
Analytics, University of Pennsylvania  Guray Erus  Center for AI and Data
Science for Integrated Diagnostics, University of Pennsylvania  Center for
Biomedical Image Computing and Analytics, University of Pennsylvania  Pratik
Chaudhari*† Department of Electrical and Systems Engineering, University of
Pennsylvania Department of Computer and Information Science, University of
Pennsylvania  Christos Davatzikos111Corresponding authors. Email:
<EMAIL_ADDRESS><EMAIL_ADDRESS>Address:
200 S 33rd Street, Department of Electrical and Systems Engineering,
University of Pennsylvania, Philadelphia, PA 19104, USA; 3700 Hamilton Walk,
7th Floor, Center for Biomedical Image Computing and Analytics, University of
Pennsylvania, Philadelphia, PA 19104, USA. These authors contributed equally
to this work. Department of Electrical and Systems Engineering, University of
Pennsylvania Center for AI and Data Science for Integrated Diagnostics,
University of Pennsylvania  Center for Biomedical Image Computing and
Analytics, University of Pennsylvania  Department of Radiology, Perelman
School of Medicine, University of Pennsylvania  222For the iSTAGING [1] and
PHENOM [2] consortia, and for the ADNI [3].

###### Abstract

Machine learning (ML) has shown great promise for revolutionizing a number of
areas, including healthcare. However, it is also facing a reproducibility
crisis, especially in medicine. ML models that are carefully constructed from
and evaluated on a training set might not generalize well on data from
different patient populations or acquisition instrument settings and
protocols. We tackle this problem in the context of neuroimaging of
Alzheimer’s disease (AD), schizophrenia (SZ) and brain aging. We develop a
weighted empirical risk minimization approach that optimally combines data
from a source group, e.g., subjects are stratified by attributes such as sex,
age group, race and clinical cohort to make predictions on a target group,
e.g., other sex, age group, etc. using a small fraction (10%) of data from the
target group. We apply this method to multi-source data of 15,363 individuals
from 20 neuroimaging studies to build ML models for diagnosis of AD and SZ,
and estimation of brain age. We found that this approach achieves
substantially better accuracy than existing domain adaptation techniques: it
obtains area under curve greater than 0.95 for AD classification, area under
curve greater than 0.7 for SZ classification and mean absolute error less than
5 years for brain age prediction on all target groups, achieving robustness to
variations of scanners, protocols, and demographic or clinical
characteristics. In some cases, it is even better than training on all data
from the target group, because it leverages the diversity and size of a larger
training set. We also demonstrate the utility of our models for prognostic
tasks such as predicting disease progression in individuals with mild
cognitive impairment. Critically, our brain age prediction models lead to new
clinical insights regarding correlations with neurophysiological tests. In
summary, we present a relatively simple methodology, along with ample
experimental evidence, supporting the good generalization of ML models to new
datasets and patient cohorts.

Keywords: distribution shift, domain generalization, domain adaptation,
neurological disorder, MRI

##  1 Introduction

Machine learning has been adopted by the medical imaging research community
for neurological disorders such as Alzheimer’s disease [4, 5], schizophrenia
[6], major depression [7], and autism [8] diagnosis, brain cancer
prognostication [9] and brain age estimation [10, 11]. Automated diagnostic
models in these domains hold a lot of promise, as they achieve, or even
surpass, human expert-level accuracy on average [12]. Importantly, in many
contexts they can provide prognostication, precision diagnostics and disease
subtyping, and therapy guidance in ways not feasible using human expert
interpretations. Finally, they are relatively inexpensive to deploy, therefore
supplementing human expert clinicians with these models might help reducing
the wide healthcare disparities in our society today. However, this promise
has been tempered by the difficulty in building effective predictive models
that work robustly and accurately across broad populations. The key reason is
that clinical data is highly heterogeneous. For neurological disorders such as
Alzheimer’s disease, the heterogeneity stems not only from diverse anatomies,
overlapping clinical phenotypes, or genomic traits of different subjects, but
also from operational, demographic and social aspects such as data acquisition
devices and protocols of different hospitals [13, 14], and paucity of data for
minorities [15]. As a consequence, machine learning models often exhibit poor
reproducibility in this context [16, 17, 18, 19, 20, 21, 22, 23, 24, 25].

Existing machine learning methods cannot adequately address such problems
primarily due to the commonly adopted one-fits-all paradigm. In particular,
learning salient features from data, which is very central to machine
learning, does not work for problems with heterogeneous data, especially when
the learned models are to be applied to conditions involving domain shifts,
such as scanner differences and demographically or genetically different
populations [15, 26]. Every method makes trade-offs, e.g., learning
representations that are invariant to the variability in the data can have
poor predictive accuracy as a trade-off for their predictions being robust
across the population [27]; transfer, meta or self-supervised learning
techniques offer the promise of building representations that can be adapted
to new domains easily but, in practice, these techniques are often ineffective
because the adaptation procedure for complicated neural architectures and
hyper-parameter regimes can be fragile [28, 29, 30, 31, 32]. In order to
develop effective automated diagnostic tools that can provide accurate and
robust decisions across broad populations, we need to address the inherent
heterogeneity in the data at different levels of granularity.

In this paper, we focus on an approach to imaging domain adaptation, by
leveraging large, diverse training sets but also providing a relatively simple
and easily implementable mechanism for adapting trained models to new
conditions using only few new datasets. We test this approach in the context
of three applications: diagnosis of Alzheimer’s disease (AD) and schizophrenia
(SZ), and prediction of brain age, which is a frequently used marker for
general brain health. We use multi-source data (magnetic resonance imaging
features, demographic, clinical variables, genetic factors, and cognitive
scores) of 15,363 individuals from several large-scale multi-modal
neuroimaging datasets. We focused on building automated diagnostic models that
are trained using data from a source group, e.g., subjects stratified into
groups based on attributes such as sex, age group, race and clinical cohort,
and ensuring that these models can make accurate predictions on another target
group, e.g., other sex, age group etc. As a baseline, we first show that
ensembles built using bagging, boosting and stacking and trained using careful
data pre-processing, hyper-parameter tuning and model selection, can predict
accurately on target groups, which is significantly better than deep neural
networks trained on the same data; these results are also substantially better
than existing domain generalization techniques on all three problems. Then, we
develop a theoretical argument for a weighted empirical risk minimization
(ERM) objective that optimally combines all data from the source group and a
small fraction of the data from the target group (in practice, as small as
10%), to further improve the predictive ability on target groups. This
weighted-ERM approach can be thought of as the simplest possible instantiation
of few-shot learning, i.e., adapting a model trained on a source group to a
new group using few samples. Surprisingly, the performance of this approach is
often comparable to, and in many cases it is even better than, that of
training on _all_ data from the target group. Overall, on all target groups,
our approach obtains area under curve greater than 0.95 for AD classification,
area under curve greater than 0.7 for SZ classification and mean absolute
error less than 5 years for brain age prediction, while offering brain age
residuals that carry clinical value.

We further show that our diagnostic models can be used for predictive tasks,
e.g., our models trained only for AD classification can be used to predict
which subjects with mild cognitive impairment will progress to AD, remain
stable, or revert to cognitively normal, in future follow-ups. Improvements in
AD classification performance on the target group using a small amount of data
also result in improvements in performance on these predictive tasks. In part
due to the improved predictive ability of our brain age model, our results
offer new evidence about the correlation between the brain age residual and
neurophysiological tests. In particular, we found stronger correlations of
brain age residuals with mini-mental state examination, digital symbol
substitution test, digit span forward/backward, and trial making tests than
those reported in the literature. In contrast to results in existing
literature, we found that correlations between brain age residuals,
cardiovascular features and lifestyle factors such as systolic and diastolic
blood pressure, and body mass index were not statistically significant,
potentially reflecting the fact that in this study we did not use brain
measurements that more directly reflect effects of cardiovascular risk
factors, such as infarcts and microbleeds.

Our results suggest that one can build machine learning models that are able
to effectively ameliorate the heterogeneity of neuroimaging data and also
provide accurate predictions on diverse groups of populations, even if some of
them are very under-represented (data disadvantaged). Furthermore, this might
indicate that diagnostic models of neurological disorders may be ready to be
evaluated in clinical settings—after necessary checks and balances are met.

##  2 Results

###  2.1 Quantifying the differences in the distribution of data from
different groups

Table 1: Summary of the data from the iSTAGING consortium (Alzheimer’s disease
and brain age) and the PHENOM consortium (schizophrenia) used in this study.

Alzheimer’s |  | ADNI-1 | ADNI-2/3 | PENN | AIBL | OASIS | Total
---|---|---|---|---|---|---|---
Disease |  | (17.37%) | (23.29%) | (25.44%) | (10.07%) | (23.82%) |
Subjects |  |  |  |  |  |  |
| Control | 173 | 261 | 228 | 119 | 276 | 1057
| Patient | 191 | 227 | 305 | 92 | 223 | 1038
Sex (%) |  |  |  |  |  |  |
| Female | 8.31 | 11.93 | 16.28 | 6.16 | 12.70 | 55.37
| Male | 9.07 | 11.36 | 9.16 | 3.91 | 11.12 | 44.63
Age (%, years) |  |  |  |  |  |  |
| 0–65 | 1.15 | 2.67 | 4.82 | 1.38 | 4.92 | 14.94
| 65–70 | 1.48 | 6.21 | 5.35 | 1.96 | 5.11 | 20.10
| 70–75 | 5.54 | 5.30 | 5.20 | 2.86 | 4.53 | 23.44
| 75–80 | 4.96 | 5.25 | 5.01 | 1.96 | 4.82 | 22.00
| > 80 | 4.25 | 3.87 | 5.06 | 1.91 | 4.44 | 19.52
Race (%) |  |  |  |  |  |  |
| White | 16.13 | 12.94 | 19.14 | 5.58 | 19.57 | 73.37
| Black | 0.91 | 0.67 | 5.11 | - | 4.06 | 10.74
| Asian | 0.24 | 0.43 | 0.48 | - | 0.19 | 1.34

Schizophrenia |  | Penn | China | Munich | Utrecht | Melbourne | Total
---|---|---|---|---|---|---|---
|  | (22.28%) | (13.94%) | (29.64%) | (20.12%) | (14.03%) |
Subjects |  |  |  |  |  |  |
| Control | 131 | 76 | 157 | 115 | 84 | 563
| Patient | 96 | 66 | 145 | 90 | 59 | 456
Sex (%) |  |  |  |  |  |  |
| Female | 11.87 | 6.77 | 7.75 | 6.97 | 4.02 | 37.39
| Male | 10.40 | 7.16 | 21.88 | 13.15 | 10.01 | 62.61
Age (%, years) |  |  |  |  |  |  |
| 0–25 | 5.79 | 4.91 | 9.42 | 10.11 | 5.89 | 36.11
| 25–30 | 6.28 | 2.36 | 7.26 | 4.12 | 2.16 | 22.18
| 30–35 | 3.53 | 2.16 | 5.99 | 2.85 | 1.37 | 15.90
| > 35 | 6.67 | 4.51 | 6.97 | 3.04 | 4.61 | 25.81
Race (%) |  |  |  |  |  |  |
| Native | 10.50 | - | - | - | - | 10.50
| Asian | 7.36 | - | - | - | - | 7.36

Brain Age |  | BIOCARD | BLSA-1.5T | BLSA-3T | CARDIA | SHIP | SPRINT | UKBB | WHIMS | WRAP | lookAHEAD | Total
---|---|---|---|---|---|---|---|---|---|---|---|---
|  | (2.12%) | (1.05%) | (7.82%) | (5.97%) | (28.54%) | (5.54%) | (34.79%) | (9.40%) | (2.09%) | (2.68%) |
Subjects |  |  |  |  |  |  |  |  |  |  |  |
| Control | 246 | 122 | 907 | 693 | 3311 | 643 | 4036 | 1090 | 242 | 311 | 11601
Sex (%) |  |  |  |  |  |  |  |  |  |  |  |
| Female | 1.31 | 0.46 | 4.37 | 2.97 | 14.63 | 1.96 | 18.12 | 9.40 | 1.46 | 1.91 | 56.56
| Male | 0.81 | 0.59 | 3.45 | 3.01 | 13.91 | 3.59 | 16.67 | - | 0.63 | 0.78 | 43.44
Age (years) |  |  |  |  |  |  |  |  |  |  |  |
| Mean | 57.61 | 67.87 | 64.06 | 51.00 | 52.81 | 68.58 | 62.81 | 69.59 | 63.57 | 58.05 | 60.14
| Min | 21 | 48 | 22 | 42 | 21 | 50 | 45 | 64 | 50 | 44 | 21
| Max | 86 | 85 | 92 | 61 | 90 | 91 | 79 | 79 | 78 | 74 | 92
Race (%) |  |  |  |  |  |  |  |  |  |  |  |
| White | 2.09 | 0.96 | 5.15 | 3.35 | - | 3.62 | 33.79 | 8.63 | 1.99 | 1.96 | 61.54
| Black | 0.02 | 0.09 | 1.97 | 2.62 | - | 1.74 | 0.21 | 0.40 | 0.04 | 0.60 | 7.69
| Asian | - | - | 0.49 | - | - | 0.05 | 0.41 | 0.13 | 0.01 | - | 1.09

We focused our analysis on groups stratified by four attributes: sex, age-
group, race and clinical study, for three different applications:
classification of Alzheimer’s disease, classification of schizophrenia, and
prediction of brain age. For all three problems, we used multi-source data
including imaging measures, demographic and clinical variables, genetic
factors, and cognitive scores. Table 1 summarizes the data used in our study.
For a number of groups, there is a large imbalance in the number of subjects,
e.g., in the iSTAGING consortium for AD, 73.37% of the subjects are European
Americans (White), 10.74% African Americans (Black), and only 1.34% are
Asians; for the PHENOM dataset for SZ, 37.4% participants are Female while the
rest (62.6%) are Male. Before we develop methods to adapt machine learning-
based diagnostic models to subjects from different groups, in order to
understand the data better, we developed a procedure based on a two-sample
test to quantify the distribution shift between data from different groups.
This involves building a multi-layer perceptron (MLP)-based model that
classifies subjects as belonging to different groups corresponding to each of
the four attributes (e.g., sex: male vs. female, age: < 65 vs. 65–70 vs. etc.,
race: White vs. Black vs. Asian, and clinical study: ADNI-1 vs. ADNI-2/3 vs.
PENN vs. etc.). The empirical two-sample test statistic on these learned
features can be used to quantify the differences in the data distribution
across groups; we use the maximum-mean discrepancy (MMD) [33] and elaborate
upon it in Section 4.4. This procedure is conducted independently for each of
the three applications.

(a)

(b)

(c)

Figure 1:  Automated and robust diagnosis of neurological disorders using
machine learning models. (a) A schematic of the framework for data pre-
processing, model development, optimization, and evaluation employed in this
paper to build machine learning models that can predict accurately on
different groups for heterogeneous neurological disorders using MR images,
demographic and clinical variables, genetic factors, and cognitive scores. (b)
Distance between leaves of this dendrogram indicates the pairwise MMD
statistic between learned features of pairs of groups, e.g., distributional
discrepancy between Male-Female groups is 0.17, while the distributional
discrepancy between < 65 years and > 80 years, or between ADNI-1 and ADNI-2/3,
is larger (0.42 and 0.26 respectively). See Section 4.4 for details of the MMD
calculation. Figs. 1(a) and 1(b) provide more details of the numerical
statistics. (c) Average AUC of Alzheimer’s disease classification for sex and
age attributes computed using five-fold nested cross-validation; see Fig. S.1
for other attributes. For both sex and age, we trained machine learning
models, a deep neural network (translucent markers) and an ensemble using
boosting, bagging and stacking (bold markers), using data from different
source groups (different colors) and evaluated this model (cross marks) on
data from different target groups (X-axis); circles denote model fitted using
our $\alpha$-weighted ERM procedure with access to 10% data from the target
group; horizontal lines denote models that are directly trained on the target
group using 80% of data (the rest for testing). All models use data from
multiple sources, namely structural measures, demographic, clinical variables,
genetic factors, and cognitive scores. In general, (i) the AUC of ensemble
models is higher than that of the neural network in all cases ($p$ < 0.01),
(ii) AUC of a model trained on a source group remains remarkably high when
evaluated on the target group (crosses), (iii) in most cases, it further
improves when one has access to a small fraction of data from the target group
(circles are higher than crosses), and (iv) often times even beyond the AUC of
a model directly trained on the target group (circles above the horizontal
lines).

In Fig. 1(b), for each attribute, we have shown a dendrogram obtained using
the pairwise MMD statistic [33] across different groups. For all attributes,
the hypothesis that different groups have the same distribution does not hold
($p<10^{-4}$). In our visualization, the angular distance between groups is
proportional to the MMD statistic which quantifies the difference between
their data distributions. Specifically, in the Alzheimer’s disease data, the
difference between distributions of features of Female and Male subjects has a
large MMD statistic (0.17). Broadly, the distance between distributions of
different age groups increases gradually as the age gap grows. Distribution
shifts due to different ethnicities and clinical studies are smaller than
those across sex and age groups. Figs. 2(b) and 3(b) show similar dendrograms
for schizophrenia and brain age data. For schizophrenia data, there is a large
difference in the distribution for groups stratified by sex and clinical study
while the differences in the distributions of groups due to age is relatively
small. For brain age data, there is a large heterogeneity in the data
corresponding to subjects from different clinical studies (e.g., SHIP and
BLSA-3T are quite different, but UKBB and CARDIA are quite similar); in
general, WHIMS and SHIP data are different from all other clinical studies
considered here. Our technique to evaluate the MMD statistic between two
distributions is a precise way to understand the differences between data from
different groups. It also provides context while interpreting the results, as
we will do often in the sequel. If two groups have similar data distributions,
then—for a well-trained model—we should expect a small deterioration when it
is trained on one group and evaluated on the other; and we should similarly
expect small improvements when such a model is adapted using a few samples
from the target group.

###  2.2 Even if machine learning models can predict very accurately on the
groups that they were trained on, they do not generalize as well to data from
other groups

We trained machine learning models using data from each group in our study
using five-fold nested cross-validation and evaluated these models on data
from other groups in the population. We report area under the receiver
operating curve (AUC) for classification tasks (AD and SZ) and mean absolute
error (MAE) for regression tasks (brain age prediction). We have developed
elaborate data pre-processing techniques and performed extensive hyper-
parameter search and model selection while fitting machine learning models,
which are detailed in Section 4, in particular Sections 4.3 and 4.2. As a
sanity check, for all three problems, neural networks trained on this pre-
processed data can predict accurately on all groups, with accuracy varying
depending on the task’s difficulty; for AD the AUC is at least 0.92, for SZ
the AUC is at least 0.7, for brain age prediction the MAE is at most 5.58
years. These results are slightly better than existing results in the
literature [34, 6, 35].

Comparing the above numbers to the translucent markers in Fig. 1(c), there is
a drop in the performance when neural networks trained on data from one group
using the same training procedure are evaluated on other groups. For example,
in AD classification, the neural network trained on Males achieves 0.928 ±
0.001 AUC on Females, which is less than training directly on Females (0.942 ±
0.010) at a significance level $p=4\times 10^{-4}$. Training only on subjects
with age $\leq$ 65 years has much higher AUC (0.935 ± 0.019) compared to
training on $\geq$ 80 years old (0.825 ± 0.013) and evaluating on $\leq$ 65
year olds; $p=1.3\times 10^{-5}$. For all three tasks, for all attributes, we
find that when a model trained on data from another group is evaluated on data
from a target specific group, it achieves a worse performance as compared to
training on data from only the target group ($p<0.01$). There are five
exceptions: Asian as target group in AD, and Female, 30–35 years age, Native
Americans and Asian as target group in SZ. This suggests that even if, with
appropriate pre-processing and model selection, neural networks trained on
multi-source data can predict accurately, they do not generalize to data from
groups that are different from the ones that were used for training.

We next constructed ensembles of machine learning models (neural networks and
different kinds of random forests) using classical techniques such as bagging,
boosting and stacking. For all three problems, for all groups, our ensembles
(bold markers in Figs. 1(c) and 1(c)) predict more accurately than the
corresponding neural networks (translucent markers) ($p<10^{-3}$). For AD
classification, the ensemble trained on Males has an AUC of 0.957 ± 0.001 on
Females whereas the neural network trained has an AUC of 0.928 ± 0.001; in
some cases the differences are extremely large, for example, models trained on
Asian show AUC 0.793 ± 0.003 (ensemble) and 0.365 ± 0.003 (neural net) on
White separately. Similarly, for SZ classification (Fig. 2(c)), the neural
network trained directly on data from Females has 0.620 ± 0.096 AUC whereas
the ensemble has an AUC of 0.740 ± 0.015; the ensemble (0.674 ± 0.068)
predicts more accurately than the neural network (0.567 ± 0.057) for Asians.
However, the ensemble also does not generalize to groups outside of the
training set. For brain age prediction (Fig. 3(c)), the ensemble trained on
Asians predicts less accurately (6.24 ± 0.0 MAE) on Whites, compared to
directly training the model on the target sub–group (3.99 ± 0.1 MAE).
Similarly, the ensemble trained on data from the WHIMS study has a much
smaller error (2.67 ± 0.1 MAE) than that of a model trained on data from
CARDIA study (15.91 ± 0.0 MAE). This suggests that even if ensembles can lead
to improvements in predictive performance as compared to neural networks,
their AUC still deteriorates when evaluated on data from groups outside of the
training set.

###  2.3 Generalization to groups outside of the training set can be improved
with access to a small amount of labeled samples

There are many methods in the current literature to enable machine learning
models to generalize better to data outside of the training set. These methods
are based on the idea that one can learn representations that are invariant to
the group and thereby, ensuring that predictions of the model are robust to
different groups [36, 37, 38, 39, 40]. When data is heterogeneous, it is
difficult to learn representations that are invariant and yet have good
predictive ability. Indeed it has been argued before that the accuracy of
output predictions of an invariant representation is limited by the domain
with the least amount of information to begin with [27].

We have developed a mathematical argument in Section 4.5 that elucidates
inevitable trade-offs in the predictive ability when invariant representations
are learned using heterogeneous data. It has two key implications. First, if
we seek good performance on the target group, it is necessary to explicitly
train the model using a small number of labeled samples from the target group
in addition to data from the source group. Second, in order to effectively
adapt the model to new data from the target group, we also need access to the
data that it was trained upon, i.e., data from the source group; just having
parameters of the pre-trained model is not sufficient. Based on this theory,
we have developed a method that uses a weighted-ERM objective to adapt to new
groups. Suppose we have $n$ labeled samples from the target group and $m$
samples from the source group. We construct a joint training set
$\left\\{(x_{i},y_{i})\right\\}_{i=1}^{m+n}$ and fit a model with parameters
$\theta$ to minimize the weighted objective:

$-\frac{(1-\alpha)}{m}\sum_{i=1}^{m}\log p_{\theta}(y_{i}\mid
x_{i})-\frac{\alpha}{n}\sum_{i=m+1}^{m+n}\log p_{\theta}(y_{i}\mid x_{i});$
(1)

for a hyper-parameter $\alpha\in[0,1]$ that depends on $m$ and $n$. The theory
gives guidelines to choose this hyper-parameter $\alpha$, e.g., if data from
the target group has a similar distribution as that of the data from the
source group and $m\sim n$, then $\alpha\sim 0.5$; if $m\gg n$, then we have
few samples from the target group and should therefore use an objective with
$\alpha\ll 1$ to predominantly use the samples from the source group for
training. In this paper, we treat $\alpha$ as a hyper-parameter and search for
its optimal value using nested cross-validation; the domain of the search is
chosen using these guidelines. For all three problems, we trained both the
neural network and the ensemble using this $\alpha$-weighted ERM objective for
two settings: when we have 10% data from the target group, and when we are
allowed to use all data from the target group (in practice, this amounts to
80% of the data because we treat the remainder as the held-out test set). The
first setting corresponds to realistic situations when it is reasonable to
procure some data from the target group for the purposes of building better
diagnostic models; the second setting when adaptation is performed using all
target data can be thought of as an upper bound on the performance.

For all three problems, for all groups, $\alpha$-weighted ERM with 10% data
improves the performance of ensemble models significantly333We discuss the
performance of the ensemble models here in the main text; neural network-based
experiments shown in the Appendix exhibit similar trends but with lower AUCs
and higher MAEs.. For AD classification in Fig. 2, the AUC for target groups
such as sex (Male), age group (< 65 and 65-70 years old), race (White), and
clinical studies (ADNI-1, ADNI-2/3, PENN and AIBL) can be improved no matter
which source group the model was adapted from. For SCZ classification in Fig.
3, all target groups show increased AUC except two: 25-30 and 30-35 years old.
For brain age prediction in Fig. 4, we also observe improved performance on
nearly all target groups except one (Black). The difference in AUC from using
10% vs. 20% data from the target group is not statistically significant
($p$-value > 0.01) or AD and SCZ classification as also MAE for brain age
prediction. More details are provided in the Appendix.

Ensemble models adapted using only 10% target data achieve comparable
performance to that of training on all data from the target group. They are
sometimes even better. For example, in AD classification, models adapted on
target groups (< 65, 70-75 years old and AIBL) have a similar AUC to that
achieved when training on all data from the target group ($p$-value > 0.01)
irrespective of the source group they were trained with. A model trained on
the 70-75 years old group achieves 0.971 ± 0.002 AUC when adapted to the 65-70
years old target group—this is much higher than training directly on all data
from the target group (0.924 ± 0.019, $p$-value 4.3$\times 10^{-3}$). For SCZ
classification, there is no statistically significant difference between
models trained on all data from target groups (Male, > 35 years old, Munich
and Utrecht) and models adapted from any of the source groups. For brain age
prediction, we found that models trained on any source group when adapted to
groups in the WHIMS study perform on par with the ones directly trained on all
WHIMS data ($p$-value > 0.01).

Figure 2:  Alzheimer’s disease classification. Markers denote the average AUC
on the target group computed using five-fold nested cross-validation for
models trained only on data from the target group (e.g., Female subjects,
denoted by the blue horizontal line), only on data from the source group
(e.g., trained on all Male subjects and evaluated on Female subjects is
denoted by the orange cross), and trained on all data from the source group
and 10% data from the target group (orange circle). Panels denote groups
stratified by one of the four attributes, namely sex, age group, race and
clinical study. Bar plots denote the proportion of subjects in these groups in
our study. All models are ensembles trained using features derived from
structural measures, demographic and clinical variables, genetic factors, and
cognitive scores. In spite of imbalances in the proportion of data in
different groups, the AUC of the ensemble is consistently high (above 0.85 in
all cases except when transferring from models built from Asians). The gap in
predictive performance of a model trained on only target data (horizontal
lines) and a model trained only on source data (crosses) can be improved with
access to as little as 10% data from the target group (circles) for Male, < 65
years, > 80 years, Asian, ADNI-1, ADNI-2/3, PENN and AIBL, when transferring
from any of other groups ($p$ < 0.005). The improvement in AUC using 10%
target data is not statistically significant for the other groups; in one case
(Female) we also see deterioration after including the target data perhaps due
to confounding factors. We observe that the AUC for the > 80 years subgroup is
low compared to other age groups even for models directly trained on this
group. This might be due to the strong normal aging effects which make it
difficult to distinguish cognitively normal individuals from AD patients.

Figure 3:  Schizophrenia classification. Markers denote the average AUC of the
ensemble on the target group computed using five-fold nested cross-validation
for models trained only on data from the target group (e.g., Female subjects,
denoted by the blue horizontal line), only on data from the source group
(crosses), and trained on all data from the source group and 10% data from the
target group (circles). Compared to Fig. 2, the AUC for schizophrenia
classification is lower in general, as expected based on respective prior
literature. We find that $\alpha$-weighted ERM using 10% data from the target
group improves the AUC of the ensemble (circles are above crosses of the same
color) in all cases except two: 25-30 years old and 30-35 years old. In most
cases, models adapted from source groups using 10% data from the target group
perform better than those trained on all target data, except when target
groups are Male, > 35 years old, Munich and Utrecht, when the difference is
statistically insignificant. We observe large performance discrepancies
between different clinical studies. Besides scanner and acquisition protocols
variations, disease severity might be playing a role here. For example, the
AUC of China cohort is large perhaps because on-site clinical cases are
usually relatively more severe clinically, largely due to cultural factors
influencing who and when will seek hospitalization.

Figure 4:  Brain age prediction. Markers denote the mean absolute error (MAE)
in years of an ensemble that predicts the brain age on different target groups
in the population computed using five-fold nested cross-validation, for models
trained only on data from the target group (e.g., Female subjects, denoted by
the blue horizontal line), only on data from the source group (crosses), and
trained on all data from the source group and 10% data from the target group
(circles). In general, the MAE of brain age prediction is remarkably small, it
is below 7 years for age and race and below 15 years in most settings when
models were trained on different clinical studies. Ensembles trained using 10%
data from the target group in addition to all data from the source group
improve the MAE in all cases (circles are below crosses) except one (when
source is White and target is Black). The third panel has 10 different
clinical studies, with very different amounts of data. Even in this case the
MAE of brain age prediction is smaller than 8 years in all cases when the
ensemble has access to some data from the target group, in some cases there
are significant improvements as compared to the corresponding crosses.
Magnetic field strength of the scanners affects the models performance
significantly. For example, only BLSA-1.5T and SHIP are acquired from 1.5T
devices and others are from 3T ones. We can see big MAE gaps between the
horizontal lines and crosses in BLSA-1.5T and SHIP studies. We also observe
that larger data size gves rise to better the performance. For example, UKBB
has the largest sample size among all studies and models trained on UKBB
usually have lower MAE when adapting to other studies.

###  2.4 Diagnostic models for Alzheimer’s disease can also be effective for
early diagnosis of stable and progressive mild cognitive impairment. Adapting
models to the target group further improves this ability.

(a) Progressive MCI v.s. stable MCI

(b) Correlation of brain age regression with neuropsychological tests

Figure 5:  Adapting diagnostic models to target groups using a small amount of
data also improves their ability to make predictions on secondary tasks. (a)
Linear discriminant analysis on the output probabilities (that determines AD
vs. cognitively normal CN) of the ensemble models trained for Alzheimer’s
disease diagnosis is used to study whether subjects with mild cognitive
impairment (MCI) progress to AD (known as pMCI) or remain stable MCI (known as
sMCI) using only the baseline scans. The AUC of pMCI vs. sMCI on the target
group is shown for three different attributes (sex, age group and race) when
models are trained only on data from the source group (crosses), using
$\alpha$-weighted ERM using all data from the source and 10% data from the
target group (circles) and with access to only all data from the target group
(horizontal lines). Improvements in the AD vs. CN AUC of these models with 10%
data translate to improvements in the ability to distinguish between pMCI and
sMCI subjects, using only baseline scans (circles above cross) except when
target groups are Black or Asian (due to very little data in these groups).
For all type of models, performance decreases as the age of the participants
increases; this is because predicting progressive MCI using baseline scans is
more and more challenging when the time difference to the target age group and
the normal aging effect increases.
(b) Pearson’s correlation between the brain age residual (predicted brain age
minus chronological age) and neuropsychological tests for two different
attributes (sex and race) for models trained only on source data (crosses),
using $\alpha$-weighted ERM on all source data and 10% target data (circles)
and only on all target data (horizontal lines). Unlike other plots, colors
denote different pairs of source and target groups. Tests (X-axis) marked in
red are expected to be negatively correlated with brain aging whereas those
marked in black are expected to be positively correlated with brain aging
according to the existing literature. Mini-mental state examination (MMSE) is
a questionnaire test that measures global cognitive impairment. Digit span
forward/backward (DSF/B) test is a way of measuring the storage capacity of a
person’s working memory. Trail making test part A/B (TMT A/B) measures a
person’s executive functioning. Digit symbol substitution test (DSST) is
another global measure of cognitive ability, requiring multiple cognitive
domains to complete effectively. In almost all cases, we observe stronger
correlations than those reported in the literature. Models trained using 10%
target data improve the correlation with these neuropsychological tests. Brain
age models trained from other groups usually have larger correlations to
cognitive scores than the ones directly trained on the target group.

Mild cognitive impairment (MCI) refers to the transition stage between the
expected cognitive changes associated with normal aging and those due to
dementia, most frequently of AD dementia. MCI progresses to AD at a rate of
approximately 15% annually (referred to as progressive MCI; pMCI), but
sometimes it remains stable (referred to as stable MCI; sMCI). Herein we adopt
the definition of progression to reflect changes occurring within three years
since the baseline visit [34]. Sometimes, MCI can also revert back to
cognitively normal state, in which case it is oftentimes called normal MCI
(nMCI). It is important to predict the future clinical progression in MCI, in
order to guide patient management and clinical trial stratification [41, 42,
43, 44, 45].

We investigated whether our machine learning models from the previous section,
which were built for diagnosing AD, can be used to predict which individuals
with MCI will progress to AD. Individuals in the MCI group typically exhibit
heterogeneous patterns with features that are intermediate between AD and CN;
this is why using models trained to diagnose AD for this task is challenging
and unusual. We are also interested in understanding whether the improvements
to these models, when adapted to data from the target group, also translate to
improvements in their predictive ability on MCI subjects. If this is the case,
then we can fruitfully improve the predictive ability on MCI subjects by
adapting models on AD data. In this study, we apply AD diagnostic models
directly on the baseline MCI subjects to check if pMCI, which has a similar
pathology as that of AD, can be distinguished from sMCI and nMCI at the
earliest clinical stage. All of our models are probabilistic classifiers and
return the (uncalibrated) probability of an AD diagnosis. Using this predicted
probability as the covariate, we perform linear discriminant analysis (LDA)
[46] using 20% of the pMCI, sMCI and nMCI retrospectively-created labels of
the baseline scans; the remainder of the MCI samples as the test set. This
experiment is repeated in a five-fold cross-validation fashion to estimate the
test error. Table S.1 summarizes the data used for this analysis.

As Fig. 5(a) shows, we find that models trained for diagnosing AD from CN can
also distinguish pMCI from sMCI accurately using only baseline scans; Fig.
7(b) shows a similar result for pMCI vs. nMCI. For example, within the Female
population, we obtained 0.891 ± 0.008 AUC for pMCI v.s. nMCI classification
and 0.800 ± 0.010 AUC for pMCI v.s. sMCI classification. When an ensemble
model trained using data from group > 80 years old is evaluated on data from
age group < 65 years old, it obtained an AUC of 0.769 ± 0.010 and 0.734 ±
0.007 for pMCI vs. nMCI and pMCI vs. sMCI tasks, respectively. These numerical
results are remarkable in the model was trained using AD vs. CN data and
refashioned to make these predictions using 20% of the data and LDA. When the
ensemble model is adapted to the target AD group using 10% of the data, we
find that the performance improves further in some cases, in particular when
the discrepancy between the data distributions the groups is large (see Fig.
1(b)); there are also situations when the AUC remains the same. For example,
in the example above (source group is > 80 years old and target group is < 65
years old) $\alpha$-weighted ERM improves the AUC to 0.804 ± 0.010 and 0.804 ±
0.011 for pMCI vs. nMCI and pMCI vs. sMCI tasks, respectively. Just like the
previous section, even in this task, we observed that sometimes the
performance of the adapted AD models surpasses that of models trained directly
on the target domain. For example, within Black population, AUC for pMCI vs.
nMCI is 0.825 ± 0.170 using a model trained only on data from this group,
while a model trained using $\alpha$-weighted ERM on AD data from the White
group along with 10% data from the Black group achieves an AUC of 0.945 ±
0.048.

Our results indicate that models which can accurately diagnose AD from CN are
also able to make predictions on whether MCI subjects will evolve into
progressive, stable or normal MCI by only using baseline scans. Access to a
small subset of the target group data for AD subjects further improves the
performance of these predictions.

###  2.5 Improved models for predicting brain chronological age lead to new
clinical conclusions about the correlation between brain age residual and
neuropsychological tests

The brain age residual (BAR) or brain age gap is the difference between the
age predicted from a brain scan and actual chronological age of the subject.
BAR is indicative of the deviation of a subject’s cognitive performance from
that of the normal population over time, a positive BAR suggests an
accelerated aging process [47]. Many recent studies [48, 49, 11, 50] have
argued that BAR can be used to quantify the brain aging process, e.g., via its
correlation with cognitive impairment [49, 51, 50]. In light of this, we
evaluate the association between BAR as predicted by our models and several
neuropsychological tests, including visual memory, visual attention, and
visual executive tests. We compute Pearson’s correlation between BAR and the
performance on mini-mental state examination (MMSE), digital symbol
substitution test (DSST), digit span forward/backward (DSF and DSB), and trial
making tests (TMT A and B). We also use cardiovascular features and lifestyle
factors such as systolic and diastolic blood pressure, and body mass index
(BMI) for association evaluation.

As Fig. 5(b) shows, we find that, in general, the BAR predicted by our
ensemble models is negatively correlated with cognitive measures that decrease
with age (e.g., MMSE and DSST) and positively correlated with those that
increase with age (e.g., TMT A and B) with $p$-values < 0.01. Interestingly,
we observe stronger correlations than those reported in the literature [49,
50]. This is perhaps due to the improved predictive ability of our BAR models.
In contrast, the correlations between the predicted BAR and cardiovascular and
lifestyle factors are not statistically significant ($p$-value > 0.01)—this
contradicts several previous studies [48, 50] (see Fig. S.8). Our BAR
prediction models are quite accurate (see Fig. S.3), so this could be
potentially reflecting the fact that our study did not use brain measurements
which more directly reflect effects of cardiovascular risk factors, such as
infarcts and microbleeds,

Similar to the previous section, we found that improving BAR predictions on
target groups using 10% data translates to improved associations with the
clinical factors, as compared to the associations computed using BAR
prediction models that did not use any target data. For example, Pearson’s
correlations between BAR and DSST/TMT A are -0.453 and 0.128 respectively when
a model trained on Asian subjects is evaluated on subjects from the White
group. If we instead improve this model using $\alpha$-weighted ERM and 10%
data from the White group, these correlations increase to -0.556 and 0.191
respectively. Just like the previous sections, models adapted to target groups
often lead to greater associations between BAR and cognitive measures compared
to those trained only on the target group. For example, within the Female
population, the correlation between BAR and MMSE is -0.161, whereas a BAR
prediction model trained using $\alpha$-weighted ERM trained on the source
Male group achieves -0.267 on the target Female group.

##  3 Discussion

#### Carefully-designed machine learning models with effective domain
adaptation techniques are necessary to ensure accurate predictions on
different groups in heterogeneous populations.

The one-fits-all paradigm, namely learning salient features from data, which
is so central to machine learning, may not work well for problems with
heterogeneous data, such as some encountered in clinical neuroscience [15, 30,
26]. In order to develop effective tools that can provide safe, robust and
unbiased decisions, we need to address the heterogeneity at its different
levels of granularity. For example, a large body of existing work builds
representations that are invariant to features of the subjects in different
groups in the population using domain generalization techniques [52, 53, 54,
36, 55, 56, 57, 58]. If the variability in data is not just nuisance
variability, but is instead correlated with the outcome, building such
invariance comes at the cost of predictive ability. For example, it has been
shown using an information-theoretic argument that the performance of such a
model is limited with the least predictive feature in the data [27]. Building
models that do not explicitly address this heterogeneity can lead to
deterioration in performance when these models are used to make predictions,
especially on groups that are under-represented in the population [21, 59,
30]. This debate has also led to concerns over the hype of precision medicine
[60, 61, 62].

Here, we provide a positive data point for the above debate. We show that
appropriately constructed machine learning models can predict both robustly
and accurately on many different target populations. We also show
theoretically that one model cannot predict accurately on all groups of a
heterogeneous population. Thus, we develop a natural technique to mitigate the
deterioration of accuracy: it is based on the employment of an
$\alpha$-weighted ERM objective using both the source and target data.
Depending upon the value of $\alpha$, the source data can _optimally_ inform
the representation that is eventually used to make predictions on the target
data in different ways. The $\alpha$-weighted ERM objective is simple to
implement, e.g., in this paper we have implemented it effectively for over 850
configurations of different source-target group pairs, amounts of target data
and machine learning models. Numerically, the AUCs and MAEs demonstrated in
this paper are equal or better than those reported in the literature on
similar datasets. But the trends are consistent with those in several recent
studies in healthcare [63, 64, 65, 66, 67, 68].

#### Training using derived imaging features performs on par with raw brain
images for classification and regression tasks.

With the recent great success of deep learning, the medical imaging community
has gradually adopted convolutional neural networks on neuroimaging data to
replace expert-designed imaging features [69, 4]. However, these expert-
informed hand-crafted imaging features can be more robust to changes in the
image intensity than raw images, e.g., those coming from different image
acquisition protocols and scanners, and hence more amenable to domain
adaptation. The diagnostic models developed in this paper are built using such
derived features; they can predict effectively on target groups, in many cases
even with no access to the target data. Another benefit of using these derived
features is that they can be combined much more easily with information from
multiple sources e.g., demographic and clinical variables, genetic factors,
and cognitive scores. In principle, this extra information can also be used
when deep networks, e.g., convolutional networks, are trained on volumetric
magnetic resonance (MR) images (as is indeed done in a number of places in the
literature) [4, 70, 71]. But it is difficult to build architectures where the
information in the multi-source features is not overwhelmed by the enormous
degree of nuisance variability in the raw MR images.

There is an increasing body of work which has recognized the value of hand-
crafted features [72, 73, 74]. For example in [74], machine learning models
trained with derived imaging features perform similar to, sometimes even
better than, those built using raw brain scans for Alzheimer’s disease,
schizophrenia, and autism spectrum disorder diagnosis tasks. It has also been
found that, with large-scale neuroimaging data, deep learning algorithms using
raw brain images do not improve the accuracy for sex and age prediction
compared to models that use such derived features [72]. These existing
results, as also our findings in this paper coming from the angle of domain
shift and adaptation, indicate that it is difficult to glean representations
from neuroimaging data using neural networks that can outperform hand-crafted
imaging features in a robust and reproducible way, and there there is a lot of
room for improvement in how deep networks are used for such data [73, 16, 19].

#### Careful data and feature preprocessing is essential for effective domain
adaptation.

Another aspect of our implementation that enables effective domain adaptation
is that we build machine learning models using carefully constructed data pre-
processing techniques and perform model selection using extensive hyper-
parameter search and cross-validation. These well-established safeguards
against poor generalization in the machine learning literature are sometimes
ignored in the modern practice of machine learning. Employing these procedures
can be computationally onerous, e.g., an estimated 10,000 GPU hours were used
to conduct the experiments in this paper. However, disregarding these
procedures can lead to deterioration in predictive ability and inadequate
generalization to new groups. For all three problems, Alzheimer’s disease
diagnosis, schizophrenia diagnosis and brain age prediction, in addition to
evaluating AUC and MAE using nested cross-validation, we also studied fairness
assessment metrics such as demographic parity differences (DPD) [75, 76] and
equalized odds difference (EOD) [77] in Fig. S.9, in order to support our
findings on generalization of machine learning models built from such large
and diverse datasets.

In summary, we have used a weighted ERM approach to mitigate the detrimental
effects of domain shifts in machine learning models derived from brain MRI,
and to develop robust and generalizable ML models for classification and
prediction of brain diseases. We have derived extensive experimental results
that demonstrate that this approach leads to ML models that work better in new
patient cohorts, in cohorts having different demographics, and under different
image acquisition conditions. We therefore provide a framework not only for
higher reproducibility of ML models, but also for increased fairness when ML
models are used on under-represented patient populations.

##  4 Methods

###  4.1 Datasets

We use 15,363 3D magnetic resonance (MR) images together with their
corresponding demographics, clinical variables, genetic factors, and cognitive
test scores from two large-scale consortia – iSTAGING [1] and PHENOM [78, 79,
80, 81, 82, 2] for Alzheimer’s disease (AD) diagnosis, schizophrenia (SZ)
diagnosis and brain age prediction tasks. This data covers a broad population
in terms of biological genders, age span, race or ethnicity diversity, scan
acquisition protocols and devices, and phenotypes of diseases. Table 1
summarizes this data.

#### iSTAGING.

For AD classification, we use four studies from iSTAGING [1] dataset including
Alzheimer’s Disease Neuroimaging Initiative (ADNI) [3], Penn Memory Center and
Aging Brain Cohort (PENN), Australian Imaging, Biomarkers and Lifestyle (AIBL)
[83], and Open Access Series of Imaging Studies (OASIS) [84]. Healthy controls
and patient groups are the two labels for the AD classification task (see
Table 1). In the brain chronological age regression experiment, we leverage
nine studies from iSTAGING [1] dataset including Biomarkers of Cognitive
Decline Among Normal Individuals (BIOCARD) [85], Baltimore Longitudinal Study
of Aging (BLSA) [86, 87], Coronary Artery Risk Development in Young Adults
(CARDIA) [88, 89], Study of Health in Pomerania (SHIP) [90], Systolic Blood
Pressure Intervention Trial (SPRINT) [91], UK Biobank (UKBB) [92], Women’s
Health Initiative Memory Study (WHIMS) [93], Wisconsin Registry for
Alzheimer’s Prevention (WRAP) [94], and Action for Health in Diabetes
(lookAHEAD) [95]. Only healthy controls are used for brain age prediction task
(see Table 1). In addition to imaging features, we use demographic information
such as gender, age, race, education level, and smoking status; clinical
variables such as diabetes, hypertension, hyperlipidemia, systolic / diastolic
blood pressure and body mass index (BMI); genetic factors such as
apolipoprotein E (APOE) alleles 2, 3 and 4; and cognitive scores such as mini-
mental state exam (MMSE) as features besides the imaging features from the
T1-weighted structural MR images in this work. Non-imaging features are
sparsely distributed among the participants in the studies.

#### PHENOM.

In the SZ diagnosis task, we use scans which are acquired from five different
sites around the world in PHENOM [78, 79, 80, 81, 82, 2] dataset, namely Penn
(United States), China, Munich, Utrecht, and Melbourne. Healthy controls and
patient groups are the two diagnosis labels for SCZ classification task (see
Table 1). To train, we use demographic information such as gender, age, race,
marital status, employment status, and handedness; and cognitive scores such
as full-scale intelligence quotient (FIQ), verbal intelligence quotient (VIQ),
and performance intelligence quotient (PIQ) besides features computed from
T1-weighted structural MR images. Again, non-imaging features are sparsely
populated in these studies.

#### Inclusion criteria.

All included subjects have passed quality control (QC) where they were
visually examined by a radiologist to screen the pre-processed scans;
distorted images were dropped. We only use the baseline (initial time point)
scans from each study. All follow-up sessions are excluded; this ensures that
there is no data leakage and the same participant is not included in the train
and test sets. For AD cohorts, we select stable cognitively normal (CN) and AD
patients based on each participant’s longitudinal diagnosis status. We only
include participants who were diagnosed as CN or AD at the baseline and stayed
stable during the follow-up sessions. For mild cognitive impairment (MCI)
cohorts, we select progressive MCI (pMCI), stable MCI (sMCI) and normal MCI
(nMCI) patients based on each participant’s longitudinal diagnosis status. We
define MCIs as follows: pMCIs are subjects who were diagnosed as MCI at
baseline and progressed to AD during the three years following the first
visit; sMCIs are subjects who were diagnosed as MCI at baseline and remained
MCI during the three years following the first visit; nMCIs are subjects who
were diagnosed as MCI at baseline and reverted back to CN during the three
years following the first visit.

###  4.2 Data pre-processing pipeline

#### Pre-processing images to obtain imaging features.

We pre-process T1-weighted MR images using a standard pipeline as follows. The
scans are first bias-field corrected [96] and skull-stripped with a multi-
atlas algorithm [97]. Then we use a multi-atlas label fusion segmentation
method [98] to obtain the anatomical region-of-interest (ROI) masks including
119 grey matter ROIs, 20 white matter ROIs, and 6 ventricle ROIs of the brain.
145 ROI volumes are calculated based on the corresponding extracted brain
masks. We further segment white matter hyperintensities (WMH) by applying a
deep learning-based algorithm [99] on the fluid-attenuated inversion recovery
(FLAIR) and T1-weighted images. The white matter lesion (WML) volumes are
obtained by summing up the WMH mask voxels.

#### Pre-processing methodology.

We pre-process and normalize imaging measures such as ROI volumes and WML
volumes, demographic, clinical and genetic features and cognitive scores.
Since many features, e.g. clinical variables, genetic factors, and cognitive
scores, are sparse, we need to impute these missing values. For continuous
features, we first impute missing values with the median of each feature and
then normalize the features to have zero mean and unit variance. We apply
quantile normalization to skewed distributions. For discrete features, we
introduce an “unknown” category that indicates missing values. For each
feature with missing values, we introduce an additional feature to indicate
whether the particular entry was missing; this helps preserve the information
of the absence. We did not remove batch or site effects from the pooled data,
i.e., we did not use any harmonization [13, 100].

###  4.3 Training and evaluation methodology

#### Creating the train and test sets.

We use nested five-fold cross-validation for all our experiments. Consider the
situation when we are training and evaluating on the same group, i.e., we
split the data into five equal-sized folds (stratified by labels), use four
for training and validation (80% data) and the fifth for testing (20% data).
All hyper-parameter tuning is performed using a further five-fold cross-
validation within the 80% data. This way, the remaining 20% data forms a
completely independent test set (which is not used for choosing any hyper-
parameters) and can be used for reporting the area under the receiver
operating curve (AUC) or the mean absolute error (MAE). When this procedure is
repeated 5 times, each time for a different test fold, we can calculate the
mean and standard deviation of the AUC or the MAE on the test set.

We constructed a similar nested cross-validation procedure for situations
where we train and evaluate on different groups. We again split target data
into five equal-sized folds (stratified by labels). Suppose we have access to
20% data from the target group. The train and validation sets together
therefore consist of all data from the source group and 20% data from the
target group; the remainder 80% data from the target group is the test set. We
further sub-divide the train set (i.e., the 100% source data and the 20%
target data) into 5 equal parts (stratified by labels, and with equal amount
of target data in each fold) to perform cross-validation and model selection.
We perform the search for the hyper-parameter $\alpha=k/(k+1)$ over
$k\in\\{1,2,\dots,10\\}$; the search grid therefore becomes finer as the value
of $\alpha$ goes closer to 1. Again, we can report the mean and standard
deviation of AUC or MAE across the entire data from the target group after
repeating this procedure for each of the five test folds. When the amount of
data used from the target group is 0%, we report the AUC and MAE on all the
data from the target group without any test folds.

Even when we have access to only 10% of the data from the target group, we
still create five equal-sized folds from the target data. In other words, our
train and validation set consists of 100% data from the source group, 10% data
from the target group and the test set consists of 80% data from the target
group. This was done for convenience: there are hundreds of experiments in
this paper across complicated multi-source datasets, each of which uses five-
fold nested cross-validation, and is conducted using multiple models such as
ensembles and neural networks—book-keeping all this information gets unwieldy
quickly.

This evaluation methodology is very expensive computationally, but it is
rigorous and allows us to glean statistically precise conclusions. The
experiments and data analysis in this study required an estimated 10,000 GPU
hours.

#### Training the neural network and the ensemble.

We use three-layer feed-forward neural network (multi-layer perceptron) with
rectified linear unit (ReLU) activations, dropout [101] and batch
normalization [102] after each layer; there is a skip-connection [103] that
connects the first and the penultimate layer. Pre-processed categorical
features are first passed to their corresponding embedding layers individually
and all these embeddings are concatenated with the pre-processed numerical
features.

Ensembling techniques such as bagging [104], boosting [105] and stacking [106]
are well-established techniques in machine learning. We used AutoGluon [107]
which is a software framework that implements these techniques efficiently,
along with parallel hyper-parameter search and model selection across CPUs and
multiple GPUs. AutoGluon also implements a novel ensembling strategy called
multi-layer stacking and repeated $k$-fold bagging [108]. This is quite useful
for the kinds of data studied in this paper and we discuss this strategy
briefly next. In multi-layer stacking, we first train models such as neural
networks, $k$-nearest neighbors, random forests, CatBoost boosted trees [109],
and LightGBM trees [110] separately. Predictions from these base models,
concatenated with features in the original data serve as input features to a
second “layer” of stacked models; this is inspired from skip-connections in
deep learning [103]. A multi-layer stacking approach can grow an ensemble in
this fashion by treating the stacked models as the base models for the next
layer; in practice to mitigate over-fitting, stacking is done only for two
levels. To make predictions, the highest-level stacked models are combined
using ensemble selection [111]; this has been found to be resilient to over-
fitting when combining high-capacity models. To curb over-fitting and to avoid
covariate shift at inference time, while fitting each layer of the stacked
model, the training set is randomly split into $k$ folds (stratified by
labels) and $k$ different models fitted on $(k-1)$ of these folds are used to
obtain out-of-fold (OOF) predictions for each sample. This $k$-fold bagging
can also be repeated several times to average the OOF predictions and further
mitigate potential over-fitting within out-of-fold predictions.

###  4.4 Learning the maximum mean discrepancy statistic

Consider the case when we have two sets of samples $D_{p}=\\{x_{i}:x_{i}\sim
p\\}_{i=1}^{n}$ and $D_{q}=\\{x^{\prime}_{i}:x^{\prime}_{i}\sim
q\\}_{i=1}^{n}$ from two probability distributions $p$ and $q$ respectively. A
two-sample test relies on finding a function (called the “statistic”) which is
large on samples drawn from $p$ and small on samples draw from $q$. The
maximum mean discrepancy (MMD) [33] statistic takes the difference between the
mean function value on the two sets of samples, and determines the two sample
sets as coming from different distributions if this maximum is large. The MMD
is defined as

$\text{MMD}[\mathcal{F},p,q]=\sup_{f\in\mathcal{F}}\left(\operatorname*{\mathbb{E}}_{x}\left[f(x)\right]-\operatorname*{\mathbb{E}}_{x^{\prime}}\left[f(x^{\prime})\right]\right),$
(2)

where $\mathcal{F}$ is a class of functions with appropriate regularity
properties, e.g., that it has uniform convergence [112]. A classical example
of such a class of functions is the reproducing kernel Hilbert space which has
the property (called the reproducing kernel property) that for any input $x$,
there exists a function $\phi_{x}\in\mathcal{F}$ (called the canonical feature
map) such that the evaluation of $f\in\mathcal{F}$ at the input $x$ can be
written as an inner product in the reproducing kernel Hilbert space (RKHS):

$\mathbb{R}\ni f(x)=\left\langle f,\phi_{x}\right\rangle_{\mathcal{F}}.$

Note that $\phi_{x}$ is a function. Observe that using the same property
above, we can write
$\phi_{x}(x^{\prime})=\left\langle\phi_{x},\phi_{x^{\prime}}\right\rangle_{\mathcal{F}}\equiv
k(x,x^{\prime})\geq 0$; here the quantity $k(x,x^{\prime})$ is called the
kernel of the RKHS and measures the similarity between the two inputs $x$ and
$x^{\prime}$. An unbiased estimate of Eq. 2 using samples is given by

$\text{MMD}_{u}^{2}[\mathcal{F},p,q]=\frac{1}{(n^{2}-n)}\sum_{i,j=1,i\neq
j}^{n}\left(k(x_{i},x_{j})+k(x_{i}^{\prime},x_{j}^{\prime})-k(x_{i},x_{j}^{\prime})-k(x_{j},x_{i}^{\prime})\right).$
(3)

Neural networks are universal function approximators and possess uniform
convergence properties [113]. Therefore, we can fruitfully set the class
$\mathcal{F}$ to be the class of functions parameterized by a multi-layer
perceptron (MLP) [114, 115]. We will next describe how we use an MLP to
estimate the differences between the probability distribution of the
covariates of subjects belonging to different groups in the Alzheimer’s
disease data. We build an MLP that takes as input the covariates $x$ and has
four different types of outputs corresponding to each of the four attributes
(sex: male vs. female, age: < 65 vs. 65–70 vs. 70-75 vs. etc., race: White vs.
Black vs. Asian, and clinical study: ADNI-1 vs. ADNI-2/3 vs. PENN vs. AIBL).
We fit one MLP to predict each of these groups using four different output
layers (for sex we have a binary classification problem, for age we have a
5-way problem, etc.) and treat its learned features as the canonical feature
map $\phi_{x}$ of the input $x$. The problem of training the MLP to minimize
of errors incurred while predicting the correct group for each attribute for
each subject, is equivalent to the maximization over the functions
$f\in\mathcal{F}$ in Eq. 2. The pairwise MMD statistic between groups for each
attribute is calculated from these learned features using Eq. 3 and reported
in a summarized form in Figs. 1(b), 2(b) and 3(b) with details provided in
Figs. 1(a), 2(a) and 3(a). We can also examine the relative difference between
the MMD statistic corresponding to different pairs of groups and attributes,
e.g., the MMD statistic between males and females in Figs. 1(a) and 1(b) is
0.17 while the statistic between ADNI-1 and ADNI-2/3 is 0.26; there is a
larger discrepancy between the probability distributions in the latter case
than the former. Note that if we had fitted different MLPs independently for
each of the attributes, then we would not have been able to examine the
relative differences because the learned feature maps would be different for
different attributes.

###  4.5 Building models that also have access to data from the target group
improves performance

Consider a dataset $D=\left\\{(x_{i},y_{i})_{i=1}^{n}\right\\}$ with $n$
samples where $x_{i}$ are the input features. For the purposes of this
argument we will think of the ground-truth $y_{i}\in\left\\{0,1\right\\}$ as
being binary but all the claims in the sequel can be formalized analogously
for multi-class classification and regression problems. We will assume that
each datum is drawn from a probability distribution $(x_{i},y_{i})\sim P$. A
machine learning model with weights $\theta\in\mathbb{R}^{p}$ is a model
$p_{\theta}(y\mid x)$ of the probability that an input $x$ belongs to a
category $y\in\left\\{0,1\right\\}$. We would like this model to generalize to
new data outside the training set, i.e., we would like to find a model

$\theta^{*}=\operatorname*{\text{argmin}}_{\theta}R(\theta):=-\operatorname*{\mathbb{E}}_{(x,y)\sim
P}\left[\log p_{\theta}(y\mid x)\right],$ (4)

which minimizes the “population risk” $R(\theta)$ which is the average
negative log-likelihood over all samples $(x,y)\sim P$. The population risk is
also called the test risk, or the test loss. In practice, we do not know the
entire distribution of data $P$ but only have access to a training set $D$
with samples from $P$. Our goal is to find a best approximation for the
weights $\theta^{*}$, which is typically done by minimizing the so-called
empirical risk (also called training loss)

$\hat{\theta}=\operatorname*{\text{argmin}}_{\theta}\hat{R}(\theta)+\Omega(\theta):=-\frac{1}{n}\sum_{i=1}^{n}\log
p_{\theta}(y_{i}\mid x_{i})+\Omega(\theta),$ (5)

where $\Omega(\theta)$ is a regularization term, e.g.,
$\Omega(\theta)=\norm{\theta}_{2}^{2}/2$, that avoids over-fitting to the
training dataset. The objective is usually minimized by the stochastic
gradient descent [116] algorithm. The discrepancy between the two, the
objective Eq. 4 that we wish to minimize and the objective in Eq. 5 that we
actually minimize, is measured using classical inequalities in machine
learning theory:

$R(\theta)\leq\hat{R}(\theta)+c\sqrt{\frac{V-\log\delta}{n}},$ (6)

which hold for all $\theta$ (in particular $\hat{\theta}$ in Eq. 5) and any
$\delta\in(0,1)$ with probability at least $1-\delta$ over independent draws
of the samples in the training dataset; here $V$ denotes the Vapnik-
Chervonenkis (VC) dimension [112] and $c$ is a constant. Roughly speaking, the
VC-dimension measures the complexity of the function class (e.g., the neural
network) parameterized by the weights. The above inequality says that larger
the complexity of the function class larger the VC-dimension and larger the
number of samples $n$ that we need in order to ensure that the population/test
risk $R(\theta)$ is small, provided the empirical risk, or the training loss,
$\hat{R}(\theta)$ is also small. One can think of the objective in Eq. 5 as
corresponding to the right-hand side of Eq. 6: the training cross-entropy loss
is equal to the empirical risk $\hat{R}(\theta)$ and the regularization term
$\Omega(\theta)$ is a proxy for the second complexity term.

#### Domain generalization can be poor if the source and target groups have
very different distributions.

If the training data and the test data are not from the same probability
distribution $P$, then we should not expect the weights $\hat{\theta}$
obtained using the training data to predict accurately on the test data, i.e.,
have a low test loss. We can quantify this deterioration using inequalities
similar to Eq. 6 as follows. Suppose the training dataset $D$ consists of
samples from a source group $P_{s}$ and we seek to minimize the population
risk on data drawn from a different target group with a probability
distribution $P_{t}\neq P_{s}$. In the context of this study, these groups
represent different sex, age, race or clinical studies. For any
$\delta\in(0,1)$ with probability at least $1-\delta$ over independent draws
of the training dataset, we have

$R_{t}(\theta)\leq
R_{s}(\theta)+c\sqrt{\frac{V-\log\delta}{n}}+\frac{d(P_{s},P_{t})}{2}+\lambda$
(7)

for all weights $\theta$; here $c$ is a constant, $R_{s}(\theta)$ is the
population risk on the source group, $R_{t}(\theta)$ is the population risk on
the target group [117] and

$\lambda=\min_{\theta}R_{s}(\theta)+R_{t}(\theta)$

is the combined risk of a model that minimizes the average population risk of
the two groups. The term $d(P_{s},P_{t})$ measures the difference between the
two probability distributions $P_{s}$ and $P_{t}$. The key point is that if
$P_{t}\neq P_{s}$, both the terms $d(P_{s},P_{t})$ and $\lambda$ are non-zero.
And therefore, even if the weights $\theta$ were to obtain a good population
risk $R_{s}(\theta)$ (which, as we discussed above, is not a given), there
would still be a gap between the population risk $R_{t}(\theta)$ due to the
discrepancy between $P_{s}$ and $P_{t}$. If the distributions of the source
and target groups $P_{s}$ and $P_{t}$ are very different, then a model trained
only on source data will perform poorly on the target group [118, 119]. This
is a fundamental hurdle in clinical applications where data is heterogeneous
[27].

#### Weighted-ERM on a combined dataset from the source and target groups.

A natural technique to mitigate the deterioration of accuracy due to the
change in data distribution between the source and target groups is to use
some data from the target group in addition to that from the source group in
the training set while fitting the weights $\theta$. Assume that the
distribution $P$ is a mixture of two groups $P_{s}$ and $P_{t}$ with
$P_{s}\neq P_{t}$. Given $m$ samples from the source sub-population $P_{s}$
and $n$ samples from the target sub-population $P_{t}$, we can construct a
combined training set $\\{(x_{i},y_{i})_{i=1}^{m+n}\\}$. We are interested in
settings where we do not have access to a large number of samples from the
target group, i.e., $n\ll m$. Consider a weighted empirical risk minimization
(ERM) objective

$\displaystyle\hat{\theta}=\operatorname*{\text{argmin}}_{\theta}\hat{R}(\theta)$
$\displaystyle:=-\frac{1-\alpha}{m}\sum_{i=1}^{m}\log p_{\theta}(y_{i}\mid
x_{i})-\frac{\alpha}{n}\sum_{i=m+1}^{m+n}\log p_{\theta}(y_{i}\mid
x_{i})+\Omega(\theta)$ (8)
$\displaystyle=(1-\alpha)\hat{R}_{s}(\theta)+\alpha\hat{R}_{t}(\theta)+\Omega(\theta),$

where we have denoted the empirical source and target risks by
$\hat{R}_{s}(\theta)$ and $\hat{R}_{t}(\theta)$ respectively. The hyper-
parameter $\alpha\in[0,1]$ controls the relative weight of these risks in
finding the optimal weights $\hat{\theta}$. To understand its effect, observe
that if $\alpha=0$, i.e., we do not use any data from the target group, we
should not expect the weights fitted using data from the source to predict
well on test data from the target group; the consequent deterioration in this
setting was discussed in Eq. 7. If we set $\alpha=1$, i.e., we do not use any
data from source group, then the small number of samples $n$ that we have from
the target group will lead to larger right-hand side in Eq. 6 and thereby a
poor predictor on the test data from the target. A naive strategy would set
$\alpha=1/2$, i.e., weigh the two sub-populations equally in the objective,
but it turns out that one often do better. Formally, for any $\delta\in(0,1)$
with probability at least $1-\delta$ over independent draws of training
samples from $P_{s}$ and $P_{t}$, it can be shown that [117]

$R_{t}(\hat{\theta})\leq
R_{t}(\theta^{*}_{t})+4\sqrt{\left(\frac{\alpha^{2}}{n}+\frac{(1-\alpha)^{2}}{m}\right)\left(V-\log\delta\right)}+2(1-\alpha)d(P_{s},P_{t}),$
(9)

where $\theta^{*}_{t}$ is the optimal, i.e., it minimizes Eq. 4 for the target
population. The third term again is a measure of the difference between the
two populations $P_{s}$ and $P_{t}$. This theorem suggests that if we choose
an appropriate value of $\alpha$ that minimizes the entire right-hand side,
then we can expect to draw maximal utility from the source data to predict
accurately on the target population. For example, observe that if
$d(P_{s},P_{t})$ is very large, then the right-hand side is small when
$\alpha\approx 1$, i.e., when the weighted-ERM objective neglects the source
data (because it is quite dissimilar to the target data). Guidelines for
picking this hyper-parameter can be computed [120]: if the number of samples
from the target population is larger than a threshold
$4(V-\log\delta)/(d(P_{s},P_{t}))^{2}$, then $\alpha=1$ minimizes the right-
hand side. In practice, it is difficult to estimate, both the VC dimension $V$
and the difference between the two probability distributions $P_{s}$ and
$P_{t}$ given by $d(P_{s},P_{t})$. But this hyper-parameter $\alpha$ can be
chosen in practice using cross-validation.

#### Relationship of weighted-ERM to other transfer learning techniques.

The weighted-ERM procedure that we have discussed above can be thought of as
an optimal mechanism to adapt a model trained on a source data to the target
data as follows. Formally, in supervised learning using a dataset
$D=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, the prediction on a test datum $x$ is made
by computing:

$\hat{y}=\operatorname*{\text{argmax}}_{y}p(y\mid x,D).$

If the class of functions that we search over to find one candidate
$p(\cdot\mid x,D)$ (say, all possible linear predictors) is large, then this
search is a computationally expensive endeavor at inference time. The
“weights” $\hat{\theta}=\operatorname*{\text{argmin}}\hat{R}(\theta)$ of a
parametric model provide a classical way to work around this issue; we can
think of the weights as a statistic of the dataset $D$, i.e.,
$\hat{\theta}\equiv\hat{\theta}(D)$. This statistic is sufficient [121] to
predict on samples within the training set $D$. Using this for predicting on
the test set by setting

$p(y\mid x,D)\equiv p_{\hat{\theta}}(y\mid x)$

entails a loss of generality and causes the gap between population risk and
empirical risk in Eq. 6 but these weights conveniently separate the training
and inference phases. This same argument holds for transfer learning where the
weights $\hat{\theta}$ are learned using a source dataset and adapted, e.g.,
using mini-batch updates, to new data from the target dataset. Explicitly
searching for the weights using both the source and target data, like we do in
weighted-ERM in Eq. 8 is at least as good as adapting the weights trained on
the source data. This is an instance of the data processing inequality [121].
In other words, we can prove that the population risk of the solution obtained
from Eq. 8 is at least as good as the one obtained using transfer learning
techniques. There are a number of theoretical and empirical works in the
machine learning literature which have argued this point for transfer learning
[122, 123], meta-learning and few-shot learning [28, 124] and continual
learning [118].

## Acknowledgments

This work was supported by the National Institute on Aging (RF1AG054409 and
U01AG068057), the National Institute of Neurological Disorders and Stroke
(U24NS130411), the National Institute of Mental Health (R01MH112070), the
National Institute on Drug Abuse (75N95019C00022), the National Science
Foundation (IIS-2145164) and cloud computing credits from Amazon Web Services.
The data used in this study are part of the iSTAGING consortium [13, 1] for
brain aging and the PHENOM consortium [2] for psychosis.

## References

  * [1] Mohamad Habes, Raymond Pomponio, Haochang Shou, Jimit Doshi, Elizabeth Mamourian, Guray Erus, Ilya Nasrallah, Lenore J Launer, Tanweer Rashid and Murat Bilgel “The Brain Chart of Aging: Machine-learning Analytics Reveals Links between Brain Aging, White Matter Disease, Amyloid Burden, and Cognition in the iSTAGING Consortium of 10,216 Harmonized MR Scans” In _Alzheimer’s & Dementia_ 17.1, 2021, pp. 89–102
  * [2] Ganesh B Chand, Dominic B Dwyer, Guray Erus, Aristeidis Sotiras, Erdem Varol, Dhivya Srinivasan, Jimit Doshi, Raymond Pomponio, Alessandro Pigoni and Paola Dazzan “Two Distinct Neuroanatomical Subtypes of Schizophrenia Revealed Using Machine Learning” In _Brain_ 143.3, 2020, pp. 1027–1038
  * [3] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula J Britson, Jennifer L. Whitwell and Chadwick Ward “The Alzheimer’s Disease Neuroimaging Initiative (ADNI): MRI Methods” In _Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine_ 27.4, 2008, pp. 685–691
  * [4] Shangran Qiu, Matthew I Miller, Prajakta S Joshi, Joyce C Lee, Chonghua Xue, Yunruo Ni, Yuwei Wang, De Anda-Duran, Phillip H Hwang and Justin A Cramer “Multimodal deep learning for Alzheimer’s disease dementia assessment” In _Nature Communications_ 13.1 Nature Publishing Group, 2022, pp. 1–17
  * [5] Rongguang Wang, Vishnu Bashyam, Zhijian Yang, Fanyang Yu, Vasiliki Tassopoulou, Sai Spandana Chintapalli, Ioanna Skampardoni, Lasya P Sreepada, Dushyant Sahoo and Konstantina Nikita “Applications of generative adversarial networks in neuroimaging and clinical neuroscience” In _NeuroImage_ Elsevier, 2023, pp. 119898
  * [6] Martin Rozycki, Theodore D Satterthwaite, Nikolaos Koutsouleris, Guray Erus, Jimit Doshi, Daniel H Wolf, Yong Fan, Raquel E Gur, Ruben C Gur and Eva M Meisenzahl “Multisite Machine Learning Analysis Provides a Robust Structural Imaging Signature of Schizophrenia Detectable across Diverse Patient Populations and within Individuals” In _Schizophrenia bulletin_ 44.5, 2018, pp. 1035–1044
  * [7] Shuang Gao, Vince D Calhoun and Jing Sui “Machine learning in major depression: From classification to treatment outcome prediction” In _CNS neuroscience & therapeutics_ 24.11 Wiley Online Library, 2018, pp. 1037–1052
  * [8] Anibal Sólon Heinsfeld, Alexandre Rosa Franco, R Cameron Craddock, Augusto Buchweitz and Felipe Meneguzzi “Identification of autism spectrum disorder using deep learning and the ABIDE dataset” In _NeuroImage: Clinical_ 17 Elsevier, 2018, pp. 16–23
  * [9] Luke Macyszyn, Hamed Akbari, Jared M Pisapia, Xiao Da, Mark Attiah, Vadim Pigrish, Yingtao Bi, Sharmistha Pal, Ramana V Davuluri and Laura Roccograndi “Imaging patterns predict patient survival and molecular subtype in glioblastoma via machine learning techniques” In _Neuro-oncology_ 18.3 Society for Neuro-Oncology, 2015, pp. 417–425
  * [10] Nicola K Dinsdale, Emma Bluemke, Stephen M Smith, Zobair Arya, Diego Vidaurre, Mark Jenkinson and Ana IL Namburete “Learning patterns of the ageing brain in MRI using deep convolutional networks” In _NeuroImage_ 224 Elsevier, 2021, pp. 117401
  * [11] Esten H Leonardsen, Han Peng, Tobias Kaufmann, Ingrid Agartz, Ole A Andreassen, Elisabeth Gulowsen Celius, Thomas Espeseth, Hanne F Harbo, Einar A Høgestøl and Ann-Marie Lange “Deep neural networks learn general and clinically relevant representations of the ageing brain” In _NeuroImage_ 256 Elsevier, 2022, pp. 119210
  * [12] Pranav Rajpurkar, Emma Chen, Oishi Banerjee and Eric J Topol “AI in health and medicine” In _Nature Medicine_ 28.1 Nature Publishing Group, 2022, pp. 31–38
  * [13] Raymond Pomponio, Guray Erus, Mohamad Habes, Jimit Doshi, Dhivya Srinivasan, Elizabeth Mamourian, Vishnu Bashyam, Ilya M Nasrallah, Theodore D Satterthwaite and Yong Fan “Harmonization of Large MRI Datasets for the Analysis of Brain Imaging Patterns throughout the Lifespan” In _NeuroImage_ 208, 2020, pp. 116450
  * [14] Pranav Rajpurkar and Matthew P Lungren “The Current and Future State of AI Interpretation of Medical Images” In _New England Journal of Medicine_ 388.21 Mass Medical Soc, 2023, pp. 1981–1990
  * [15] Christos Davatzikos “Machine Learning in Neuroimaging: Progress and Challenges” In _NeuroImage_ 197, 2019, pp. 652
  * [16] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge and Felix A Wichmann “Shortcut learning in deep neural networks” In _Nature Machine Intelligence_ 2.11 Nature Publishing Group, 2020, pp. 665–673
  * [17] Narmin Ghaffari Laleh, Daniel Truhn, Gregory Patrick Veldhuizen, Tianyu Han, Marko Treeck, Roman D Buelow, Rupert Langer, Bastian Dislich, Peter Boor and Volkmar Schulz “Adversarial attacks and adversarial robustness in computational pathology” In _Nature Communications_ 13.1 Nature Publishing Group, 2022, pp. 1–10
  * [18] Agostina J Larrazabal, Nicolás Nieto, Victoria Peterson, Diego H Milone and Enzo Ferrante “Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis” In _Proceedings of the National Academy of Sciences_ 117.23 National Acad Sciences, 2020, pp. 12592–12594
  * [19] Alex J DeGrave, Joseph D Janizek and Su-In Lee “AI for radiographic COVID-19 detection selects shortcuts over signal” In _Nature Machine Intelligence_ 3.7 Nature Publishing Group, 2021, pp. 610–619
  * [20] Manoj Kumar Kanakasabapathy, Prudhvi Thirumalaraju, Hemanth Kandula, Fenil Doshi, Anjali Devi Sivakumar, Deeksha Kartik, Raghav Gupta, Rohan Pooniwala, John A Branda and Athe M Tsibris “Adaptive adversarial neural networks for the analysis of lossy and domain-shifted datasets of medical images” In _Nature Biomedical Engineering_ 5.6 Nature Publishing Group, 2021, pp. 571–585
  * [21] Frederick M Howard, James Dolezal, Sara Kochanny, Jefree Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufunmilayo I Olopade and Jakob N Kather “The impact of site-specific digital histology signatures on deep learning model accuracy and bias” In _Nature Communications_ 12.1 Nature Publishing Group, 2021, pp. 1–13
  * [22] Oualid Benkarim, Casey Paquola, Bo-yong Park, Valeria Kebets, Seok-Jun Hong, Reinder Wael, Shaoshi Zhang, BT Thomas Yeo, Michael Eickenberg and Tian Ge “Population heterogeneity in clinical cohorts affects the predictive accuracy of brain imaging” In _PLoS Biology_ 20.4 Public Library of Science San Francisco, CA USA, 2022, pp. e3001627
  * [23] Jingwei Li, Danilo Bzdok, Jianzhong Chen, Angela Tam, Leon Qi Rong Ooi, Avram J Holmes, Tian Ge, Kaustubh R Patil, Mbemba Jabbi and Simon B Eickhoff “Cross-ethnicity/race generalization failure of behavioral prediction from resting-state functional connectivity” In _Science Advances_ 8.11 American Association for the Advancement of Science, 2022, pp. eabj1812
  * [24] Laleh Seyyed-Kalantari, Haoran Zhang, Matthew McDermott, Irene Y Chen and Marzyeh Ghassemi “Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations” In _Nature Medicine_ 27.12 Nature Publishing Group, 2021, pp. 2176–2182
  * [25] Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey and Olivier Gevaert “Disparities in dermatology AI performance on a diverse, curated clinical image set” In _Science Advances_ 8.31 American Association for the Advancement of Science, 2022, pp. eabq6147
  * [26] Jakub Kopal, Lucina Q Uddin and Danilo Bzdok “The end game: Respecting major sources of population diversity” In _Nature Methods_ Nature Publishing Group US New York, 2023, pp. 1–7
  * [27] Daniel Moyer and Polina Golland “Harmonization and the worst scanner syndrome” In _arXiv preprint arXiv:2101.06255_ , 2021
  * [28] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran and Stefano Soatto “A Baseline for Few-Shot Image Classification” In _Proc. of International Conference of Learning and Representations (ICLR)_ , 2020
  * [29] Hao Li, Pratik Chaudhari, Hao Yang, Michael Lam, Avinash Ravichandran, Rahul Bhotika and Stefano Soatto “Rethinking the Hyper-Parameters for Fine-Tuning” In _Proc. of International Conference of Learning and Representations (ICLR)_ , 2020
  * [30] Abigail S Greene, Xilin Shen, Stephanie Noble, Corey Horien, C Alice Hahn, Jagriti Arora, Fuyuze Tokoglu, Marisa N Spann, Carmen I Carrión and Daniel S Barron “Brain–phenotype models fail for individuals who defy sample stereotypes” In _Nature_ Nature Publishing Group, 2022, pp. 1–10
  * [31] Scott Marek, Brenden Tervo-Clemmens, Finnegan J Calabro, David F Montez, Benjamin P Kay, Alexander S Hatoum, Meghan Rose Donohue, William Foran, Ryland L Miller and Timothy J Hendrickson “Reproducible brain-wide association studies require thousands of individuals” In _Nature_ Nature Publishing Group, 2022, pp. 1–7
  * [32] Elvisha Dhamala, BT Thomas Yeo and Avram J Holmes “One size does not fit all: methodological considerations for brain-based predictive modeling in psychiatry” In _Biological Psychiatry_ 93.8 Elsevier, 2023, pp. 717–728
  * [33] Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf and Alexander Smola “A Kernel Two-Sample Test” In _Journal of Machine Learning Research_ 13.Mar, 2012, pp. 723–773
  * [34] Junhao Wen, Elina Thibeau-Sutre, Mauricio Diaz-Melo, Jorge Samper-González, Alexandre Routier, Simona Bottani, Didier Dormont, Stanley Durrleman, Ninon Burgos and Olivier Colliot “Convolutional Neural Networks for Classification of Alzheimer’s Disease: Overview and Reproducible Evaluation” In _Medical Image Analysis_ , 2020, pp. 101694
  * [35] Vishnu M Bashyam, Guray Erus, Jimit Doshi, Mohamad Habes, Ilya M Nasrallah, Monica Truelove-Hill, Dhivya Srinivasan, Liz Mamourian, Raymond Pomponio and Yong Fan “MRI Signatures of Brain Age and Disease over the Lifespan Based on a Deep Brain Network and 14 468 Individuals Worldwide” In _Brain_ 143.7, 2020, pp. 2312–2324
  * [36] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand and Victor Lempitsky “Domain-Adversarial Training of Neural Networks” In _The Journal of Machine Learning Research_ 17.1, 2016, pp. 2096–2030
  * [37] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani and David Lopez-Paz “Invariant Risk Minimization” In _arXiv preprint arXiv:1907.02893_ , 2019
  * [38] Qingyu Zhao, Ehsan Adeli and Kilian M Pohl “Training Confounder-Free Deep Learning Models for Medical Applications” In _Nature Communications_ 11.1, 2020, pp. 1–9
  * [39] Daniel Moyer, Greg Ver Steeg, Chantal MW Tax and Paul M Thompson “Scanner Invariant Representations for Diffusion MRI Harmonization” In _Magnetic Resonance in Medicine_ , 2020
  * [40] Nicola K Dinsdale, Mark Jenkinson and Ana IL Namburete “Deep learning-based unlearning of dataset bias for MRI harmonisation and confound removal” In _NeuroImage_ 228 Elsevier, 2021, pp. 117689
  * [41] Clifford R Jack, David S Knopman, William J Jagust, Leslie M Shaw, Paul S Aisen, Michael W Weiner, Ronald C Petersen and John Q Trojanowski “Hypothetical model of dynamic biomarkers of the Alzheimer’s pathological cascade” In _The Lancet Neurology_ 9.1 Elsevier, 2010, pp. 119–128
  * [42] Reisa A Sperling, Paul S Aisen, Laurel A Beckett, David A Bennett, Suzanne Craft, Anne M Fagan, Takeshi Iwatsubo, Clifford R Jack Jr, Jeffrey Kaye and Thomas J Montine “Toward defining the preclinical stages of Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease” In _Alzheimer’s & dementia_ 7.3 Elsevier, 2011, pp. 280–292
  * [43] Wenyong Zhu, Liang Sun, Jiashuang Huang, Liangxiu Han and Daoqiang Zhang “Dual attention multi-instance deep learning for Alzheimer’s disease diagnosis with structural MRI” In _IEEE Transactions on Medical Imaging_ 40.9 IEEE, 2021, pp. 2354–2366
  * [44] Esther E Bron, Stefan Klein, Janne M Papma, Lize C Jiskoot, Vikram Venkatraghavan, Jara Linders, Pauline Aalten, Peter Paul De Deyn, Geert Jan Biessels and Jurgen AHR Claassen “Cross-cohort generalizability of deep and conventional machine learning for MRI-based diagnosis and prediction of Alzheimer’s disease” In _NeuroImage: Clinical_ 31 Elsevier, 2021, pp. 102712
  * [45] Carolyn Lou, Mohamad Habes, Nicholas A Illenberger, Ali Ezzati, Richard B Lipton, Pamela A Shaw, Alisa J Stephens-Shields, Hamed Akbari, Jimit Doshi and Christos Davatzikos “Leveraging machine learning predictive biomarkers to augment the statistical power of clinical trials with baseline magnetic resonance imaging” In _Brain communications_ 3.4 Oxford University Press, 2021, pp. fcab264
  * [46] Suresh Balakrishnama and Aravind Ganapathiraju “Linear discriminant analysis-a brief tutorial” In _Institute for Signal and information Processing_ 18.1998 Mississippi, 1998, pp. 1–8
  * [47] James H Cole and Katja Franke “Predicting age using neuroimaging: innovative brain ageing biomarkers” In _Trends in neurosciences_ 40.12 Elsevier, 2017, pp. 681–690
  * [48] Stephen M Smith, Diego Vidaurre, Fidel Alfaro-Almagro, Thomas E Nichols and Karla L Miller “Estimation of brain age delta from brain imaging” In _Neuroimage_ 200 Elsevier, 2019, pp. 528–539
  * [49] Benedikt Atli Jónsson, Gyda Bjornsdottir, TE Thorgeirsson, Lotta María Ellingsen, G Bragi Walters, DF Gudbjartsson, Hreinn Stefansson, Kari Stefansson and MO Ulfarsson “Brain age prediction using deep learning uncovers associated sequence variants” In _Nature communications_ 10.1 Nature Publishing Group UK London, 2019, pp. 5409
  * [50] Zhen Zhou, Hongming Li, Dhivya Srinivasan, Ahmed Abdulkadir, Ilya M Nasrallah, Junhao Wen, Jimit Doshi, Guray Erus, Elizabeth Mamourian and Nick R Bryan “Multiscale functional connectivity patterns of the aging brain learned from harmonized rsfMRI data of the multi-cohort iSTAGING study” In _NeuroImage_ 269 Elsevier, 2023, pp. 119911
  * [51] Rory Boyle, Lee Jollans, Laura M Rueda-Delgado, Rossella Rizzo, Görsev G Yener, Jason P McMorrow, Silvin P Knight, Daniel Carey, Ian H Robertson and Derya D Emek-Savaş “Brain-predicted age difference score is related to specific cognitive functions: a multi-site replication analysis” In _Brain imaging and behavior_ 15 Springer, 2021, pp. 327–345
  * [52] Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko and Trevor Darrell “Deep Domain Confusion: Maximizing for Domain Invariance” In _arXiv preprint arXiv:1412.3474_ , 2014
  * [53] Mingsheng Long, Yue Cao, Jianmin Wang and Michael Jordan “Learning Transferable Features with Deep Adaptation Networks” In _International Conference on Machine Learning_ , 2015, pp. 97–105
  * [54] Baochen Sun, Jiashi Feng and Kate Saenko “Return of Frustratingly Easy Domain Adaptation” In _Thirtieth AAAI Conference on Artificial Intelligence_ , 2016
  * [55] Eric Tzeng, Judy Hoffman, Kate Saenko and Trevor Darrell “Adversarial Discriminative Domain Adaptation” In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 7167–7176
  * [56] Yen-Cheng Liu, Yu-Ying Yeh, Tzu-Chien Fu, Sheng-De Wang, Wei-Chen Chiu and Yu-Chiang Frank Wang “Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation” In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8867–8876
  * [57] Qingjie Meng, Daniel Rueckert and Bernhard Kainz “Learning Cross-Domain Generalizable Features by Representation Disentanglement” In _arXiv preprint arXiv:2003.00321_ , 2020
  * [58] Nicola K Dinsdale, Mark Jenkinson and Ana IL Namburete “Unlearning Scanner Bias for MRI Harmonisation” In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , 2020, pp. 369–378
  * [59] María Agustina Ricci Lara, Rodrigo Echeveste and Enzo Ferrante “Addressing fairness in artificial intelligence for medical imaging” In _Nature Communications_ 13.1 Nature Publishing Group, 2022, pp. 1–6
  * [60] Jack Wilkinson, Kellyn F Arnold, Eleanor J Murray, Maarten Smeden, Kareem Carr, Rachel Sippy, Marc Kamps, Andrew Beam, Stefan Konigorski and Christoph Lippert “Time to reality check the promises of machine learning-powered precision medicine” In _The Lancet Digital Health_ 2.12 Elsevier, 2020, pp. e677–e680
  * [61] Samuel G Finlayson, Adarsh Subbaswamy, Karandeep Singh, John Bowers, Annabel Kupke, Jonathan Zittrain, Isaac S Kohane and Suchi Saria “The clinician and dataset shift in artificial intelligence” In _The New England Journal of Medicine_ 385.3 NIH Public Access, 2021, pp. 283
  * [62] Michael Roberts, Derek Driggs, Matthew Thorpe, Julian Gilbey, Michael Yeung, Stephan Ursprung, Angelica I Aviles-Rivero, Christian Etmann, Cathal McCague and Lucian Beer “Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans” In _Nature Machine Intelligence_ 3.3 Nature Publishing Group, 2021, pp. 199–217
  * [63] Yan Gao and Yan Cui “Deep Transfer Learning for Reducing Health Care Disparities Arising from Biomedical Data Inequality” In _Nature communications_ 11.1, 2020, pp. 1–8
  * [64] Yeping Lina Qiu, Hong Zheng, Arnout Devos, Heather Selby and Olivier Gevaert “A Meta-Learning Approach for Genomic Survival Analysis” In _Nature communications_ 11.1, 2020, pp. 1–11
  * [65] Dani Kiyasseh, Tingting Zhu and David Clifton “A clinical deep learning framework for continually learning from cardiac signals across diseases, time, modalities, and institutions” In _Nature Communications_ 12.1 Nature Publishing Group, 2021, pp. 1–11
  * [66] Matthias Perkonigg, Johannes Hofmanninger, Christian J Herold, James A Brink, Oleg Pianykh, Helmut Prosch and Georg Langs “Dynamic memory to alleviate catastrophic forgetting in continual learning with medical imaging” In _Nature Communications_ 12.1 Nature Publishing Group, 2021, pp. 1–12
  * [67] Rongguang Wang, Pratik Chaudhari and Christos Davatzikos “Embracing the disharmony in medical imaging: A simple and effective framework for domain adaptation” In _Medical Image Analysis_ 76 Elsevier, 2022, pp. 102309
  * [68] Tong He, Lijun An, Pansheng Chen, Jianzhong Chen, Jiashi Feng, Danilo Bzdok, Avram J Holmes, Simon B Eickhoff and BT Yeo “Meta-matching as a simple framework to translate phenotypic predictive models from big to small data” In _Nature Neuroscience_ Nature Publishing Group, 2022, pp. 1–10
  * [69] Shangran Qiu, Prajakta S Joshi, Matthew I Miller, Chonghua Xue, Xiao Zhou, Cody Karjadi, Gary H Chang, Anant S Joshi, Brigid Dwyer and Shuhan Zhu “Development and validation of an interpretable deep learning framework for Alzheimer’s disease classification” In _Brain_ 143.6 Oxford University Press, 2020, pp. 1920–1933
  * [70] Lei Xu, Hui Wu, Chunming He, Jun Wang, Changqing Zhang, Feiping Nie and Lei Chen “Multi-modal sequence learning for Alzheimer’s disease progression prediction with incomplete variable-length longitudinal data” In _Medical Image Analysis_ 82 Elsevier, 2022, pp. 102643
  * [71] Hong-Yu Zhou, Yizhou Yu, Chengdi Wang, Shu Zhang, Yuanxu Gao, Jia Pan, Jun Shao, Guangming Lu, Kang Zhang and Weimin Li “A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics” In _Nature Biomedical Engineering_ Nature Publishing Group UK London, 2023, pp. 1–13
  * [72] Marc-Andre Schulz, BT Thomas Yeo, Joshua T Vogelstein, Janaina Mourao-Miranada, Jakob N Kather, Konrad Kording, Blake Richards and Danilo Bzdok “Different Scaling of Linear Models and Deep Learning in UKBiobank Brain Images versus Machine-Learning Datasets” In _Nature communications_ 11.1, 2020, pp. 1–15
  * [73] Benoit Dufumier “Representation learning in neuroimaging: Transferring from big healthy data to small clinical cohorts” In _Université Paris-Saclay_ , 2022
  * [74] Rongguang Wang, Pratik Chaudhari and Christos Davatzikos “Bias in Machine Learning Models Can Be Significantly Mitigated by Careful Training: Evidence from Neuroimaging Studies” In _Proceedings of the National Academy of Sciences_ 120.6, 2023, pp. e2211613120
  * [75] Moritz Hardt, Eric Price and Nati Srebro “Equality of opportunity in supervised learning” In _Advances in neural information processing systems_ 29, 2016
  * [76] Alekh Agarwal, Miroslav Dudík and Zhiwei Steven Wu “Fair regression: Quantitative definitions and reduction-based algorithms” In _International Conference on Machine Learning_ , 2019, pp. 120–129 PMLR
  * [77] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger and Suresh Venkatasubramanian “Certifying and removing disparate impact” In _proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining_ , 2015, pp. 259–268
  * [78] Theodore D Satterthwaite, Daniel H Wolf, James Loughead, Kosha Ruparel, Jeffrey N Valdez, Steven J Siegel, Christian G Kohler, Raquel E Gur and Ruben C Gur “Association of Enhanced Limbic Response to Threat with Decreased Cortical Facial Recognition Memory Response in Schizophrenia” In _American Journal of Psychiatry_ 167.4, 2010, pp. 418–426
  * [79] Daniel H Wolf, Theodore D Satterthwaite, Jacob J Kantrowitz, Natalie Katchmar, Lillie Vandekar, Mark A Elliott and Kosha Ruparel “Amotivation in Schizophrenia: Integrated Assessment with Behavioral, Clinical, and Imaging Measures” In _Schizophrenia bulletin_ 40.6, 2014, pp. 1328–1337
  * [80] Tianhao Zhang, Nikolaos Koutsouleris, Eva Meisenzahl and Christos Davatzikos “Heterogeneity of Structural Brain Changes in Subtypes of Schizophrenia Revealed Using Magnetic Resonance Imaging Pattern Analysis” In _Schizophrenia bulletin_ 41.1, 2015, pp. 74–84
  * [81] Jiajia Zhu, Chuanjun Zhuo, Feng Liu, Lixue Xu and Chunshui Yu “Neural Substrates Underlying Delusions in Schizophrenia” In _Scientific reports_ 6.1, 2016, pp. 1–10
  * [82] Chuanjun Zhuo, Xiaolei Ma, Hongru Qu, Lina Wang, Feng Jia and Chunli Wang “Schizophrenia Patients Demonstrate Both Inter-Voxel Level and Intra-Voxel Level White Matter Alterations” In _PLoS One_ 11.9, 2016, pp. e0162656
  * [83] Kathryn A Ellis, Christopher C Rowe, Victor L Villemagne, Ralph N Martins, Colin L Masters, Olivier Salvado, Cassandra Szoeke, David Ames and AIBL Research Group “Addressing Population Aging and Alzheimer’s Disease through the Australian Imaging Biomarkers and Lifestyle Study: Collaboration with the Alzheimer’s Disease Neuroimaging Initiative” In _Alzheimer’s & dementia_ 6.3, 2010, pp. 291–296
  * [84] Pamela J LaMontagne, Tammie LS Benzinger, John C Morris, Sarah Keefe, Russ Hornbeck, Chengjie Xiong, Elizabeth Grant, Jason Hassenstab, Krista Moulder and Andrei G Vlassenko “OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease” In _MedRxiv_ Cold Spring Harbor Laboratory Press, 2019
  * [85] Marilyn Albert, Anja Soldan, Rebecca Gottesman, Guy McKhann, Ned Sacktor, Leonie Farrington, Maura Grega, Raymond Turner, Yi Lu and Shanshan Li “Cognitive changes preceding clinical symptom onset of mild cognitive impairment and relationship to ApoE genotype” In _Current Alzheimer Research_ 11.8 Bentham Science Publishers, 2014, pp. 773–784
  * [86] Susan M Resnick, Dzung L Pham, Michael A Kraut, Alan B Zonderman and Christos Davatzikos “Longitudinal magnetic resonance imaging studies of older adults: a shrinking brain” In _Journal of Neuroscience_ 23.8 Soc Neuroscience, 2003, pp. 3295–3301
  * [87] Nicole M Armstrong, Yang An, Lori Beason-Held, Jimit Doshi, Guray Erus, Luigi Ferrucci, Christos Davatzikos and Susan M Resnick “Predictors of neurodegeneration differ between cognitively normal and subsequently impaired older adults” In _Neurobiology of aging_ 75 Elsevier, 2019, pp. 178–186
  * [88] Gary D Friedman, Gary R Cutter, Richard P Donahue, Glenn H Hughes, Stephen B Hulley, David R Jacobs Jr, Kiang Liu and Peter J Savage “CARDIA: study design, recruitment, and some characteristics of the examined subjects” In _Journal of clinical epidemiology_ 41.11 Elsevier, 1988, pp. 1105–1116
  * [89] Karina Davidson, Bruce S Jonas, Kim E Dixon and Jerome H Markovitz “Do depression symptoms predict early hypertension incidence in young adults in the CARDIA study?” In _Archives of internal medicine_ 160.10 American Medical Association, 2000, pp. 1495–1500
  * [90] Henry Völzke, Dietrich Alte, Carsten Oliver Schmidt, Dörte Radke, Roberto Lorbeer, Nele Friedrich, Nicole Aumann, Katharina Lau, Michael Piontek and Gabriele Born “Cohort profile: the study of health in Pomerania” In _International journal of epidemiology_ 40.2 Oxford University Press, 2011, pp. 294–307
  * [91] Ilya M Nasrallah, Nicholas M Pajewski, Alexander P Auchus, Gordon Chelune, Alfred K Cheung, Maryjo L Cleveland, Laura H Coker, Michael G Crowe, William C Cushman and Jeffrey A Cutler “Association of intensive vs standard blood pressure control with cerebral white matter lesions” In _Jama_ 322.6 American Medical Association, 2019, pp. 524–534
  * [92] Cathie Sudlow, John Gallacher, Naomi Allen, Valerie Beral, Paul Burton, John Danesh, Paul Downey, Paul Elliott, Jane Green and Martin Landray “UK biobank: an open access resource for identifying the causes of a wide range of complex diseases of middle and old age” In _PLoS medicine_ 12.3 Public Library of Science, 2015, pp. e1001779
  * [93] LH Coker, PE Hogan, NR Bryan, LH Kuller, KL Margolis, Kerstin Bettermann, RB Wallace, Z Lao, R Freeman and ML Stefanick “Postmenopausal hormone therapy and subclinical cerebrovascular disease: the WHIMS-MRI Study” In _Neurology_ 72.2 AAN Enterprises, 2009, pp. 125–134
  * [94] Sterling C Johnson, Rebecca L Koscik, Erin M Jonaitis, Lindsay R Clark, Kimberly D Mueller, Sara E Berman, Barbara B Bendlin, Corinne D Engelman, Ozioma C Okonkwo and Kirk J Hogan “The Wisconsin Registry for Alzheimer’s Prevention: a review of findings and current directions” In _Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring_ 10 Elsevier, 2018, pp. 130–142
  * [95] Mark A Espeland, Kirk Erickson, Rebecca H Neiberg, John M Jakicic, Thomas A Wadden, Rena R Wing, Lisa Desiderio, Guray Erus, Meng-Kang Hsieh and Christos Davatzikos “Brain and white matter hyperintensity volumes after 10 years of random assignment to lifestyle intervention” In _Diabetes care_ 39.5 Am Diabetes Assoc, 2016, pp. 764–771
  * [96] Nicholas J Tustison, Brian B Avants, Philip A Cook, Yuanjie Zheng, Alexander Egan, Paul A Yushkevich and James C Gee “N4ITK: Improved N3 Bias Correction” In _IEEE transactions on medical imaging_ 29.6, 2010, pp. 1310–1320
  * [97] Jimit Doshi, Guray Erus, Yangming Ou, Bilwaj Gaonkar and Christos Davatzikos “Multi-Atlas Skull-Stripping” In _Academic radiology_ 20.12, 2013, pp. 1566–1576
  * [98] Jimit Doshi, Guray Erus, Yangming Ou, Susan M Resnick, Ruben C Gur, Raquel E Gur, Theodore D Satterthwaite, Susan Furth, Christos Davatzikos and Alzheimer’s Neuroimaging Initiative “MUSE: MUlti-atlas Region Segmentation Utilizing Ensembles of Registration Algorithms and Parameters, and Locally Optimal Atlas Selection” In _Neuroimage_ 127, 2016, pp. 186–195
  * [99] Jimit Doshi, Guray Erus, Mohamad Habes and Christos Davatzikos “DeepMRSeg: A convolutional deep neural network for anatomy and abnormality segmentation on MR images” In _arXiv preprint arXiv:1907.02110_ , 2019
  * [100] Rongguang Wang, Pratik Chaudhari and Christos Davatzikos “Harmonization with Flow-Based Causal Inference” In _Proc. of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)_ , 2021
  * [101] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” In _JMLR_ 15.1, 2014, pp. 1929–1958
  * [102] Sergey Ioffe and Christian Szegedy “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift” In _International Conference on Machine Learning_ , 2015, pp. 448–456
  * [103] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun “Deep Residual Learning for Image Recognition” In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778
  * [104] Leo Breiman “Bagging Predictors” In _Machine learning_ 24.2, 1996, pp. 123–140
  * [105] Peter Bartlett, Yoav Freund, Wee Sun Lee and Robert E Schapire “Boosting the margin: A new explanation for the effectiveness of voting methods” In _The annals of statistics_ 26.5 Institute of Mathematical Statistics, 1998, pp. 1651–1686
  * [106] David H Wolpert “Stacked generalization” In _Neural networks_ 5.2 Elsevier, 1992, pp. 241–259
  * [107] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li and Alexander Smola “AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data” In _arXiv preprint arXiv:2003.06505_ , 2020
  * [108] Bambang Parmanto, Paul W Munro and Howard R Doyle “Reducing variance of committee prediction with resampling techniques” In _Connection Science_ 8.3-4 Taylor & Francis, 1996, pp. 405–426
  * [109] Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush and Andrey Gulin “CatBoost: unbiased boosting with categorical features” In _Advances in neural information processing systems_ 31, 2018
  * [110] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye and Tie-Yan Liu “LightGBM: A highly efficient gradient boosting decision tree” In _Advances in neural information processing systems_ 30, 2017
  * [111] Rich Caruana, Alexandru Niculescu-Mizil, Geoff Crew and Alex Ksikes “Ensemble Selection from Libraries of Models” In _ICML_ , 2004
  * [112] Bernhard Schölkopf and Alexander J Smola “Learning with Kernels”, 2002
  * [113] George Cybenko “Approximation by Superpositions of a Sigmoidal Function” In _Mathematics of control, signals and systems_ 2.4, 1989, pp. 303–314
  * [114] Bharath K Sriperumbudur, Arthur Gretton, Kenji Fukumizu, Bernhard Schölkopf and Gert RG Lanckriet “Hilbert Space Embeddings and Metrics on Probability Measures” In _The Journal of Machine Learning Research_ 11, 2010, pp. 1517–1561
  * [115] Shai Ben-David, John Blitzer, Koby Crammer and Fernando Pereira “Analysis of Representations for Domain Adaptation” In _Advances in neural information processing systems_ 19, 2006
  * [116] Léon Bottou “Large-Scale Machine Learning with Stochastic Gradient Descent” In _Proceedings of COMPSTAT’2010_ , 2010, pp. 177–186
  * [117] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira and Jennifer Wortman Vaughan “A Theory of Learning from Different Domains” In _Machine learning_ 79.1, 2010, pp. 151–175
  * [118] Rahul Ramesh and Pratik Chaudhari “Model Zoo: A Growing "Brain" That Learns Continually” In _Proc. of International Conference of Learning and Representations (ICLR)_ , 2022
  * [119] Shai Ben-David and Reba Schuller “Exploiting Task Relatedness for Multiple Task Learning” In _Learning Theory and Kernel Machines_ , 2003, pp. 567–580
  * [120] Ashwin De Silva, Rahul Ramesh, Carey E Priebe, Pratik Chaudhari and Joshua T Vogelstein “The Value of Out-of-Distribution Data” In _Proc. of International Conference of Machine Learning (ICML)_ , 2023
  * [121] Robert W Keener “Theoretical Statistics: Topics for a Core Course”, 2010
  * [122] Yansong Gao, Rahul Ramesh and Pratik Chaudhari “Deep Reference Priors: What Is the Best Way to Pretrain a Model?” In _Proc. of International Conference of Machine Learning (ICML)_ , 2022
  * [123] Yansong Gao and Pratik Chaudhari “An Information-Geometric Distance on the Space of Tasks” In _Proc. of International Conference of Machine Learning (ICML)_ , 2021
  * [124] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto and Alexander J Smola “Meta-Q-Learning” In _Proc. of International Conference of Learning and Representations (ICLR)_ , 2020

##  Appendix A Additional Tables and Figures

Table S.1: Summary of the data from the iSTAGING consortium used for early
diagnosis of stable and progressive mild cognitive impairment.

Mild Cognitive |  | ADNI-1 | ADNI-2/3 | PENN | AIBL | Total
---|---|---|---|---|---|---
Impairment |  | (44.60%) | (47.69%) | (2.31%) | (5.40%) |
Subjects |  |  |  |  |  |
| Progressive MCI | 197 | 113 | 5 | 12 | 327
| Stable MCI | 79 | 151 | 10 | 11 | 251
| Normal MCI | 13 | 45 | - | 12 | 70
Sex (%) |  |  |  |  |  |
| Female | 15.90 | 21.30 | 1.08 | 2.01 | 40.28
| Male | 28.70 | 26.39 | 1.23 | 3.40 | 59.72
Age (%, years) |  |  |  |  |  |
| 0–65 | 5.86 | 10.80 | 0.62 | 0.15 | 17.44
| 65–70 | 5.86 | 11.27 | 0.46 | 1.39 | 18.98
| 70–75 | 11.57 | 11.73 | 0.46 | 1.85 | 25.62
| 75–80 | 10.19 | 11.73 | 0.62 | 1.39 | 21.60
| > 80 | 11.11 | 4.48 | 0.15 | 0.62 | 16.36
Race (%) |  |  |  |  |  |
| White | 42.75 | 43.83 | 2.16 | 2.16 | 90.90
| Black | 0.77 | 0.93 | 0.15 | - | 1.85
| Asian | 1.08 | 0.62 | - | - | 1.70

(a)

(b)

(c)

Figure S.1:  Discrepancy in the data and AUC of diagnostic models of
Alzheimer’s disease on different groups in the population. (a-b) Distance
between leaves of this dendrogram in (b) indicates the pairwise MMD statistic
in (a) between learned features of pairs of groups, e.g., distributional
discrepancy between Male-Female groups is 0.17, while the distributional
discrepancy between < 65 years and > 80 years, or between ADNI-1 and ADNI-2/3,
is larger (0.42 and 0.26 respectively). (c) Average AUC of Alzheimer’s disease
classification computed using five-fold nested cross-validation. We trained a
machine learning model, either a deep neural network (translucent markers) or
an ensemble using boosting, bagging and stacking (bold markers), using data
from different source groups (different colors) and evaluated this model
(cross marks) on data from different target groups (X-axis); circles denote
model fitted using our $\alpha$-weighted empirical risk minimization (ERM)
procedure with access to 10% data from the target group; horizontal lines
denote models that are directly trained on the target group using 80% of data
(the rest for testing). All models use data from multiple sources, namely
structural measures, demographic, clinical variables, genetic factors, and
cognitive scores.

(a)

(b)

(c)

Figure S.2:  Discrepancy in the data and AUC of diagnostic models of
schizophrenia on different groups in the population. (a-b) Distance between
leaves of this dendrogram in (b) indicates the pairwise MMD statistic in (a)
between learned features of pairs of groups. (c) Average AUC of schizophrenia
classification computed using five-fold nested cross-validation. We trained a
deep neural network (translucent markers) and an ensemble using boosting,
bagging and stacking (bold markers), using data from different source groups
(different colors) and evaluated this model (cross marks) on data from
different target groups (X-axis); circles denote model fitted using our
$\alpha$-weighted empirical risk minimization (ERM) procedure with access to
10% data from the target group; horizontal lines denote models that are
directly trained on the target group using 80% of data (the rest for testing).
Similar to diagnostic models of Alzheimer’s disease, in general, (i) the AUC
of ensemble models is higher than that of the neural network in all cases ($p$
< 0.01), (ii) AUC of a model trained on a source group remains remarkably high
when evaluated on the target group (crosses), (iii) in almost all cases for
the ensemble, it further improves when one has access to a small fraction of
data from the target group (circles are higher than crosses), and (iv) some
times even beyond the AUC of a model trained only on the target group (circles
above horizontal lines).

(a)

(b)

(c)

Figure S.3:  Discrepancy in the data and MAE (in years) models that predict
the brain age. (a-b) Distance between leaves of this dendrogram in (b)
indicates the pairwise MMD statistic in (a) between learned features of pairs
of groups. (c) Average AUC of schizophrenia classification computed using
five-fold nested cross-validation. We trained a deep neural network
(translucent markers) and an ensemble using boosting, bagging and stacking
(bold markers), using data from different source groups (different colors) and
evaluated this model (cross marks) on data from different target groups
(X-axis); circles denote model fitted using our $\alpha$-weighted empirical
risk minimization (ERM) procedure with access to 10% data from the target
group; horizontal lines denote models that are directly trained on the target
group using 80% of data (the rest for testing). Similar to diagnostic models
of Alzheimer’s disease and schizophrenia, in general, (i) the MAE of ensemble
models is lower than that of the neural network in all cases ($p$ < 0.01), and
(ii) in almost all cases for the ensemble, the MAE improves when one has
access to a small fraction of data from the target group (circles are lower
than crosses).

Figure S.4:  Average AUC of Alzheimer’s disease classification computed using
five-fold nested cross-validation. We trained a machine learning model, either
a deep neural network (translucent markers) or an ensemble using boosting,
bagging and stacking (bold markers), using data from different source groups
(different colors); circles and triangles denote model fitted using our
$\alpha$-weighted empirical risk minimization (ERM) procedure with access to
10% and 20% data respectively from the target group; horizontal lines denote
models that are directly trained on the target group using 80% of data (the
rest for testing). All models use data from multiple sources, namely
structural measures, demographic, clinical variables, genetic factors, and
cognitive scores. In some cases, the AUC using 20% data is better (triangles
above circles) but overall the AUC of models trained with 20% target data is
statistically the same as that of using 10% data. This indicates that we can
fruitfully predict the Alzheimer’s disease diagnosis for subjects from
different groups in the population with access to as little as 10% data from
those groups.

Figure S.5:  Average AUC of schizophrenia classification computed using five-
fold nested cross-validation. We trained a machine learning model, either a
deep neural network (translucent markers) or an ensemble using boosting,
bagging and stacking (bold markers), using data from different source groups
(different colors); circles and triangles denote model fitted using our
$\alpha$-weighted empirical risk minimization (ERM) procedure with access to
10% and 20% data respectively from the target group; horizontal lines denote
models that are directly trained on the target group using 80% of data (the
rest for testing). All models use data from multiple sources, namely
structural measures, demographic, clinical variables, genetic factors, and
cognitive scores. In some cases, the AUC using 20% data is better (triangles
above circles) but overall the AUC of models trained with 20% target data is
statistically the same as those trained with 10% data. This indicates that we
can fruitfully predict the schizophrenia diagnosis for subjects from different
groups in the population with access to as little as 10% data from those
groups.

Figure S.6:  Average mean average error (MAE) in years of predicting the brain
age, computed using five-fold nested cross-validation. The setup is the same
as that of Figs. S.4 and S.5. For brain age prediction, again, we see that the
MAE obtained using 20% data is some times better than of 10% data (triangles
below circles) but overall the two settings achieve the same MAE
statistically. This indicates that we can fruitfully predict the brain age of
subjects from different groups in the population with access to as little as
10% data from those groups.

(a)

(b)

Figure S.7:  Linear discriminant analysis on the output probabilities (that
determines AD vs. control) of the ensemble models trained for Alzheimer’s
disease diagnosis is used to study whether subjects with mild cognitive
impairment (MCI) progress to AD (known as pMCI) or remain stable MCI (known as
sMCI) in (a), and pMCI vs. nMCI in (b) where the latter refers to MCI subjects
diagnosed as normal within three years of the first clinical visit. The AUC of
pMCI vs. sMCI on the target group is shown for three different attributes
(sex, age group and race) when models are trained only on data from the source
group (crosses), using $\alpha$-weighted ERM using all data from the source
and 10% data from the target group (circles) and with access to only all data
from the target group (horizontal lines). Improvements in the AD vs. control
AUC of these models with 10% data translate to improvements in the ability to
distinguish between pMCI and sMCI subjects, using only baseline scans (circles
above cross) except when target groups are Black or Asian (due to very little
data in these groups).

Figure S.8:  Correlation of brain age regression with cardiovascular and
lifestyle factors. Pearson’s correlation between the brain age residual
(predicted brain age minus chronological age) and clinical factors for two
different attributes (sex and race) for models trained only on source data
(crosses), using $\alpha$-weighted ERM on all source data and 10% target data
(circles) and only on all target data (horizontal lines). Unlike other plots,
colors denote different pairs of source and target groups. Tests (X-axis)
marked in red are expected to be negatively correlated with brain aging
whereas those marked in black are expected to be positively correlated with
brain aging according to the existing literature. Body mass index (BMI) is an
individual’s health indicator based on tissue mass (muscle, fat, and bone) and
height. Systolic and diastolic blood pressures are the measures of heart
functioning. In almost all cases, we observe weak correlations which are
statistically insignificant ($p$-value > 0.01).

(a) Alzheimer’s disease

(b) Schizophrenia

(c) Brain age prediction

Figure S.9:  Fairness assessment of machine learning models with respect to
sensitive attributes including sex, age group, race, and clinical study.
Violin plots denote the test demographic parity differences (DPD) and
equalized odds difference (EOD) on five different held-out subsets of data.
Translucent gray denotes the performance of a baseline deep network while
translucent colors indicate the ensemble models. Solid colors indicate using
$\alpha$-weighted ERM on all source data and 10% target data. White dots
denote the average performance of each group. A model is perfectly fair if the
fairness disparity (DPD or EOD) is zero. In general, we observe that our
$\alpha$-weighted ERM models have lower disparities for all sensitive
attributes in both metrics compared to the neural net and ensemble models.