# Challenges for machine learning in clinical translation of big data imaging
studies

Nicola K. Dinsdale  Wellcome Centre for Integrative Neuroimaging, FMRIB,
Nuffield Department of Clinical Neurosciences, University of Oxford, UK
Institute of Biomedical Engineering, Department of Engineering Science,
University of Oxford, UK  Australian Institute for Machine Learning (AIML),
School of Computer Science, University of Adelaide, Adelaide, Australia  South
Australian Health and Medical Research Institute (SAHMRI), North Terrace,
Adelaide, Australia  Emma Bluemke  Vaanathi Sundaresan  Mark Jenkinson
Stephen M Smith  Ana I. L. Namburete

###### Abstract

The combination of deep learning image analysis methods and large-scale
imaging datasets offers many opportunities to imaging neuroscience and
epidemiology. However, despite the success of deep learning when applied to
many neuroimaging tasks, there remain barriers to the clinical translation of
large-scale datasets and processing tools. Here, we explore the main
challenges and the approaches that have been explored to overcome them. We
focus on issues relating to data availability, interpretability, evaluation
and logistical challenges, and discuss the challenges we believe are still to
be overcome to enable the full success of big data deep learning approaches to
be experienced outside of the research field.

###### keywords:

Neuroimaging, Deep Learning, Clinical Translation

††journal: Neuron

##  1 Introduction

Across neuroimaging, the majority of datasets have been limited to small-scale
collections, typically focusing on a specific research question or clinical
population of interest. Recently, however, large scale ‘big data’ collections
have begun to be collated, many of which are openly available to researchers,
meaning that, if the acquisition protocol, demographic and non-imaging
information of the data meets the requirements of the study, novel research
can be completed without having to acquire new scans. The sharing of these
large-scale datasets has had many benefits: not only do they enable the
exploration of new research questions, they also enable reproducibility and
quicker method prototyping.

Existing large-scale datasets have been curated to explore different research
questions with varying numbers of subjects and imaging sites across studies.
For instance, if the research question were about lifespan and ageing,
datasets to consider would include: IXI111 https://brain-development.org/ixi-
dataset ($n=581$), NKI-RS (Nooner et al., 2012) ($n>1000$), UK Biobank (Sudlow
et al., 2015) ($n=100,000$), CamCAN (Taylor et al., 2017) ($n=700$) and
Lifespan HCP in Ageing (Bookheimer et al., 2018) ($n>1200$). Similarly, if one
was interested in early development, available datasets include: dHCP (Hughes
et al., 2017) ($n>1000$) and ABCD (Marek et al., 2019) ($n>10,000$), or for
research on young adults, one could consider: HCP Young Adult (Van Essen et
al., 2013) (n=1200), GSP (Holmes et al., 2015) ($n=1070$) and CoRR (Zuo et
al., 2015) ($n=1629$). Datasets also exist that explore specific clinical
groups, such as Alzheimer’s (OASIS 3 (Marcus et al., 2007) ($n=1098$) and ADNI
(Jack et al., 2008) ($n>3500$)), Autism (ABIDE (Di Martino et al., 2013)
($n=1112$)), and schizophrenia and bipolar disorder (CANDI (Frazier et al.,
2008) ($n=103$)). These datasets allow the exploration of questions that would
not be possible with traditional small-scale studies where, for instance, the
statistical power would not be sufficient to find significant results. For
instance, if we are interested in exploring individual differences from the
average trajectory rather than differences within-subject or group, we are
likely to require a larger sample size afforded by the large-scale datasets
(Madan, 2021). Large-scale studies have also enabled the characterisation of
potential subtypes within patient samples – for example, (Young et al., 2018)
demonstrated heterogeneity and subtypes in atrophy patterns due to Alzheimer’s
disease using data from ADNI.

UK Biobank (Sudlow et al., 2015) is the largest of these studies, with a goal
of collecting brain imaging data from 100,000 volunteers, including 6 MRI
modalities, to study structure, function, and connectivity. UK Biobank also
contains large quantities of lifestyle, genetic and health measures, which
allow researchers to create models of population ageing and model how genetic
and environmental factors interplay with this process. For instance, the
atrophy of the hippocampus is a well-validated biomarker for Alzheimer’s
disease, and so, using the UK Biobank, a nomogram of hippocampal volume with
normal ageing has been created (Nobis et al., 2019), illustrating the
progression with age and percentiles of expected volume across the population.

For population-based models to have maximal impact they need to be
translatable – that is, they need to have genuine clinical impact beyond the
research field. Having created a model of healthy ageing, when a patient
arrives in the clinic we wish to be able compare their MRI scan to normative
distributions. This would then allow us to identify whether the patient was
ageing differently to the population average, and potentially identify the
need for intervention. Unfortunately, comparing a scan to the normative model
is not as simple as merely acquiring a scan and making that comparison. First,
research data differs from clinical data in terms of quality, acquisition and
purpose and, further, MRI data acquired on different scanners or with
different protocols may differ in characteristics to such an extent that
comparisons with the model are no longer valid. These ‘batch effects’ also
cause a harmonisation problem within many large studies, where data collected
on different scanners within the study contain bias introduced by the effects
of the particular scanner hardware and acquisition protocol on the image
characteristics. Second, the populations may differ so much that the patient
falls outside of the population modelled, and it cannot be known if the models
will extrapolate. For instance, as UK Biobank spans subjects from 45-85 years
of age, any model developed on this data is unlikely to extrapolate well to
subjects much younger than that range. Third, logistical challenges make the
deployment of the models difficult in a clinical setting, potentially to the
point where it is not currently possible for them to be used outside of the
research world, for instance the dependence on GPUs for processing which are
unlikely to be available. Further, many research studies utilise functional
and diffusion MRI which are often unavailable clinically due to the lack of
expertise and equipment, and long acquisition times being impractical (e.g.,
to match the quality or time taken over research scans). Functional
connectivity analyses, for instance, have become a prominent approach for
examining individual differences, driven by the availability of high quality
data from the large-scale studies. Ultimately, if this data cannot be acquired
clinically, the utility of any model is limited. Solutions to these problems,
however, are beginning to be developed and deep learning offers potential
opportunities.

Due to the growth in size of the datasets, deep learning models are now an
option for neuroimaging analysis, enabling us to explore new questions in a
data-driven manner. Applications of deep learning techniques to neuroimaging
data have been explored in a research setting, with increasing numbers of
novel methods proposed year on year. Powered by their ability to learn
complex, non-linear relationships and patterns from data, deep learning
methods have been applied to a wide range of applications and have found
success in previously unsolved problems. However, challenges for applying deep
learning models to the clinical domain remain. These challenges limit the
impact that big datasets such as the UK Biobank are currently able to have on
patient care, and work must be undertaken to allow models to extend beyond the
research domain. Recent developments in deep learning have begun to tackle the
problems faced, but further developments are needed. Here, we will discuss the
challenges being faced and current approaches being developed to mitigate
them, covering challenges around data availability, interpretability, model
evaluation and logistical challenges including data privacy. We will also
identify and explore the barriers we believe still need to be overcome.

Figure 1: An example network architecture for a convolutional neural network
(CNN) for a classification or regression task.

###  1.1 Deep Learning Background

To understand the challenges for clinical translatability of deep learning
methods, we first require an overview of how these methods approach problems –
for a more detailed introduction see, for example, (LeCun et al., 2015). We
will only consider convolutional neural networks (CNNs), which form the vast
majority of deep learning methods currently applied in medical imaging, an
example architecture of which is shown in Fig. 1. The majority are supervised
approaches (LeCun et al., 2015), meaning that to explore the research
question, we need access to a dataset of images, $\bm{X}$, and the set of
known true labels, $\bm{y}$, for the task we wish to explore. This requires an
understanding of the information that we expect to be encoded within the
images and an understanding of which questions are of interest, defined as
domain knowledge. Examples for the $\bm{X}$ and $\bm{y}$ data could be a
structural scan with an associated segmentation mask, or multiple modality
data for $\bm{X}$, with the label being disease prognosis.

Having curated and appropriately preprocessed the data (for instance, skull
stripping, bias field correction, see e.g. (Manjón, 2017)) the task is then to
design a neural network architecture which we expect to be able to map from
$\bm{X}$ to $\bm{y}$ through learning a highly non-linear mapping function
$f(\bm{X},\bm{y};\bm{W})$, where $\bm{W}$ are the trainable weights of the
neural network. The choice of architecture is highly influenced by a variety
of factors: the task being explored, the quantity of data available, and the
computational power available, and is again highly influenced by domain
knowledge.

Nevertheless, most networks are formed through the same basic building blocks.
The first are convolutional filters, which learn features of interest from the
data, encoding spatial relationships between pixels by learning the image
features from small patches of the input data, determined by the filter’s
kernel size. These layers, then, complete feature extraction. They contain the
weights and biases which need to be learned during the optimisation process,
$\bm{W}=\\{\bm{w},b\\}$, and stacks of these layers are placed at different
spatial resolutions in order for a range of different features to be extracted
at each level of abstraction, allowing us to create a rich understanding of
the input data. During the forward pass of the back propagation training
procedure (LeCun et al., 1990a), each filter is convolved across the width and
height of the input volume. This means that the network learns filters which
activate when specific features are detected, and the exact nature of the
features is learned through a network optimisation procedure that updates the
filter weights, in order to find features that are useful contributors to the
overall goal of predicting $\bm{y}$.

Figure 2: Example extracted features, $\bm{z}_{l}$, for brain age prediction
(see (Dinsdale et al., 2021a)) at different network depths and spatial
resolutions. It can be seen that several different features are extracted and
some features are repeated, showing redundancy in filters.

The next components are the activation functions which play a fundamental role
in the model training. The activation functions apply a non-linear
transformation to the data, after it has been weighted by the convolutional
layers. This non-linearity provides a distinct edge to CNNs, allowing them to
learn the complex non-linear relationships (or mapping) between the input and
the output. Without the non-linear activation functions, CNNs would be
rendered as only linear models, despite the flexibility to learn the weights.
Commonly used activation functions include rectified linear units (ReLU, e.g.,
zeroing negative values and keeping positive ones unchanged) (Glorot et al.,
2011) and sigmoid (e.g., squashing large values down to ceilings). Therefore,
the features, $\bm{z}_{l}$, at layer depth $l$ are given by
$\bm{z}_{l}=ReLU(\bm{z}_{(l-1)}*\bm{w}_{l}+b_{l})$ and we can see that, due to
the CNN’s sequential data flow, the features at a given depth are a non-linear
combination of the previous features and the weights and biases. Therefore,
despite the sequential nature of CNNs, without the activation functions we
would only be able to train linear models. Example extracted features,
$\bm{z}_{l}$, after the activation function, can be seen in Fig. 2.

The networks then learn features at different spatial resolutions through the
inclusion of pooling blocks, which reduce the input over a certain region to a
single value, thus sub-sampling the data and condensing information. Pooling
condenses the intensity-based information, provides a basic invariance to
rotations and translations, and has been demonstrated to improve the object
detection capability of convolutional networks. Learning features at different
resolutions allows the network to create a rich understanding of the input
image, aiding the mapping between the input data, $\bm{X}$, and the output
label, $\bm{y}$.

The final key components of neural networks are fully connected layers, which
are essential to many classification or regression architectures. They are
normally placed at the end of a network, after the convolutional layers have
extracted features from the data, and then the fully connected layers learn
how to classify these features. The activation map from the final layer of the
feature extraction is reshaped into a long vector instead of a volume tensor.
In fully connected layers, all nodes in one layer are connected to all the
nodes in the next, meaning that they are much more computationally powerful
and expensive than convolutional layers.

By feeding the data through the network, we are able to create an output
prediction. To make these predictions accurate, the weights of the network
must be optimised using a process called back propagation (LeCun et al.,
1990a), and normalisation techniques such as batch normalisation are added to
minimise the impact of outliers and help result generalisation (i.e.
performance on new or unseen datasets)(Ioffe and Szegedy, 2015). To this end,
we evaluate a loss or cost function which determines the error in the network
prediction by comparing the prediction ($\bm{\hat{y}}$) and the true label
($\bm{y}$). The choice of the loss function is task-dependent, and plays a
crucial role in the final network performance. The optimisation algorithm is
often stochastic gradient decent or a similar algorithm, which updates the
weights such that we reach a minimum in the loss space: that is, a model which
minimises the loss. The aim is to find the global minimum of the loss space,
which is the place in the loss space which most minimises the loss function.
However, the optimisation can get stuck in local minima, which are spots of
low loss due to the non-convex nature of the loss function, and so the
learning rate (the step size taken during the optimisation process) must be
chosen to best help the model find the global minimum.

Figure 3: When Learning to classify samples (e.g., pixels or subjects -
circles and squares), there is likely to be a continuum between the classes
and the decision boundary chooses where one class begins and another ends. For
subjects close to this decision boundary, it becomes critical as it determines
the classification given.

Therefore, we have an optimisation problem, the performance of which is highly
dependent on two factors: first, the design decisions made about the network
architecture and the loss function; second, the data available to train the
network. This optimisation process determines the location of the decision
boundary, that is which locations in the feature space will be allocated to
which class and so which data points are allocated to each class, as
illustrated in Fig. 3.

In computer vision, where most of the deep learning techniques have been
developed, very large datasets are available, with datasets commonly
consisting of many millions of data points (e.g., 2D training images with
labels), and being relatively easy to curate, for instance, by scraping the
internet for images. In neuroimaging in general, data has to be labelled by a
domain expert rather than mechanically produced. This is one of many
differences between working in medical imaging and the general computer vision
field, and therefore, although many of the methods used were developed in
vision-related fields, there are challenges specific to working in the medical
imaging or neuroimaging domain.

##  2 Data Availability

For clinical translatability of deep learning techniques, data availability is
a major limitation. Despite the growth in the size of available datasets, the
largest are still only of the order of tens of thousands of imaging subjects,
with a thousand images being commonly regarded as a large dataset. For many
specific tasks, datasets exist only in the order of hundreds of subjects, due
to many factors such as the monetary and time cost of acquiring data, the
difficulties in sharing and/or pooling data across sites, and the fact that,
for some conditions, there are insufficient patients to create a dataset of
any great size (Morid et al., 2021). For instance, the frequently explored
Brain Tumour Segmentation (BraTS) dataset (Menze et al., 2014) only has data
from 369 subjects222Training Data for 2020 challenge available for training,
which is in stark contrast to the popular datasets from computer vision such
as ImageNet (Deng et al., 2009) (1,281,167 training examples), CIFAR-10
(Krizhevsky et al., 2009) (50,000 training examples) and MNIST (Lecun et al.,
1998) (60,000 training examples) where many of the methods are being
developed. By simply looking at dataset sizes, it is clear that we are likely
to be underpowered for training neural networks. Highly parameterised, deep
neural networks are very dependent on the amount of available training data
(Cho et al., 2015; He et al., 2020), with performance generally improving as
the number of data points is increased, and they are far more affected by the
amount of available training data than classical machine learning techniques,
due to the need to learn the useful features as well as the decision boundary
(He et al., 2020).

###  2.1 Maximising the impact of available data

There has, therefore, been an increasing focus on developing techniques to
facilitate the most effective use of the data available. A commonly used
technique from computer vision is the use of large natural image datasets
(Raghu et al., 2019; Talo et al., 2018; Mehmood et al., 2021), with ImageNet
(Deng et al., 2009) being the most popular (Cheplygina et al., 2019), to pre-
train the network. Pre-training involves training the weights on a related
task with more available data, such that the optimisation starts from an
informed place, rather than a random initialisation. We can see why this might
be useful by considering the information learned by the network at the
different stages (Olah et al., 2018): the early layers learn features such as
edges and simple textures, largely resembling Gabor filters – see Fig. 2 – and
are therefore very general and applicable across different images, regardless
of the target tasks (Yosinski et al., 2014). The deeper convolutional layers
then tend to learn features such as more complex textures and object parts,
and the final layers learn features which are far more task- and dataset-
specific (e.g. fully connected layers learn discriminative features for a
classification task). Therefore, we can take a network pre-trained on the
large, canonical dataset, and either use the network to extract features which
we then pass to a classifier, requiring only the classifier to be trained, or,
more commonly, we can fine-tune (re-train) the deeper layers to the specific
task. This requires less data, as not only are we starting the optimisation
process from an informed point in the parameter space, but also the very
earliest layers can often be frozen (kept at their value and not updated
during training), greatly reducing the number of weights in the model that
need to be optimised. This process is referred to as transfer learning and is
a step frequently used to allow networks to be trained with lower amounts of
training data. Transfer learning can be performed across data domain
(dataset), task, or both, depending on the datasets available for pre-
training.

The standard practice is to use the very large datasets of natural images for
the pre-training. However, natural images have very different characteristics
from many medical images, and thus the features learned are not necessarily
the most appropriate for the tasks we need to consider (Raghu et al., 2019).
For instance, natural images are often stored as RGB images, whereas MR images
are encoded as greyscale. Similarly, in medical images the location of
structures could be informative, which is rarely true in natural images.
Creating pre-trained networks for medical images has therefore been a focus,
with Model Genesis (Zhou et al., 2019) creating a flexible architecture
trained to complete multiple tasks, to create features which aim to generalise
across medical imaging tasks. Similarly, some works pre-train on large
datasets such as UK Biobank for tasks such as age or sex prediction, where
obtaining labels is relatively trivial (Lu et al., 2021) or on datasets for
the same task with a dataset where more labels are available (Kushibar et al.,
2019; Bashyam et al., 2020; Abrol et al., 2020). Here again, the aim is to
learn features from the prediction task which are useful for the task we are
interested in: that is, features that generalise across tasks.

Other studies utilise self-supervised approaches, such as contrastive
representation learning, where general features of a dataset are learned,
without labels, by teaching the model which data points are similar or
different. These then act as the starting point for model training on the
smaller target dataset, rather than pre-training the model on a different
dataset. An example approach is presented in (Chen et al., 2020), where the
data has been augmented (small transformations applied to increase the size of
the dataset, discussed below) and then the network is trained to encode the
original image and the augmented image into the same location in the feature
space (i.e., they make the same output prediction), using a contrastive loss
function (Hadsell et al., 2006), that learns features describing the
similarity between images. Different self-supervised methods and contrastive
loss approaches have been developed (Henaff, 2020; Grill et al., 2020), and
have begun to be applied for medical imaging applications (Zhang et al.,
2020a; Chaitanya et al., 2020), including for segmentation of MRI scans of the
brain (Zhuang et al., 2019b; Chen et al., 2019), increasing the performance on
tasks where a small amount of training data is available.

###  2.2 Data Augmentation

Figure 4: Example augmentations that might be applied to a MRI image. Standard
augmentations, those that come directly from computer vision approaches, and
domain specific augmentations for neuroimaging which focus on variation that
would be likely to be seen within MR images.

Convolutional neural networks, however, still ultimately require a reasonable
amount of data (100s or 1000s) in the target data domain, as at least some of
the network parameters must be fine-tuned to optimise the network for the
specific dataset and task. Even though the amount of data required is likely
to be reduced (the degree to which it is reduced will be determined by the
similarity between the proxy task and target task (He et al., 2019; Kornblith
et al., 2019)) the amount still required may remain greater than is available
for the task we are exploring. In this circumstance, data augmentation is
normally applied (Simard et al., 1998; Chapelle et al., 2001), artificially
increasing the size and diversity of the training dataset through applying
transformations, creating slightly perturbed versions of the data.

These augmentations can take the form of basic transformations such as flips
and rotations (Krizhevsky et al., 2012; Simonyan and Zisserman, 2015) as
standardly applied in computer vision tasks, to more extreme examples such as
mixup (Zhang et al., 2017) which merges images from different classes together
to form hybrid classes, or generative networks such as conditional Generative
Adversarial Networks (GANs), which are networks trained to generate simulated
data (Mirza and Osindero, 2014). Some example augmentations can be seen in
Fig. 4. While the vast majority of deep learning studies apply data
augmentation during training, some studies explore this for neuroimaging
specifically, and its effect on model performance. For instance augmentation
can be achieved through GANs being used to generate additional meaningful
datapoints (Wu et al., 2020; Chaitanya et al., 2019; Zhuang et al., 2019a) or
registration to templates (Nguyen et al., 2020; Lyu et al., 2021), which
generate biologically plausible transformations of the data. Similarly, they
can be produced by identifying augmentations which are plausible across sites
and scanners (Pérez-García et al., 2020; Billot et al., 2020a), such as
applying bias field (scaling intensities by a smoothly-varying random gain).

Existing literature suggests that performing augmentations, even
transformations which create images beyond realistic variation (Billot et al.,
2020a), helps the network to generalise better to unseen data at test time.
Data augmentation must be handled with caution, however, so that the
transformations applied do not change the validity of the label associated
with the image. Consider, for instance, when trying to classify Alzheimer’s
disease from structural MRI: the key indicator could be the atrophy of the
hippocampus and so, if any transformations are applied during the augmentation
process that affect this region (e.g. local elastic deformations), care must
be taken to ensure that the perceived level of atrophy is not affected and,
thus, the true label changed. Ensuring this requires high levels of specific
domain knowledge and therefore, for certain scenarios, limits the
augmentations which can be applied.

Other approaches to solving the shortage of available training data focus on
breaking the input data down into patches e.g. (Wachinger et al., 2018; Lee et
al., 2020a) or slices (where 3D data is available) with many studies treating
MRI data as 2D inputs e.g. (Livne et al., 2019; Henschel et al., 2020; Sinha
and Dolz, 2020). This approach can vastly increase the amount of available
data and can be especially effective for segmentation tasks where we have
voxelwise labels. However, by fragmenting the image, these approaches lead to
the loss of global information. Although this can be compensated for to some
degree using optical flow (Zitnick et al., 2005) or conditional random fields
(Kamnitsas et al., 2017b), patch-wise or slice-wise approaches cannot
necessarily be applied to classification tasks, where a single label is
provided for the whole image (which may not be valid for all slices) on a
given patch or slice of the image (Khagi et al., 2018). When these approaches
can be applied, say for a segmentation task, care must be taken when combining
the patches at the output to reconstruct a single output image, so that we do
not suffer from artefacts at boundaries. Fully 3D networks have been found in
most cases to provide better results when they can be implemented (Kamnitsas
et al., 2017b).

###  2.3 Differences between datasets or data domain shift

Having sufficient data to train the model, however, is only the first
difficulty being faced by clinical application of these methods. Deep learning
methods have a very high degree of freedom which enables them to learn very
complicated and highly non-linear mappings between the input images and the
labels, but this same high degree of flexibility comes at a cost: deep
learning methods are prone to overfitting to the training data (Srivastava et
al., 2014). Furthermore, while, a well-trained model should interpolate well
to data which falls within the same distribution as that seen during training,
the performance degrades quickly once it has to extrapolate to out-of-
distribution data, and even perturbations which are not noticeable by the
human eye can cause the network performance to collapse (Papernot et al.,
2017). Considerable research effort within computer vision has focused on
generalisability from the dataset seen during training to the dataset only
seen during testing, where both datasets are drawn from the same distribution.
For clinical translatability, we would need generalisability from the training
set to all other reasonable datasets, including those which have not yet been
collected.

If, for instance, we consider multisite datasets, such as from the ABIDE study
(Di Martino et al., 2013), where attempts have been made to harmonise
acquisition protocols and to use identical phantoms across imaging sites,
there is still an increase in non-biological variance when we pool the data
across the sites and scanners (Yu et al., 2018). A demonstration of this
variance leading to performance degradation for a segmentation task is shown
in Fig. 5. Multiple studies have confirmed this variation, identifying causes
(batch effects) from scanner and acquisition differences, including scanner
manufacturer (Han et al., 2006; Takao et al., 2013), scanner upgrade (Han et
al., 2006), scanner drift (Takao et al., 2011), scanner strength (Han et al.,
2006), and gradient non-linearities (Jovicich et al., 2006).

Figure 5: To demonstrate the effect of the difference between domain datasets
or domain shift, we completed tissue segmentation on data from three sites
collected as part of the ABIDE (Di Martino et al., 2013) multisite dataset.
Although the data was collected as part of the same study, there are
differences between the data collected at different sites due to being
collected on different scanners. The architecture used was a 3D UNet (Cicek et
al., 2016) with T1 as the input image, and only images from one site were used
during training. The predictions can be seen for example images for three
sites, the one seen during training and two unseen. It is clear that the
segmentation for the site seen during training is good but suffers significant
degradation when applied to the unseen sites, despite them being collected for
the same study and having similar (normalised) voxel intensities,
demonstrating the potential difficulties caused by domain shift.

The removal of this scanner-induced variance is therefore vital for many
neuroimaging studies. The majority of deep learning approaches try either to
produce harmonised images (Cetin Karayumak et al., 2019; St-Jean et al., 2019;
Dewey et al., 2019; Zhao et al., 2019), or to remove the scanner-related
information from the features used to produce the predictions (Moyer et al.,
2020; Dinsdale et al., 2021b). Both approaches aim for any results obtained
for use downstream to be invariant to the acquisition scanner and protocol.
These methods succeed in removing the scanner effects from the predictions,
but hold no guarantees for scanners not seen during training, and, as the
results are very hard to verify without ‘travelling heads datasets’ (images
from the same subjects acquired on the scanners to be harmonised) the results
obtained from the generated harmonised images are hard to validate (Moyer et
al., 2020).

The domain shift experienced when we have multisite data is much less than
might be expected when we move between research data and clinical data, or
even just two datasets collected independently. The domain shift here can come
from two clear sources: the scanner and acquisition, and the demographics (or
pathologies) of the studies. Firstly, MRI scans collected for research are
often at a higher resolution and higher field-strength than clinical scans. On
the other hand, clinical scans are designed to be much more time efficient –
both in terms of acquisition time and in time required for visual inspection –
and are often collected at a lower resolution, often at a lower field
strength. Research scans frequently also have isotropic voxel sizes, whereas
anisotropic voxels are still the norm in the clinic and are present in the
vast majority of legacy data (Iglesias et al., 2020a). Unfortunately, due to
the lack of training data already discussed, we are unlikely to be able to
train sophisticated models directly on clinical data.

Therefore, methods are being developed that consider this domain shift (e.g.
between clinical and research data), which focus either on domain adaptation
approaches to create shared feature representations for the different
datasets, or in synthesising data to enable us to use the clinical domain.
Unlike the harmonisation paradigm, such approaches typically do not wish to
allow us to combine the datasets, but simply to be able to harness the shared
information from one domain for use in the other. Domain adaptation techniques
normally consider the situation where we have a large source dataset – say a
research dataset such as UK Biobank (Sudlow et al., 2015) – and a much smaller
target dataset – say the clinical dataset that we are actually interested in
(Ben-David et al., 2010; Ganin et al., 2015). Domain adaptation then asks the
question: can a shared embedding be found which is discriminative for the task
of interest, while invariant to the domain of the data? This can take the form
of a fully supervised problem, where task labels are available for both
datasets, semi-supervised (Valverde et al., 2019), where only a small number
of labels are available for the target dataset, or unsupervised (Kamnitsas et
al., 2017a; Perone et al., 2019), where no labels are available for the target
dataset. The vast majority of approaches perform domain adaptation across
domains only; however, some also consider adaptation across related tasks
(Tzeng et al., 2015) and have been applied for segmentation (Kamnitsas et al.,
2017a; Valverde et al., 2019; Perone et al., 2019; Sundaresan et al., 2021)
and classification problems (Guan et al., 2020). These methods are clearly
closely related to transfer learning from a large dataset, but a single
feature representation is found for both source and target dataset, rather
than creating a new one for the target dataset. These methods perform well on
the target data, but further work is required to enable them to adapt reliably
to higher numbers of datasets simultaneously.

Domain adaptation methods, at the extreme, essentially have the end goal that
the network would work regardless of the acquisition, which is an active area
of research (Billot et al., 2020b; Thakur et al., 2020; Billot et al., 2020c).
The other approach which has been explored is to use generative methods to
convert the data from one domain to the other (Iglesias et al., 2020b), such
that the transformed data can be used in the existing model. Any generated
images must be carefully validated to ensure that they convey the same
information as the original and that the outcomes are the same.

Finally, research data is generally cleaner than clinical data. For instance,
many developed algorithms require multiple input modalities and therefore,
cannot be applied to a subject if they do not have all the scanning modalities
available (Chen et al., 2018). Missing modalities, different fields of view,
and incidental findings would all potentially lead to the performance of the
model being significantly degraded or the model simply not being applicable.
Approaches to deal with missing modalities exist (Zhou et al., 2020), but
generally still result in a significant degradation in performance, compared
to when all modalities are present. Similarly, models are likely to suffer
performance degradation or exhibit unexpected behaviour when presented with
unexpected pathologies that were not present in the training set (Guha Roy et
al., 2018).

###  2.4 Data Composition and Algorithmic Biases

Finally, and potentially most concerning, is the consideration that the
demographics of study data frequently do not fully represent the population as
a whole, and so a domain shift is experienced when we attempt to move from the
research domain to the clinical domain. Research data is usually acquired with
a certain study question in mind and therefore subjects are selected so as to
try to allow targeted exploration of that question. Therefore, research
datasets rarely contain subjects with co-morbidities, and subjects with
incidental findings would possibly be excluded from the study. For example,
patients with advanced Alzheimer’s disease are unlikely to be recruited for a
general imaging study, due to the ethical implications (Clement et al., 2019),
such as the inability to consent and the potential trauma of being scanned.

In addition, there exists a strong selection bias, both in relation to the
people who volunteer for studies and those who see the study through to the
end, with studies having demonstrated associations to age, education,
ancestry, geographic location, and health status (Karlawish et al., 2008;
Clement et al., 2019). Furthermore, people with family connections to a given
condition are more likely to volunteer for a study as a healthy control,
leading to certain genetic markers being more prevalent in a study dataset
than in the population as a whole (Hostage et al., 2013). Therefore,
associations learned when considering research data may not generalise, and
care must be taken in extrapolating any model trained on these datasets to
clinical populations.

Therefore, any trained model will suffer from algorithmic bias: that is, the
outcomes of the model will potentially be systematically less favourable to,
or have systematically lower performance on, individuals within a particular
group, where there is no relevant difference between groups that justifies
such effects (Paulus and Kent, 2020). Erroneous or unsuitable outcomes will
likely be produced for the groups who are less likely to be represented in the
training data. Algorithmic bias is therefore a function of the creator and the
creation process, and fundamentally, of the data which drove the model
training. When considering bias, there are two issues that must be considered:
first, does the algorithm have different accuracy rates for different
demographic groups? Second, does the algorithm make vastly different decisions
when applied to different populations? As networks simply learn the patterns
in the data, any bias, such as racial bias (Williams and Wyatt, 2015), in the
data may be learnt and encoded into the models.

Inevitably, when considering complicated questions with extremely
heterogeneous populations, the datasets used to train the deep learning
methods will be incomplete and insufficient in terms of spanning all the
possible modes of variability (Ning et al., 2020). For instance, pathologies
will occur against a background of normal ageing, with differences being
present between individuals due to both processes. Sufficiently encompassing
all this variation is infeasible, not only due to the number of subjects which
would be required, but also to the difficulty in recruiting subjects from some
specific groups. There is, therefore, often inadequate data from minorities,
and consequently any model found through the optimisation process (which is
usually selected as the model which performs best on the average subject in
the validation dataset) will probably be inadequate for these groups,
especially for conditions with higher rates in these groups (Wong et al.,
2015). When models are developed for clinical translation, therefore, the
limitations of the models must be understood: wherever groups are under-
represented, the appropriateness of the application of the model must be
considered, and any limitations identified. Where these limitations mean that
minorities will receive a lower standard of care, the models are
inappropriate.

##  3 Interpretability

Figure 6: If we take a model trained to predict brain age (Dinsdale et al.,
2021a) from T1 structural images and present the network with an image of
random noise, as it is unable to output an unknown class, the network
predicted the random noise to have an age of 65, around the average age of the
subjects of the dataset. While we clearly can identify the random noise image,
there are many situations where the model could be presented with an image
outside of the distribution it was trained on, but would still output a
(meaningless) result, which would be much harder for a user to identify.

The performance degradation experienced with domain shift would be potentially
less problematic were it not for another problem associated with deep learning
methods: that is, models will output a prediction for any data fed in, but
that prediction may not necessarily be meaningful. Lacking a ‘do not know’
option, given any image of the correct input dimension, a neural network will
output a prediction, even if that prediction is meaningless or the input is
nonsensical. For instance, if a random noise image is fed into a network
trained to predict brain age, the network will predict an apparently valid age
for the random noise (see Fig. 6). While in this example, visually identifying
the pure-noise image is trivial, were the network trained for a more
complicated classification task, identifying erroneous results is more
difficult and requires large amounts of clinical and domain knowledge. This
therefore leads to the critical question, can the results be trusted?

Despite the assertion of Geoffery Hinton, head of GoogleBrain, that there is
no need for AI to be interpretable333https://www.wired.com/story/googles-ai-
guru-computers-think-more-like-brains/, the majority would agree that, were
deep learning and AI methods to be used to determine patient care, they need
to be interpretable and interrogable. Interpretability is often defined as
‘the ability to provide explanations in understandable terms to a human’. The
explanations should, therefore, be logical decision rules which lead to a
given diagnosis or patient care being chosen, and the understandable terms
need to be from the domain knowledge related to the task. This is especially
important because neural networks have no semantic understanding of the
problem. That is, they have no understanding of the problem they are being
asked to solve. Rather, they are blunt tools which, given $\bm{X}$ and
$\bm{y}$, learn a mapping between the two. If there exists spurious
information in $\bm{X}$ which can aid in this mapping (or confounders), then
this information will probably be used, misleading the predictive potential of
the network. Consider for instance, the case where all subjects with a given
pathology were collected on the same scanner. A network could then achieve
$100\%$ recall (correctly identify all examples) for this pathology by fitting
to the scanner signal, rather than learning any information about the
pathology (Winkler et al., 2019). It would then, in all probability, identify
a healthy control from the same scanner as having the same pathology.

The effect of confounders would not be observed without further probing of the
behaviour of the trained model, and the probing of networks is non-trivial.
This has led to neural networks being commonly described as ‘blackbox’
methods. There is therefore a need for interpretable networks, allowing both
understanding and scrutiny of any decision made. Approaches have broadly
focused on two main areas: visualisation and uncertainty.

###  3.1 Visualisation

Figure 7: Schematic of the limitation of using saliency: when identifying the
presence of white matter hyper-intensities, the neural network might only need
to focus on a few of them to make the prediction. This would lead to not all
of the white matter hyper-intensities being indicated in the saliency map and
so the prediction not matching the clinician’s expectation.

Visualisation methods generally attempt to show which aspects of the input
image led to the given classification, often by creating a ‘heat map’ of
importance within the input image. Many of these methods are post-hoc, taking
a pre-trained model and trying to understand how the decision was made. Most
commonly, these methods analyse the gradients or activations of the network
for a given input image, such as saliency maps (Simonyan et al., 2014),
GradCAM (Ramprasaath et al., 2017), integrated gradients (Sundararajan et al.,
2017), and layerwise relevance propagation (Binder et al., 2016). They have
been applied in a range of MRI analysis tasks to explain decision-making, such
as in Alzheimer’s disease classification (Böhle et al., 2019), brain age
prediction (Dinsdale et al., 2021a), and brain tumour detection and
segmentation (Mitra et al., 2017; Takacs and Manno-Kovacs, 2018). Other
methods are occlusion- (or perturbation-) based, where parts of the image are
removed or altered in the input image, then heat maps are generated which
evaluate the effect of this perturbation on the network’s performance (Zeiler
and Fergus, 2014; Abrol et al., 2020). Most of these methods, however, provide
coarse and low resolution attribution maps and can be extremely
computationally expensive (Bass et al., 2020), especially when working with 3D
medical images.

These posthoc methods do not require any model training in addition to the
original network, however, they have been shown to be, in some instances,
unable to identify all of the salient regions of a class, especially in
medical imaging applications (Bass et al., 2020; Baumgartner et al., 2018). It
has been shown that classifiers base their results on certain salient regions,
rather than the object as a whole, and therefore, a classifier may ignore a
region if the information there is redundant – i.e. if it can be provided by a
different region of the image, which is sufficient to minimise the loss
function. Therefore, the regions of interest highlighted by these methods may
not fully match the expectations of a clinician (see Fig. 7), and also the
prediction results might be virtually unchanged if the network was retrained
with supposedly-salient areas removed. Generally, although many methods have
been developed to produce saliency or ‘heatmaps’ from CNNs, limited effort has
been focused on their evaluation with actual end-users (Alqaraawi et al.,
2020). Furthermore, these methods, at their best, only highlight the important
content of the image, rather than uncovering the internal mechanisms of the
model.

Attention gates are a component of the network, which aim to focus a CNN on
the target region of the image (the salient regions) by suppressing irrelevant
feature responses in feature maps during training rather than post-hoc. This
focuses the attention of the network onto the information critical for the
specific task, rather than learning non-useful information in the background
or surrounding areas (Park et al., 2018; Wang et al., 2018; Hu et al., 2018).
These methods provide the user with attention maps, which again highlight the
regions of the input image driving the network predictions. However, these
methods, similarly to saliency or gradient based methods, may not highlight
all of the expected regions in the image. Attention gates have been applied to
a range of medical image analysis tasks, both for classification (Dinsdale et
al., 2021a; Schlemper et al., 2019) and segmentation (Schlemper et al., 2019;
Zhuang et al., 2019b; Zhang et al., 2020). Other methods have been developed
to allow the visualisation of the differences between classes directly, rather
than analysing the model post-hoc (Bass et al., 2020; Lee et al., 2020b).
Other methods aim to increase their interpretability by breaking down the task
into smaller, more understandable tasks, such as first segmenting a region
known to be a biomarker for a given condition, and then classifying based on
this region (Liu et al., 2020).

The methods discussed so far enable the visualisation of the regions of the
input image which drive the predictions, but they do not provide insight into
how the underlying filters of the network create decision boundaries. In
addition, in brain imaging, the class phenotypes are typically heterogenous
and any changes they cause probably occur simultaneously, with significant
amounts of healthy and normal variation in shape and appearance, meaning that
interpretation of feature attribution maps to understand network predictions
is often difficult. Given the millions of parameters in many deep learning
networks, despite our ability to visualise individual filters and weights
helping us to understand the hierarchical image composition, it is very
difficult to interrogate how decision boundaries are formed. Therefore, there
is a need to create networks which complete their predictions in more
understandable ways, without restricting the complex non-linearities in
behaviour that are necessary for good prediction performance.

###  3.2 Uncertainty

Figure 8: Methods which use Monte Carlo dropout have dropout applied at
training and test time, where weights in the convolutional kernels are
removed, which is approximated to represent the distribution of possible model
architectures at test time. To demonstrate this we trained a standard 3D UNet
to complete hippocampal segmentation, with a dropout value of 0.5 applied on
all convolutional layers. The HarP dataset (Frisoni and Jack, 2015) was used
in this experiment, preprocessed as in (Dinsdale et al., 2019). For each
subject, we obtained a mean prediction and a uncertainty map, indicating the
regions where the predictions between models were the most varied and so
approximated to be the least certain.

The use of uncertainties, then, is an approach which aims to address the
problem that, regardless of the input image, neural networks will always
output a prediction, even when they are very unsure of the prediction. An
example is when the data falls far away from the domain of data the network
was trained on. Furthermore, the softmax values frequently output by a neural
network are not true probabilities (Gal and Ghahramani, 2016), and networks
often output high, incorrect softmax values, especially when presented with
noisy or ambiguous data, or when the data presented to them differs from the
distribution of the training data. Therefore, the development of models aware
of the uncertainty in their predictions is key for providing confidence and
trust in systems – and this is not provided by traditional deep learning
algorithms.

Uncertainties in deep learning can be split into two distinct groups (Kendall
and Gal, 2017): aleatoric uncertainty, the uncertainty due to the ambiguity
and noise in the data, and epistemic uncertainty, which is due to the model
parameters. The majority of methods in the literature focus on epistemic
uncertainty, using Bayesian approaches to quantify the degree of uncertainty.
The goal here is to estimate the posterior distribution of the model
parameters. However, due to the very high dimensional parameter space,
analytically computing the posterior directly is infeasible. Therefore, most
methods use Monte Carlo dropout (Gal and Ghahramani, 2016), demonstrated in
Fig. 8, where dropout (Srivastava et al., 2014) is applied to each of the
convolutional layers and kept at test time, and thus the model architecture is
random at test time. Although other methods exist, in the majority of
approaches for medical imaging, the uncertainty is then quantified through the
variance of the predictive distribution, resulting from multiple iterations of
the prediction stage with dropout present at test time, as demonstrated in
Fig. 8. This approach can be simply applied to existing convolutional neural
networks and in medical imaging has primarily been used for segmentation tasks
(Roy et al., 2018; Kwon et al., 2020), where the segmentation is predicted
alongside an uncertainty map. Other works have studied disease prediction,
where the uncertainty is associated with the predicted class (Jungo et al.,
2018; Tousignant et al., 2019) and image registration (Bian et al., 2020).
However, care must be taken with choice of the hyperparameters (the dropout
probability and the number of iterations that the variation is calculated
over) to ensure that the model assumptions are reasonable.

Some methods focus on the aleatoric uncertainty instead, which is estimated by
having augmentation at test time (Ayhan and Berens, 2018; Wang et al., 2019).
Understanding of the uncertainty introduced by data varying from the training
distribution is vital for clinical translation of deep learning techniques:
with the degree of variation present in clinical data between sites and
scanners, it is vital to understand what this variation adds to predictions,
both to allow it to be mitigated against, and to provide users with confidence
in the predictions. Unsurprisingly, there is a correlation between erroneous
predictions and high uncertainties, and so, this could be used to improve the
eventual predictions (Jungo et al., 2018; Herzog et al., 2020).

There is, however, need for further development of these methods to ensure
that the uncertainties produced are meaningful in all the circumstances in
which they could be deployed. For instance, further study is required to
ensure that the uncertainty is meaningful for all possible dataset shifts and
to provide a calibration for the uncertainty values so that they are
comparable across methods (Thagaard et al., 2020; Laves et al., 2020).
Furthermore, the uncertainty values are, also, only as good as the trained
model, the assumptions behind the uncertainty model, and are only meaningful
alongside a well-validated model which is sufficiently powerful to
discriminate the class of interest.

###  3.3 Interrogating the Decision Boundary

For many applications in medical imaging, the output of the deep learning
algorithm, if applied clinically, could potentially directly influence patient
care and outcomes. Therefore, there is a clear need to be able to interrogate
how decisions were made (Shah et al., 2019). For instance, if we reconsider
Fig. 3, it is clear that the location of the decision boundary could impact
highly on the care for the patient if the classification was the presence or
absence of white matter hyper intensities. While visualisation methods allow
inspection of which regions of the image influenced the prediction, and
uncertainties grant us an insight to the confidence we should place in a given
prediction, for many applications we need to know precisely which
characteristics led to a given prediction. This is also important to help with
the identification of algorithmic bias influencing the decision making.

One method of understanding the decision boundary is counterfactual analysis,
which, given a supervised model where the desired prediction has not been
achieved, shows what would have happened if the input to the model were
altered slightly (Verma et al., 2020). In other words, it identifies what
altered characteristics would have led to a different model prediction.
However, applications to neuroimaging (Pawlowski et al., 2020), and even
medical imaging (Major et al., 2020; Singla et al., 2021) more generally, are
currently few and the utility across neuroimaging tasks needs to be explored.

##  4 Evaluation

###  4.1 Availability of Training Labels

The evaluation of metrics also requires labels – that is, a ground truth label
created by a domain expert. In medical imaging, the ground truth is regarded
as labels created by domain experts and these labels are key for training
models, but do not necessarily form part of standard clinical practice.
Firstly, the labels are required for evaluation of the model performance, and
secondly, they are required to allow the training of supervised methods. This,
therefore, exacerbates the problem of the shortage of data as we need not only
large amounts of data, but we also need equal amounts of correct manual
labels. These labels are expensive to obtain, requiring large allocation of
expert time to curate and expert domain knowledge. Therefore, there is a need
to develop methods which can work in data domains where low numbers of
labelled data points are available.

Few- and zero-shot learning methods work in very low data regimes and are
beginning to be applied to medical imaging problems (Feyjie et al., 2020).
They are, however, very unlikely to generalise well to images from other sites
and scanners, as the variation seen will not span the expected variation of
the data but they can help begin to learn clusters within subjects where few
labels are available. Unsupervised domain adaptation has been applied more
widely, including for neuroimaging problems, to help cope with a lack of
labels, with information from one dataset being leveraged to help us perform
the same or a related task on another dataset (Perone et al., 2019; Sundaresan
et al., 2021).

Other methods to overcome the lack of available labels focus on working with
approximations for labels, which are cheaper to acquire (Tajbakhsh et al.,
2020). For instance, many methods propose pre-training the network using
auxiliary labels generated using automatic tools (e.g., traditional image
segmentation methods) and then fine-tuning the model on the small number of
manual labels (Guha Roy et al., 2018; Wang et al., 2020), or registration of
an atlas to propagate labels from the atlas to the subject space (Zhu et al.,
2019). Other approaches are weakly supervised, utilising quick annotations
such as image level labels (Feng et al., 2017), bounding box annotations
(Rajchl et al., 2017), scribbles (Dorent et al., 2020; Luo et al., 2021) or
point labels (McEver and Manjunath, 2020).

Other approaches to allow us to utilise deep learning when we have access to
limited numbers of training labels include active learning and omnisupervised
learning, both trying to make the most effective use of the limited number of
labels available for a task. Active learning aims to minimise the quantity of
labelled data required to train the network by prompting a human labeller only
to produce additional manual labels where they might provide the greatest
performance improvements. This minimises the total number of annotations that
need to be provided, and provides better performance than randomly annotating
the same number of samples (Yang et al., 2017; Nath et al., 2021). In
omnisupervised learning (Radosavovic et al., 2018) automatically generated
labels are created to improve predictions, starting from a small labelled
training set. By combining data diversity through applying data augmentation,
and model diversity through the use of multiple different models, a consensus
of labels is produced, which can be used to train the final model. For both
approaches, the labels used are chosen using various different approaches such
as uncertainty (Gal et al., 2017; Venturini et al., 2020).

The difficulty in acquiring good quality manual labels, is of course,
exacerbated by the variance caused when we pool data, as discussed above. This
increase in variance limits the impact any produced labels can have, and so
again, methods of pooling data across datasets without getting an increase in
variance due to the scanner effects will be necessary.

The labels themselves, however, will provide another source of variance
(Cabitza et al., 2019): when working with medical images, the labels are
frequently complicated and ambiguous (Shwartzman et al., 2020), often open to
interpretation or with subjects having multiple labels that could be
attributed due to co-morbidities (Graber, 2013) but despite this we usually
assume them to be $100\%$ accurate (Cabitza et al., 2020) – the ‘gold’
standard. If there is no objective answer, we cannot expect networks to
provide an objective answer. Furthermore, this also leads to inter-rater
variability between the people creating the labels, which leads to a degree of
uncertainty in the produced ground truth. The effect that this variability has
on the predictions of the network needs to be understood and mitigated
against, with the challenge being that obtaining labels for multiple raters
for datasets is deeply infeasible, and has been shown to limit the performance
of algorithms (Svensson et al., 2015). The uncertainty in the labels is also
amplified by the lack of available data for rare conditions, which are
therefore less represented in datasets, resulting in raters having less
experience assessing them. This is especially problematic if trying to
quantify longitudinal changes with different raters (Visser et al., 2019).

Approaches need to consider three factors (Cabitza et al., 2020): agreement –
the degree to which raters agree on a given label; confidence – how certain a
rater is in their label, and competence – how accurate a given rater is.
Research directions into the effect of rater variance have largely focused on
either quantifying the reliability of the labels (Cabitza et al., 2020), or
quantifying its effect on network performance (Shwartzman et al., 2020; Visser
et al., 2019; Haarburger et al., 2020). Before any algorithm is deployed in
practice, the limitations due to the labels must be understood, and its
consideration become a standard part of any deployment pipeline, remembering
always the “garbage in, garbage out” principle, for instance by producing the
range of segmentations that might be produced when training with multiple
raters (Baumgartner et al., 2019).

###  4.2 Choice of Loss Function

When training networks and evaluating the model performance, we have to choose
a loss, or cost, function which the network aims to minimise. Although some
works design bespoke, task-specific cost functions, the majority are based on
standard functions, such as categorical cross entropy for classification and
segmentation tasks, Dice (a measure of overlap between the predicted and true
label) for segmentation, and mean square error (MSE) for regression-based
tasks.

These metrics are normally chosen because of their well-understood and
characterised behaviour (Maier-Hein et al., 2018), and, especially in the
cases of segmentation, their ability to produce visibly acceptable
predictions. For clinical translation of deep learning methods, however, we
rather need to consider which measures are most important for the clinical
application (Shah et al., 2019; Keane and Topol, 2018). Metrics only tell us
part of the story and it is important that we ensure that all vital
information for clinical assessment is provided by the reported metrics. For
instance, in many cases, false negatives are more problematic than false
positives, resulting in a patient failing to receive the necessary care.
Similarly, when considering tasks such as tumour segmentation, the
classification of the center of the tumour is trivial and the utility for a
clinician would be identification of boundaries, where the classification is
harder or ambiguous. Therefore, a metric such as the Dice overlap metric is
not necessarily a good indicator of performance from a clinician’s point of
view, with the boundary having little impact on the metric if the region of
interest is large, while being potentially critical for patient care.
Developing networks and loss functions with the specific application in mind
is vital and has been explored for some applications, such as focusing
segmentations to the boundary (Guha Roy et al., 2018; Hatamizadeh et al.,
2019; Zhang et al., 2020b), but needs to be considered for each application in
turn.

Furthermore, when training neural networks, we generally maximise the
performance averaged over the batch (a subset of the training data). In
practice, however, we are more likely to care about the performance on the
hardest examples being acceptable, than the performance on the easiest set of
examples being improved slightly (Shu et al., 2020). If we, for instance,
consider a segmentation task, a given level of performance could be achieved
by easier examples being perfectly segmented, and more difficult examples
being completely missed. We would, however, probably prefer trading-off a
small amount of performance on the easier examples in return for better
performance on the harder examples, which may give the same average
performance overall. Therefore, the standard practice of minimising the
average performance may not be appropriate when the aim is clinical
deployment, rather than the development of methods.

##  5 Logistical Challenges

###  5.1 Computational Power

The final category of challenges are far more logistical. Firstly, most deep
learning models are developed using GPUs (graphical processing units),
hardware that is rarely available in a clinical setting, and producing
predictions on standard hardware often takes infeasible amounts of time or
requires prohibitive amounts of computational power. Many of the most
successful methods applied in imaging challenges involve large ensemble models
such as (Khosla et al., 2019; Isensee et al., 2021), leading to a very large
amount of parameters and therefore calculations that must be stored and
computed.

Therefore, for clinical translatability, methods need to be developed with
consideration of the computational limitations that will be present on
deployment, and seeking to create solutions which work within these
constraints. Student-teacher networks (Hinton et al., 2015; Ghafoorian et al.,
2018) and model distillation (Murugesan et al., 2020; Vaze et al., 2020) aim
to create smaller networks which are able to mimic the performance of the
original large model, thus reducing the number of parameters in the final
network which is deployed. Other approaches use separable convolutions which
drastically reduce the number of parameters in the network. Model pruning
(LeCun et al., 1990b; Molchanov et al., 2016; Li et al., 2017) acknowledges
that the parameters in neural networks are very sparse and, therefore, by
removing those that are contributing the least to the final prediction, we can
reduce the size of the model architecture whilst having limited impact on the
final performance of the model.

###  5.2 Data Sharing and Data Privacy

Figure 9: Illustration of a centralised federated learning framework. In the
framework, the data for training the model is stored in local servers and not
shared with the central server to ensure data security. On the other hand,
while the global model is available in the central server, the model
parameters are shared with the local nodes 1,2… N (indicated by blue arrows)
where training and parameter updates happen. The updated weights are then
received at the central server (indicated by yellow arrows), where the
incoming updates are aggregated and applied to the global model. This learning
and update happens in an iterative manner, and both up and down transfer of
model parameters are encrypted for data security.

Clearly from the above discussion of data and labels, if we want CNNs which
work for patients in real clinical applications, we need to be able to train
our models on medical data which is relevant, realistic, and representative.
Many current approaches focus on pooling anonymised data from across sites and
patient groups, and to be able to do this concentrate on removing identifiable
features such as name, birth data and faces from the images. However, neural
networks are still capable of extracting identifiable features from these
anonymised images such as age and sex (Pawlowski and Glocker, 2019), which, in
combination with other features such as hospital location and illness, could
be identifying (Sweeney, 2002). As neural network research continues to
develop, it is probable that their ability to extract information despite the
best efforts at anonymisation will increase. Furthermore, a proportion of
identification risk comes from the presence of other auxiliary information,
for instance, in neuroimaging, the scanner used to acquire the image. This is
known as linkage attack, and is increasingly difficult to protect against
across fields using classic anonymisation techniques (Sweeney, 2002; Gentili
et al., 2017; Bindschaedler et al., 2018).

While de-identifying this data may just seem like an extra task for medical
researchers, there are parties whose core business model is to de-anonymize
medical data that has been sold for research and sell that information to
insurance companies (Tanner, 2017). De-anonymisation research is a rapidly
advancing field: for instance, reconstructing the faces of defaced medical
images (Abramian and Eklund, 2019; Vidanage et al., 2020). Therefore, in order
to avoid data privacy problems in the future, approaches avoiding the
aggregation of private medical information are valuable.

Fortunately, medical research is not the only field to face difficulties
regarding the handling of sensitive and personal information. For instance,
banking and mobile phone companies have faced this problem before: how can one
extract only the desired answer, without also collecting information that
could be used for harm? Therefore, we can take advantage of the privacy-
preserving data analysis techniques that have rapidly developed in recent
years. The field of privacy-preserving data analysis consists of a collection
of techniques that allow models to be trained without having direct access to
the data, and that prevent these models from inadvertently storing sensitive
information about the data. The most popular of these techniques are federated
learning, differential privacy, and various forms of encrypted computation
(Al-Rubaie and Chang, 2019; Kaissis et al., 2021). Here we will focus on
federated learning and differential privacy, as they currently show the most
practical relevance in a neuroscience research setting (Rieke et al., 2020);
for a broader review see, for instance, (Kaissis et al., 2021).

Federated learning means training or testing your model (or performing your
statistical query) on data that is stored on different devices or servers
across the world, without having to centrally collect the data samples into
one local aggregate dataset (Li et al., 2020; Yang et al., 2019). Instead of
moving the data to the model, copies of the global model are sent to where the
data is located; the data remains on the hospital server. A model is sent to
the device and trained on the local data, after which the newly improved model
with its update is sent back to the main server to be aggregated with the main
model. This preserves privacy in the sense that the data has not been moved
from the device, and therefore is gaining popularity in various healthcare
applications (Rieke et al., 2020) – for instance (Huang et al., 2019; Sheller
et al., 2019; Li et al., 2019). However, federated learning is limited by the
fact that the content of the local data can sometimes be inferred from the
weight updates or improvements in the models (Wang et al., 2019). In addition,
deep neural networks are often over-parametrised, meaning that they can encode
more information than is necessary for the prediction task, resulting in a
model that can potentially, inadvertently memorize individual samples.
Therefore, to prevent the possibility of inferring personal characteristics
from the data, further techniques need to be employed, such as differential
privacy (Dwork and Roth, 2014).

Differential privacy techniques work by injecting a controlled amount of
statistical noise to obscure the data contributions from individuals in the
dataset (Dwork and Roth, 2014), and has been applied to medical imaging
(Ziller et al., 2021). This is performed while ensuring that the model still
gains insight into the overall population, and thus provides predictions that
are accurate enough to be useful. Differential privacy is a mathematical
framework, and thus enables the degree of privacy loss to be calculated and
evaluated based on the concept of a privacy ‘budget’. Therefore, ultimately
the use of differential privacy is a careful trade-off between privacy
preservation and model utility (Dwork and Roth, 2014; Shokri and Shmatikov,
2015; Abadi et al., 2016). Importantly, a critical aspect of differential
privacy is its inherent robustness to linkage attacks (Sweeney, 2002).

##  6 Conclusion

The combination of deep learning based methods and large scale imaging
datasets, such as UK Biobank, offers many opportunities to neuroimaging.
Clearly, however, for the full impact of these methods to be experienced in
the clinical domain there are challenges that must still be overcome.
Ultimately, for models to be able to be deployed successfully, the clinical
needs and limitations must be considered centrally to model design, so that
the models produced are robust, reliable, and able to improve patient
outcomes.

The code for the examples in this paper can be found at
github.com/nkdinsdale/challenges_review.

##  7 Acknowledgements

This work was supported in parts by funding from the Engineering and Physical
Sciences Research Council (EPSRC) and Medical Research Council (MRC) [grant
number EP/L016052/1] (ND and EB), by the Clarendon Scholarship fund (EB), and
and a Wellcome Trust Strategic Award (215573/Z/19/Z) (SS). VS is supported by
the Wellcome Centre for Integrative Neuroimaging [Grant number 203139/Z/16/Z].
A.N. is grateful for support from the UK Royal Academy of Engineering under
the Engineering for Development Research Fellowships scheme. M.J. is supported
by the National Institute for Health Research (NIHR) and the Oxford Biomedical
Research Centre (BRC). The Wellcome Centre for Integrative Neuroimaging is
supported by core funding from the Wellcome Trust (203139/Z/16/Z).

This research has been conducted in part using the UK Biobank Resource under
Application Number 8107. We are grateful to UK Biobank for making the data
available, and to all UK Biobank study participants, who generously donated
their time to make this resource possible. Analysis was carried out on the
clusters at the Oxford Biomedical Research Computing (BMRC) facility and FMRIB
(part of the Wellcome Centre for Integrative Neuroimaging). BMRC is a joint
development between the Wellcome Centre for Human Genetics and the Big Data
Institute, supported by Health Data Research UK and the NIHR Oxford Biomedical
Research Centre. The views expressed are those of the author(s) and not
necessarily those of the NHS, the NIHR or the Department of Health.

The primary support for the ABIDE dataset by Adriana Di Martino was provided
by the (NIMH K23MH087770) and the Leon Levy Foundation. Primary support for
the work by Michael P. Milham and the INDI team was provided by gifts from
Joseph P. Healy and the Stavros Niarchos Foundation to the Child Mind
Institute, as well as by an NIMH award to MPM ( NIMH R03MH096321).

## References

  * Abadi et al. (2016) Abadi, M., Chu, A., Goodfellow, I., McMahan, H.B., Mironov, I., Talwar, K., Zhang, L., 2016.  Deep learning with differential privacy, in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, p. 308–318.  doi:10.1145/2976749.2978318.
  * Abramian and Eklund (2019) Abramian, D., Eklund, A., 2019\.  Refacing: Reconstructing anonymized facial features using GANS, in: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pp. 1104–1108.  doi:10.1109/ISBI.2019.8759515.
  * Abrol et al. (2020) Abrol, A., Bhattarai, M., Fedorov, A., Du, Y., Plis, S., Calhoun, V., 2020\.  Deep residual learning for neuroimaging: An application to predict progression to Alzheimer’s disease.  Journal of Neuroscience Methods 339, 108701.
  * Al-Rubaie and Chang (2019) Al-Rubaie, M., Chang, J.M., 2019\.  Privacy-preserving machine learning: Threats and solutions.  IEEE Security Privacy 17, 49–58.  doi:10.1109/MSEC.2018.2888775.
  * Alqaraawi et al. (2020) Alqaraawi, A., Schuessler, M., Weiß, P., Costanza, E., Bianchi-Berthouze, N., 2020.  Evaluating saliency map explanations for convolutional neural networks: A user study, in: IUI ’20: 25th International Conference on Intelligent User Interfaces, pp. 275–285.  doi:10.1145/3377325.3377519.
  * Ayhan and Berens (2018) Ayhan, M., Berens, P., 2018\.  Test-time data augmentation for estimation of heteroscedastic aleatoric uncertainty in deep neural networks, in: Proceedings of Medical Imaging with Deep Learning (2018).
  * Bashyam et al. (2020) Bashyam, V., Erus, G., Doshi, J., Habes, M., Nasralah, I., Truelove-Hill, M., Srinivasan, D., Mamourian, L., Pomponio, R., Fan, Y., Launer, L., Masters, C., Maruff, P., Zhuo, C., Volzke, H., Johnson, S., Fripp, J., Koutsouleris, N., Satterthwaite, T., Davatzikos, C., 2020\.  MRI signatures of brain age and disease over the lifespan based on a deep brain network and 14 468 individuals worldwide.  Brain : a journal of neurology 143\.  doi:10.1093/brain/awaa160.
  * Bass et al. (2020) Bass, C., Silva, M., Sudre, C., Tudosiu, P., Smith, S., Robinson, E., 2020\.  ICAM: Interpretable classification via disentangled representations and feature attribution mapping, in: 34th Conference on Neural Information Processing Systems (NeurIPS 2020).  doi:arxiv.org/abs/2006.08287.
  * Baumgartner et al. (2018) Baumgartner, C.F., Koch, L.M., Tezcan, K.C., Ang, J.X., Konukoglu, E., 2018.  Visual feature attribution using wasserstein GANs, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8309–8319.  doi:10.1109/CVPR.2018.00867.
  * Baumgartner et al. (2019) Baumgartner, C.F., Tezcan, K.C., Chaitanya, K., Hötker, A.M., Muehlematter, U.J., Schawkat, K., Becker, A.S., Donati, O., Konukoglu, E., 2019.  PHiSeg: Capturing uncertainty in medical image segmentation, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pp. 119–127.  doi:10.1007/978-3-030-32245-814.
  * Ben-David et al. (2010) Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J., 2010\.  A theory of learning from different domains.  Machine Learning 79, 151–175.
  * Bian et al. (2020) Bian, C., Yuan, C., Wang, J., Li, M., Yang, X., Yu, S., Ma, K., Yuan, J., Zheng, Y., 2020.  Uncertainty-aware domain alignment for anatomical structure segmentation.  Medical Image Analysis 64, 101732\.  doi:https://doi.org/10.1016/j.media.2020.101732.
  * Billot et al. (2020a) Billot, B., Bocchetta, M., Todd, E., Dalca, A.V., Rohrer, J.D., Iglesias, J.E., 2020a.  Automated segmentation of the hypothalamus and associated subunits in brain MRI.  NeuroImage 223, 117287\.  doi:https://doi.org/10.1016/j.neuroimage.2020.117287.
  * Billot et al. (2020b) Billot, B., Greve, D., Van Leemput, K., Fischl, B., Iglesias, J., Dalca, A., 2020b.  A learning strategy for contrast-agnostic MRI segmentation, in: Proceedings of Machine Imaging with Deep Learning (MIDL) 2020.  doi:arXiv:2003.01995v3.
  * Billot et al. (2020c) Billot, B., Robinson, E., Dalca, A.V., Iglesias, J.E., 2020c.  Partial volume segmentation of brain MRI scans of any resolution and contrast, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, pp. 177–187.  doi:10.1007/978-3-030-59728-318.
  * Binder et al. (2016) Binder, A., Lapuschkin, S., Montavon, G., Müller, K.R., Samek, W., 2016.  Layer-wise relevance propagation for deep neural network architectures, in: Information Science and Applications (ICISA) 2016, pp. 913–922.  doi:10.1007/978-981-10-0557-2_87.
  * Bindschaedler et al. (2018) Bindschaedler, V., Grubbs, P., Cash, D., Ristenpart, T., Shmatikov, V., 2018.  The tao of inference in privacy-protected databases.  Proc. VLDB Endow. 11, 1715–1728.  doi:10.14778/3236187.3236217.
  * Bookheimer et al. (2018) Bookheimer, S., Salat, D., Terpstra, M., Ances, B., Barch, D., Buckner, R., Burgess, G., Curtiss, S., Diaz-Santos, M., Elam, J., Fischl, B., Greve, D., Hagy, H., Harms, M., Hatch, O., Hedden, T., Hodge, C., Japardi, K., Kuhn, T., Yacoub, E., 2018\.  The lifespan human connectome project in aging: An overview.  NeuroImage 185.  doi:10.1016/j.neuroimage.2018.10.009.
  * Böhle et al. (2019) Böhle, M., Eitel, F., Weygandt, M., Ritter, K., 2019\.  Layer-wise relevance propagation for explaining deep neural network decisions in MRI-based Alzheimer’s disease classification.  Frontiers in Aging Neuroscience 11, 194.  doi:10.3389/fnagi.2019.00194.
  * Cabitza et al. (2020) Cabitza, F., Campagner, A., Albano, D., Aliprandi, A., Bruno, A., Chianca, V., Corazza, A., Di Pietto, F., Gambino, A., Gitto, S., Messina, C., Orlandi, D., Pedone, L., Zappia, M., Sconfienza, L.M., 2020.  The elephant in the machine: Proposing a new metric of data reliability and its application to a medical case to assess classification reliability.  Applied Sciences 10.  doi:10.3390/app10114014.
  * Cabitza et al. (2019) Cabitza, F., Ciucci, D., Rasoini, R., 2019.  A giant with feet of clay: On the validity of the data that feed machine learning in medicine, in: Cabitza, F., Batini, C., Magni, M. (Eds.), Organizing for the Digital World, pp. 121–136.  doi:10.1007/978-3-319-90503-710.
  * Cetin Karayumak et al. (2019) Cetin Karayumak, S., Bouix, S., Ning, L., James, A., Crow, T., Shenton, M., Kubicki, M., Rathi, Y., 2019\.  Retrospective harmonization of multi-site diffusion MRI data acquired with different acquisition parameters.  NeuroImage 184, 180–200.  doi:10.1016/j.neuroimage.2018.08.073.
  * Chaitanya et al. (2020) Chaitanya, K., Erdil, E., Karani, N., Konukoglu, E., 2020\.  Contrastive learning of global and local features for medical image segmentation with limited annotations .
  * Chaitanya et al. (2019) Chaitanya, K., Karani, N., Baumgartner, C.F., Becker, A., Donati, O., Konukoglu, E., 2019\.  Semi-supervised and task-driven data augmentation, in: Information Processing in Medical Imaging, pp. 29–41.  doi:10.1007/978-3-030-20351-1s3.
  * Chapelle et al. (2001) Chapelle, O., Weston, J., Bottou, L., Vapnik, V., 2001\.  Vicinal risk minimization, in: Leen, T., Dietterich, T., Tresp, V. (Eds.), Advances in Neural Information Processing Systems, MIT Press.
  * Chen et al. (2018) Chen, H., Dou, Q., Yu, L., Qin, J., Heng, P.A., 2018\.  VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images.  NeuroImage 170, 446–455.  doi:https://doi.org/10.1016/j.neuroimage.2017.04.041.
  * Chen et al. (2019) Chen, L., Bentley, P., Mori, K., Misawa, K., Fujiwara, M., Rueckert, D., 2019\.  Self-supervised learning for medical image analysis using image context restoration.  Medical Image Analysis 58, 101539\.  doi:https://doi.org/10.1016/j.media.2019.101539.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., Hinton, G., 2020\.  A simple framework for contrastive learning of visual representations.  ArXiv .
  * Cheplygina et al. (2019) Cheplygina, V., de Bruijne, M., Pluim, J.P., 2019.  Not-so-supervised: A survey of semi-supervised, multi-instance, and transfer learning in medical image analysis.  Medical Image Analysis 54, 280–296.  doi:https://doi.org/10.1016/j.media.2019.03.009.
  * Cho et al. (2015) Cho, J., Lee, K., Shin, E., Choy, G., Do, S., 2015\.  How much data is needed to train a medical image deep learning system to achieve necessary high accuracy.  arXiv: Learning doi:arXiv:1511.06348.
  * Cicek et al. (2016) Cicek, O., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O., 2016.  3D U-Net: Learning dense volumetric segmentation from sparse annotation, in: Medical Image Computing and Computer-Assisted Intervention - MICCAI 2016, pp. 424 – 432\.  doi:10.1007/978.3.319.46723.8.49.
  * Clement et al. (2019) Clement, C., Selman, L., Kehoe, P., Howden, B., Lane, A., Horwood, J., 2019\.  Challenges to and facilitators of recruitment to an Alzheimer’s disease clinical trial: A qualitative interview study.  Journal of Alzheimer’s Disease 69, 1–9.  doi:10.3233/JAD-190146.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L., Kai Li, Li Fei-Fei, 2009\.  ImageNet: A large-scale hierarchical image database, in: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255.  doi:10.1109/CVPR.2009.5206848.
  * Dewey et al. (2019) Dewey, B., Zhao, C., Reinhold, J., Carass, A., Fitzgerald, K., Sotirchos, E., Saidha, S., Oh, J., Pham, D., Calabresi, P., Zijl, P., Prince, J., 2019\.  DeepHarmony: A deep learning approach to contrast harmonization across scanner changes.  Magnetic Resonance Imaging 64.
  * Di Martino et al. (2013) Di Martino, A., Yan, C.G., Li, Q., Denio, E., Castellanos, F., Alaerts, K., Anderson, J., Assaf, M., Bookheimer, S., Dapretto, M., Deen, B., Delmonte, S., Dinstein, I., Birgit, E.W., Fair, D., Gallagher, L., Kennedy, D., Keown, C., Keysers, C., Milham, M., 2013\.  The Autism Brain Imaging Data Exchange: Towards large-scale evaluation of the intrinsic brain architecture in autism.  Molecular psychiatry 19.  doi:10.1038/mp.2013.78.
  * Dinsdale et al. (2021a) Dinsdale, N.K., Bluemke, E., Smith, S., Arya, Z., Vidaurre, D., Jenkinson, M., Namburete, A.I., 2021a.  Learning patterns of the ageing brain in MRI using deep convolutional network.  Neuroimage 224.  doi:10.1016/j.neuroimage.2020.117401.
  * Dinsdale et al. (2021b) Dinsdale, N.K., Jenkinson, M., Namburete, A.I., 2021b.  Deep learning-based unlearning of dataset bias for MRI harmonisation and confound removal.  NeuroImage 228, 117689\.  doi:https://doi.org/10.1016/j.neuroimage.2020.117689.
  * Dinsdale et al. (2019) Dinsdale, N.K., Jenkinson, M., Namburete, A.I.L., 2019.  Spatial warping network for 3D segmentation of the hippocampus in MR images, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2019, pp. 284–291.  doi:10.1007/978-3-030-32248-9.32.
  * Dorent et al. (2020) Dorent, R., Joutard, S., Shapey, J., Bisdas, S., Kitchen, N., Bradford, R., Saeed, S., Modat, M., Ourselin, S., Vercauteren, T., 2020\.  Scribble-based domain adaptation via co-segmentation, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, pp. 479–489.  doi:10.1007/978-3-030-59710-8-47.
  * Dwork and Roth (2014) Dwork, C., Roth, A., 2014.  The algorithmic foundations of differential privacy.  Found. Trends Theor. Comput. Sci. 9, 211–407.  doi:10.1561/0400000042.
  * Feng et al. (2017) Feng, X., Yang, J., Laine, A., Angelini, E., 2017.  Discriminative localization in CNNs for weakly-supervised segmentation of pulmonary nodules, pp. 568–576.  doi:10.1007/978-3-319-66179-765.
  * Feyjie et al. (2020) Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J., 2020\.  Semi-supervised few-shot learning for medical image segmentation.  arXiv doi:arXiv:2003.08462.
  * Frazier et al. (2008) Frazier, J., Hodge, S., Breeze, J., Giuliano, A., Terry, J., Moore, C., Kennedy, D., Lopez-Larson, M., Caviness, V., Seidman, L., Zablotsky, B., Makris, N., 2008\.  Diagnostic and sex effects on limbic volumes in early-onset bipolar disorder and schizophrenia.  Schizophrenia bulletin 34, 37–46.  doi:10.1093/schbul/sbm120.
  * Frisoni and Jack (2015) Frisoni, G., Jack, C., 2015\.  HarP: The EADC-ADNI harmonized protocol for manual hippocampal segmentation. a standard of reference from a global working group.  Alzheimer’s and Dementia 11.  doi:10.1016/j.jalz.2014.05.1761.
  * Gal and Ghahramani (2016) Gal, Y., Ghahramani, Z., 2016\.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning, in: Proceedings of The 33rd International Conference on Machine Learning, pp. 1050–1059.  doi:10.5555/3045390.3045502.
  * Gal et al. (2017) Gal, Y., Islam, R., Ghahramani, Z., 2017.  Deep Bayesian active learning with image data, in: Proceedings of the 34th International Conference on Machine Learning, pp. 1183–1192.
  * Ganin et al. (2015) Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., Lempitsky, V., 2015\.  Domain-adversarial training of neural networks.  Journal of Machine Learning Research .
  * Gentili et al. (2017) Gentili, M., Hajian, S., Castillo, C., 2017.  A case study of anonymization of medical surveys, in: Proceedings of the 2017 International Conference on Digital Health, p. 77–81.  doi:10.1145/3079452.3079490.
  * Ghafoorian et al. (2018) Ghafoorian, M., Teuwen, J., Manniesing, R., Leeuw, F.E., Ginneken, B., Karssemeijer, N., Platel, B., 2018.  Student beats the teacher: Deep neural networks for lateral ventricles segmentation in brain MR, in: Proceedings Volume 10574, Medical Imaging 2018: Image Processing; 105742U (2018).  doi:doi.org/10.1117/12.2293569.
  * Glorot et al. (2011) Glorot, X., Bordes, A., Bengio, Y., 2011.  Deep sparse rectifier neural networks, in: Gordon, G., Dunson, D., Dudík, M. (Eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 315–323.
  * Graber (2013) Graber, M.L., 2013.  The incidence of diagnostic error in medicine.  BMJ Quality & Safety 22, ii21–ii27.  doi:10.1136/bmjqs-2012-001615.
  * Grill et al. (2020) Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., Piot, B., kavukcuoglu, k., Munos, R., Valko, M., 2020\.  Bootstrap your own latent - a new approach to self-supervised learning, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (Eds.), Advances in Neural Information Processing Systems, pp. 21271–21284.
  * Guan et al. (2020) Guan, H., Yang, E., Yap, P.T., Shen, D., Liu, M., 2020\.  Attention-guided deep domain adaptation for brain dementia identification with multi-site neuroimaging data, in: Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning, pp. 31–40.  doi:10.1007/978-3-030-60548-34.
  * Guha Roy et al. (2018) Guha Roy, A., Conjeti, S., Navab, N., Wachinger, C., 2018\.  QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy.  NeuroImage 186.  doi:10.1016/j.neuroimage.2018.11.042.
  * Haarburger et al. (2020) Haarburger, C., Müller-Franzes, G., Weninger, L., Kuhl, C., Truhn, D., Merhof, D., 2020\.  Radiomics feature reproducibility under inter-rater variability in segmentations of CT images.  Scientific Reports 10.  doi:10.1038/s41598-020-69534-6.
  * Hadsell et al. (2006) Hadsell, R., Chopra, S., Lecun, Y., 2006.  Dimensionality reduction by learning an invariant mapping, pp. 1735 – 1742.  doi:10.1109/CVPR.2006.100.
  * Han et al. (2006) Han, X., Jovicich, J., Salat, D., Kouwe, A., Quinn, B., Czanner, S., Busa, E., Pacheco, J., Albert, M., Killiany, R., Maguire, R., Rosas, H., Makris, N., Dale, A., Dickerson, B., Fischl, B., 2006\.  Reliability of MRI-derived measurements of human cerebral cortical thickness: The effects of field strength, scanner upgrade and manufacturer.  NeuroImage 32, 180–94.  doi:10.1016/j.neuroimage.2006.02.051.
  * Hatamizadeh et al. (2019) Hatamizadeh, A., Terzopoulos, D., Myronenko, A., 2019.  End-to-End Boundary Aware Networks for Medical Image Segmentation.  pp. 187–194.  doi:10.1007/978-3-030-32692-0_22.
  * He et al. (2019) He, K., Girshick, R., Dollar, P., 2019.  Rethinking ImageNet pre-training, in: 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4917–4926.  doi:10.1109/ICCV.2019.00502.
  * He et al. (2020) He, T., Kong, R., Holmes, A.J., Nguyen, M., Sabuncu, M.R., Eickhoff, S.B., Bzdok, D., Feng, J., Yeo, B.T., 2020\.  Deep neural networks and kernel regression achieve comparable accuracies for functional connectivity prediction of behavior and demographics.  NeuroImage 206, 116276\.  doi:doi.org/10.1016/j.neuroimage.2019.116276.
  * Henaff (2020) Henaff, O., 2020.  Data-efficient image recognition with contrastive predictive coding, in: Proceedings of the 37th International Conference on Machine Learning, pp. 4182–4192.
  * Henschel et al. (2020) Henschel, L., Conjeti, S., Estrada, S., Diers, K., Fischl, B., Reuter, M., 2020\.  FastSurfer - a fast and accurate deep learning based neuroimaging pipeline.  NeuroImage 219, 117012\.  doi:doi.org/10.1016/j.neuroimage.2020.117012.
  * Herzog et al. (2020) Herzog, L., Murina, E., Dürr, O., Wegener, S., Sick, B., 2020.  Integrating uncertainty in deep neural networks for MRI based stroke analysis.  Medical Image Analysis 65, 101790\.  doi:https://doi.org/10.1016/j.media.2020.101790.
  * Hinton et al. (2015) Hinton, G., Vinyals, O., Dean, J., 2015.  Distilling the knowledge in a neural network.  ArXiv doi:arXiv:1503.02531.
  * Holmes et al. (2015) Holmes, A., Hollinshead, M., O’Keefe, T., Petrov, V., Fariello, G., Wald, L., Fischl, B., Rosen, B., Mair, R., Roffman, J., Smoller, J., Buckner, R., 2015\.  Brain Genomics Superstruct Project initial data release with structural, functional, and behavioral measures.  Scientific data 2, 150031\.  doi:10.1038/sdata.2015.31.
  * Hostage et al. (2013) Hostage, C., Choudhury, K., Doraiswamy, P., Petrella, J., 2013\.  Dissecting the gene dose-effects of the APOE $\eta$4 and $\eta$2 alleles on hippocampal volumes in aging and alzheimer’s disease.  PloS one 8, e54483.  doi:10.1371/journal.pone.0054483.
  * Hu et al. (2018) Hu, J., Shen, L., Sun, G., 2018.  Squeeze-and-excitation networks, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132–7141.  doi:10.1109/CVPR.2018.00745.
  * Huang et al. (2019) Huang, L., Shea, A.L., Qian, H., Masurkar, A., Deng, H., Liu, D., 2019.  Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records.  Journal of Biomedical Informatics 99, 103291.  doi:10.1016/j.jbi.2019.103291.
  * Hughes et al. (2017) Hughes, E., Winchman, T., Padormo, F., Teixeira, R.P., Wurie, J., Sharma, M., Fox, M., Hutter, J., Cordero-Grande, L., Price, A., Allsop, J., Bueno-Conde, J., Tusor, N., Arichi, T., Edwards, D., Rutherford, M., Counsell, S., Hajnal, J., 2017\.  A dedicated neonatal brain imaging system.  Magnetic Resonance in Medicine 78, C1–C1.  doi:10.1002/mrm.26844.
  * Iglesias et al. (2020a) Iglesias, J., Billot, B., Balbastre, Y., Tabari, A., Conklin, J., Alexander, D., Golland, P., Edlow, B., Fischl, B., 2020a.  Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI exams with scans of different orientation, resolution and contrast.  ArXiv .
  * Iglesias et al. (2020b) Iglesias, J., Billot, B., Balbastre, Y., Tabari, A., Conklin, J., Alexander, D., Golland, P., Edlow, B., Fischl, B., 2020b.  Joint super-resolution and synthesis of 1 mm isotropic MP-RAGE volumes from clinical MRI exams with scans of different orientation, resolution and contrast.  ArXiv doi:arXiv:2012.13340.
  * Ioffe and Szegedy (2015) Ioffe, S., Szegedy, C., 2015\.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Proceedings of the 32nd International Conference on Machine Learning, PMLR. pp. 448–456.  URL: http://proceedings.mlr.press/v37/ioffe15.html.
  * Isensee et al. (2021) Isensee, F., Jaeger, P., Kohl, S., Petersen, J., Maier-Hein, K., 2021.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature Methods 18, 1–9.  doi:10.1038/s41592-020-01008-z.
  * Jack et al. (2008) Jack, C., Bernstein, M., Fox, N., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P., Whitwell, J., Ward, C., Dale, A., Felmlee, J., Gunter, J., Hill, D., Killiany, R., Schuff, N., Fox-Bosetti, S., Lin, C., Studholme, C., Weiner, M., 2008\.  The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods.  Journal of magnetic resonance imaging : JMRI 27, 685–91.  doi:10.1002/jmri.21049.
  * Jovicich et al. (2006) Jovicich, J., Czanner, S., Greve, D., Haley, E., Kouwe, A., Gollub, R., Kennedy, D., Schmitt, F., Brown, G., Macfall, J., Fischl, B., Dale, A., 2006\.  Reliability in multi-site structural MRI studies: Effects of gradient non-linearity correction on phantom and human data.  NeuroImage 30, 436–43.  doi:10.1016/j.neuroimage.2005.09.046.
  * Jungo et al. (2018) Jungo, A., McKinley, R., Meier, R., Knecht, U., Vera, L., Pérez-Beteta, J., Molina-García, D., Pérez-García, V.M., Wiest, R., Reyes, M., 2018.  Towards uncertainty-assisted brain tumor segmentation and survival prediction, pp. 474–485.  doi:10.1007/978-3-319-75238-9_40.
  * Kaissis et al. (2021) Kaissis, G., Ziller, A., Passerat-Palmbach, J., Ryffel, T., Usynin, D., Trask, A., Lima, I., Mancuso, J., Jungmann, F., Steinborn, M.M., Saleh, A., Makowski, M., Rueckert, D., Braren, R., 2021\.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  Nature Machine Intelligence , 1–12doi:10.1038/s42256-021-00337-8.
  * Kamnitsas et al. (2017a) Kamnitsas, K., Baumgartner, C., Ledig, C., Newcombe, V., Simpson, J., Kane, A., Menon, D., Nori, A., Criminisi, A., Rueckert, D., Glocker, B., 2017a.  Unsupervised domain adaptation in brain lesion segmentation with adversarial networks, in: Information Processing in Medical Imaging, pp. 597–609.  doi:10.1007/978-3-319-59050-947.
  * Kamnitsas et al. (2017b) Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B., 2017b.  Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation.  Medical Image Analysis 36, 61–78.  doi:https://doi.org/10.1016/j.media.2016.10.004.
  * Karlawish et al. (2008) Karlawish, J., Cary, M.S., Rubright, J., TenHave, T., 2008\.  How redesigning AD clinical trials might increase study partners’ willingness to participate.  Neurology 71, 1883–1888.  doi:10.1212/01.wnl.0000336652.05779.ea.
  * Keane and Topol (2018) Keane, P., Topol, E., 2018.  With an eye to AI and autonomous diagnosis.  npj Digital Medicine 1.  doi:10.1038/s41746-018-0048-y.
  * Kendall and Gal (2017) Kendall, A., Gal, Y., 2017.  What uncertainties do we need in bayesian deep learning for computer vision?, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, p. 5580–5590.  doi:10.5555/3295222.3295309.
  * Khagi et al. (2018) Khagi, B., Lee, C.G., Kwon, G.R., 2018.  Alzheimer’s disease classification from brain MRI based on transfer learning from CNN, in: 2018 11th Biomedical Engineering International Conference (BMEiCON), pp. 1–4.  doi:10.1109/BMEiCON.2018.8609974.
  * Khosla et al. (2019) Khosla, M., Jamison, K., Kuceyeski, A., Sabuncu, M.R., 2019\.  Ensemble learning with 3D convolutional neural networks for functional connectome-based prediction.  NeuroImage 199, 651–662.  doi:https://doi.org/10.1016/j.neuroimage.2019.06.012.
  * Kornblith et al. (2019) Kornblith, S., Shlens, J., Le, Q.V., 2019.  Do better ImageNet models transfer better?, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * Krizhevsky et al. (2009) Krizhevsky, A., Nair, V., Hinton, G., 2009.  CIFAR-10 (Canadian institute for advanced research).  Technical Report URL: http://www.cs.toronto.edu/~kriz/cifar.html.
  * Krizhevsky et al. (2012) Krizhevsky, A., Sutskever, I., Hinton, G.E., 2012.  ImageNet classification with deep convolutional neural networks, in: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems.
  * Kushibar et al. (2019) Kushibar, K., Valverde, S., González-Villà, S., Bernal, J., Cabezas, M., Oliver, A., Llado, X., 2019.  Supervised domain adaptation for automatic sub-cortical brain structure segmentation with minimal user interaction.  Scientific Reports 9.  doi:10.1038/s41598-019-43299-z.
  * Kwon et al. (2020) Kwon, Y., Won, J., Kim, B., Paik, M., 2020.  Uncertainty quantification using bayesian neural networks in classification: Application to biomedical image segmentation.  Computational Statistics and Data Analysis 142\.  doi:10.1016/j.csda.2019.106816.
  * Laves et al. (2020) Laves, M.H., Ihler, S., Fast, J.F., Kahrs, L.A., Ortmaier, T., 2020.  Well-calibrated regression uncertainty in medical imaging with deep learning, in: Medical Imaging with Deep Learning.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., Hinton, G., 2015.  Deep learning.  Nature 521, 436–44.  doi:10.1038/nature14539.
  * LeCun et al. (1990a) LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., Jackel, L., 1990a.  Handwritten digit recognition with a back-propagation network, in: Touretzky, D. (Ed.), Advances in Neural Information Processing Systems.
  * Lecun et al. (1998) Lecun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998\.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE 86, 2278–2324.  doi:10.1109/5.726791.
  * LeCun et al. (1990b) LeCun, Y., Denker, J.S., Solla, S.A., 1990b.  Optimal brain damage, in: Advances in Neural Information Processing Systems 2, Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. p. 598–605.
  * Lee et al. (2020a) Lee, B., Yamanakkanavar, N., Choi, J.Y., 2020a.  Automatic segmentation of brain MRI using a novel patch-wise U-Net deep architecture.  PLOS ONE 15, 1–20.  doi:10.1371/journal.pone.0236493.
  * Lee et al. (2020b) Lee, H.Y., Tseng, H.Y., Mao, Q., Huang, J.B., Lu, Y.D., Singh, M., Yang, M.H., 2020b.  DRIT++: Diverse image-to-image translation via disentangled representations.  International Journal of Computer Vision 128\.  doi:10.1007/s11263-019-01284-z.
  * Li et al. (2017) Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H., 2017.  Pruning filters for efficient ConvNets, in: International Conference on Learning Representations.
  * Li et al. (2020) Li, T., Sahu, A.K., Talwalkar, A., Smith, V., 2020\.  Federated learning: Challenges, methods, and future directions.  IEEE Signal Processing Magazine 37, 50–60.  doi:10.1109/MSP.2020.2975749.
  * Li et al. (2019) Li, W., Milletarì, F., Xu, D., Rieke, N., Hancox, J., Zhu, W., Baust, M., Cheng, Y., Ourselin, S., Cardoso, M.J., Feng, A., 2019.  Privacy-preserving federated brain tumour segmentation, in: Machine Learning in Medical Imaging, pp. 133–141.  doi:10.1007/978-3-030-32692-0-16.
  * Liu et al. (2020) Liu, M., Li, F., Yan, H., Wang, K., Ma, Y., Shen, L., Xu, M., 2020.  A multi-model deep convolutional neural network for automatic hippocampus segmentation and classification in Alzheimer’s disease.  NeuroImage 208, 116459\.  doi:https://doi.org/10.1016/j.neuroimage.2019.116459.
  * Livne et al. (2019) Livne, M., Rieger, J., Aydin, O.U., Taha, A.A., Akay, E.M., Kossen, T., Sobesky, J., Kelleher, J.D., Hildebrand, K., Frey, D., Madai, V.I., 2019.  A U-Net deep learning framework for high performance vessel segmentation in patients with cerebrovascular disease.  Frontiers in Neuroscience 13.  doi:10.3389/fnins.2019.00097.
  * Lu et al. (2021) Lu, B., Li, H.X., Chang, Z.K., Li, L., Chen, N.X., Zhu, Z.C., Zhou, H.X., Li, X.Y., Wang, Y.W., Cui, S.X., Deng, Z.Y., Fan, Z., Yang, H., Chen, X., Thompson, P.M., Castellanos, F.X., Yan, C.G., 2021.  A practical Alzheimer disease classifier via brain imaging-based deep learning on 85,721 samples.  ArXiv doi:10.1101/2020.08.18.256594.
  * Luo et al. (2021) Luo, X., Wang, G., Song, T., Zhang, J., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., Zhang, S., 2021\.  MIDeepSeg: Minimally interactive segmentation of unseen objects from medical images using deep learning.  Medical Image Analysis , 102102doi:https://doi.org/10.1016/j.media.2021.102102.
  * Lyu et al. (2021) Lyu, I., Bao, S., Hao, L., Yao, J., Miller, J.A., Voorhies, W., Taylor, W.D., Bunge, S.A., Weiner, K.S., Landman, B.A., 2021.  Labeling lateral prefrontal sulci using spherical data augmentation and context-aware training.  NeuroImage 229, 117758\.  doi:https://doi.org/10.1016/j.neuroimage.2021.117758.
  * Madan (2021) Madan, C., 2021.  Scan once, analyse many: Using large open-access neuroimaging datasets to understand the brain.  Neuroinformatics doi:10.1007/s12021-021-09519-6.
  * Maier-Hein et al. (2018) Maier-Hein, L., Eisenmann, M., Reinke, A., Onogur, S., Stankovic, M., Scholz, P., Arbel, T., Bogunović, H., Bradley, A., Carass, A., Feldmann, C., Frangi, A., Full, P., Ginneken, B., Hanbury, A., Honauer, K., Kozubek, M., Landman, B., März, K., Kopp-Schneider, A., 2018\.  Why rankings of biomedical image analysis competitions should be interpreted with care.  Nature Communications 9.  doi:10.1038/s41467-018-07619-7.
  * Major et al. (2020) Major, D., Lenis, D., Wimmer, M., Sluiter, G., Berg, A., Bühler, K., 2020\.  Interpreting medical image classifiers by optimization based counterfactual impact analysis, in: 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pp. 1096–1100.  doi:10.1109/ISBI45749.2020.9098681.
  * Manjón (2017) Manjón, J.V., 2017.  MRI Preprocessing. Cham.  pp. 53–63.  doi:10.1007/978-3-319-43504-6_5.
  * Marcus et al. (2007) Marcus, D., Wang, T., Parker, J., Csernansky, J., Morris, J., Buckner, R., 2007\.  Open access series of imaging studies (OASIS): Cross-sectional MRI data in young, middle aged, nondemented, and demented older adults.  Journal of cognitive neuroscience 19, 1498–507.  doi:10.1162/jocn.2007.19.9.1498.
  * Marek et al. (2019) Marek, S., Tervo-Clemmens, B., Nielsen, A., Wheelock, M., Miller, R., Laumann, T., Earl, E., Foran, W., Cordova, M., Doyle, O., Perrone, A., Miranda Dominguez, O., Feczko, E., Sturgeon, D., Graham, A., Hermosillo, R., Snider, K., Galassi, A., Nagel, B., Dosenbach, N., 2019\.  Identifying reproducible individual differences in childhood functional brain networks: An ABCD study.  Developmental Cognitive Neuroscience 40, 100706.  doi:10.1016/j.dcn.2019.100706.
  * McEver and Manjunath (2020) McEver, R., Manjunath, B., 2020\.  PCAMs: Weakly supervised semantic segmentation using point supervision.  ArXiv .
  * Mehmood et al. (2021) Mehmood, A., Yang, S., Feng, Z., Wang, M., Ahmad, A.S., Khan, R., Maqsood, M., Yaqub, M., 2021\.  A transfer learning approach for early diagnosis of Alzheimer’s disease on MRI images.  Neuroscience 460, 43–52.  doi:https://doi.org/10.1016/j.neuroscience.2021.01.002.
  * Menze et al. (2014) Menze, B., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahaniy, K., Kirby, J., Burren, Y., Porz, N., Slotboomy, J., Wiest, R., Lancziy, L., Gerstnery, E., Webery, M.A., Arbel, T., Avants, B., Ayache, N., Buendia, P., Collins, L., Cordier, N., Van Leemput, K., 2014\.  The multimodal brain tumor image segmentation benchmark (BRATS).  IEEE Transactions on Medical Imaging 99\.  doi:10.1109/TMI.2014.2377694.
  * Mirza and Osindero (2014) Mirza, M., Osindero, S., 2014\.  Conditional generative adversarial nets.  ArXiv doi:arXiv:1411.1784.
  * Mitra et al. (2017) Mitra, S., Banerjee, S., Hayashi, Y., 2017.  Volumetric brain tumour detection from MRI using visual saliency.  PLOS ONE 12, 1–14.  doi:10.1371/journal.pone.0187209.
  * Molchanov et al. (2016) Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J., 2016.  Pruning convolutional neural networks for resource efficient transfer learning, in: International Conference on Learning Representations.
  * Morid et al. (2021) Morid, M.A., Borjali, A., Del Fiol, G., 2021.  A scoping review of transfer learning research on medical image analysis using ImageNet.  Computers in Biology and Medicine 128, 104115.  doi:https://doi.org/10.1016/j.compbiomed.2020.104115.
  * Moyer et al. (2020) Moyer, D., Ver Steeg, G., Tax, C., Thompson, P., 2020\.  Scanner invariant representations for diffusion MRI harmonization.  Magnetic Resonance in Medicine doi:10.1002/mrm.28243.
  * Murugesan et al. (2020) Murugesan, B., Vijayarangan, S., Sarveswaran, K., Ram, K., Sivaprakasam, M., 2020.  KD-MRI: A knowledge distillation framework for image reconstruction and image restoration in mri workflow, in: Proceedings of the Third Conference on Medical Imaging with Deep Learning, PMLR. pp. 515–526.
  * Nath et al. (2021) Nath, V., Yang, D., Landman, B.A., Xu, D., Roth, H.R., 2021.  Diminishing uncertainty within the training pool: Active learning for medical image segmentation.  ArXiv doi:abs/2101.02323.
  * Nguyen et al. (2020) Nguyen, K.P., Fatt, C.C., Treacher, A., Mellema, C., Trivedi, M.H., Montillo, A., 2020\.  Anatomically informed data augmentation for functional MRI with applications to deep learning, in: Medical Imaging 2020: Image Processing, pp. 172 – 177.  doi:10.1117/12.2548630.
  * Ning et al. (2020) Ning, L., Bonet-Carne, E., Grussu, F., Sepehrband, F., Kaden, E., Veraart, J., Blumberg, S.B., Khoo, C.S., Palombo, M., Kokkinos, I., Alexander, D.C., Coll-Font, J., Scherrer, B., Warfield, S.K., Karayumak, S.C., Rathi, Y., Koppers, S., Weninger, L., Ebert, J., Merhof, D., Moyer, D., Pietsch, M., Christiaens, D., Gomes Teixeira, R.A., Tournier, J.D., Schilling, K.G., Huo, Y., Nath, V., Hansen, C., Blaber, J., Landman, B.A., Zhylka, A., Pluim, J.P., Parker, G., Rudrapatna, U., Evans, J., Charron, C., Jones, D.K., Tax, C.M., 2020.  Cross-scanner and cross-protocol multi-shell diffusion MRI data harmonization: Algorithms and results.  NeuroImage 221, 117128\.  doi:https://doi.org/10.1016/j.neuroimage.2020.117128.
  * Nobis et al. (2019) Nobis, L., Manohar, S.G., Smith, S.M., Alfaro-Almagro, F., Jenkinson, M., Mackay, C.E., Husain, M., 2019.  Hippocampal volume across age: Nomograms derived from over 19,700 people in UK Biobank.  NeuroImage: Clinical 23, 101904\.  doi:10.1016/j.nicl.2019.101904.
  * Nooner et al. (2012) Nooner, K., Colcombe, S., Tobe, R., Mennes, M., Benedict, M., Moreno, A., Panek, L., Brown, S., Zavitz, S., Li, Q., Sikka, S., Gutman, D., Bangaru, S., Schlachter, R., Kamiel, S., Anwar, A., Hinz, C., Kaplan, M., Rachlin, A., Milham, M., 2012\.  The NKI-Rockland sample: A model for accelerating the pace of discovery science in psychiatry.  Frontiers in neuroscience 6, 152\.  doi:10.3389/fnins.2012.00152.
  * Olah et al. (2018) Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L., Ye, K., Mordvintsev, A., 2018.  The building blocks of interpretability.  Distill doi:10.23915/distill.00010.
  * Papernot et al. (2017) Papernot, N., McDaniel, P., Goodfellow, I., Jha, S., Celik, Z.B., Swami, A., 2017\.  Practical black-box attacks against machine learning, in: Association for Computing Machinery 2017, pp. 506–519.  doi:10.1145/3052973.3053009.
  * Park et al. (2018) Park, J., Woo, S., Lee, J.Y., Kweon, I., 2018.  BAM: Bottleneck attention module, in: Proceedings of the British Machine Vision Conference (BMVC) 2018.  doi:arXiv:1807.06514.
  * Paulus and Kent (2020) Paulus, J., Kent, D., 2020.  Predictably unequal: understanding and addressing concerns that algorithmic clinical prediction may increase health disparities.  npj Digital Medicine 3, 99\.  doi:10.1038/s41746-020-0304-9.
  * Pawlowski et al. (2020) Pawlowski, N., Coelho de Castro, D., Glocker, B., 2020.  Deep structural causal models for tractable counterfactual inference, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.F., Lin, H. (Eds.), Advances in Neural Information Processing Systems, pp. 857–869.
  * Pawlowski and Glocker (2019) Pawlowski, N., Glocker, B., 2019\.  Is texture predictive for age and sex in brain {mri}?, in: International Conference on Medical Imaging with Deep Learning – Extended Abstract Track.  doi:openreview.net/forum?id=BJxgXfab94.
  * Pérez-García et al. (2020) Pérez-García, F., Sparks, R., Ourselin, S., 2020.  TorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.  arXiv:2003.04696 [cs, eess, stat] URL: http://arxiv.org/abs/2003.04696. arXiv: 2003.04696.
  * Perone et al. (2019) Perone, C.S., Ballester, P., Barros, R.C., Cohen-Adad, J., 2019\.  Unsupervised domain adaptation for medical imaging segmentation with self-ensembling.  NeuroImage 194, 1–11.  doi:https://doi.org/10.1016/j.neuroimage.2019.03.026.
  * Radosavovic et al. (2018) Radosavovic, I., Dollar, P., Girshick, R., Gkioxari, G., He, K., 2018.  Data distillation: Towards omni-supervised learning, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4119–4128.  doi:10.1109/CVPR.2018.00433.
  * Raghu et al. (2019) Raghu, M., Zhang, C., Kleinberg, J., Bengio, S., 2019\.  Transfusion: Understanding transfer learning with applications to medical imaging, in: 33rd Conference on Neural Information Processing Systems (NeurIPS 2019).
  * Rajchl et al. (2017) Rajchl, M., Lee, M.C.H., Oktay, O., Kamnitsas, K., Passerat-Palmbach, J., Bai, W., Damodaram, M., Rutherford, M.A., Hajnal, J.V., Kainz, B., Rueckert, D., 2017.  DeepCut: Object segmentation from bounding box annotations using convolutional neural networks.  IEEE Transactions on Medical Imaging 36, 674–683.  doi:10.1109/TMI.2016.2621185.
  * Ramprasaath et al. (2017) Ramprasaath, R.S., Cogswell, M., Vedantam, R., Parikh, D., Batra, D., 2017.  Grad-CAM: Visual explanations from deep networks via gradient-based localization, in: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 618–626.  doi:10.1109/ICCV.2017.74.
  * Rieke et al. (2020) Rieke, N., Hancox, J., Milletari, F., Roth, H., Albarqouni, S., Bakas, S., Galtier, M., Landman, B., Maier-Hein, K., Ourselin, S., Sheller, M., Summers, R., Trask, A., Xu, D., Baust, M., Cardoso, M.J., 2020\.  The future of digital health with federated learning.  npj Digital Medicine volume doi:10.1038/s41746-020-00323-1.
  * Roy et al. (2018) Roy, A.G., Conjeti, S., Navab, N., Wachinger, C., 2018\.  Inherent brain segmentation quality control from fully ConvNet Monte Carlo sampling, in: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pp. 664–672.  doi:10.1007/978-3-030-00928-1_75.
  * Schlemper et al. (2019) Schlemper, J., Oktay, O., Schaap, M., Heinrich, M., Kainz, B., Glocker, B., Rueckert, D., 2019.  Attention gated networks: Learning to leverage salient regions in medical images.  Medical Image Analysis 53.  doi:10.1016/j.media.2019.01.012.
  * Shah et al. (2019) Shah, N.H., Milstein, A., Bagley, PhD, S.C., 2019.  Making Machine Learning Models Clinically Useful.  JAMA 322, 1351–1352.  doi:10.1001/jama.2019.10306.
  * Sheller et al. (2019) Sheller, M.J., Reina, G.A., Edwards, B., Martin, J., Bakas, S., 2019.  Multi-institutional deep learning modeling without sharing patient data: A feasibility study on brain tumor segmentation, in: Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries, pp. 92–104.  doi:10.1007/978-3-030-11723-8-9.
  * Shokri and Shmatikov (2015) Shokri, R., Shmatikov, V., 2015\.  Privacy-preserving deep learning, in: 2015 53rd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 909–910.  doi:10.1109/ALLERTON.2015.7447103.
  * Shu et al. (2020) Shu, M., Liu, C., Qiu, W., Yuille, A., 2020.  Identifying model weakness with adversarial examiner.  Proceedings of the AAAI Conference on Artificial Intelligence 34, 11998–12006.  doi:10.1609/aaai.v34i07.6876.
  * Shwartzman et al. (2020) Shwartzman, O., Gazit, H., Shelef, I., Riklin-Raviv, T., 2020\.  The worrisome impact of an inter-rater bias on neural network training.  ArXiv doi:arXiv:1906.11872.
  * Simard et al. (1998) Simard, P., Lecun, Y., Denker, J., 1998.  Transformation invariance in pattern recognition tangent distance and tangent propagation.  Neural networks: tricks of the trade .
  * Simonyan et al. (2014) Simonyan, K., Vedaldi, A., Zisserman, A., 2014.  Deep inside convolutional networks: Visualising image classification models and saliency maps, in: Workshop at International Conference on Learning Representations.
  * Simonyan and Zisserman (2015) Simonyan, K., Zisserman, A., 2015\.  Very deep convolutional networks for large-scale image recognition, in: International Conference on Learning Representations.
  * Singla et al. (2021) Singla, S., Pollack, B., Wallace, S., Batmanghelich, K., 2021\.  Explaining the black-box smoothly- a counterfactual approach.  ArXiv .
  * Sinha and Dolz (2020) Sinha, A., Dolz, J., 2020.  Multi-scale self-guided attention for medical image segmentation.  IEEE Journal of Biomedical and Health Informatics PP, 1–1.  doi:10.1109/JBHI.2020.2986926.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research 15, 1929–1958.  doi:10.5555/2627435.2670313.
  * St-Jean et al. (2019) St-Jean, S., Viergever, M., Leemans, A., 2019.  Harmonization of diffusion MRI datasets with adaptive dictionary learning.  bioRXiv doi:10.1101/824706.
  * Sudlow et al. (2015) Sudlow, C., Gallacher, J., Allen, N., Beral, V., Burton, P., Danesh, J., Downey, P., Elliott, P., Green, J., Landray, M., Liu, B., Matthews, P., Ong, G., Pell, J., Silman, A., Young, A., Sprosen, T., Peakman, T., Collins, R., 2015.  UK biobank: An open access resource for identifying the causes of a wide range of complex diseases of middle and old age 12, e1001779.
  * Sundararajan et al. (2017) Sundararajan, M., Taly, A., Yan, Q., 2017.  Axiomatic attribution for deep networks, in: Proceedings of the 34th International Conference on Machine Learning.  doi:10.5555/3305890.3306024.
  * Sundaresan et al. (2021) Sundaresan, V., Zamboni, G., Dinsdale, N.K., Rothwell, P.M., Griffanti, L., Jenkinson, M., 2021\.  Comparison of domain adaptation techniques for white matter hyperintensity segmentation in brain MR images.  bioRxiv doi:10.1101/2021.03.12.435171.
  * Svensson et al. (2015) Svensson, C., Hübler, R., Figge, M., 2015.  Automated classification of circulating tumor cells and the impact of interobsever variability on classifier training and performance.  Journal of Immunology Research 2015, 1–9.  doi:10.1155/2015/573165.
  * Sweeney (2002) Sweeney, L., 2002.  K-anonymity: A model for protecting privacy.  Int. J. Uncertain. Fuzziness Knowl.-Based Syst. 10, 557–570.  doi:10.1142/S0218488502001648.
  * Tajbakhsh et al. (2020) Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X., 2020.  Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation.  Medical Image Analysis 63, 101693\.  doi:https://doi.org/10.1016/j.media.2020.101693.
  * Takacs and Manno-Kovacs (2018) Takacs, P., Manno-Kovacs, A., 2018\.  MRI brain tumor segmentation combining saliency and convolutional network features, in: 2018 International Conference on Content-Based Multimedia Indexing (CBMI), pp. 1–6.  doi:10.1109/CBMI.2018.8516544.
  * Takao et al. (2011) Takao, H., Hayashi, N., Ohtomo, K., 2011.  Effect of scanner in longitudinal studies of brain volume changes.  Journal of magnetic resonance imaging : JMRI 34, 438–44.  doi:10.1002/jmri.22636.
  * Takao et al. (2013) Takao, H., Hayashi, N., Ohtomo, K., 2013.  Effects of study design in multi-scanner voxel-based morphometry studies.  NeuroImage 84.  doi:10.1016/j.neuroimage.2013.08.046.
  * Talo et al. (2018) Talo, M., Baloglu, U., yıldırım, O., Acharya, U.R., 2018\.  Application of deep transfer learning for automated brain abnormality classification using MRimages.  Cognitive Systems Research doi:10.1016/j.cogsys.2018.12.007.
  * Tanner (2017) Tanner, A., 2017.  Our Bodies, Our Data: How Companies Make Billions Selling Our Medical Records.  Beacon Press.
  * Taylor et al. (2017) Taylor, J.R., Williams, N., Cusack, R., Auer, T., Shafto, M.A., Dixon, M., Tyler, L.K., Cam-CAN, Henson, R.N., 2017.  The Cambridge Centre for Ageing and Neuroscience (Cam-CAN) data repository: Structural and functional MRI, MEG, and cognitive data from a cross-sectional adult lifespan sample.  NeuroImage 144, 262–269.  doi:https://doi.org/10.1016/j.neuroimage.2015.09.018.
  * Thagaard et al. (2020) Thagaard, J., Hauberg, S., van der Vegt, B., Ebstrup, T., Hansen, J.D., Dahl, A.B., 2020\.  Can you trust predictive uncertainty under real dataset shifts in digital pathology?, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, Cham. pp. 824–833.  doi:10.1007/978-3-030-59710-8-80.
  * Thakur et al. (2020) Thakur, S., Doshi, J., Pati, S., Rathore, S., Sako, C., Bilello, M., Ha, S., Shukla, G., Flanders, A., Kotrotsou, A., Milchenko, M., Liem, S., Alexander, G., Lombardo, J., Palmer, J., LaMontagne, P., Nazeri, A., Talbar, S., Kulkarni, U., Bakas, S., 2020\.  Brain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training.  NeuroImage 220, 117081\.  doi:10.1016/j.neuroimage.2020.117081.
  * Tousignant et al. (2019) Tousignant, A., Lemaître, P., Precup, D., Arnold, D.L., Arbel, T., 2019.  Prediction of disease progression in multiple sclerosis patients using deep learning analysis of MRI data, in: Proceedings of The 2nd International Conference on Medical Imaging with Deep Learning, pp. 483–492.
  * Tzeng et al. (2015) Tzeng, E., Hoffman, J., Darrell, T., Saenko, K., 2015\.  Simultaneous deep transfer across domains and tasks, in: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4068–4076.  doi:10.1109/ICCV.2015.463.
  * Valverde et al. (2019) Valverde, S., Salem, M., Cabezas, M., Pareto, D., Vilanova, J.C., Ramió-Torrentà, L., Àlex Rovira, Salvi, J., Oliver, A., Lladó, X., 2019\.  One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks.  NeuroImage: Clinical 21, 101638\.  doi:https://doi.org/10.1016/j.nicl.2018.101638.
  * Van Essen et al. (2013) Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E., Yacoub, E., Ugurbil, K., 2013\.  The WU-Minn Human Connectome Project: An overview.  NeuroImage 80, 62–79.  doi:https://doi.org/10.1016/j.neuroimage.2013.05.041.
  * Vaze et al. (2020) Vaze, S., Xie, W., Namburete, A.I.L., 2020.  Low-memory CNNs enabling real-time ultrasound segmentation towards mobile deployment.  IEEE Journal of Biomedical and Health Informatics 24, 1059–1069.  doi:10.1109/JBHI.2019.2961264.
  * Venturini et al. (2020) Venturini, L., Papageorghiou, A.T., Noble, J.A., Namburete, A.I.L., 2020\.  Uncertainty estimates as data selection criteria to boost omni-supervised learning, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, pp. 689–698.  doi:10.1007/978-3-030-59710-867.
  * Verma et al. (2020) Verma, S., Dickerson, J., Hines, K., 2020.  Counterfactual explanations for machine learning: A review .
  * Vidanage et al. (2020) Vidanage, A., Christen, P., Ranbaduge, T., Schnell, R., 2020\.  A graph matching attack on privacy-preserving record linkage, in: Proceedings of the 29th ACM International Conference on Information and Knowledge Management, p. 1485–1494.  doi:10.1145/3340531.3411931.
  * Visser et al. (2019) Visser, M., Müller, D., Duijn, R., Smits, M., Verburg, N., Hendriks, E., Nabuurs, R., Bot, J., Eijgelaar, R., Witte, M., Herk, M., Barkhof, F., De Witt Hamer, P., De Munck, J., 2019\.  Inter-rater agreement in glioma segmentations on longitudinal MRI.  NeuroImage: Clinical 22, 101727\.  doi:10.1016/j.nicl.2019.101727.
  * Wachinger et al. (2018) Wachinger, C., Reuter, M.R., Klein, T., 2018.  DeepNAT: Deep convolutional neural network for segmenting neuroanatomy.  NeuroImage 170, 434–445.  doi:doi.org/10.1016/j.neuroimage.2017.02.035.
  * Wang et al. (2019) Wang, G., Li, W., Aertsen, M., Deprest, J., Ourselin, S., Vercauteren, T., 2019.  Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks.  Neurocomputing 338, 34–45.  doi:https://doi.org/10.1016/j.neucom.2019.01.103.
  * Wang et al. (2020) Wang, L., Guo, D., Wang, G., Zhang, S., 2020.  Annotation-efficient learning for medical image segmentation based on noisy pseudo labels and adversarial learning.  IEEE Transactions on Medical Imaging , 1–1doi:10.1109/tmi.2020.3047807.
  * Wang et al. (2018) Wang, X., Girshick, R., Gupta, A., He, K., 2018\.  Non-local neural networks, in: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7794–7803.  doi:10.1109/CVPR.2018.00813.
  * Williams and Wyatt (2015) Williams, D.R., Wyatt, R., 2015\.  Racial Bias in Health Care and Health: Challenges and Opportunities.  JAMA 314, 555–556.  doi:10.1001/jama.2015.9260.
  * Winkler et al. (2019) Winkler, J.K., Fink, C., Toberer, F., Enk, A., Deinlein, T., Hofmann-Wellenhof, R., Thomas, L., Lallas, A., Blum, A., Stolz, W., Haenssle, H.A., 2019.  Association Between Surgical Skin Markings in Dermoscopic Images and Diagnostic Performance of a Deep Learning Convolutional Neural Network for Melanoma Recognition.  JAMA Dermatology 155, 1135–1141.  doi:10.1001/jamadermatol.2019.1735.
  * Wong et al. (2015) Wong, W., Laveist, T., Sharfstein, J.M., 2015.  Achieving health equity by design.  JAMA 313.  doi:10.1001/jama.2015.2434.
  * Wu et al. (2020) Wu, W., Lu, Y., Mane, R., Guan, C., 2020.  Deep learning for neuroimaging segmentation with a novel data augmentation strategy, in: 2020 42nd Annual International Conference of the IEEE Engineering in Medicine Biology Society (EMBC), pp. 1516–1519.  doi:10.1109/EMBC44109.2020.9176537.
  * Yang et al. (2017) Yang, L., Zhang, Y., Chen, J., Zhang, S., Chen, D.Z., 2017\.  Suggestive annotation: A deep active learning framework for biomedical image segmentation, in: Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, pp. 399–407.
  * Yang et al. (2019) Yang, Q., Liu, Y., Chen, T., Tong, Y., 2019.  Federated machine learning: Concept and applications.  ACM Trans. Intell. Syst. Technol. 10\.  doi:10.1145/3298981.
  * Yosinski et al. (2014) Yosinski, J., Clune, J., Bengio, Y., Lipson, H., 2014\.  How transferable are features in deep neural networks?, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc.
  * Young et al. (2018) Young, A., Marinescu, R., Oxtoby, N., Bocchetta, M., Yong, K., Firth, N., Cash, D., Thomas, D., Moore, K., Cardoso, M.J., Swieten, J., Borroni, B., Galimberti, D., Masellis, M., Tartaglia, M., Rowe, J., Graff, C., Tagliavini, F., Frisoni, G., Alexander, D., 2018\.  Uncovering the heterogeneity and temporal complexity of neurodegenerative diseases with subtype and stage inference.  Nature Communications 9.  doi:10.1038/s41467-018-05892-0.
  * Yu et al. (2018) Yu, M., Linn, K., Cook, P., Phillips, M., McInnis, M., Fava, M., Trivedi, M., Weissman, M., Shinohara, R., Sheline, Y., 2018.  Statistical harmonization corrects site effects in functional connectivity measurements from multisite fMRI data.  Human Brain Mapping 39.  doi:10.1002/hbm.24241.