# DeepLesionBrain: Towards a broader deep-learning generalization for multiple
sclerosis lesion segmentation

Reda Abdellah Kamraoui
Univ. Bordeaux, Bordeaux INP,
CNRS, LaBRI, UMR5800, PICTURA,
F-33400 Talence, France
&Vinh-Thong Ta
Univ. Bordeaux, Bordeaux INP,
CNRS, LaBRI, UMR5800, PICTURA,
F-33400 Talence, France
&Thomas Tourdias
Service de Neuroimagerie Diagnostique et Thérapeutique,
Univ. Bordeaux, INSERM, Neurocentre Magendie,
U1215, F-3300 Bordeaux, France
&Boris Mansencal
Univ. Bordeaux, Bordeaux INP,
CNRS, LaBRI, UMR5800, PICTURA,
F-33400 Talence, France
&José V Manjon
ITACA, Universitat Politècnica de València,
46022 Valencia, Spain
&Pierrick Coupé
Univ. Bordeaux, Bordeaux INP,
CNRS, LaBRI, UMR5800, PICTURA,
F-33400 Talence, France

###### Abstract

Recently, segmentation methods based on Convolutional Neural Networks (CNNs)
showed promising performance in automatic Multiple Sclerosis (MS) lesions
segmentation. These techniques have even outperformed human experts in
controlled evaluation conditions such as Longitudinal MS Lesion Segmentation
Challenge (ISBI Challenge). However state-of-the-art approaches trained to
perform well on highly-controlled datasets fail to generalize on clinical data
from unseen datasets. Instead of proposing another improvement of the
segmentation accuracy, we propose a novel method robust to domain shift and
performing well on unseen datasets, called DeepLesionBrain (DLB). This
generalization property results from three main contributions. First, DLB is
based on a large group of compact 3D CNNs. This spatially distributed strategy
ensures a robust prediction despite the risk of generalization failure of some
individual networks. Second, DLB includes a new image quality data
augmentation to reduce dependency to training data specificity (e.g.,
acquisition protocol). Finally, to learn a more generalizable representation
of MS lesions, we propose a hierarchical specialization learning (HSL). HSL is
performed by pre-training a generic network over the whole brain, before using
its weights as initialization to locally specialized networks. By this end,
DLB learns both generic features extracted at global image level and specific
features extracted at local image level. DLB generalization was validated in
cross-dataset experiments on MSSEG’16, ISBI challenge, and in-house datasets.
During experiments, DLB showed higher segmentation accuracy, better
segmentation consistency and greater generalization performance compared to
state-of-the-art methods. Therefore, DLB offers a robust framework well-suited
for clinical practice.

_K_ eywords Multiple Sclerosis Segmentation $\cdot$ Neuroimaging $\cdot$ Deep
Learning $\cdot$ Domain Generalization

##  1 Introduction

In recent years, the medical imaging community has witnessed a rapid increase
in image processing methods based on Deep Learning (DL). These novel
techniques came with remarkable performance in many tasks including Multiple
Sclerosis (MS) lesion segmentation. Some automated algorithms have even
reached human level performance in controlled evaluation conditions (see
[Carass et al., 2017]). Unlike over-controlled conditions where most DL
approaches have been validated, real world data exhibit high diversity.
Consequently, clinical use of MS lesion segmentation based on DL is still
limited mainly because of their poor generalization on new data coming from
medical sites that have not been covered during training (unseen domains).
This lack of generalization of DL methods can result from several factors such
as the selected solution during optimization, the diversity of the training
dataset or the genericity of the extracted features.

DL is based on the assumption that training and test data are independent but
come from the same distribution. This assumption is usually not respected in
medical imaging data especially for Magnetic Resonance Imaging (MRI) where
acquisition protocols, MRI scanner, patient populations, and software
processing may vary depending on the clinical center or the cohort. As a
result of these differences of data distribution (covariate shift), a decrease
in performance is observed between the training data (source domain) and the
test data derived from different distributions (target domain). This is known
as the domain shift.

An intuitive method to reduce this problem is to train on a wider and more
heterogeneous dataset (as shown by [Mårtensson et al., 2020]). However this
requires a large dataset annotated by experts which is rarely available and
tedious to produce. Some deal with this phenomenon by applying extensive data
augmentation (such as [Zhang et al., 2020b]). Others use few available labeled
images from the target domain in order to reduce the covariate shift, such as
few-shot and one-shot learning strategies (see [Snell et al., 2017, Valverde
et al., 2019]).

Besides, DL requires the tuning of a large number of parameters relatively to
the number of training data samples. Thus, it usually ends up converging to
one of the many possible local minima as opposed to the theoretical best
parameter configuration which leads to the global minimum. Consequently, the
generalization ability of the model depends on the selected solution. The
selection of the best generalizing local minimum is still an open question. On
one hand, some works have proposed to select it using the local
characteristics (e.g., flatness) of the loss function (see [Keskar et al.,
2016, Wu et al., 2017] ). On the other hand, an alternative strategy consists
in combining several local minima to improve the generalization capability of
the method. This can be done by averaging several local minima of one model
(e.g., snapshot ensemble [Huang et al., 2017]) or by combining outputs of
different models trained independently (e.g., classical ensemble [Zhang et
al., 2019] and spatially distributed networks [Coupé et al., 2020, Huo et al.,
2019]).

Unlike classical methods that use hand crafted features, Convolutional Neural
Networks (CNNs) automatically extract the most suitable set of features
automatically for a particular task. Although this strategy is very efficient
to extract relevant features for a particular source domain, this set of
features may not generalize well for the target domain. Some works proposed to
learn invariant features that coexist across different source domains [Motiian
et al., 2017, Muandet et al., 2013, Yang and Gao, 2013]. They tried to apply a
regularization in order to learn an abstract representation of the specific
computer vision task (i.e., just like humans understand high level concepts).
Indeed, the extraction of generalizing features lies between the freedom of
the optimization process to find the optimal combination from data, and the
constraints used for minimizing domain bias.

The successful deployment of DL based methods for MS lesion segmentation
requires generalization capabilities that can guarantee high performance for
unseen domains. First, such methods should ensure the convergence of the DL
model to generalizing minima. Second, the training process should anticipate
the reduction of the covariate shift. Moreover, the method should be enforced
to learn MS lesion generalizing features from the source domain, in order to
effectively delineate lesions despite the target domain distribution. Finally,
this solution should not require additional annotation in case of processing
unseen domains.

Recently, many works have been proposed for MS lesion segmentation using CNNs.
First, [Brosch et al., 2016] proposed a deep 3D encoder-decoder network, with
joint training of the encoder and the decoder. The authors used shortcut
connections between the two interconnected pathways for integrating high and
low level features. This pioneer work demonstrated the high potential of deep
learning for MS lesion segmentation.
[Valverde et al., 2017] proposed a cascade of two patch-wise 3D CNNs, composed
of a first sensitive network to reveal possible lesion candidates followed by
a second network to reduce misclassified voxels. This cascade allows refined
segmentation but it uses a small receptive field that prevents capturing the
global context. Later, the authors [Valverde et al., 2019] improved their
method by proposing an one-shot domain adaptation model which uses transfer
learning and partial fine-tuning. However, this domain adaptation needs a
labeled example from the new domain. Moreover, such strategies lead to
different versions of the method after each adaptation, this results in
discrepancies in the segmentation.
[Hashemi et al., 2018] considered the problem of data imbalance (i.e., the
under-sampling of the lesion class) by using an asymmetric similarity loss
function based on Tversky index to train a 3D CNN that performed better than
Dice or cross entropy measures. This result suggests that further work should
be done on choosing adequate loss function. Although the proposed loss can be
tuned for the optimal trade-off between precision and recall in a particular
domain, the generalization to unseen domains have to be proven.
[Zhang et al., 2019] used a fully convolutional densely connected network for
MS lesion segmentation. They stacked adjacent 2D slices of different
modalities with a channel-wise concatenation, before forwarding this stack
through a 2D CNN. The final segmentation is based on a majority vote along
different orientations. While this method showed competitive performance on a
well-controlled challenge, the stacking using only the two directly adjacent
slices gives a weak insight on the 3D nature of the data. Moreover, 2D
features may not be considered as generalizing features when processing 3D
volumes and can result in the limited generalization of the method.
[Aslani et al., 2019] proposed an end-to-end encoder-decoder 2D network with
multiple downsampling branches, one for each input modality, and a decoder
part where features from the different modalities are put together at multiple
scales. This separation in encoder branches enables the model to encode
information efficiently from each modality, before combining them in a later
stage. However this 2D approach does not combine features based on axial,
coronal, and sagittal orientations that may greatly reduce its generalization
on 3D images.
[Feng et al., 2019] considered MRI modality unavailability during segmentation
by introducing sequence dropout. This is an important point since the
availability of all the modalities is not always ensured between datasets that
can greatly reduce the generalization capacity of a method. This framework
randomly drops specific MRI sequences during training, with the intent to
learn the intrinsic information of each sequence. This technique showed it can
produce acceptable segmentation even in the absence of one or two modalities.
Nonetheless, it is less efficient than other state-of-the-art methods when all
modalities are available (will be detailed in section 3.3).
[Aslani et al., 2020] tackled the problem of generalization to new domains by
integrating a regularization network to the traditional encoder-decoder
network. The regularizer penalizes the network when the later learns features
which allow the prediction of MRI scanning sites. However, [Li et al., 2018]
have argued that such strategies suffer from overfitting, the obtained
representation could well generalize for all the source domains but poorly for
the unknown target domains.

All the cited MS methods ([Brosch et al., 2016, Valverde et al., 2017, Feng et
al., 2019, Hashemi et al., 2018, Zhang et al., 2019, Aslani et al., 2019, Feng
et al., 2019]) focused on obtaining accurate segmentation within a same domain
evaluation. However, the use of out-of-domain datasets is essential to ensure
a good evaluation of the generalization capabilities of a method. This
question is right now a hot topic (see [Mårtensson et al., 2020], and [Bron et
al., 2020]) and an important recommendation from the clinical world (see
[Omoumi et al., 2021]). Experiments using training and testing images derived
from the same domain are known to be biased [[Omoumi et al., 2021]] and do not
ensure generalization. Therefore, a model used in clinical conditions should
produce accurate segmentation for new domain images without the need of
retraining with expert segmentation on the new domain.

In this paper we propose DeepLesionBrain (DLB), a novel method for MS lesion
segmentation robust to domain shift, validated on out-of-domain testing
(cross-dataset testing).
First, we use a large group of compact 3D CNNs spatially distributed over the
brain with overlapping receptive fields between regions. By associating a
distinct network with each region of the brain, the spatially distributed
networks (see [Coupé et al., 2020, Huo et al., 2019]) strategy simplifies the
MS lesion segmentation from a single complex task on the whole brain to
multiple simpler sub-tasks on each region. Moreover, the overlapping regions
ensure consistent and stable consensus. Studies highlighted that such
consensus-based strategy helps the convergence towards generalizing minima
(see [Breiman, 2001, Izmailov et al., 2018, Olson et al., 2018]). Moreover,
applied to brain segmentation, this strategy demonstrated good generalization
on unseen domains (e.g., child’s brain or patients with Alzheimer’s disease)
when trained on healthy adult brains [[Coupé et al., 2020]].
Second, DLB is trained with a novel Image Quality Data Augmentation (IQDA)
method, which mimics real world data diversity by adding realistic alterations
to the training images. As shown in the works of [Zhang et al., 2020a], such a
type of regularization technique aims to reduce covariate shift. IQDA proposes
specific augmentations that constrain task learning to be independent from
source data acquisition resolution, data contrast, or data quality.
Consequently, the proposed augmentation strategy enables domain shift
robustness.
Third, to ensure good features generalization, we focus on feature learning
strategy. We consider that a generalizing model should learn two types of
features: first global and generic features, and second local and specific
features. Therefore, we propose Hierarchical Specialization Learning (HSL) to
efficiently extract those features in two steps. In the first step, a single
network (the generic network) is trained on all brain regions. In the second
step, each network of the spatially distributed networks is initialised with
the generic network weights and specialized for a specific region of the
brain.
Finally, we propose a method using only two modalities (T1w and FLAIR) to
ensure its compatibility with a large number of datasets. Most of the methods
([Feng et al., 2019, Brosch et al., 2016, Valverde et al., 2017, Zhang et al.,
2019]) optimize their segmentation using T1w, FLAIR, PD, and T2 modalities.
However in clinical conditions, not all these sequences are always available.
Therefore we focused our work on developing a robust approach using only two
modalities.

##  2 Method and Material

Figure 1: The two steps training process of DeepLesionBrain (see Sect. 2.1.3
for more details)

###  2.1 Method Overview

####  2.1.1 Spatially Distributed Networks Strategy

DeepLesionBrain (DLB) uses a group of compact networks, each network is
specialized in a particular region of the brain, and processes a sub-volume of
the global volume. The receptive fields of the neighboring networks overlap
with one another, and the final segmentation of the whole volume is obtained
with a majority vote of the local predictions. Employing our spatially
distributed compact networks is equivalent to a big network with more filters
and a higher receptive field (see Fig. 1). This particular configuration with
overlapping receptive fields also permits domain generalization and robustness
to failure of an individual network. Indeed, employing a group of spatially
distributed networks collectively ensures a stable segmentation. The majority
vote on overlapped regions produces a consensus more robust to domain shift
even if the prediction of some networks taken separately may be affected by
covariate shift.

####  2.1.2 Image Quality Data Augmentation (IQDA)

The quality of the MRI greatly varies between datasets. In fact, the quality
of the images depends on several factors such as signal to noise ratio,
contrast to noise ratio, resolution or slice thickness.
To address this issue, we propose a data augmentation strategy which considers
image quality disparity. During training, we simulate “on the fly” altered
versions of 3D patches. We randomly introduce at each iteration either blur,
edge enhancement, or axial subsampling distortion (2D FLAIR are usually
acquired along the axial direction). For the blur, a gaussian kernel is used
with a randomly selected standard deviation ranging between $[0.5,1.75]$. For
edge enhancement, we use unsharp masking with the inverse of the blur filter.
For axial subsampling distortion, we simulate acquisition artifacts that can
result from the varying slice thickness. We use a uniform filter (a.k.a mean
filter) on the axial direction with a size of $[1\times 1\times sz]$ where
$sz\in{2,3,4}$. Ground truth is kept the same as the original version. This
process reduces the domain bias when learning to extract relevant features
caused by data variability.

####  2.1.3 Hierarchical Specialization Learning (HSL)

In order to achieve generalization for MS lesion segmentation, we aim to
extract generalizing lesion features. These features can be grouped into two
categories. First, some lesion characteristics are considered generic and
shared among lesion types. Such features are independent from lesion
localization. They have a common and inherent significance at the global scale
of the brain volume, we will refer to them in this paper as “generic global
features”. Second, other relevant features for MS lesions depend on brain
structure and some distinct regions (see [Filippi et al., 2019]). In this work
we refer to these features as “specialized local features”.

On the one hand, training each specialized network on a specific sub-region of
the brain (see Fig. 1 right) would prevent the efficient learning of “generic
global features”, since each specialized member of our group would be trained
on a particular region of the whole brain. On the other hand, using a single
3D CNN to learn "specialized local features" over the whole brain volume would
require a large model which may not fit into memory and a large dataset to
train it.

To overcome this limitation and in order to learn generalizing features, we
propose a novel Hierarchical Specialization Learning (HSL). This two step
learning process shown in Fig. 1 is equivalent to a cascade of networks with
the first layer of the cascade focusing on "generic global features", and the
second one on "specialized local features". First, the “generic network” is
trained with data samples from all over brain regions to learn general
knowledge about lesions by extracting “generic global features”. Second, each
network in the spatially distributed strategy is specialized over a specific
sub-volume of the brain.

The generic network is used as an initialization for each network of our
spatially distributed networks, by transferring the generic network weights to
each individual specialized network. The knowledge gained from this transfer
learning transmits the ability to extract “generic global features”, while the
specialized network training will specialize them in extracting local
“specialized local features”.

In our ablation study, we will show that this hierarchical specialization
learning of the specialized networks performs better than training a single
network over the whole brain, or training the specialized networks without
HSL.

####  2.1.4 Selection of the Required Modalities

In order to use a trained model for MS lesion segmentation with optimal
performance, this usually requires to use the same set of modalities that have
been chosen during training. DLB proposes a method that needs only T1w and
FLAIR sequences in order to be compatible with all benchmark MS datasets and
most already available MS patients data.

Our method is built with the purpose to generalize on unseen datasets, thus it
uses the minimum necessary modalities. Indeed, increasing the number of
sequences requires longer scan acquisition time. Besides, it needs more
complex processing which may be prone to error, such as multimodal image
registration. Furthermore, the use of more sequences during the training on a
dataset may reduce the generalization to other image domains.

In addition to the wide use of T1w and FLAIR for MS diagnosis, the choice of
these modalities have also been motivated by the fact that FLAIR is the most
relevant sequence for revealing most of MS lesions (see [Narayana et al.,
2020]), while T1w can provide complementary structural information needed for
accurate segmentation.

Figure 2: Illustration of the considered U-Net architecture. The number of
input channels (NC) depends on the modality number (i.e., NC= 2, for using T1w
and FLAIR ). Each block is composed of group normalization (GN), Convolution
(Conv) and Rectified Linear Unit (ReLU) activation.

###  2.2 Implementation Details

The network architecture used in our spatially distributed strategy is based
on 3D U-Net composed of a downsampling part and an upsampling one, linked with
one another by skip connections at the multiple scales. This 3D CNN
architecture, shown in Figure 2, has been used for all the networks in our
approach. Dropout with 0.5 rate is used after max-pooling layers to prevent
the overfitting of our model to the training data. Due to GPU memory
constraints, we trained with a batch size of 1, and so we used Group
Normalization (GN) [Wu and He, 2018] with 8 groups before each convolution. We
have chosen Rectified Linear Units (ReLu) to introduce non linearity after
convolution layers.

DLB is optimized with Adam [Kingma and Ba, 2014] using a learning rate of
0.0001 and a momentum of 0.9. The experiments have been performed with Keras
2.2.4 [Chollet et al., 2015] and Tensorflow 1.12.0 [Abadi et al., 2016] on
NVIDIA Titan Xp 12 GB GPU.

###  2.3 Method Description

To obtain sub-volumes for each image, we first divide our whole MRI registered
into the MNI space into multiple overlapping 3D patches. We perform a cropping
operation over the whole image using a sliding window of the size
$(Px,Py,Pz)$, and a stride of $(Sx,Sy,Sz)$. We take $Sx<Px$, $Sy<Py$, $Sz<Pz$
to ensure the overlapping.

Sub-volumes from different images, that represent the same receptive field
into the MNI space (the same sub-volume region of the whole volume), are
grouped together. They are used for training a network specialized for that
particular region.

In this work, we explored many combinations of sub-volume sizes and numbers.
We chose a configuration with 125 sub-volumes by taking experimentally
$Px=Py=Pz=96$, $Sx=Sy=76$, and $Sz=67$ as a good trade-off between the overall
performance and computation resources.

####  2.3.1 Symmetrical Training

To limit redundant training and to use the most possible data for a particular
brain region, we choose to train specialized networks only on one hemisphere.
By flipping (mirroring) the sub-volumes of the second hemisphere, it is
possible to train a single specialized network for both sides. Thus, we can
use twice the amount of data for each region while reducing the number of
networks to train to nearly a half (due to sub-volume overlapping, the plane
of symmetry cuts through the median networks which cover equivalent
symmetrical regions from both hemispheres). Consequently, unlike previous
works with spatially distributed networks (i.e., [Coupé et al., 2020]),
instead of using 125 networks we use only 75 specialized networks. We
experimentally verified that using 125 networks or only 75 trained with twice
the number of patches, produced similar segmentation accuracy.

####  2.3.2 Loss Function

MS lesion segmentation task suffers from class imbalance since lesion volume
is considerably low compared to healthy volume. Thus, we use a smooth version
of the Generalized Jaccard Loss (GJL), which considers this issue [Manjon et
al., 2020].

$GJL=1-\frac{\sigma+\sum_{c=1}^{Nc}w_{c}\sum_{i=1}^{N}p_{ci}t_{ci}}{\sigma+\sum_{c=1}^{Nc}w_{c}\left(\sum_{i=1}^{N}(p_{ci}+t_{ci})-\sum_{i=1}^{N}p_{ci}t_{ci}\right)}$
(1)

Where $w_{c}=1/(1+\sum_{i=1}^{N}t_{ci})$, $\sigma$ is the smoothness factor,
$N$ is the number of voxels, $Nc$ is the number of classes, $p_{ci}$ and
$t_{ci}$ are respectively the predicted probability and the ground truth
probability of the vowel $i$ for the class $c$.

During inference, we combine the overlapping predictions in a straightforward
aggregation majority vote technique. The class of each voxel (either lesion or
healthy tissue) is chosen based on the most predicted class among the networks
which cover that voxel.

###  2.4 Datasets

To assess the robustness of a model, it is crucial to test its ability to
generalize on unseen domains. Therefore, DLB has been trained and validated
using different datasets to assess its domain generalization ability (see 3.2
). These datasets exhibit high heterogeneity in terms of resolution, data
processing, acquisition sites, delineation protocols, and they also cover a
large variety of clinical scenarios.

####  2.4.1 ISBI Longitudinal Multiple Sclerosis Lesion

The ISBI dataset [Carass et al., 2017] consists of five subjects for training,
fourteen subjects for testing, with a mean of 4.4 time-points per subject (21
images for training and 61 images for testing). Two human expert raters
delineated MS lesions, from the four available modalities acquired on 3.0
Tesla MRI scanner: 3D MPRAGE $T_{1}-$weighted (T1w) of $0.82\times 0.82\times
1.17$ $mm^{3}$ voxel size, 2D $T_{2}-$weighted (T2), 2D $T_{2}-$weighted fluid
attenuated inversion recovery (FLAIR), and 2D Proton Density weighted (PD), of
$0.82\times 0.82\times 2.2$ $mm^{3}$ voxel size each.

For the training, we used the ISBI training dataset with available annotations
from the two experts. For test and evaluation, we segmented the test data with
no available expert annotation, and submitted our results to the ISBI
challenge website111https://smart-stats-tools.org/lesion-challenge-upload-
results. The ISBI pipeline already included preprossessing. Each first time-
point T1w was inhomogeneity-corrected using N4 [[Tustison et al., 2010]],
skull-stripped [Carass et al., 2007], dura stripped [Shiee et al., 2014],
followed by a second N4 inhomogeneity correction, and rigid registration to a
1 $mm^{3}$ isotropic MNI template. Then, this image was used as a target for
the remaining T1w time-points and all modalities for the same subject. These
images were N4 corrected and then rigidly registered to the T1w in the MNI
space. The skull and dura-stripped mask from the target T1w was applied, which
were then N4 corrected again. We added an intensity normalization step using
kernel density estimation for all images.

####  2.4.2 MICCAI2016 MS Challenge Dataset

The MSSEG’16 training dataset [Commowick et al., 2016] contains 15 patients
from 3 different clinical sites. Five modalities are available for each
patient: 3D FLAIR, 3D T1w, 3D T1w GADO, 2D PD, and 2D T2. The images were
acquired on 1.5T and 3T MRI scanners with multiple resolutions:

3D FLAIR modalities ranging from $1\times 0.5\times 0.5$ to $1.25\times
1.04\times 1.04$ $mm^{3}$, and 3D T1w sequences between $0.85\times 0.74\times
0.74$ and $1.08\times 1.08\times 0.9$ $mm^{3}$.

Seven human experts have manually segmented the multiple sclerosis lesions.
Each patient modalities have been preprocessed with the same pipeline. First,
each sequence was denoised using the non local means algorithm [Coupé et al.,
2008]. Second, a rigid registration of each modality on the FLAIR was
performed [Commowick et al., 2012]. Then, skull stripping of T1w was performed
using the volBrain platform [Manjón and Coupé, 2016], the same mask is applied
to other modalities. Finally, bias field correction was applied using the N4
algorithm [Tustison et al., 2010]. In addition to these steps that have been
performed on the available images, each modality was registered to the MNI
space for our experiments. Similarly to the ISBI images, we used kernel
density estimation for the normalization step.

####  2.4.3 In-house Dataset

For further evaluation of our approach, we used an In-house 3D MRI dataset,
with 3D T1w and 3D FLAIR modalities [Coupé et al., 2018]. This dataset
contains 43 subjects diagnosed with MS. The images were acquired with
different scanners and multiple resolutions ($0.6\times 0.6\times 0.65$
$mm^{3}$, $0.5\times 0.5\times 0.9$ $mm^{3}$, and $1\times 1\times 1$
$mm^{3}$).

The dataset lesion masks have been obtained by human experts manual
delineation. The images were pre-processed using the lesionBrain pipeline from
the volBrain platform [[Manjón and Coupé, 2016]]. First, it included denoising
of each modality [[Coupé et al., 2008]]. Second, an affine registration to MNI
space was performed on the T1w , then the FLAIR was registered to the
transformed T1w. Skull stripping and bias correction have been performed on
the modalities, followed by a second denoising. Finally, the intensities have
been normalized.

Table 1: Description of datasets used in this work. | FLAIR resolution | Site | # Subjects | # Raters | Modalities
---|---|---|---|---|---
ISBI train-set | 2D | Mono | 5, multiple time-points | 2 | T1w, FLAIR, T2, PD
MSSEG’16 | 3D | Multi | 15 | 7 | T1w, T1w GADO, FLAIR, PD, T2
In-house | 3D | Multi | 43 | 1 | T1w, FLAIR

####  2.4.4 Datasets Summary

Table 1 summarizes the main differences between the 3 datasets. We focused
specifically on the resolution of FLAIR due to its known relevance in MS
lesion segmentation. To summarize, ISBI train-set contains multiple time
points of only five subjects, acquired in a single clinical site with two
human expert segmentations. Except 3D MPRAGE T1w, the other three modalities
are in 2D. MSSEG’16 dataset is a multi-site database comprising 15 patients,
with seven human segmentations. This dataset contains 5 available modalities
with 3D FLAIR. Finally, In-house dataset is the largest dataset with 43
patients, and multi-site 3D modalities, segmented by a single human rater and
validated by a second one.

Figure 3: FLAIR examples from the considered three datasets in the MNI space
and after intensity normalization. From left to right, the two images are from
ISBI, MSSEG’16, our in-house data, respectively.

Figure 3 shows examples from the three presented datasets, each image
represents a sagittal section of the FLAIR modality in the MNI space after
intensity normalization. The two images on the left are examples from the ISBI
dataset. We notice blurring effects which makes it hard to distinguish
precisely brain structures. This blur comes from 2D low resolution
acquisitions. In the middle, the two examples come from the MSSEG’16 dataset.
These 3D FLAIR are noticeably of higher resolution than the other images.
Therefore, lesion boundaries are more easily delineated and main structures
are clearly apparent. The final two images on the right are from our In-house
3D dataset. The 3D resolution enables the differentiation of white matter,
gray matter, and shows the lesions clearly.

In terms of FLAIR images, we notice that both MSSEG’16 and In-house dataset
(3D FLAIR) propose better visual quality than ISBI dataset (2D FLAIR).

###  2.5 Validation Framework

####  2.5.1 Evaluation Metrics

The assessment of a segmentation method is usually measured by a similarity
metric between the predicted segmentation and the human expert ground truth.

First, we use several complementary metrics in order to assess segmentation
performance. Namely, we use the Dice similarity coefficient, the Positive
Predictive Value (PPV or the precision), true positive rate (TPR, known as
recall or Sensitivity), and Pearson’s correlation coefficient (CORR).

$PPV=\frac{TP}{TP+FP}\,\mathrm{,}\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,TPR=\frac{TP}{TP+FN}\,\mathrm{,}$
(2)

$Dice=\frac{2\times TP}{(TP+FN)+(TP+FP)}\,\mathrm{,}$ (3)

where TP, FN, FP represent respectively true positives, false negatives, and
false positives.

Second, recent works (i.e., [Commowick et al., 2018]) question the relevance
of classic metrics (Dice) compared to detection metrics, which are used for MS
diagnostic and clinical evaluation of the patient evolution. Thus, in addition
to the voxel-wise metrics, we also use lesion-wise metrics that focus on the
lesion count. Such as, Lesion False Positive Rate (LFPR) and Lesion True
Positive Rate (LTPR).

$LTPR=\frac{LTP}{LTP+LFN}\,\mathrm{,}\quad\quad
LFPR=\frac{LFP}{LTP+LTN}\,\mathrm{,}$ (4)

where LTP, LFN, LFP represent respectively lesion true positives, lesion false
negatives, and lesion false positives.

ground truth volumes.

Moreover, [García-Lorenzo et al., 2013] pointed out that even though Dice is
commonly used and simplifies method comparison, multiple complementary metrics
are needed to provide a better understanding of the performance. Recently,
international challenges took into consideration several metrics ([Carass et
al., 2017] and [Commowick et al., 2018]). Consequently, we decided to evaluate
our methods using Hybrid score proposed by [Carass et al., 2017]. This metric
combines voxel-wise segmentation, lesion-wise detection, and volumetric
metrics. It is defined as:

$Hybrid=\frac{Dice}{8}+\frac{PPV}{8}+\frac{(1-LFPR)}{4}+\frac{LTPR}{4}+\frac{CORR}{4}$
(5)

Finally, we also use the ISBI Submission Score for the evaluation of ISBI
test-set segmentations. [Carass et al., 2017] defined it as the average of the
hybrid scores of all image examples with the different human raters and with
inter rater variability taken into consideration. This score is computed after
submitting the segmentation to ISBI’s challenge website 222https://smart-
stats-tools.org/lesion-challenge. Obtaining an ISBI score of 90 or higher with
a segmentation technique indicates that this method is similar to the human
raters.

####  2.5.2 Cross-dataset Experiments

In our context, the domain is characterised by acquisition hardware, data
processing, and annotation protocol. Thus, source and target domains are
related but slightly different, restricting the applicability of traditional
supervised learning models which rely on the assumption that train and
inference data come from the same distribution .

Both domain generalization and adaptation are concerned with reducing dataset
bias. The difference between these strategies is that for domain adaptation,
some unlabeled data or even a few labeled data from the target domain are
exploited to capture properties of the target domain for model adaptation.
However, in domain generalization no samples of any kind are used from the
target domain.

Domain generalization has been proposed to address the problem of
unavailability of target domain samples by leveraging the labeled data to
learn a universal representation in order to generalize for any target domain
and without any prior insight from that domain.

In this work, we emphasize on testing the domain generalization of our
approach with cross-dataset evaluation. Unlike domain adaptation such as one-
shot domain adaptation [Valverde et al., 2019], DLB does not need expert
segmentation from the target domain. Our testing conditions draw a clear
distinction between training data containing source domains and testing data
containing unseen target domain.

####  2.5.3 Reference Methods

During experiments, our method was compared to three publicly available state-
of-the-art approaches. We performed training and validation for all three
compared methods, in the same conditions regarding datasets and preprocessing.
The reference methods are nicMSlesion by [Valverde et al., 2019], DeepMedic by
[Kamnitsas et al., 2017], and 2.5D Tiramisu by [Zhang et al., 2019]. These
methods have been selected for the availability of the authors source code and
the relevance of their contributions in the MS segmentation community.

nicMSlesion: This method is based on a cascade of two 3D patch-wise CNNs. The
first one is trained to be sensitive in order to reveal lesion candidates. The
second one is trained to reduce the misclassified voxels from the first
network. Training is performed on $11\times 11\times 11$ patches randomly
augmented with flipping and rotations. Therefore, nicMSlesion involves
classical data augmentation. In the first network, the negative class is
under-sampled to the same number of existing lesion voxels. It is composed of
patches extracted from all of the available lesion voxels and a random
selection of normal appearing tissue voxels. Afterwards, an evaluation of the
first CNN model is computed by performing inferences on the same train-set and
identifying negative voxels that have been misclassified as lesions (False
Positives). Finally, the second model is trained using a balanced set composed
of all the lesion voxels and a random selection from the identified False
Positives in the previous step.

DeepMedic: This method is based on a 11-layers deep dual pathway 3D CNN
designed for brain lesion segmentation. In order to incorporate both local and
larger contextual information, the dual pathway architecture processes the
input images at multiple scales simultaneously. To overcome the computational
burden, the authors use a hybrid dense training scheme processing adjacent
image patches into one pass through the network. To refine the network
segmentation and remove false positives, a 3D fully connected conditional
random field is used. The training includes data augmentation with sagittal
reflections.

2.5D Tiramisu: This method is based on a fully convolutional densely connected
network. The model uses stacked slices from all three anatomical planes to
achieve a 2.5D based method. Individual slices from a given orientation
provide global context along the plane and the stack of adjacent slices adds
local context. The training also includes flipping and rotations of the 2D
patches for data augmentation. Therefore, 2.5D Tiramisu involves classical
data augmentation. For each stack of 2D $128\times 128$ slices composed of a
center slice and its 2 adjacent slices, the model produces the segmentation of
the center slice. Then, the inference results along the different orientations
are combined via majority vote to output the final segmentation.

For both these methods, we use the implementations provided publicly by the
authors (see 333https://github.com/sergivalverde/nicMSlesions and
444https://github.com/MedICL-VU/LesionSeg ).

####  2.5.4 Statistical Test

To assert the advantage of a technique obtaining the highest average score, we
conducted a Wilcoxon test over the lists of hybrid scores measured at image
level (for the consistency of the segmentations section we took the lists of
dice indices between the two segmentations). The significance of the test is
established for a p-value below 0.05. In the following tables, * indicates a
significantly better average score when compared with the rest of the other
approaches.

##  3 Results

###  3.1 Ablation Study

To demonstrate the impact of each proposed contribution on domain
generalization, we measured separately their effects on different metrics. To
show both the effect on accuracy improvement and the domain shift robustness,
we propose out-of-domain and in-domain ablation study. First, we trained each
method configuration on ISBI challenge train-set, then we validated on both
ISBI test-set (see Table 2) and MSSEG’16 (see Table 3). To ensure a fair
comparison, each configuration is trained until convergence. Specifically, we
used an early stopping criterion of 50 epochs (i.e., the training stops if the
loss function does not improve on the validation set during 50 epochs) with a
maximum number of 500 epochs. We verified that none of the configurations
reached this maximum number.

Table 2: Ablation study results with different variants of our approach
trained on ISBI challenge train-set and tested on ISBI test-set.
DeepLesionBrain (DLB) refers to using our spatially distributed specialized
networks, each network in charge of segmenting a sub-volume. The generic
network represents the variant of DLB with a single network (without the
spatially distributed strategy). Hierarchical Specialized Learning (HSL)
indicates that we initialized the “specialized networks” with the “generic
Network”. To evaluate the performance of the proposed Data Augmentation, we
compared variants with IQDA (previously defined in 2.1.2) and without IQDA.
For each metric, the bold values indicate the best result. In hybrid score
column, * indicates a significantly better score than the other approaches
using the Wilcoxon test.

Method | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | Submission Score
---|---|---|---|---|---|---|---
DLB with HSL and IQDA | 0.747* | 0.646 | 0.888 | 0.545 | 0.131 | 0.486 | 92.849
DLB with HSL and without IQDA | 0.732 | 0.677 | 0.849 | 0.603 | 0.192 | 0.489 | 92.383
DLB without HSL and with IQDA | 0.710 | 0.576 | 0.892 | 0.453 | 0.121 | 0.360 | 91.713
DLB with models genesis init. and IQDA | 0.718 | 0.621 | 0.867 | 0.513 | 0.187 | 0.438 | 91.885
DLB with AssemblyNet init. and IQDA | 0.723 | 0.628 | 0.885 | 0.515 | 0.140 | 0.406 | 92.109
The generic network with IQDA | 0.736 | 0.668 | 0.859 | 0.585 | 0.178 | 0.489 | 92.491
The generic network without IQDA | 0.688 | 0.654 | 0.502 | 0.869 | 0.162 | 0.468 | 92.425

Table 2 shows the effect of each contribution to segmentation accuracy, when
trained on ISBI challenge train-set and tested on ISBI test-set. First, the
best performing combination is DLB with HSL and IQDA, it obtained an ISBI
Score of 92.849. Second, both the versions of DLB without IQDA and DLB without
HSL are less accurate. They obtained respectively ISBI scores of 92.383 and
91.713. The later comparison shows the impact of HSL on the accuracy of
segmentations. Moreover, the generic network is less accurate than our
spatially distributed approach used in DLB. The variant of generic Network
with IQDA obtained a score of 92.491, whereas the variant without IQDA
obtained a hybrid score of 92.425. Finally, we compare HSL with other weight
initialization strategies. Specifically, HSL is compared with the neighbor
transfer learning from AssemblyNet proposed by [Coupé et al., 2020] and models
genesis proposed by [Zhou et al., 2021]. Although both variants obtained a
better score compared to DLB without HSL, both initialization strategies gave
a lower score than DLB with HSL and IQDA.

Table 3: Ablation study results with different variants of our approach
trained on ISBI challenge train-set and tested on MSSEG’16 (see caption of
Table 2 for details).

Method | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR
---|---|---|---|---|---|---
DLB with HSL and IQDA | 0.684* | 0.639 | 0.768 | 0.608 | 0.319 | 0.700
DLB with HSL and without IQDA | 0.673 | 0.669 | 0.728 | 0.671 | 0.416 | 0.725
DLB without HSL and with IQDA | 0.648 | 0.562 | 0.806 | 0.489 | 0.320 | 0.629
DLB with models genesis init. and IQDA | 0.623 | 0.593 | 0.737 | 0.576 | 0.436 | 0.665
DLB with AssemblyNet init. and IQDA | 0.610 | 0.541 | 0.708 | 0.537 | 0.466 | 0.705
The generic Network with IQDA | 0.672 | 0.665 | 0.721 | 0.673 | 0.413 | 0.727
The generic network without IQDA | 0.626 | 0.625 | 0.763 | 0.588 | 0.449 | 0.611

Table 3 shows the effect of each contribution to domain shift robustness, when
trained on ISBI challenge train-set and tested on MSSEG’16. First, the most
robust combination is DLB with HSL and IQDA, it obtained a hybrid score of
0.684. Second, both the variants of DLB without IQDA and DLB without HSL are
less accurate. They obtained hybrid scores of 0.673 and 0.648 respectively.
Moreover, the generic network is less robust than our spatially distributed
approach with DLB. The variant of the generic network with IQDA obtained a
score of 0.672, whereas the variant without IQDA obtained a hybrid score of
only 0.626. The later comparison shows the impact of IQDA on robustness even
without the spatially distributed networks. Finally, HSL is compared with
Assembynet [Coupé et al., 2020] and model genesis [Zhou et al., 2021]
initialization strategies. The variants with model genesis and AssemblyNet
initialization methods obtained respectively hybrid scores of 0.623 and
0.6103.

###  3.2 Cross-dataset Testing

In this section, we assess the cross-dataset robustness and generalization
ability of our proposed approach. We chose to compare our method with two
state-of-the-art approaches: nicMSlesion [Valverde et al., 2019], DeepMedic
[Kamnitsas et al., 2017], and 2.5D Tiramisu [Zhang et al., 2019].
During the proposed validation, all the methods have been trained on exactly
the same dataset (i.e., same preprocessing, same number of modalities, etc.)
to ensure a fair comparison of method performance. Although refernece methods
have been originally proposed with a specific number of modalities (i.e.,
Tiramisu 2.5D and nicMSlesion were tested with 4 and 3 modalities
respectively), their implementation is independent of the number of modalities
since all modalities are concatenated and feeded to the CNN.

Besides, their official open source implementations support the usage of only
T1w and FLAIR sequences. Thus, in this evaluation all methods are trained
using only these two modalities. The following cross-dataset testing (cross-
domain testing) consists in training each technique on one dataset at each
time. Afterwards, the obtained models are evaluated on the other datasets
which contain unseen domains. We verified the average inference time per image
for each method on the same machine: 57.353s for nicMSlesion, 17.547s for
DeepMedic, 47.471s for 2.5D Tiramisu, and 38.014s for DLB (this time does not
include preprocessing since similar for all the methods). Unlike using a
single network to segment patches coming from the entire image, DLB uses
multiple networks, one network for each brain area. These networks are loaded
one by one to enable the use of a common GPU hardware solution (e.g., NVIDIA
Titan Xp with 12 GB in our setup). Even though DLB requires sequential loading
of multiple networks on GPU, the inference time over the whole image is
similar to using a single network since network weights loading time is
negligible compared to patch segmentation time.

The ISBI score is returned by the challenge website only for ISBI test-set
evaluation, and thus this metric is not available (NA) for testing on other
datasets.

####  3.2.1 Trained on ISBI

Table 4: Results of the different approaches trained on the ISBI training
dataset, with T1w and FLAIR modalities. For each metric, the bold values
indicate the best result. In hybrid score column, * indicates a significantly
better score than the three other approaches using the Wilcoxon test. Red
values indicate hybrid scores lower than 0.5 or Dice index below 0.25.

Trained on ISBI | Approach | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | CORR | Submission Score
---|---|---|---|---|---|---|---|---|---
MSSEG’16 | nicMSlesion | 0.537 | 0.442 | 0.614 | 0.423 | 0.504 | 0.629 | 0.495 | NA
DeepMedic | 0.510 | 0.476 | 0.542 | 0.560 | 0.829 | 0.850 | 0.509 | NA
2.5D Tiramisu | 0.711 | 0.664 | 0.741 | 0.658 | 0.284 | 0.695 | 0.730 | NA
DLB | 0.684 | 0.639 | 0.768 | 0.608 | 0.319 | 0.700 | 0.650 | NA
In-house dataset | nicMSlesion | 0.419 | 0.204 | 0.727 | 0.129 | 0.309 | 0.361 | 0.158 | NA
DeepMedic | 0.523 | 0.536 | 0.633 | 0.499 | 0.805 | 0.765 | 0.549 | NA
2.5D Tiramisu | 0.654 | 0.545 | 0.871 | 0.410 | 0.204 | 0.476 | 0.635 | NA
DLB | 0.696* | 0.675 | 0.850 | 0.564 | 0.342 | 0.644 | 0.718 | NA

Table 4 shows the results of segmentation when training the different
approaches using T1w and FLAIR modalities, on the ISBI training dataset (2D
resolution FLAIR).
When validating the methods on MSSEG’16, we report that 2.5D Tiramisu obtained
slightly better results (not significantly) than DLB, in terms of hybrid score
whereas nicMSlesion and DeepMedic performed relatively worse with 0.537 and
0.51 respectively.
On our in-house dataset, DLB performed significantly better with a hybrid
score of 0.696 while 2.5D Tiramisu, DeepMedic, and nicMSlesion obtained
respectively 0.654, 0.523, and 0.419. We can notice that nicMSlesion offers
poor cross domain performance on 3D FLAIR when trained with 2D FLAIR dataset.

####  3.2.2 Trained on MSSEG’16

Table 5: Results of the different approaches trained on the MSSEG’16 dataset,
with T1w and FLAIR modalities. For each metric, the bold values indicate the
best result. In hybrid score column, * indicates a significantly better score
than the three other approaches using the Wilcoxon test. Red values indicate
hybrid scores lower than 0.5 or Dice index below 0.25.

Trained on MSSEG’16 | Approach | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | CORR | Submission Score
---|---|---|---|---|---|---|---|---|---
ISBI test-set | nicMSlesion | 0.555 | 0.398 | 0.717 | 0.292 | 0.368 | 0.206 | 0.822 | 87,173
DeepMedic | 0.547 | 0.378 | 0.801 | 0.265 | 0.416 | 0.298 | 0.717 | 87.344
2.5D Tiramisu | 0.462 | 0.165 | 0.937 | 0.096 | 0.075 | 0.160 | 0.212 | 86,686
DLB | 0.618* | 0.535 | 0.697 | 0.471 | 0.353 | 0.373 | 0.835 | 89.043
In-house dataset | nicMSlesion | 0.669 | 0.686 | 0.689 | 0.705 | 0.467 | 0.717 | 0.737 | NA
DeepMedic | 0.597 | 0.645 | 0.647 | 0.670 | 0.721 | 0.811 | 0.650 | NA
2.5D Tiramisu | 0.664 | 0.706 | 0.766 | 0.694 | 0.432 | 0.801 | 0.552 | NA
DLB | 0.697* | 0.746 | 0.681 | 0.847 | 0.478 | 0.754 | 0.799 | NA

Table 5 shows the results of segmentation when training the different
approaches on the MSSEG’16 dataset comprising 3D T1w and 3D FLAIR modalities.
First, we notice that our approach obtained significantly better hybrid scores
for both ISBI test and In-house datasets.
Second, when validating on ISBI, the obtained submission score is 89.043 for
DLB (the closest to human performance), 87.344 for DeepMedic, 87.173 for
nicMSlesion, and 86.686 for 2.5D Tiramisu (the farest from human performance).
In the same conditions, 2.5D Tiramisu obtained the average Dice of 0.165 that
indicates a failure of the method and thus a lack of generalization in this
scenario (when trained on high quality 3D FLAIR and tested on low quality 2D
FLAIR).
Finally for In-house dataset, DLB produced significantly better segmentation
than other methods. DLB obtained a hybrid score of 0.697 while nicMSlesion
obtained 0.669, 2.5D Tiramisu obtained 0.664, and DeepMedic obtained the
lowest score of 0.597.

####  3.2.3 Trained on In-house

Table 6: Results of the different approaches trained on In-house dataset, with
T1w and FLAIR modalities. For each metric, the bold values indicate the best
result. In hybrid score column, * indicates a significantly better score than
the three other approaches, using the Wilcoxon test. Red values indicate
hybrid scores lower than 0.5 or Dice index below 0.25.

Trained on In-house | Approach | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | CORR | Submission Score
---|---|---|---|---|---|---|---|---|---
MSSEG’16 | nicMSlesion | 0.700 | 0.650 | 0.822 | 0.586 | 0.150 | 0.607 | 0.607 | NA
DeepMedic | 0.717 | 0.694 | 0.750 | 0.701 | 0.345 | 0.782 | 0.709 | NA
2.5D Tiramisu | 0.745 | 0.665 | 0.741 | 0.687 | 0.164 | 0.720 | 0.722 | NA
DLB | 0.741 | 0.719 | 0.735 | 0.744 | 0.209 | 0.671 | 0.776 | NA
ISBI test-set | nicMSlesion | 0.453 | 0.131 | 0.644 | 0.075 | 0.338 | 0.050 | 0.712 | 84,512
DeepMedic | 0.523 | 0.385 | 0.807 | 0.273 | 0.388 | 0.215 | 0.670 | 86.810
2.5D Tiramisu | 0.608 | 0.355 | 0.938 | 0.231 | 0.065 | 0.160 | 0.689 | 89.289
DLB | 0.638* | 0.476 | 0.877 | 0.348 | 0.104 | 0.193 | 0.787 | 89.843

Table 6 shows the results of segmentation when training on our In-house
dataset with 3D FLAIR. First, the obtained results when testing on MSSEG’16
indicates a close segmentation accuracy for DLB and 2.5D Tiramisu in terms of
hybrid score (0.745 and 0.741), and a slightly lower performance from
nicMSlesion and DeepMedic (0.7 and 0.717). Second, we notice that our approach
obtained a significantly higher hybrid score when validating on the ISBI
testing dataset, with a submission score of 89.843 compared to 2.5 Tiramisu,
DeepMedic, and nicMSlesion with 89.289, 86.810, and 84.512 respectively. In
this scenario, nicMSlesion obtained the worst score with a dice of 0.131
indicating a failure of the method.

####  3.2.4 Cross-dataset Testing Summary

First, it is noteworthy that when our approach obtained a better score, the
superiority was statistically significant. On the contrary, when one of the
other approaches obtained a higher score, the advantage was not significant
using the Wilcoxon test. Second, it should be pointed out that in all the
considered cross-domain cases, DLB did not degenerate not even once while
maintaining high scores. We reported for nicMSlesion trained on ISBI and
validated on the In-house dataset a hybrid score of 0.419. We also recall the
low performance of 2.5D Tiramisu trained on MSSEG’16 and tested on ISBI (0.462
hybrid score). This shows the cross-domain robustness of the proposed
strategy.

Table 7 sums up cross-dataset experiments results. This table presents the
average score estimated over all the images obtained during the three
experiments presented in Table 4, Table 5, and Table 6 (61 images for ISBI
test-set, 43 images for In-house, 15 images for MSSEG’16). We notice that DLB
obtains the highest hybrid score and Dice index by a large margin compared to
2.5D tiramisu, DeepMedic, and nicMSlesion.

Table 7: Summary of the cross-dataset experiment. The table represent the
average of cross-dataset experiment results (see Table 4, Table 5, and Table
6) based on the number of images for each dataset. For each metric, the bold
values indicate the best result. In hybrid score column, * indicates a
significantly better score than the three other approaches using the Wilcoxon
test.

Strategy | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | CORR
---|---|---|---|---|---|---|---
nicMSlesion | 0.526 | 0.365 | 0.695 | 0.308 | 0.362 | 0.338 | 0.595
DeepMedic | 0.554 | 0.483 | 0.725 | 0.429 | 0.556 | 0.520 | 0.649
2.5 D Tiramisu | 0.608 | 0.443 | 0.870 | 0.368 | 0.179 | 0.402 | 0.537
DLB | 0.663* | 0.601 | 0.775 | 0.550 | 0.299 | 0.484 | 0.780

###  3.3 Same Domain Validation

Despite the previously mentioned limitations of in-domain validation, we also
provide experiments using the same domain as complementary results. First,
Table 8 shows the results of DLB, nicMSlesion, DeepMedic, and 2.5D Tiramisu on
ISBI test-set after being trained on ISBI train-set (same domain), with T1w
and FLAIR modalities. The three approaches give close results with submission
scores of 92.923 for 2.5D Tiramisu, 92.849 for DLB, and 92.161 for
nicMSlesion. DeepMedic comes last with a submission score of 90.866.

Second, Table 9 shows the current top performing methods on the ISBI challenge
website. 2.5D Tiramisu [Zhang et al., 2019] is the best ranked method with the
current highest ISBI Score of 93.21, followed in second place by nnUnet
[Isensee et al., 2019] with 93.09. Both approaches rely on 4 modalities (T1w,
FLAIR, T2, PD). Our approach comes in third place using only 2 modalities, and
obtained the ISBI submission score of 92.85. Although DLB uses a lower number
of modalities, it obtained better results than IMAGINE [Hashemi et al., 2018],
Self-adaptive network [Feng et al., 2019], and Multi-branch [Aslani et al.,
2019] that obtained respectively the scores of 92.49, 92.41, and 92.12.

Table 8: Results of the different approaches trained on the ISBI training
dataset and tested on ISBI test-set, with T1w and FLAIR modalities. For each
metric, the bold values indicate the best result. In hybrid score column, *
indicates a significantly better score than the three other approaches using
the Wilcoxon test.

Approach | Hybrid Score | Dice | PPV | TPR | LFPR | LTPR | CORR | Submission Score
---|---|---|---|---|---|---|---|---
nicMSlesion | 0.724 | 0.639 | 0.853 | 0.541 | 0.144 | 0.432 | 0.863 | 92.161
DeepMedic | 0.649 | 0.643 | 0.827 | 0.557 | 0.408 | 0.530 | 0.873 | 90.866
2.5D Tiramisu | 0.750 | 0.672 | 0.865 | 0.592 | 0.150 | 0.513 | 0.868 | 92.923
DLB | 0.748 | 0.646 | 0.888 | 0.545 | 0.131 | 0.486 | 0.868 | 92.849

Table 9: State-of-the-art published results for the ISBI challenge.

Approach | Modalities | CNN type | Submission Score
---|---|---|---
2.5D Tiramisu [Zhang et al., 2019] | T1w, FLAIR, T2, PD | 2D | 93.21
nnUnet [Isensee et al., 2019] | T1w, FLAIR, T2, PD | 2D and 3D | 93.09
DLB [ours] | T1w, FLAIR | 3D | 92.85
IMAGINE [Hashemi et al., 2018] | T1w, FLAIR, T2, PD | 3D | 92.49
Self-adaptive network [Feng et al., 2019] | T1w, FLAIR, T2, PD | 3D | 92.41
Multi-branch [Aslani et al., 2019] | T1w, FLAIR, T2 | 2D | 92.12

The high accuracy of the results were expected as both the training and
testing sets share the same domain (same acquisition conditions, and same
scanner…). By tuning and adapting a method to this specific domain conditions,
we can obtain artificially higher performance (e.g., DLB with 4 modalities
obtained a score of 92.92, and a 2D version of DLB obtained 93.14
555https://smart-stats-tools.org/lesion-challenge). However, in our opinion,
results reported in the same domain experiment do not truly reflect methods
performances. For instance, the best performing method of this section (2.5D
Tiramisu) failed when trained on different datasets (obtained submission
scores of 89.043 and 89.289 in Table 5 and Table 6). The limitation of such a
validation strategy is one of the main messages of our paper. Hence, we
consider that cross-dataset evaluation with diverse images from different
domains is a better alternative for method assessment.

###  3.4 Corss-dataset Segmentation Consistency

Finally, a usually under-investiguated method property is its cross-dataset
segmentation consistency. In order to assess the consistency of our model
segmentation, we decided to compare the segmentation produced by each approach
on the same data when the model is trained on different datasets. We compute
the Dice between the different segmentations of a method as a similarity index
to quantify the prediction consistency. Table 10 shows the segmentation
consistency for each approach in our cross-dataset setting.

First, we analyzed the segmentations on In-house when the models are trained
respectively on ISBI train-set and and MSSEG’16. In this case, DLB obtained
the best score of 0.647, followed by 2.5D Tiramisu and DeepMedic with 0.6261
and 0.602 respectively. Lastly, nicMSlesion obtained a score of 0.217. Second,
we analyzed the segmentations on MSSEG’16 when the models are trained
respectively on ISBI train-set and In-house. In this case, we obtained close
consistency scores for 2.5D Tiramisu and DLB with Dice scores around 0.72
while DeepMedic and nicMSlesion are less consistent with 0.537 and 0.514
respectively. Finally, we analyzed the segmentations on ISBI test-set when
comparing the models trained on ISBI train-set, the models trained on In-
house, and the models trained on MSSEG’16. For all settings, DLB was
significantly more consistent than both other methods with a Dice ranging from
0.63 to 0.649. 2.5D Tiramisu segmentation consistency index varies from 0.217
to 0.485. DeepMedic consistency index fluctuates from 0.49 to 0.602.
nicMSlesion is the least consistent with scores ranging from 0.177 to 0.512.
During our cross-dataset consistency experiment, DLB was the only method
capable of ensuring segmentation consistency independent of the training
dataset. Both other methods failed several times as indicated with red color
in Table 10.

Table 10: The consistency of the segmentations for each approach in the cross-
dataset setting. The consistency index represents the test-set average of Dice
values, each Dice is computed between two segmentations produced by the same
method when trained on two different train-sets. Higher values indicate better
consistency in the segmentations. The bold values indicate the best result and
red values indicate consistency lower than 0.5. * indicates a significantly
better segmentation consistency score than the three other approaches, using
the Wilcoxon test.

Test-set | In-house | MSSEG’16 | ISBI Test-set
---|---|---|---
|  | ISBI Train-set | ISBI Train-set | ISBI Train-set | In-house | ISBI Train-set
Train-sets |  Dataset 1 vs. Dataset 2 | MSSEG’16 | In-house | MSSEG’16 | MSSEG’16 | In-house
| nicMSlesion | 0.217 | 0.514 | 0.512 | 0.250 | 0.177
| DeepMedic | 0.602 | 0.537 | 0.490 | 0.602 | 0.496
| 2.5D Tiramisu | 0.615 | 0.726 | 0.217 | 0.485 | 0.460
The consistency of the model predictions when trained on Dataset1 vs. Dataset2 | DLB | 0.647 | 0.719 | 0.630* | 0.637* | 0.649*

Figure 4: Part A (left) axial sections of multi-modal MRI (T1w and FLAIR) from
In-house dataset, and its respective expert consensus segmentations for MS
lesion segmentation. Part B (right) cross dataset segmentation of the image
section shown in Part A. The first and second rows illustrate the
segmentations of methods when trained respectively on ISBI dataset, and
MSSEG’16 datasets. First, second, third, and fourth columns represent
respectively the segmentations of nicMSlesion, DeepMedic, 2.5D Tiramisu, and
DeepLesionBrain.

Figure 5: Part A (left) axial sections of multi-modal MRI (T1w and FLAIR) from
MSSEG’16 dataset, and its respective expert consensus segmentations for MS
lesion segmentation. Part B (right) cross dataset segmentation of the image
section shown in Part A. The first and second rows illustrate the
segmentations of methods when trained respectively on ISBI challenge, and In-
house datasets. First, second, third, and fourth columns represent
respectively the segmentations of nicMSlesion, DeepMedic, 2.5D Tiramisu, and
DeepLesionBrain.

Figure 6: Part A (left) axial sections of multi-modal MRI (T1w and FLAIR) from
ISBI challenge dataset, and its respective raters segmentations for MS lesion
segmentation. Part B (right) cross dataset segmentation of the image section
shown in Part A. The first and second rows illustrate the segmentations of
methods when trained respectively on MSSEG’16, and In-house datasets. First,
second, third, and fourth columns represent respectively the segmentations of
nicMSlesion, DeepMedic, 2.5D Tiramisu, and DeepLesionBrain.

Figure 4 represents an image from the In-house dataset and the segmentation of
the different methods when trained on ISBI challenge and MSSEG’16 datasets.
First, both nicMSlesion and 2.5D Tiramisu fail to segment the majority of
lesions when trained on ISBI challenge dataset. This exhibits the limitation
of the robustness of these methods to domain shift, especially for 2.5D
Tiramisu currently considered as the state-of-the-art approach on the ISBI
challenge. Second, DLB detects almost all the lesions in the same conditions.
Third, although DeepMedic also detects most of the lesions, it is more prone
to false positives compared to the other methods. Finally, when choosing
MSSEG’16 as a training dataset, DLB produces the most similar segmentation to
expert annotation.

Figure 5 represents an image from MSSEG’16 dataset and the segmentation of the
different methods when trained on ISBI challenge and In-house datasets. First,
when trained on ISBI dataset, the segmentations of 2.5D Tiramisu, DeepMedic,
and DLB are more accurate than nicMSlesion segmentation, although all
techniques missed a large portion of the central lesion (False Negative)
located around the midsagittal plane. These common voxels misclassification
can result from the subjectivity of raters between training and testing
datasets. Second, when trained on In-house, DLB delineates successfully most
of the lesions. Especially in the case of small lesions, DLB misses only one
lesion, whereas both nicMSlesion and 2.5 D Tiramissu miss four lesions, and
DeepMedic misses two lesions.

Figure 6 represents an image from the ISBI challenge and the segmentation of
the different methods when trained on MSSEG’16 and In-house datasets. From the
four methods, DLB had the most consistent segmentation across different
conditions of training domains. In this case, nicMSlesion produced a decent
segmentation for this example only when trained on MSSEG’16. Likewise, 2.5D
Tiramisu produced better segmentation when trained on In-house than on
MSSEG’16. Although DeepMedic is consistent for this case, the produced
segmentation is less precise and prone to false positives compared to the
other methods.

##  4 Discussion and Conclusion

###  4.1 Discussion

Segmentation models trained with supervised learning can be sensitive to
domain shift, that leads to the generalization failure. Such a domain shift
may be caused by hardware and pre-processing diversity, difference in
acquisition protocol or annotation protocol, that results in difference
between the distributions of training and testing datasets. Besides, we also
have to acknowledge the subjectivity of raters in training datasets. Indeed,
the disagreement between expert segmentations, both in the same dataset and
across different datasets, can make it difficult to train a generalizing
model. Our experiments showed the limited generalization capability of state-
of-the-art approaches, whereas DLB was able to adapt across different domains.
Our study emphasizes the importance of cross-dataset validation, particularly
when considering clinical application of machine learning.

DLB uses a group of several separately trained networks, each network is
specialized in a particular region of the brain. This strategy makes our model
less sensitive to domain shift, since the final segmentation is based on
consensus of diversified and specialized networks. In the ablation study (see
Tables 2 and 3) , our spatially distributed networks strategy showed better
generalization and higher accuracy than using a single model.

Automated MS lesion segmentation should be able to render the most accurate
segmentation with the minimum number of modalities, in order to be efficiently
adopted in clinical conditions and to limit inter-modality dependence. Many
experts agree that FLAIR is the most important modality for MS lesion
delineation. Moreover, T1w modality can provide complementary information for
better white-matter, gray-matter, and cerebrospinal-fluid distinction. FLAIR
and T1w are the most available modalities for MS patients and in all MS
benchmark datasets. Our method achieved a competitive performance using these
two modalities even on unseen domains.

In this paper, we proposed a novel data augmentation technique to reduce
domain shift introduced by the variability of image resolution and quality.
IQDA simulates different acquisition conditions in order to reduce covariate
shift. Our ablation study (see Tables 2 and 3) showed IQDA as a solid
contribution to segmentation accuracy and cross-domain generalization. Indeed,
while other methods (nicMSlesion, DeepMedic, and 2.5D Tiramisu) involve usual
data augmentation (rotation and flipping), such simple strategies failed to
ensure good generalization on unseen dataset.

In our work we considered both specialized local features, and generic global
features of MS lesions. The hierarchical specialization learning proposes an
alternative to network cascades (i.e., [Valverde et al., 2017]). Instead of
using cascades which are prone to error propagation, we suggested a logical
hierarchy during learning based on data selection and transfer learning. The
ablation study (see Tables 2 and 3) exhibits the contribution of HSL to
accuracy and domain generalization compared to DLB without HSL.

In this paper, the proposed method was validated using an out-of-domain cross-
dataset evaluation. This strategy ensures that the performance obtained is not
biased by the training dataset domain information. Indeed, the use of testing
and training images from the same domain is questionable and does not reflect
the generalization ability. The community should start considering this issue
for both the validation and the comparison of proposed methods.

In section 3.2, we reported that the best performances of DLB have been
obtained when using high resolution 3D FLAIR datasets and multi-rater
consensus ground truth for training. The resulting model is able to render
more accurate segmentations for both 2D and 3D image resolution data, even
across unseen domains. This observation led us to believe that to efficiently
train 3D CNN based models for domain generalization, it may be desirable to
optimize the model using high resolution training data.

With current available technology, it is unfeasible to exploit 3D CNNs with
equivalent depth and kernel size as state-of-the-art 2D CNNs. Consequently,
many neuroimaging automated pipelines are still using 2D CNNs despite
processing 3D data. Our results suggest that using multiple compact networks
can approximate a larger and more stable model since the sum of features
extracted by the group of specialized networks and the features of a
hypothetical big network may be equivalent in terms of relevant information
for MS lesion segmentation. In our work, we have chosen to break down the
complexity of the task spatially, based on the sub-volume division of the
whole brain volume. One other advantage of this distribution is the ability to
train networks in parallel, since network weights and images of each region
are independent. It is possible to use several GPUs for parallel training.

###  4.2 Conclusion

DeepLesionBrain is a deep learning framework for MS lesion segmentation
designed for domain generalization. First, we use a spatially distributed
strategy of multiple compact 3D CNNs with large overlapping receptive fields,
in order to produce consensus based segmentation robust to domain shift.
Second, we propose a novel image quality data augmentation to increase
training data variability in a realistic way. Furthermore, we train our method
using hierarchical specialization learning to efficiently incorporate both
generic and specialized features. Finally, we use only T1w and FLAIR
modalities to propose a method compatible with a large number of datasets.

The ablation study showed the impact of each contribution on segmentation
accuracy and domain generalization. The out-of-domain cross-dataset testing is
suggested as an alternative for method evaluation in areas which are sensitive
to domain bias (i.e., medical imaging). Our validation showed the
generalization ability of our method and its robustness to domain shift. We
also proved experimentally that DLB produces consistent segmentations compared
to other state-of-the-art approaches regardless of the training data domain.

##  5 Acknowledgements

This work benefited from the support of the project DeepvolBrain of the French
National Research Agency (ANR-18-CE45-0013). This study was achieved within
the context of the Laboratory of Excellence TRAIL ANR-10-LABX-57 for the
BigDataBrain project. Moreover, we thank the Investments for the future
Program IdEx Bordeaux (ANR-10-IDEX-03-02, HL-MRI Project), Cluster of
excellence CPU and the CNRS/INSERM for the DeepMultiBrain project. This study
has also been supported by the DPI2017-87743-R grant from the Spanish
Ministerio de Economia, Industria Competitividad. The authors gratefully
acknowledge the support of NVIDIA Corporation with their donation of the TITAN
Xp GPU used in this research.

## References

  * [Abadi et al., 2016] Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016).  Tensorflow: A system for large-scale machine learning.  In 12th $\\{$USENIX$\\}$ Symposium on Operating Systems Design and Implementation ($\\{$OSDI$\\}$ 16), pages 265–283.
  * [Aslani et al., 2019] Aslani, S., Dayan, M., Storelli, L., Filippi, M., Murino, V., Rocca, M. A., and Sona, D. (2019).  Multi-branch convolutional neural network for multiple sclerosis lesion segmentation.  NeuroImage, 196:1–15.
  * [Aslani et al., 2020] Aslani, S., Murino, V., Dayan, M., Tam, R., Sona, D., and Hamarneh, G. (2020).  Scanner invariant multiple sclerosis lesion segmentation from MRI.  In 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI), pages 781–785. IEEE.
  * [Breiman, 2001] Breiman, L. (2001).  Random forests.  Machine learning, 45(1):5–32.
  * [Bron et al., 2020] Bron, E. E., Klein, S., Papma, J. M., Jiskoot, L. C., Venkatraghavan, V., Linders, J., Aalten, P., De Deyn, P. P., Biessels, G. J., Claassen, J. A., et al. (2020).  Cross-cohort generalizability of deep and conventional machine learning for mri-based diagnosis and prediction of alzheimer’s disease.  arXiv preprint arXiv:2012.08769.
  * [Brosch et al., 2016] Brosch, T., Tang, L. Y., Yoo, Y., Li, D. K., Traboulsee, A., and Tam, R. (2016).  Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation.  IEEE transactions on medical imaging, 35(5):1229–1239.
  * [Carass et al., 2017] Carass, A., Roy, S., Jog, A., Cuzzocreo, J. L., Magrath, E., Gherman, A., Button, J., Nguyen, J., Prados, F., Sudre, C. H., et al. (2017).  Longitudinal multiple sclerosis lesion segmentation: resource and challenge.  NeuroImage, 148:77–102.
  * [Carass et al., 2007] Carass, A., Wheeler, M. B., Cuzzocreo, J., Bazin, P.-L., Bassett, S. S., and Prince, J. L. (2007).  A joint registration and segmentation approach to skull stripping.  In 2007 4th IEEE international symposium on biomedical imaging: from nano to macro, pages 656–659. IEEE.
  * [Chollet et al., 2015] Chollet, F. et al. (2015).  Keras.  https://keras.io.
  * [Commowick et al., 2016] Commowick, O., Cervenansky, F., and Ameli, R. (2016).  Msseg challenge proceedings: Multiple sclerosis lesions segmentation challenge using a data management and processing infrastructure.
  * [Commowick et al., 2018] Commowick, O., Istace, A., Kain, M., Laurent, B., Leray, F., Simon, M., Pop, S. C., Girard, P., Ameli, R., Ferré, J.-C., et al. (2018).  Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure.  Scientific reports, 8(1):1–17.
  * [Commowick et al., 2012] Commowick, O., Wiest-Daesslé, N., and Prima, S. (2012).  Block-matching strategies for rigid registration of multimodal medical images.  In 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI), pages 700–703. IEEE.
  * [Coupé et al., 2020] Coupé, P., Mansencal, B., Clément, M., Giraud, R., de Senneville, B. D., Ta, V.-T., Lepetit, V., and Manjon, J. V. (2020).  AssemblyNet: A large ensemble of cnns for 3d whole brain mri segmentation.  NeuroImage, page 117026.
  * [Coupé et al., 2018] Coupé, P., Tourdias, T., Linck, P., Romero, J. E., and Manjón, J. V. (2018).  Lesionbrain: an online tool for white matter lesion segmentation.  In International Workshop on Patch-based Techniques in Medical Imaging, pages 95–103. Springer.
  * [Coupé et al., 2008] Coupé, P., Yger, P., Prima, S., Hellier, P., Kervrann, C., and Barillot, C. (2008).  An optimized blockwise nonlocal means denoising filter for 3-d magnetic resonance images.  IEEE transactions on medical imaging, 27(4):425–441.
  * [Feng et al., 2019] Feng, Y., Pan, H., Meyer, C., and Feng, X. (2019).  A self-adaptive network for multiple sclerosis lesion segmentation from multi-contrast mri with various imaging sequences.  In 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019), pages 472–475. IEEE.
  * [Filippi et al., 2019] Filippi, M., Preziosa, P., Banwell, B. L., Barkhof, F., Ciccarelli, O., De Stefano, N., Geurts, J. J., Paul, F., Reich, D. S., Toosy, A. T., et al. (2019).  Assessment of lesions on magnetic resonance imaging in multiple sclerosis: practical guidelines.  Brain, 142(7):1858–1875.
  * [García-Lorenzo et al., 2013] García-Lorenzo, D., Francis, S., Narayanan, S., Arnold, D. L., and Collins, D. L. (2013).  Review of automatic segmentation methods of multiple sclerosis white matter lesions on conventional magnetic resonance imaging.  Medical image analysis, 17(1):1–18.
  * [Hashemi et al., 2018] Hashemi, S. R., Salehi, S. S. M., Erdogmus, D., Prabhu, S. P., Warfield, S. K., and Gholipour, A. (2018).  Asymmetric loss functions and deep densely-connected networks for highly-imbalanced medical image segmentation: Application to multiple sclerosis lesion detection.  IEEE Access, 7:1721–1735.
  * [Huang et al., 2017] Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q. (2017).  Snapshot ensembles: Train 1, get m for free.  arXiv preprint arXiv:1704.00109.
  * [Huo et al., 2019] Huo, Y., Xu, Z., Xiong, Y., Aboud, K., Parvathaneni, P., Bao, S., Bermudez, C., Resnick, S. M., Cutting, L. E., and Landman, B. A. (2019).  3D whole brain segmentation using spatially localized atlas network tiles.  NeuroImage, 194:105–119.
  * [Isensee et al., 2019] Isensee, F., Petersen, J., Kohl, S. A., Jäger, P. F., and Maier-Hein, K. H. (2019).  NNU-Net: Breaking the spell on successful medical image segmentation.  arXiv preprint arXiv:1904.08128, 1:1–8.
  * [Izmailov et al., 2018] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. (2018).  Averaging weights leads to wider optima and better generalization.  arXiv preprint arXiv:1803.05407.
  * [Kamnitsas et al., 2017] Kamnitsas, K., Ledig, C., Newcombe, V. F., Simpson, J. P., Kane, A. D., Menon, D. K., Rueckert, D., and Glocker, B. (2017).  Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation.  Medical image analysis, 36:61–78.
  * [Keskar et al., 2016] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2016).  On large-batch training for deep learning: Generalization gap and sharp minima.  arXiv preprint arXiv:1609.04836.
  * [Kingma and Ba, 2014] Kingma, D. P. and Ba, J. (2014).  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * [Li et al., 2018] Li, H., Jialin Pan, S., Wang, S., and Kot, A. C. (2018).  Domain generalization with adversarial feature learning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400–5409.
  * [Manjón and Coupé, 2016] Manjón, J. V. and Coupé, P. (2016).  volbrain: an online mri brain volumetry system.  Frontiers in neuroinformatics, 10:30.
  * [Manjon et al., 2020] Manjon, J. V., Romero, J. E., and Coupe, P. (2020).  DeepHIPS: A novel deep learning based hippocampus subfield segmentation method.  arXiv preprint arXiv:2001.11789.
  * [Mårtensson et al., 2020] Mårtensson, G., Ferreira, D., Granberg, T., Cavallin, L., Oppedal, K., Padovani, A., Rektorova, I., Bonanni, L., Pardini, M., Kramberger, M. G., et al. (2020).  The reliability of a deep learning model in clinical out-of-distribution mri data: a multicohort study.  Medical Image Analysis, page 101714.
  * [Motiian et al., 2017] Motiian, S., Piccirilli, M., Adjeroh, D. A., and Doretto, G. (2017).  Unified deep supervised domain adaptation and generalization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 5715–5725.
  * [Muandet et al., 2013] Muandet, K., Balduzzi, D., and Schölkopf, B. (2013).  Domain generalization via invariant feature representation.  In International Conference on Machine Learning, pages 10–18.
  * [Narayana et al., 2020] Narayana, P. A., Coronado, I., Sujit, S. J., Sun, X., Wolinsky, J. S., and Gabr, R. E. (2020).  Are multi-contrast magnetic resonance images necessary for segmenting multiple sclerosis brains? a large cohort study based on deep learning.  Magnetic resonance imaging, 65:8–14.
  * [Olson et al., 2018] Olson, M., Wyner, A., and Berk, R. (2018).  Modern neural networks generalize on small data sets.  In Advances in Neural Information Processing Systems, pages 3619–3628.
  * [Omoumi et al., 2021] Omoumi, P., Ducarouge, A., Tournier, A., Harvey, H., Kahn, C. E., Louvet-de Verchère, F., Dos Santos, D. P., Kober, T., and Richiardi, J. (2021).  To buy or not to buy—evaluating commercial ai solutions in radiology (the eclair guidelines).  European Radiology, pages 1–11.
  * [Shiee et al., 2014] Shiee, N., Bazin, P.-L., Cuzzocreo, J. L., Ye, C., Kishore, B., Carass, A., Calabresi, P. A., Reich, D. S., Prince, J. L., and Pham, D. L. (2014).  Reconstruction of the human cerebral cortex robust to white matter lesions: method and validation.  Human brain mapping, 35(7):3385–3401.
  * [Snell et al., 2017] Snell, J., Swersky, K., and Zemel, R. (2017).  Prototypical networks for few-shot learning.  In Advances in neural information processing systems, pages 4077–4087.
  * [Tustison et al., 2010] Tustison, N. J., Avants, B. B., Cook, P. A., Zheng, Y., Egan, A., Yushkevich, P. A., and Gee, J. C. (2010).  N4itk: improved N3 bias correction.  IEEE transactions on medical imaging, 29(6):1310–1320.
  * [Valverde et al., 2017] Valverde, S., Cabezas, M., Roura, E., González-Villà, S., Pareto, D., Vilanova, J. C., Ramió-Torrentà, L., Rovira, À., Oliver, A., and Lladó, X. (2017).  Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach.  NeuroImage, 155:159–168.
  * [Valverde et al., 2019] Valverde, S., Salem, M., Cabezas, M., Pareto, D., Vilanova, J. C., Ramió-Torrentà, L., Rovira, À., Salvi, J., Oliver, A., and Lladó, X. (2019).  One-shot domain adaptation in multiple sclerosis lesion segmentation using convolutional neural networks.  NeuroImage: Clinical, 21:101638.
  * [Wu et al., 2017] Wu, L., Zhu, Z., et al. (2017).  Towards understanding generalization of deep learning: Perspective of loss landscapes.  arXiv preprint arXiv:1706.10239.
  * [Wu and He, 2018] Wu, Y. and He, K. (2018).  Group normalization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 3–19.
  * [Yang and Gao, 2013] Yang, P. Y. and Gao, W. (2013).  Multi-view discriminant transfer learning.
  * [Zhang et al., 2019] Zhang, H., Valcarcel, A. M., Bakshi, R., Chu, R., Bagnato, F., Shinohara, R. T., Hett, K., and Oguz, I. (2019).  Multiple sclerosis lesion segmentation with tiramisu and 2.5 D stacked slices.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 338–346. Springer.
  * [Zhang et al., 2020a] Zhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. (2020a).  How does mixup help with robustness and generalization?  arXiv preprint arXiv:2010.04819.
  * [Zhang et al., 2020b] Zhang, L., Wang, X., Yang, D., Sanford, T., Harmon, S., Turkbey, B., Wood, B. J., Roth, H., Myronenko, A., Xu, D., et al. (2020b).  Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation.  IEEE Transactions on Medical Imaging.
  * [Zhou et al., 2021] Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., and Liang, J. (2021).  Models genesis.  Medical image analysis, 67:101840.