# Improving Uncertainty-based Out-of-Distribution detection for medical image
segmentation

###### Abstract

Deep Learning models are easily disturbed by variations in the input images
that were not seen during training, resulting in unpredictable behaviours.
Such Out-of-Distribution (OOD) images represent a significant challenge in the
context of medical image analysis, where the range of possible abnormalities
is extremely wide, including artifacts, unseen pathologies, or different
imaging protocols. In this work, we evaluate various uncertainty frameworks to
detect OOD inputs in the context of Multiple Sclerosis lesions segmentation.
By implementing a comprehensive evaluation scheme including 14 sources of OOD
of various nature and strength, we show that methods relying on the predictive
uncertainty of binary segmentation models often fails in detecting outlying
inputs. On the contrary, learning to segment anatomical labels alongside
lesions highly improves the ability to detect OOD inputs.

Index Terms—  Deep Learning, brain MRI, Uncertainty, Anomaly Detection,
Multiple Sclerosis.

##  1 Introduction

###  1.1 Motivations

Out-of-distribution (OOD) images are defined as being far from the training
dataset distribution. Deep Learning (DL) models tend to behave inconsistently
for this type of inputs, making OOD image detection crucial to avoid hidden
model deficiencies [1]. It is especially required in real-world automated
pipelines, where input images may not be visually inspected before running the
analysis. In the context of computerized medical-images analysis, a large
variety of phenomenons in the input images can disturb a DL model and lead to
unpredictable behaviours: noise, artifacts, variations in the imaging
acquisition protocol and device, or pathological cases that were not included
in the training dataset.

###  1.2 Related Works

OOD image detection frameworks can roughly be divided into two categories [2]:
methods that build a model specifically dedicated to the OOD task, and methods
that rely on the prediction uncertainty of a trained model to detect abnormal
inputs.

Within the first category, the most straightforward manner is to build a
classifier directly on input images for the detection of OOD ones. For this, a
Convolutional Neural Network (CNN) can be trained in a supervised manner,
using an annotated dataset containing various types of real-world OOD [3].
Alternatively, Unsupervised Anomaly Detection (UAD) proposes to model the
appearance of normal images by training an Auto-Encoder network (AE) to
reconstruct in-distribution (ID) samples. At test-time, reconstruction is
expected to be degraded for OOD samples, allowing for their detection [4, 5].
Finally, in the context of semantic image segmentation, the Image Resynthesis
framework [6, 7] proposes to train a Generative Adversarial Network (GAN) to
reconstruct the input image from the predicted segmentation. As the automated
segmentation of OOD images is expected to be poor, the resynthesized input
image should be distant from the true (unseen) input image.

While the former methods require the training of an auxiliary model (CNN, AE,
or GAN) dedicated to the OOD detection task, uncertainty quantification (UQ)
approaches propose to detect OOD inputs directly from the outputs of the
trained predictive neural network. They rely on the hypothesis that the
uncertainty of the deployed model should be higher when confronted to an
unusual image, compared to the one of ID images. Based on this principle, OOD
detection can be performed by monitoring the output uncertainty of the model.
Various frameworks have been proposed to quantify uncertainty, and we present
four of the most popular approaches below.

The baseline approach to estimate the uncertainty of a trained prediction
model is to consider its output softmax probabilities. Following the intuition
that the higher the probability, the more certain is the model, the _Maximum
Softmax Probability_ (MSP) can be used as an estimator of uncertainty [8].
Another standard UQ methodology consists in producing a set of diverse and
plausible predictions for the same input image. To achieve that, the _MC
dropout_ approach [9] consists in training a segmentation model with dropout
and keep it activated during inference. At test time, each input image is
forwarded multiple times through the model, with randomly sampled Dropout
masks, yielding to multiple predictions for the same input image.
Alternatively, _Deep Ensemble_ (DE) [10] proposes to train the same neural
network multiple times. As the weights are randomly initialized at each
training, the models reach different optimums. At test time, each individual
model processes the input image independently, which produces a set of
predictions. For both the MC dropout model and DE, uncertainty can then be
estimated by computing the variance among the predictions. Finally,
_Deterministic Uncertainty Methods_ (DUM) analyze the intermediate feature
maps of a trained model to detect OOD inputs [11]. It is based on the
assumption that the hidden activations of the model should be different
between an ID sample and an OOD one. In [12], authors propose to analyze the
signature of an input image in the latent space of a segmentation model to
detect OOD samples. At test time, an image OOD score is obtained by computing
the distance between the signature of the test image, and the signatures of
all training images.

As mentioned, OOD detection in medical image segmentation is critical,
particularly in a clinical setting. Although uncertainty-based OOD detection
has been applied for medical image segmentation [12, 13, 14], its evaluation
is usually limited to extreme OOD cases (e.g. different imaging modality,
organ of interest or imaging center). While it is necessary to robustly detect
such cases, we believe that the detection of more subtle artifacts (e.g. image
artifacts, truncation or skipped preprocessing) is also crucial in clinical
practice.

###  1.3 Contributions

In this work, we propose to further explore uncertainty-based OOD detection in
the context of medical image segmentation. As a use-case, we study OOD
detection in input MRIs in the context of Multiple Sclerosis (MS) lesions
segmentation. We develop a robust evaluation framework containing 14 synthetic
or real abnormalities, representative of the abnormalities encountered in
clinical routine. We use this framework to compare four of the most popular
UQ-based OOD-detection models. Each model, initially binary, is additionally
derived as a multiclass model that also segments anatomical regions of the
brain in parallel with the MS-lesion segmentation. Indeed, we hypothesize that
binary models trained to solely segment lesions tend to discard information
not relevant for the task at hand, which may leave out key information for OOD
detection. To verify this intuition, we compare the binary and multiclass
versions of each segmentation model with respect to the OOD detection task. We
demonstrate that the retention of all additional information in multiclass
models highly improves the OOD performance of uncertainty-based techniques, as
compared to binary models.

Fig. 1: In and Out-of-distribution data used in the experiments. In black: ID
data and associated ground truth; blue: synthetic OOD; red: real OOD.

##  2 Materials

Data were collected to train the segmentation models, to evaluate the OOD
detection, as well as to evaluate the segmentation performance of the models.

###  2.1 In-distribution data

We collect data from several open-source MS datasets: the MSSEG-1 Challenge
[15], the WMH 2017 Challenge [16], and the database gathered by the University
of Ljubljana, Slovenia (MSLUB) [17]. For each of these datasets, T1-weighted
and T2-weighted FLAIR MRI sequences are provided, as well as the ground truth
segmentation of White Matter Hyperintensities (WMH). From the total of 143
subjects, we randomly set aside 50 subjects. The remaining 93 subjects are
used to train the segmentation models while the 50 set-aside subjects are
included in the OOD-detection evaluation database. These 50 subjects sharing
the same origin than the 93 training images, they are considered as ID images.

We first co-register FLAIR and T1 sequences, followed by a rigid registration
of both sequences on a 170 × 204 × 170 template with an isotropic resolution
of 1mm. We use FastSurfer on the T1 sequences in order to obtain an anatomical
atlas of the brain composed of 95 classes for each patient [18], which we
further simplify into 6 brain regions: supratentorial white and grey matter,
cerebrospinal fluid, infratentorial white and grey matter, and brain stem. We
skull-strip the MRIs using the computed brain mask. We finally superpose the
ground truth WMH delineations to the anatomical labels to obtain the final
training masks composed of 7 foreground classes (6 anatomical classes plus 1
lesion class). Only the FLAIR sequences are used for the training of the
segmentation models.

###  2.2 Out-of-distribution data

OOD data were collected or generated to build the OOD detection evaluation
database (Figure 1). The same data were also used to evaluate the segmentation
performance of the models, together with the 50 set-aside ID samples.

####  2.2.1 Synthetic Out-of-distribution data

A common type of OOD in MRI consists is artifacts. However, finding real
images with a controlled amount of artifacts to allow evaluation of OOD
detection methods is difficult. We therefore generate realistic synthetic
artifacted images from the set of ID evaluation images. We used the TorchIO
Data Augmentation library [19] to generate _Downsample_ , _Bias_ , _Motion_ ,
_Spikes_ , _Gaussian Noise_ , _Ghost_ , and _Scaling_ artifacts. We also
implement a _Truncation_ operation that randomly removes the top and bottom
slices of the MRI to mimic errors in the file transfer. From each ID
evaluation image, we generate one OOD image per type of synthetic artefact.
Having implemented 8 types of synthetic artifacts, we obtain 400 synthetically
artifacted images.

####  2.2.2 Real Out-of-distribution data

We also propose several real-world OOD scenarios to validate our approaches.
We use the set-aside T1-weighted sequences of the 50 test subjects, as well
post-contrast T1-weighted (T1-Gd) sequences from the MSSEG-1 dataset to
simulate mistakes in the input MRI sequences of an automated pipeline (T1 or
T1-Gd sequences used as input, instead of expected FLAIR series). We use 50
FLAIR sequences from the BraTS 2020 dataset [20] containing brain tumors to
test how the models behave on a pathology not observed during training. We
also construct a version of the test dataset for which no skull-stripping is
applied to mimic an error in the image preprocessing pipeline. Lastly, we use
two other MS datasets that are not used during training to simulate domain-
shift due to different acquisition protocols. We used 21 scans from the ISBI
2015 MS dataset [21] and 60 scans from the MSSEG-2 dataset [22]. All data
follow the same preprocessing steps presented in 2.1, except for the samples
which are deliberately not skull-stripped.

##  3 Method

We train standard and MC dropout segmentation models in 2 settings: binary
(with only WMH being segmented) and multiclass (WMH as well as the 6 brain
anatomical regions are segmented). MSP is obtained from the softmax
probabilities of the standard models, and DUM is implemented using the
features of their penultimate layers [12]. Each type of segmentation model is
trained 5 times, allowing to obtain robust statistics for MSP, MC Dropout and
DUM models, as well as to build DE models.

While DUM directly provides image-wise scores, the other implemented UQ
methods (MSP, MC dropout and DE) provide one uncertainty score per voxel. For
these methods, we compute the average of all the volume voxel uncertainties
(estimated using the variance for MC dropout and DE) to get a single score per
MRI [12].

We define OOD detection as a binary classification task, with ID images
defined as negative samples, and OOD images as positive samples [8]. For each
OOD dataset, we compute the image-level uncertainty scores and compare them
with the scores obtained for the ID set of test images. We use the area under
the ROC curve (AUROC) to estimate the ability to distinguish between ID and
OOD images based on the image-level uncertainty scores. When the ground truth
of WMH is available (i.e. not for the BraTS and MSSEG-2 data), we also report
the WMH segmentation performances of MSP, MC dropout and DE, evaluated using
the Dice score (DSC).

| MSP | MC dropout | DE | DUM
---|---|---|---|---
| Binary | Multiclass | Binary | Multiclass | Binary | Multiclass | Binary | Multiclass
| AUROC | DSC | AUROC | DSC | AUROC | DSC | AUROC | DSC | AUROC | DSC | AUROC | DSC | AUROC | AUROC
Test | - | 0.67 $\pm$ 0.00 | - | 0.68 $\pm$ 0.00 | - | 0.66 $\pm$ 0.01 | - | 0.66 $\pm$ 0.01 | - | 0.68 | - | 0.68 | - | -
Downsample | 0.50 $\pm$ 0.02 | 0.62 $\pm$ 0.00 | 0.94 $\pm$ 0.01 | 0.63 $\pm$ 0.00 | 0.52 $\pm$ 0.02 | 0.61 $\pm$ 0.01 | 0.84 $\pm$ 0.03 | 0.62 $\pm$ 0.01 | 0.53 | 0.63 | 0.88 | 0.64 | 0.88 $\pm$ 0.03 | 0.94 $\pm$ 0.02
Bias | 0.58 $\pm$ 0.11 | 0.55 $\pm$ 0.01 | 0.74 $\pm$ 0.02 | 0.59 $\pm$ 0.01 | 0.51 $\pm$ 0.04 | 0.50 $\pm$ 0.02 | 0.71 $\pm$ 0.02 | 0.58 $\pm$ 0.02 | 0.58 | 0.56 | 0.74 | 0.59 | 0.94 $\pm$ 0.01 | 0.92 $\pm$ 0.02
Motion | 0.29 $\pm$ 0.16 | 0.63 $\pm$ 0.00 | 0.99 $\pm$ 0.01 | 0.64 $\pm$ 0.01 | 0.40 $\pm$ 0.01 | 0.63 $\pm$ 0.01 | 0.91 $\pm$ 0.03 | 0.63 $\pm$ 0.01 | 0.46 | 0.64 | 0.96 | 0.65 | 0.98 $\pm$ 0.02 | 0.99 $\pm$ 0.00
Spikes | 0.15 $\pm$ 0.16 | 0.19 $\pm$ 0.04 | 1.00 $\pm$ 0.00 | 0.15 $\pm$ 0.04 | 0.51 $\pm$ 0.14 | 0.31 $\pm$ 0.05 | 1.00 $\pm$ 0.00 | 0.26 $\pm$ 0.07 | 0.57 | 0.18 | 1.00 | 0.18 | 1.00 $\pm$ 0.00 | 1.00 $\pm$ 0.00
Noise | 0.40 $\pm$ 0.36 | 0.64 $\pm$ 0.00 | 0.98 $\pm$ 0.01 | 0.64 $\pm$ 0.01 | 0.38 $\pm$ 0.03 | 0.64 $\pm$ 0.01 | 0.93 $\pm$ 0.05 | 0.64 $\pm$ 0.02 | 0.57 | 0.65 | 0.97 | 0.65 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.00
Ghost | 0.41 $\pm$ 0.20 | 0.65 $\pm$ 0.00 | 0.97 $\pm$ 0.03 | 0.65 $\pm$ 0.01 | 0.48 $\pm$ 0.02 | 0.64 $\pm$ 0.01 | 0.94 $\pm$ 0.03 | 0.64 $\pm$ 0.01 | 0.52 | 0.66 | 0.96 | 0.66 | 0.96 $\pm$ 0.01 | 0.99 $\pm$ 0.04
Truncation | 0.56 $\pm$ 0.03 | 0.64 $\pm$ 0.00 | 0.15 $\pm$ 0.01 | 0.63 $\pm$ 0.00 | 0.52 $\pm$ 0.01 | 0.62 $\pm$ 0.01 | 0.49 $\pm$ 0.04 | 0.62 $\pm$ 0.01 | 0.52 | 0.65 | 0.67 | 0.65 | 0.89 $\pm$ 0.01 | 0.96 $\pm$ 0.01
Scale | 0.48 $\pm$ 0.11 | 0.61 $\pm$ 0.00 | 0.95 $\pm$ 0.03 | 0.61 $\pm$ 0.01 | 0.59 $\pm$ 0.03 | 0.59 $\pm$ 0.01 | 0.95 $\pm$ 0.01 | 0.60 $\pm$ 0.01 | 0.58 | 0.62 | 0.99 | 0.63 | 0.99 $\pm$ 0.00 | 0.99 $\pm$ 0.00
T1 | 0.46 $\pm$ 0.21 | 0.00 $\pm$ 0.00 |  1.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.78 $\pm$ 0.12 | 0.00 $\pm$ 0.00 | 0.99 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.86 | 0.00 | 1.00 | 0.00 | 0.96 $\pm$ 0.0 | 0.99 $\pm$ 0.00
T1-Gd | 0.31 $\pm$ 0.20 | 0.00 $\pm$ 0.00 |  1.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.42 $\pm$ 0.12 | 0.00 $\pm$ 0.00 | 1.00 $\pm$ 0.00 | 0.00 $\pm$ 0.00 | 0.49 | 0.00 | 1.00 | 0.00 | 0.98 $\pm$ 0.01 | 1.00 $\pm$ 0.00
BraTS | 0.54 $\pm$ 0.17 | - |  1.00 $\pm$ 0.00 | - | 0.93 $\pm$ 0.02 | - | 1.00 $\pm$ 0.00 | - | 0.96 | - | 1.00 | - | 0.99 $\pm$ 0.00 | 1.00 $\pm$ 0.00
Skull | 0.35 $\pm$ 0.17 | 0.51 $\pm$ 0.01 | 1.00 $\pm$ 0.00 | 0.52 $\pm$ 0.02 | 0.58 $\pm$ 0.06 | 0.49 $\pm$ 0.01 | 1.00 $\pm$ 0.00 | 0.54 $\pm$ 0.02 | 0.68 | 0.53 | 1.00 | 0.56 | 0.97 $\pm$ 0.01 | 0.99 $\pm$ 0.00
ISBI 2015 | 0.34 $\pm$ 0.20 | 0.62 $\pm$ 0.01 | 1.00 $\pm$ 0.00 | 0.65 $\pm$ 0.02 | 0.46 $\pm$ 0.05 | 0.65 $\pm$ 0.02 | 0.99 $\pm$ 0.01 | 0.65 $\pm$ 0.02 | 0.45 | 0.62 | 1.00 | 0.67 | 0.97 $\pm$ 0.02 | 0.99 $\pm$ 0.00
MSSEG-2 | 0.62$\pm$ 0.05 | - | 0.94 $\pm$ 0.01 | - | 0.51 $\pm$ 0.02 | - | 0.93 $\pm$ 0.04 | - | 0.50 | - | 0.99 | - | 0.95 $\pm$ 0.02 | 0.99 $\pm$ 0.00

Table 1: AUROC and DSC scores obtained for each ID and OOD datasets. For all
methods except DE, we present the performance averaged over the 5 runs, as
well as the standard deviation. The best OOD detection performance for each
dataset is highlighted in bold.

##  4 Implementation details

Our framework is implemented using PyTorch [23]. We use a 3D Residual UNet
[24] as the backbone of the segmentation models. For MC dropout, we apply a 3D
dropout with a rate of $20\%$ in each layer of the encoder and decoder. The
models are trained using 3D patches of $160\times 192\times 32$ with the Dice
loss [25]. We use a simple Data Augmentation pipeline during training composed
of flipping, rotation, and contrast alteration. At inference, we perform
$T=20$ inferences per image to obtain the final segmentation and uncertainty
estimates for MC dropout models.

##  5 Results

The AUROC and DSC scores for each OOD dataset and UQ method are presented in
Table 1. As expected, DSC scores are inferior on OOD data (_e.g._ _T1-Gd_
dataset), meaning that the inability to detect OOD could result in
catastrophic outcomes in the context of a real world automated pipeline. This
is the case with binary implementations of MSP, MC dropout and DE that reach
low OOD detection performance on most datasets, being even worst than chance
($\text{AUROC}<0.5$) in some settings (_e.g._ , MSP Binary on the _Spikes_
dataset). Exceptions are made for the MC dropout Binary and DE Binary that
show satisfying OOD detection on the T1 and BraTS datasets, which are however
the most obvious types of OOD. On the contrary, the multiclass versions
demonstrate high improvement with respect to the OOD task, being able to
perfectly distinguish between ID and OOD samples in many settings (_e.g._ ,
$\text{AUROC}=1.00$ for DE Multiclass on the _Skull_ dataset). Interestingly,
DUM models reach high detection accuracy in both settings, binary and
multiclass.

##  6 Discussion and Conclusion

In this work, we investigate the use of UQ approaches to detect OOD inputs in
the context of medical image segmentation. We elaborate an evaluation
framework with a set of 14 various synthetic or real OODs to evaluate several
popular UQ methods for OOD detection. Our experimental results demonstrate
that standard UQ approaches such as MSP, MC dropout and DE fail to detect OOD
inputs when applied on binary segmentation models, even on extreme OOD cases
(_e.g._ _T1-Gd_ dataset). Overall, binary segmentation models tend to be
overconfident on abnormal inputs.

Second, our experiments show that UQ methods applied on models that were
trained to segment anatomical labels additionally to lesions (multiclass)
achieve excellent OOD detection. Intuitively, this additional segmentation
task enforces the model to take into account the entirety of the available
brain information to make its prediction, increasing the quantity of
information available to decide whether the input is abnormal or not.

Finally, DUM models show excellent OOD detection performances, with DUM
Multiclass being a little more consistent than the Binary version. This
suggests that the hidden activations of a trained model are a powerful
indicator of the distance of a sample to the training set. Interestingly, DUM
is very computationally efficient, as they only require a single trained
model, in contrast to the MC dropout and DE that require either performing
multiple predictions with the same network, or training several models. These
approaches are thus particularly interesting for the implementation of OOD-
detection tasks in a clinical setting.

While multiclass seems beneficial for OOD detection, it requires the
generation of anatomical atlases to train the model, which may not always be
feasible. In that case, the DUM approach is the most suitable, as it achieves
strong OOD detection results even on binary models.

Future work will investigate the impact of the number of learned anatomical
labels on the OOD detection and compare the performance of uncertainty-based
method to the performance of dedicated models such as AE or GANs. We will also
investigate the impact of preprocessing choices (_e.g._ skullstripping,
debiasing) on OOD detection performance.

##  7 Acknowledgments

BL, AT, SD are employees of the Pixyl company. MD and FF serve on Pixyl
scientific advisory board.

##  8 Compliance with Ethical Standards

This research study was conducted retrospectively using human subject data
made available in open access. Ethical approval was not required as confirmed
by the license attached with the open access data.

## References

  * [1] Jingkang Yang et al.,  “Generalized out-of-distribution detection: A survey,”  arXiv preprint arXiv:2110.11334, 2021.
  * [2] Christoph Berger et al.,  “Confidence-based out-of-distribution detection: a comparative study and analysis,”  Unsure 2021, pp. 122–132, 2021.
  * [3] Simona Bottani et al.,  “Automatic quality control of brain t1-weighted magnetic resonance images for a clinical data warehouse,”  Medical Image Analysis, vol. 75, pp. 102219, 2022.
  * [4] Dong Gong et al.,  “Memorizing normality to detect anomaly: Memory-augmented deep autoencoder for unsupervised anomaly detection,”  ICCV, pp. 1705–1714, 2019.
  * [5] Hyunjong Park et al.,  “Learning memory-guided normality for anomaly detection,”  CVPR, pp. 14372–14381, 2020.
  * [6] Krzysztof Lis et al.,  “Detecting the unexpected via image resynthesis,”  ICCV, pp. 2152–2161, 2019.
  * [7] Yingda Xia et al.,  “Synthesize then compare: Detecting failures and anomalies for semantic segmentation,”  European Conference on Computer Vision, pp. 145–161, 2020.
  * [8] Dan Hendrycks and Kevin Gimpel,  “A baseline for detecting misclassified and out-of-distribution examples in neural networks,”  ICLR 2017, 2017.
  * [9] Yarin Gal and Zoubin Ghahramani,  “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning,”  ICML, vol. 48, pp. 1050–1059, 2016.
  * [10] Balaji Lakshminarayanan et al.,  “Simple and scalable predictive uncertainty estimation using deep ensembles,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [11] Janis Postels et al.,  “On the practicality of deterministic epistemic uncertainty,”  ICML, vol. 162, pp. 17870–17909, 2022.
  * [12] Davood Karimi and Ali Gholipour,  “Improving calibration and out-of-distribution detection in deep models for medical image segmentation,”  IEEE Transactions on Artificial Intelligence, 2022.
  * [13] Alireza Mehrtash et al.,  “Confidence calibration and predictive uncertainty estimation for deep medical image segmentation,”  IEEE transactions on medical imaging, vol. 39, no. 12, pp. 3868–3878, 2020.
  * [14] Zhaoshuo Diao et al.,  “A unified uncertainty network for tumor segmentation using uncertainty cross entropy loss and prototype similarity,”  Knowledge-Based Systems, vol. 246, pp. 108739, 2022.
  * [15] Olivier Commowick et al.,  “MSSEG challenge proceedings: multiple sclerosis lesions segmentation challenge using a data management and processing infrastructure,”  MICCAI, 2016.
  * [16] Hugo J Kuijf et al.,  “Standardized assessment of automatic segmentation of white matter hyperintensities and results of the wmh segmentation challenge,”  IEEE transactions on medical imaging, vol. 38, no. 11, pp. 2556–2568, 2019.
  * [17] Žiga Lesjak et al.,  “A novel public mr image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus,”  Neuroinformatics, vol. 16, no. 1, pp. 51–63, 2018.
  * [18] Leonie Henschel et al.,  “Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline,”  NeuroImage, vol. 219, pp. 117012, 2020.
  * [19] Pérez-García et al.,  “Torchio: a python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning,”  Computer Methods and Programs in Biomedicine, vol. 208, pp. 106236, 2021.
  * [20] Bjoern H Menze et al.,  “The multimodal brain tumor image segmentation benchmark (brats),”  IEEE transactions on medical imaging, vol. 34, no. 10, pp. 1993–2024, 2014.
  * [21] Aaron Carass et al.,  “Longitudinal multiple sclerosis lesion segmentation: resource and challenge,”  NeuroImage, vol. 148, pp. 77–102, 2017.
  * [22] Olivier Commowick et al.,  “MSSEG-2 challenge proceedings: Multiple sclerosis new lesions segmentation challenge using a data management and processing infrastructure,”  MICCAI, pp. 1–118, 2021.
  * [23] Adam Paszke et al.,  “Pytorch: An imperative style, high-performance deep learning library,”  Advances in neural information processing systems, vol. 32, 2019\.
  * [24] Eric Kerfoot et al.,  “Left-ventricle quantification using residual u-net,”  STACOM, pp. 371–380, 2018.
  * [25] Fausto Milletari et al.,  “V-net: Fully convolutional neural networks for volumetric medical image segmentation,”  3DV, pp. 565–571, 2016.