# Automated computed tomography and magnetic resonance imaging segmentation
using deep learning: a beginner’s guide
††thanks: We thank the following research support foundations: CAPES, CNPQ,
and grant #2019/21964-4, São Paulo Research Foundation (FAPESP).

Diedre Carmo§ MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>Gustavo Pinheiro§ MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>Lívia Rodrigues§ MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>Thays Abreu  MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>Roberto Lotufo  MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>Letícia Rittner  MICLab
University of Campinas (UNICAMP)
Campinas, Brazil
<EMAIL_ADDRESS>
###### Abstract

Medical image segmentation is an increasingly popular area of research in
medical imaging processing and analysis. However, many researchers who are new
to the field struggle with basic concepts. This tutorial paper aims to provide
an overview of the fundamental concepts of medical imaging, with a focus on
Magnetic Resonance and Computerized Tomography. We will also discuss deep
learning algorithms, tools, and frameworks used for segmentation tasks, and
suggest best practices for method development and image analysis. Our tutorial
includes sample tasks using public data, and accompanying code is available on
GitHub (https://github.com/MICLab-Unicamp/Medical-Imaging-Tutorial). By
sharing our insights gained from years of experience in the field and learning
from relevant literature, we hope to assist researchers in overcoming the
initial challenges they may encounter in this exciting and important area of
research.

§§footnotetext: Equal contribution

##  I Introduction

It is not a mystery that medical images are a great tool in medicine. Besides
being non-invasive, they are useful for diagnosing, evaluating, and predicting
diseases. Also, many physicians use medical images for research purposes. The
history of medical imaging begins in 1895, with the discovery of X-rays by
Wilhelm Rontgen. It did not take long, the technique was being used by
physicists to analyze medical issues. However, only in the later 1960’s
Hounsfield, an EMI Limited researcher, started to study x-rays in a 3D form.
In 1972, Hounsfield and Dr. James Ambrose could diagnose a tumor using
Computerized Tomography (CT) due to different tissue contrasts. By the same
time, in 1973, Paul Lauterbur demonstrated that Nuclear Magnetic Resonance
(NMR) could be used to create an image [1]. Yet, only in 1977 the first human
image using MR was acquired, taking 5 hours for the acquisition[2].

With the popularization of imaging methods, research centers are dealing with
an increasing amount of data that are both time and monetary costly to analyze
manually. Using computational methods, engineers and computer science
professionals can help physicians diminish these costs. In the early days,
these computational methods were mainly for image enhancement, based on
classical imaging algorithms such as morphology and filtering. More complex
tasks, such as pattern recognition through machine learning algorithms, became
possible as the area evolved. Although the first work on artificial
intelligence (AI) began in the 1950s, AI only started to be applied to medical
issues in the 1970s [3].

By the decade of 1990, there were several segmentation methods applied to MR
images, and it was common to find methods based on the image characteristics,
such as region-based algorithms. After a while, machine learning (ML) methods
started to appear, and it became more usual to extract features from the
images and use them on algorithms such as Support Vector Machines or Decision
Trees. Finally, deep learning (DL) has become central over the last few years.
Unlike the previous methods applied to medical images, DL can compute features
from the raw input data without requiring manual feature extraction. The
feature extraction step is done by the first layers of the architecture, and
it is not necessarily interpretable by humans since they are abstract
representations of the input image. However, DL algorithms require a large
amount of data, which is a challenge in the medical field.

DL algorithms are widely applied to various medical image problems, such as
regression, classification, and segmentation. In classification problems, DL
classifies a sample into one of the N possible labels. Many applications in
the literature focused on helping medical research, such as brain tumor
classification [4, 5] and covid $\times$ non-covid affected lungs method [6].
In regression problems, the model is trained to predict a continuous value.
Bounding box prediction for structures detection [7, 8] and MRI reconstruction
from K-space images [9] are examples of regression used on medical images.
Finally, segmentation methods contour the border of a specific structure or
area to be studied. Physicians may use the segmentation to study brain lesions
[10], lung findings [11], and subtle changes in specific structures. Several
DL networks have been developed for this purpose [12, 13] and are applied to
segment different structures of the brain [14, 15] and body [16, 17].

This paper will focus on Magnetic Resonance (MR) and Computed Tomography (CT)
segmentation using DL algorithms. First, we will give a brief overview on the
data, its acquisition, and intrinsic characteristics. Then, we will shortly
introduce DL for image analyses and the computational environment required.
Finally, we will describe the usual workflow and the most common statistical
analysis, and give some recommendations and useful tips.

##  II Medical images

A medical image can be understood as any image that represents aspects of the
biological tissue. Medical images can be classified by the technique that is
used for acquisition, also called modalities, such as ultrasound, magnetic
resonance, and X-ray computed tomography. They can also be classified by their
dimension: planar (or 2D), volumetric (or 3D), time series (4D), or by their
range of values: single-channel scalar; multi-channel (e.g. dermatoscopic
image) scalar; tensorial (e.g.: diffusion tensor imaging).

Among the great variety of medical image types, the vast majority is composed
of scalar measures. The images are made of a collection of voxels (volume
elements) representing a single scalar value. Thus, the images are a scalar
field in a $\mathbb{Z}^{3}$ space.

As for any finite discrete scalar field, or array, the images are defined by
geometrical parameters that are directly related to the image quality. Field
of View (FOV), spatial resolution, Voxel Size (Fig. 1), and radial resolution
are some of the most relevant parameters. The FOV is the size of the image in
real-world dimensions, the spatial resolution is the number of voxels in each
image dimension, the voxel size is the measurement of the voxel dimensions,
and the radial resolution is the number of possible scalar values that the
voxel can assume. This scalar value is the representative of the tissue for
each voxel.

Figure 1: Parameters of a 3D image: Fiel of View (FOV), resolution, and voxel
size.

As medical images are usually acquired in 2D slices, the 3D image can be seen
as a stack of slices, and a 3D image gives us the freedom to look at the image
from three views (Fig. 2): axial (or transversal), coronal (or frontal), and
sagittal (or longitudinal).

Figure 2: Slice view orientation in 3D medical images: Axial, Coronal, and
Sagittal

The voxel is a volumetric element expressed in a single scalar value, and it
represents an average of what is inside the defined space. Consequently, the
voxel value suffers from the partial volume, which happens when more than one
biological tissue is represented in a single voxel. This effect is minimized
by reducing the voxel size and increasing the resolution for the same FOV. The
partial volume can affect both manual and automatic segmentation methods since
the fuzziness of the values could confuse the algorithms.

Depending on the application, the image acquisition parameters could vary
considerably. For example, in the research field, the images using research-
grade parameters tend to have higher resolution, better contrast, and less
noise at the cost of longer acquisition time or better quality equipment. On
the other hand, for clinical purposes, where resources such as time are
scarce, the images acquired in clinical settings usually have a less spatial
resolution (sometimes even skipping slices) and lower quality in general.

Among all 3D medical imaging types, CT and MR imaging are two of the most
popular imaging-based diagnostic modalities used in different clinical
conditions for diagnosis, follow-up, image-guided procedures, and medical
research. Studying these images allows the analysis and segmentation of
different body structures. Annotations can be used for different purposes such
as volume measurements, statistics of a population in medical research,
localization of abnormal tissue according to an underlying pathological
process, and disease of the patient [18]. Since manual analysis is very time-
consuming and poorly reproducible, there is an interest in automated
processing of these images [19].

###  II-A X-ray CT

In the X-ray based CT modality, the image is reconstructed from various X-ray
acquisitions around the patient. Several parameters guide the acquisition and
reconstruction of signals recorded by the CT scanner into the final image and
can be found on the image’s header. These parameters include, among others:
the spacing between body slices, which can range from less than 1 mm to more
than 1 cm; slice resolution, which is commonly very high with a pixel
representing only 0.5mm²; reconstruction kernel or filter, which controls the
frequencies present in the image and can generate from very smooth to very
noisy images; and many others. One of the most important parameters to be
considered when processing CT images is the Hounsfield Units (HU) window. HU
directly maps to specific tissues, air, and water (Fig 3). Other variants of
CT acquisition, such as PET-CT are out of the scope of this manuscript.

Figure 3: Hounsfield Unit scale, mapping values to the represented tissue, and
an axial slice of a CT scan showing the inside of the lung.

Since HU values go beyond the traditional 8-bit (256 values) representation of
gray images in a monitor, applying a window (clipping) to HU values will
improve the contrast and visualization of specific tissues. In addition, the
values contained in the digital reconstruction may have been rescaled to a
different value. This rescaling is represented by a linear mapping $ax+b$
where $a$ is the rescale slope and $b$ is the rescale intercept. These are
commonly applied to remove negative numbers from the intensities, allowing
unsigned storage. Section IV-A5 will review some recommendations to leverage
these proprieties of CT images and avoid common mistakes when preprocessing
and using CT images.

###  II-B MRI

MR Imaging is a technique that creates images by the interference between a
high-intensity magnetic field, radio frequency pulses, and the field generated
by the spin of the protons in the tissues inside the MR scanner. The signal
strength measured by the scanner coils is responsible for generating the
contrast for each voxel of the three-dimensional image.

MR images play a key role in the clinical environment as it is proven to be a
fast, safe (non-ionizing radiation), and non-intrusive way to look inside the
body. To support different types of exams, there are several MR imaging pulse
sequences, each one presenting a different image contrast, exploiting specific
characteristics of biological tissues. Due to a qualitative similarity to an
anatomical slice, the sequence known as T1-weighted is one of the most
popular. It is also one of the fastest-acquired sequences and relatively
simple to analyze.

Besides T1-weighted images, many other MR image sequences (Fig. 4) are used
for various applications, including structural and functional. For example,
FLAIR [20] is used for investigating brain lesions, Diffusion sequences are
used to measure the water diffusion in the tissue and infer about the fiber
organization [21], Functional MR imaging [22] is used to measure brain
activity, and spectroscopy [23] to measure metabolites.

Figure 4: Examples of MRI images from different acquisition sequences:
respectively T1-weighted, FLAIR, Diffusion Tensor Imaging, and Functional MRI.
Images from different subjects and non-correspondent slices.

###  II-C Medical Imaging Processing

The field of medical imaging processing has considerably changed over the last
few years. At first, methods were based on image characteristics and often
mimic the process done by human experts, such as region-based algorithms.
After a while, machine learning (ML) methods started appearing, and extracting
features from the images and using them in the algorithms became more usual.
Finally, over the last few years, deep learning (DL) has taken the central
place [24].

Complementary to previous machine learning methods, Deep Learning can learn
how to generate the features automatically. A Deep Learning model can perform
well using only raw data as input, without the necessity of a previously
manually engineered feature extraction section, even requiring less technical
knowledge about the problem in question to achieve baseline results [25].

In the medical imaging context, convolutional kernel parameters are the
weights adjusted in training. The application of these convolutional kernels
is invariant to translation, which allows learning from different positions
throughout the image. Optimizing the kernel weights takes place by minimizing
a loss function [26] expressing the model performance, through the application
of optimization methods using variations of stochastic gradient descent [27].
In addition, these convolutions are usually supported by normalization layers
[28] and nonlinear activations [29]. Normalization and non-linearity were some
of the main breakthroughs that allowed deep learning networks to thrive in
many applications, significantly accelerating the convergence of training. The
Deep Learning architectures that use these techniques are called Convolutional
Neural Networks (CNN). Note that CNNs are not the only way to use deep
learning for imaging applications, with Tranformers [30] also being a current
competitive architecture paradigm in natural and medical imaging processing.

There is an infinite amount of possible combinations of convolutional and
other types of layers to define a CNN architecture. However, some
combinations, or architectures, are already defined and well-known for certain
applications. For example, U-Net [13] is one of the most used architectures in
the field of medical imaging for image segmentation due to its compact
encoder-decoder design that propagates multi-resolution features from the
encoder to the decoder, allowing for fewer parameters when compared to famous
large natural image classification architectures such as ResNet [31],
EfficientNet [32], and ConvNext [33]. The encoder acts as a feature extractor
and the decoder as a reconstructor of the intended output. It is important to
stress that those features automatically extracted by the encoder are not
necessarily visually interpretable since they are abstract representations of
the input image that maximize a given objective. From the seminal UNet paper,
many variations of the encoder-decoder segmentation architecture have been
proposed [34, 35], and the encoder-decoder design continues to be the de-facto
approach for supervised automated medical imaging segmentation.

##  III Environment Recommendations

Software and hardware recommendations can be sensitive since different people
and groups will have different experiences. In this Section, we will recommend
what has worked for us in recent years, focusing on DL over medical imaging
processing. However, these recommendations can be extrapolated using different
processing methodologies. Firstly, regarding hardware, although having a good
amount of RAM and a good CPU helps, we recommend distributing your budget,
focusing on the best possible GPU with a decent amount of video memory. We
also suggest paying attention to how much storage your work needs since some
medical images can use a large amount of storage. If possible, Use a high-
speed (SSD) storage for preprocessed data and low-speed storage for
raw/original data.

From our experience, we recommend using Python and PyTorch as the programming
language and DL framework, respectively. PyTorch 111https://pytorch.org/
follows an object-oriented programming approach and is currently being used by
a large part of the DL community. A top-level framework that simplifies some
of the “engineering” code necessary to use PyTorch is PyTorch Lightning
222https://pytorchlightning.ai/. The libraries you will use for processing
will vary depending on the needs of your project. Some libraries commonly used
for reading and dealing with medical imaging include NumPy, SimpleITK,
Nibabel, and Pydicom. Both Windows and Ubuntu work well with Python in terms
of operational systems. Jupyter Notebooks are a good tool for a more
interactive programming approach, especially for prototyping and proof of
concept. As a workflow, having separate Python scripts that are imported in a
jupyter notebook or a main command line script is interesting for controlling
input arguments. Finally, we recommend that you use logging frameworks for
experiments and do not implement logging from scratch. Many useful tools are
available to log experiment parameters and results, and they are essential for
organizing your research and recalling experiments in the future. Some
examples include Neptune.ai333https://neptune.ai/ and
TensorBoard444https://www.tensorflow.org/tensorboard.

Storage requirements are one of the main things to keep in mind when dealing
with digital medical imaging. Unlike other imaging processing areas that deal
with 2D images, medical imaging is frequently three-dimensional, with high
resolution. This results in uncompressed storage of one volume, sometimes
using hundreds of megabytes of space. Therefore, you need to reserve gigabytes
or even terabytes of space to store both original and preprocessed copies of
your data. We recommend that preprocessed data be stored in fast storage for
faster processing, such as SSDs, and original data be stored in slower HDDs or
even a separate computer. For deep learning training, using GPUs is mandatory,
given that the training process consists of parallel multiplications and sums
that can be performed optimally in GPUs. We recommend using a modern Nvidia
GPU with CUDA support, due to it being the norm in the field currently, with
most frameworks using CUDA. For RAM, We recommend a minimum of 16 GB of RAM,
with more being beneficial especially for training 3D networks. Having more
RAM is not essential but beneficial, increasing the amount of data that can be
cached during training, and reducing the possibility of bottlenecks in
parallel data loading. There is no need to have the most expensive CPU, but a
top tier CPU will be of benefit for data processing and loading during
training. Finally, it is very important to highlight that the GPU usage during
training should be close to 100%. If not, that means that data loading is a
bottleneck, either due to slow readings from storage, slow/not enough RAM, or
a slow CPU. Always pay attention to hardware cooling as modern Nvidia GPUs
should stay below 83 degrees Celsius while training. Finally, try to use
parallel data loading and optimize your code. Badly optimized data loading
code can bring even the most expensive hardware to a halt in training.

##  IV Workflow

This section will go through recommendations and tips on all steps of the DL
for medical imaging analysis workflow (Fig. 5).

Figure 5: Illustration of a generic workflow for training DL models for
medical imaging analysis, from data pre-processing to model training and
evaluation.

###  IV-A Data

Dataset | Annotation | Description | Number of images
---|---|---|---
CT
LTRC [36] | – | Lung Tissue Research Consortium | 1200 patients
LIDC-IDRI[37] | Nodule Detection | Lung Image Database Consortium Image Collection | 1018 scans
LOLA11[38] | Lobe masks (external) | LObe and Lung Analysis Challenge | 55 scans
LUNA16[39] | Pulmonary nodules | Challenge for pulmonary nodule segmentation | 888 scans
IEE CCAP[40] | – | COVID-19 low dose scans | 154 scans
MSC[41] | Lung and COVID-19 findings | COVID-19 patients scans | 100 scans
MOSMED[42] | COVID-19 findings | COVID-19 patients scans | 860 slices
CoronaCases [43] | Lung and COVID-19 findings | COPD and COVID-19 patients scans | 20 scans
MSD [43] | Pulmonary nodules segmentation | Challenge with multiple tasks including lung cancer | 2633 slices
MEDGift[44] | ILD annotations | Multimedia dataset of ILD cases | 128 patients
Empire10[45] | – | Registration of thoracic CT data | 30 scan pairs
VESSEL12[46] | Lung and vessel | VESsel SEgmentation in the Lung Challenge | 30 scans
VISCERAL[47] | Various modalities including lung Lung | Visual Concept Extraction Challenge in Radiology | 80 volumes
DSB[48] | Cancer | Data Science Bowl 2017 Kaggle Competition | 2101 scans
Kaggle[49] | Lung annotations | Kaggle lung segmentation challenge | 267 slices
NSCLC[50] | Pulmonary nodule annotations | Non-small cell lung cancer patients | 422 patients
EXACT09[51] | Airway annotations | Extraction of Airways from CT | 40 scans
ImageCLEF[52] | Lung masks | Tuberculosis severity scoring challenge | 335 scans
SARS-CoV-2[53] | Lung | COVID-19 patients scans | 2482 scans
TCIA[54] | Lung and COVID-19 findings | COVID-19 patients scans | 753 patientes
MRI
ADNI [55] | – | Patient (Alzheimer and mild cognitive impairment) and controls | 1821 scans
OASIS [56] | – | T1-w, T2-w, FLAIR, ASL, SWI, time of flight, resting-state BOLD, and DTI |  $>$2000 scans
HCP [57] | – | Human Connectome Project: Structural and diffusion data | 1200 patients
IXI [58] | – | Information eXtraction from Images | 581 patients
ABIDE [59] | – | Autism Brain Imaging Data Exchange | 1112 patients
Kirby21 [60] | – | Test-retest dataset on 21 subjects | 42 scans
HARP [61] | Hippocampus mask | EADC-ADNI Harmonized Protocol for manual hippocampal segmentation on magnetic resonance | 135 patients
CC359 [62] | Automated and manual Skull Stripping | Healthy subjects T1-w MRI | 359 patients
BRATS [63] | Brain tumor manual segmentation | The multimodal brain tumor image segmentation benchmark | 65 scans
MiLI [64] | Automated and Manual hypothalamus masks | MICLab-LNI Initiative: T1-w MRI | 452 patients
Thalamus  | Automated and Manual thalamus masks | T1-w and Diffusion MRI | 1063 patients
LBPA40 [65] | 56 brain structure masks | 3D probabilistic atlas of human cortical structures | 40 patients
NFBS [66] | Manual skull stripping | Neurofeedback Skull-stripped: T1-w MRI | 125 scans
IBSR [67] | 43 structures of the brain | Internet Brain Segmentation Repository | 18 patients
LGG [68] | Manual FLAIR abnormality segmentation masks | Brain MRI from patients with lower-grade gliomas | 110 patients
IBSD[69] | Subcortical segmentation | Indian Brain Segmentation Dataset | 114 patients

TABLE I: Examples of public datasets of MRI or CT scans, some with included
voxel-wise segmentation annotations.

Data curation is one of the most important steps of the training workflow.
Here, we list five steps you need to consider before training your network.

####  IV-A1 Understanding your problem and your data

The first thing you need to do is to understand the data you are using. For
instance, it will be hard to classify lung lesions in CT images if you cannot
locate the lung or if you have not heard about Hounsfield units. The more you
understand the problem, the easier it is to find computational solutions for
it. For example, if you are trying to segment a small structure, you may need
to use some specific loss function or pay attention to the depth of the
network. In this first step, you need to talk with physicians, radiologists,
and specialists about the problem you are working on. You need to understand
the primary goal of your project. Are you trying to decrease the processing
time of existing methods, to improve metrics regardless of the processing
time, or maybe, to develop a more generic application, even if it means a loss
in performance? For instance, if your application is meant to work only in one
imaging center, always using the same type of acquisition, you may train your
network using only one dataset. However, if you intend to develop an
application that can be used by different research centers, it is imperative
to mix different data sources to increase generalization.

By understanding your problem early in development, you will be able to find
the most effective data for your application and avoid wasting time re-
training in the future.

####  IV-A2 Labels

Here, you need to define what will be your ground truth. Usually, semi-
supervised and supervised applications will require trustful labels to deliver
good predictions. For medical images, the gold standard is usually defined as
a specialist annotation. Also, the ideal scenario is to have more than one
specialist annotating your data to avoid bias. However, the cost of manual
labels is high in terms of both time and money. So, the first thing you need
to do before starting a project is to analyze if you have data and annotation.

In terms of segmentation, if you have only a few labeled data, you may try
some tricks:

  * •

Silver Standards: Manual annotation is time-consuming, so you may create a
dataset using established automated methods (Tab. II). To deal with different
types of segmentation and reduce label noise, it is recommended to use label-
cleaning strategies such as STAPLE and majority voting. Souza et al. [62]
defined this as silver standard labels.

  * •

Synthetic Images: Recent studies have proved the generalization ability of a
network trained with synthetic images. This means that by creating synthetic
datasets using GAN [70] or probabilistic models [71], your network can predict
on real images.

TABLE II:  Some examples of automated segmentation tools. Tool | Structure
---|---
FreeSurfer [72] | whole brain
FSL [73] | Brain tissues
BrainSuite [74] | skull stripping
VolBrain [75] | subcortical
SLANT [76] | whole brain
SynthSeg [77] | whole brain
FastSurfer [78] | whole brain
QuickNAT [79] | whole brain
DeepNAT [80] | whole brain
Pulmonary Toolkit 555https://github.com/tomdoel/pulmonarytoolkit | lung
Lungmask [81] | lung
MEDSeg [11] | airway, lung and findings

####  IV-A3 Checking the data

You defined your problem, talked to physicians, and understood what data you
need. Now it is time to look at it. Once again, It may sound obvious, however,
if you are working with medical imaging, you must visually inspect the images
before training. There are three moments during your application development
when you need to look at a sample of the data. First, before any
preprocessing. Check your raw data, see if the labels are coherent and if the
images follow the same orientation acquisition. If you are dealing with
diverse datasets, check the difference between them. Use this time to
understand which preprocessing algorithms you will need to run on your image
and to analyze if you understand the structure(s) you are dealing with. You
may use visualization tools such as: ITK-SNAP [82]; Freeview [72]; 3D Slicer
[83]; or DSI Studio [84] to see your images; or load them on your python
script, using libraries such as: Nibabel [85]; PyDicom
666https://pydicom.github.io/; SimpleITK [86]; or MedPY
777https://loli.github.io/medpy/. If you have doubts about the labels, this is
the moment to clarify them, or you may have to train your network again.
Second, look at your data after preprocessing. You must ensure that the
network will receive exactly what you intended to deliver. Finally, check your
predictions according to the input data, if the metrics make sense, or if you
need some post-processing algorithm to improve your results.

####  IV-A4 Ethics on data

It is crucial to understand that medical imaging is sensitive data, especially
if you deal with patient images and/or pediatric images. To use the data, you
need permission from the patients or from their legal representatives. It is
imperative to anonymize the data by hiding personal information. Also, it is
interesting to run a defacing algorithm in some cases to prevent face
reconstruction from 3D images [87]. If you are using a public dataset, it is
important to check if the data attend the ethics committee requirements. You
will also need permission from the owner to use the data.

####  IV-A5 Preprocessing

Before inserting the image into the model, several pre-processing steps can be
done to the image, such as bias field correction, registration, skull
stripping (in the case of brain images), normalization, and clipping.
Registration is required to place multiple images in the same space,
especially when working with multi-modality data, and it can be inter or
intra-subject. For example, T1-weighted and Diffusion MR images usually have
different resolutions and sometimes even acquisition directions. The
registration process will transport one of the images to the space of the
other, making each voxel have an exact correspondence in both images. A few of
the most used tools to perform the registration are FSL, FreeSurfer, and Dipy.
They are able to perform linear registration, a method that can basically
manipulate the scales and orientation of the images, and nonlinear
registration, which can elastically deform the images to adjust for punctual
deformations. Intensity inhomogeneity, or bias field, is a low-frequency noise
that represents intensity variability on a tissue and is caused during the
acquisition process of the MR image [88]. Hence, as preprocessing, it is often
necessary to apply a bias field correction such as the nonuniform
normalization (N3) algorithm [89]. Skull-stripping (SS), on the other hand, is
a step that separates brain tissue from the skull in MR images of the head. It
is often used when the skull worsens the task results and it is mandatory when
comparing brain structure volumes since the volumes need to be normalized by
the total intracranial volume. When working with CT, clipping usually is a
common pre-processing step. It reduces the gray level window to a specific
intensity range highlighting desired structures (Fig. 3). Finally, the network
input is commonly normalized to avoid exploding gradients - usually into the
[0,1] or [-1,1] range.

Figure 6: Example of CT image after clipping and normalization. On the left,
you can see the original image. On the center, the image after a [-1024,600]
clipping: It is possible to see changes in the image contrast. On the right,
image after minmax normalization: Visually, there are no changes, however, the
minimum and maximum intensity are different.

###  IV-B Model Training

####  IV-B1 Data Loading

After data checking and pre-processing are performed, you should have your
data stored in a preprocessed state somewhere, such as preprocessed 3D volumes
or preprocessed 2D images. Since medical images are usually multidimensional
and large and DL requires a significant amount of data, you will need to pay
attention to certain optimization tactics in data loading that are essential
for training to be feasible, mainly related to how you save and read those
preprocessed files. With some machine learning applications, you could have
all the data stored as a variable in RAM, and fit your model there, but that
is not viable with the large size of medical images. Therefore, your data
reading logic will need to save some kind of index to your images and read
each item in real-time during training. PyTorch recommends having a Dataset
class that manages indexing all your data files and application of data
augmentation if necessary and feeding that Dataset class to a Dataloader,
which optimizes the loading process using parallelization. The goal in DL
training is to use 100% of your GPU at all times. Therefore, the parallel
loading logic implemented by the dataloader is sometimes essential for optimal
training speeds. Monitoring your GPU usage during training is a good way to
check if your data loading is bottlenecking your training.

There are many ways to optimize storing and reading your pre-processed data,
and the exact way this will be done depends on the nature of your data, your
hardware environment, and your goals. For example, using compressed formats
such as .npz can lead to less storage use, but more CPU load in decompressing.
Uncompressed .h5py files can make organization easier and have less CPU usage,
but it can lead to the need for more storage. Even if you have unlimited
storage, using uncompressed formats such as .npy can be slower due to the slow
read speeds of slow hard drives. An example of how to store processed data
from CT slices for 2D training using compressed .npz is in our case studies
(Section VI), but should not be taken as the ”correct” way. Experimentation
must be done in your environment to determine the best storage and reading
format for your case.

####  IV-B2 Data Augmentation

When dealing with medical imaging processing, the generalization capacity of
the model is of major importance. Once we have several scan manufacturers and
different acquisition protocols, each database has its own particularities,
especially when compared with data from different medical centers. One could
train their model using only one dataset, however, this model would have a
loss of performance when applied to data from different centers. Focusing on
increasing image variability and improving the generalization of the methods,
most authors resort to data augmentation (Fig. 7). There are several types of
augmentation you may use in your method. It is really important to understand
which will suit you more or the wrong use of this technique may worsen your
results. A few examples of data augmentation focused on medical images are
random crop, random rotation, elastic transformation, noise insertion, and
intensity transformations (contrast, bright). When dealing with segmentation,
it is important to apply the same transforms on the images and labels. Some
data augmentation libraries may help, such as Torchvision, Albumentations, and
Torch.io.

Figure 7: Example of MR images after data augmentation (RandomRotation and
RandomCrop). In red, it is possible to see the label overlaying the image.

####  IV-B3 Deep Networks

Deep Networks are a class of Machine learning methods that are able to extract
the features of the data without the need for feature engineering done by a
specialist. Instead, the architecture is able to learn not only the task but
also the feature extraction on the first convolutional layers.

Usually, for medical images, specialists are focused on the application, not
on designing the architecture. Thankfully, there is a large amount of CNN
architectures available for general and medical imaging use, in both 2D and 3D
formats. Although intuitive, the 3D CNN approach is not always the best
performer compared to stacking 2D prediction of each slice. It is important to
emphasize that even 2D slices have a thickness in addition to the voxel planar
dimension, so the information of each voxel comes from a unity of volume.

We could split the CNN into many different groups, but in this tutorial, we
are restricted to two CNN types, one for segmentation and another for
classification. Each type of CNN has its characteristics accordingly to the
application it is performing. For example, architectures designed for
segmentation, usually fully convolutional (every layer is convolutional), are
able to generate an output image with the same dimensions as the input. For
this type of CNN, the ground truth is composed of masks of the same size as
the input images. On the other hand, architectures designed for classification
present a dense layer after the feature extraction section, and these dense
sections have one output for each of the considered classes. In these
problems, the ground truth is a single value that represents the class of the
input image.

In both types of applications, there are many available architectures. For
example, in segmentation, some of the most popular or most performing
architectures are the U-Net [13], V-Net [90] and QuickNAT [12]. For
applications focused on classification, some example architectures are the
Resnet [31], Inception [91], DenseNet [92], MobileNet [93] and EfficientNet
[32].

####  IV-B4 Loss Function

The loss function is defined by a relation between the current output of the
model and the desired output, defined by the ground truth. By computing the
loss function for the current state of the model, the training framework can
define the gradient directions in which the weights are going to be moved to
minimize the prediction error of the model.

The loss function must be properly defined depending on the problem the CNN is
solving. For segmentation problems, loss functions based on overlap measures
are commonly used [26].

It is important to notice that the loss function must be continuous in order
to be differentiable, allowing for the optimizer to minimize it, therefore
minimizing the error expressed by the loss. However, many problems would
require metrics that do not fit this requirement, so the loss function must be
properly defined depending on the problem the model is solving. For example,
some classification problems search for higher accuracy, which is a non-
differentiable function. In this case, a loss based on cross entropy is a
valid option. For segmentation problems, a similarity metric such as Dice
Coefficient [26] is not differentiable in its original set based definition.
Therefore, most Dice based losses alter the metric definition to allow for
probabilistic inputs to be compared with the binary groundtruth, which also
has the benefit of smoothing convergence. This version of Dice is commonly
called Dice Loss and can be defined as:

$DiceLoss=1-2\frac{\sum_{i}^{N}p_{i}t_{i}}{\sum_{i}^{N}p_{i}^{2}+\sum_{i}^{N}t_{i}^{2}}$
(1)

where $p$ is the probabilistic output of the network between 0 and 1, usually
from sigmoid or softmax activations, and $t$ are binary targets. Note that
minimizing equation (1) by definition maximizes the value of the Dice metric
(Section IV-D).

####  IV-B5 Optimizer

Neural network training performs adjustments to weights $w$ present in the
network, which in the case of a CNN are convolutional kernel values. These
weights are changed based on values returned by the loss functions, which
measure how ”wrong” an output is in relation to a target, which is assumed to
be the expected output. The gradient $\delta_{w}$ of a weight represents the
change in loss caused by a change in the weights, in other words, a
derivation, and guides the training process in the direction of minimizing the
loss. The gradient for each weight is commonly calculated using
backpropagation in the form of PyTorch’s Autograd [94]. The optimization
process is controlled by optimizers, with the most common ones being
Stochastic gradient descent (SGD) and Adaptative momentum estimation (ADAM).

With the SGD optimizer, the derivation of an updated weight $w_{t}$ for a
current discrete time $t$ as a function of past weight $w_{t-1}$ can be
expressed as:

$w_{t}=w_{t-1}-\alpha\Delta_{w}{L(O,T)}$ (2)

for a loss function $L$ over outputs $O$ and targets $T$.

SGD can also be implemented with momentum, where past weights are also taken
into consideration. The learning rate (LR) controls the speed of the
optimization process. Finding the correct learning rate is a key factor in
training a CNN. A high learning rate can make the model skip states where the
minimum loss is achieved, whereas a low learning rate can lead to very slow
convergence, and to the model being stuck in a local minimum. Some learning
rate schedulers change the learning rate during training with a function of
the number of epochs passed. Some other optimizers try to change the learning
rate adaptively, such as ADAM. ADAM computes a learning rate per parameter,
instead of using a global learning rate, and takes into consideration a moving
average of gradients. More details can be found in its original publication in
[95]. ADAM and its variations [96] are a good initial choice of optimizer due
to their adaptability. A weight update for ADAM can be expressed as:

$w_{t}=w_{t-1}-\eta\frac{m_{t}}{\sqrt{v_{t}}+\epsilon}$ (3)

where $\eta$ is the step-size, that can vary between iterations. $m_{t}$ and
$v_{t}$ are bias corrected first and second momentums, respectively. These
momentums are a function of gradients and the square of the gradients, also in
relation to an input and loss function. More details can be found in its
original publication in [95]. The usage of momentum and moving averages avoids
the next step to be completely determined by the current batch of data since
batches can be very randomized. The momentum keeps pointing to a general
direction of minimization, where the current gradient points to the
minimization for the current batch. The final step can be defined by a
combination of the two.

###  IV-C Post processing

Post-processing is the process of editing the network output to enhance the
results. There are several post-processing options based on the end result you
want to achieve. For instance, the analysis of connected components (CC) over
the three dimensions. By checking the CC on a segmentation output, it is
possible to remove miss-classified voxels (Fig. 8). The threshold analysis is
also used to improve a network outcome. It needs to be done on the validation
set and the main idea is to find a threshold that better improves the final
result. For this matter, it is necessary to vary the thresholding by analyzing
the metric you want to improve. This may be applied to different applications
such as classification and segmentation. Seung-lab
888https://github.com/seung-lab/connected-components-3d provides a useful 3D
connected components generation method which supports images containing many
different labels, not just binary images. It also supports continuously valued
images such as grayscale microscope images with an algorithm that joins
together nearby values. The benefit of this package is that it labels all
connected components in one shot, improving performance by one or more orders
of magnitude.

Figure 8: Prediction noise pointed by yellow arrows (left image) can be
reduced (right image) using post-processing algorithms such as CC filters.

###  IV-D Evaluation Metrics

Concerning segmentation models, different metrics should be used to evaluate
results due to their complementarity in expressing how well the method’s
output is similar to the ground truth [97]. Some examples include the Dice
coefficient (Dice), Hausdorff Distance (HD), Hausdorff Average Distance (AVD),
and Volume Simmetry, among others. Considering A as the model prediction and M
as the label, we may define:

####  IV-D1 Dice Coefficient

The DC is an overlap measure defined as follows:

$DC=\frac{2*\left|M\cap A\right|}{|M|+|A|}$ (4)

DC is sensitive to small segmentation and does not identify boundary errors.
However, it can be used as a measure of reproducibility and is widely used for
medical imaging segmentation analysis, being the most used metric in the
medical imaging segmentation field [97]. DC results may be in the [0,1] range,
where 1 a perfect DC

####  IV-D2 Hausdorff Distance

The HD measures the distance between two sets of points.

$HD(A,M)=max(h(A,M),h(M,A))$ (5)

where:

$h(A,M)=\max_{a\epsilon A}\min_{m\epsilon M}\left\|a-m\right\|$ (6)

####  IV-D3 Average Hausdorff Distance

As the name suggests, AVD is the averaged Hausdorff Distance over all points.

$AVD(A,M)=max(d(A,M),d(M,A))$ (7)

where:

$d(A,M)=\frac{1}{N}\sum_{a\epsilon A}\min_{m\epsilon M}\left\|a-m\right\|$ (8)

Similar to HD, it is also a spatial distance metric that is robust to small
structures. However, on average, it is less sensitive to outliers [97]. The
smaller the AVD between manual and automated segmentation, the better the
automated segmentation.

####  IV-D4 Volume Similarity

Finally, the VS calculates the similarity between the two samples.

$VS=1-\frac{\left|\left|A\right|-\left|M\right|\right|}{\left|A\right|+\left|M\right|}$
(9)

being $\left|X\right|$ the module of X. Although it ignores borders and
overlaps, VS is a good metric for analyzing the segmentation volume when
determining the volume of the structure is the main goal [97]. VS results may
be in a [0,1] range, being 1 a perfect VS

##  V Statistic Analysis

Statistical analysis is an important task to analyze the reliability of the
obtained results obtained. The main idea is to compare groups and verify if
they are statistically equivalent or different, thus obtaining a significant
answer or not for the problem in question.

At first, it is important to stipulate which hypotheses are being tested. In a
hypothesis test, the Null Hypothesis ($H_{0}$) is the equality hypothesis,
which argues that the groups being analyzed are statistically equal. The
alternative hypothesis, ($H_{1}$), is the complement of the null hypothesis
and states that the effect between the groups under study exists, that is,
that the groups are statistically different.

A statistical test can be one-tailed or two-tailed. In the two-tailed test,
the alternative hypothesis is a hypothesis of inequality ($\neq$), not taking
into account if values are less or greater ($<$ or $>$).

There are several statistical tests and the correct choice will depend on the
analysis you want to perform. Our focus will be to describe step by step how
to perform a statistical test for two groups with quantitative response
variables, which are the most used analyses in medical data.

###  V-A Check if the groups are independent or dependent

Two groups are independent (unpaired) if the sample selected from one of the
populations is not related to the sample selected from the second population.
For example, if we want to compare patients $\times$ control.

Two groups are dependent (paired) if each member in a sample correspondent
with a member from another sample. For example, collected data from the same
cohort, before and after treatment.

###  V-B Check data normality

To perform a confident analysis, it is important to use the statistical test
that best represents the data, so before choosing the test it is important to
check the distribution of the data. Normality is a characteristic of the data
in which the majority (higher frequency) of the sample values are close to the
mean value of all samples. Normality can be visualized through the scatter
histogram and the box-plot plot. The data are normally distributed if their
scatter histogram is bell-shaped. A boxplot with many outliers is a
characteristic of data without distribution, that is, of non-parametric data.
However, it is often difficult to define normality just by visualizing the
data. According to [98], the most powerful statistical test to verify data
normality is the Shapiro Wilk test. The Shapiro Wilk test uses the following
hypothesis:

$H_{0}$ : data distribution = normal $\rightarrow$ $p>\alpha$

$H_{1}$ : data distribution $\neq$ normal $\rightarrow$ $p\leq\alpha$ where
alpha is the significance level stipulated as a priory, a limit established
for the rejection or note null hypothesis based on the value of p. In the
medical area, it is often used $\alpha=0.05$.

###  V-C Use the appropriate statistical test to compare two groups

Parametric tests require a normal data distribution, while non-parametric
tests do not require normal distribution.

####  V-C1 Independent t test:

The t-test for two independent samples (also called independent Student’s
t-test), allows comparing means between two independent groups. The dependent
variable is numerical with normal distribution and the independent variable is
categorical with two categories. Before performing the independent t-test, it
is important to verify whether the variances of the groups are homogeneous or
not using the Levene test [99]. The hypotheses of the independent t-test are:

$H_{0}$ : mean of the groups are same $\rightarrow$ $p>\alpha$

$H_{1}$ : mean of the groups are different $\rightarrow$ $p\leq\alpha$

####  V-C2 Mann-Whitney test:

Mann-Whitney test compares two independent groups. It is a non-parametric
alternative to the independent t-test. The Mann-Whitney test is used for data
that do not have a normal distribution. That is, the mean is not a good
representation of the data set and the median is the measure that best
represents the data [100]. The test hypotheses are:

$H_{0}$ : median of the groups are same $\rightarrow$ $p>\alpha$

$H_{1}$ : median of the groups are different $\rightarrow$ $p\leq\alpha$

An important observation is that we can find the Mann-Whitey test as a test
that compares the medians, but deep down, it is not just comparing the medians
of the two groups, it is comparing the distributions [101]. It may happen that
the same median is found in both groups under study and the Mann-Whitney test
is significant, that is, this significance indicates that there is a
difference in the distributions.

####  V-C3 Paired t test:

Paired Student’s t-test is a type of hypothesis test that allows comparing the
mean of two paired groups. The dependent variable must be numerical, and the
independent variable is composed of two paired groups. To use this test the
data must be represented by a normal or approximately normal distribution. The
paired t-test hypotheses are the same as the independent t-test.

####  V-C4 Wilcoxon test:

The Wilcoxon test, also known as the Wilcoxon signed-rank test, is based on
ranks and allows comparing two paired samples[102]. It is a nonparametric test
corresponding to the paired t-test. This method considers the size of the
differences in the case under study. The Wilcoxon signed-rank test is used to
test differences in population. They are generally used for data that do not
have a normal distribution. This type of data is rarely well represented by
the mean and the median is the measure of central tendency that best
represents the data. The hypotheses of this test are the same as in the Mann
Whitney test.

##  VI Case Studies

We have prepared a practical demonstration of most principles showcased in
this paper for a CT and an MR image case using public data [103, 104] in the
following repository: https://github.com/MICLab-Unicamp/Medical-Imaging-
Tutorial.

##  VII Conclusion

In this manuscript we have gone through a brief introduction to the field of
medical imaging analysis with deep learning, focusing on segmentation of
images. A workflow guidance is provided, with tips and good practices for all
phases commonly present in this field of research. In addition, we provide
hands-on examples using public data, which will be used in our tutorial
session.

## Acknowledgment

We thank MICLab students Álvaro Capelo, Beatriz Vicente, Bruno Santos, Gabriel
Dias, Jean Ribeiro, and Joany Rodrigues, for participating in our internal
seminars. Livia Rodrigues and Thays Abreu thanks the Higher Education
Personnel Improvement Coordination (CAPES). Diedre Carmo thanks grant
#2019/21964-4, São Paulo Research Foundation (FAPESP). Gustavo Pinheiro,
Roberto Lotufo, and Leticia Rittner thank the National Scientific and
Technological Development Council (CNPq).

## References

  * [1] P. C. Lauterbur, “Image formation by induced local interactions: examples employing nuclear magnetic resonance,” _nature_ , vol. 242, no. 5394, pp. 190–191, 1973.
  * [2] R. R. Edelman, “The history of MR imaging as seen through the pages of radiology,” _Radiology_ , vol. 273, no. 2S, pp. S181–S200, 2014.
  * [3] V. Kaul, S. Enslin, and S. A. Gross, “History of artificial intelligence in medicine,” _Gastrointestinal endoscopy_ , vol. 92, no. 4, pp. 807–812, 2020\.
  * [4] N. F. Aurna, M. A. Yousuf, K. A. Taher, A. Azad, and M. A. Moni, “A classification of MRI brain tumor based on two stage feature level ensemble of deep cnn models,” _Computers in Biology and Medicine_ , vol. 146, p. 105539, 2022.
  * [5] S. Deepak and P. Ameer, “Brain tumor classification using deep CNN features via transfer learning,” _Computers in biology and medicine_ , vol. 111, p. 103345, 2019.
  * [6] D. Carmo, I. Campiotti, L. Rodrigues, I. Fantini, G. Pinheiro, D. Moraes, R. Nogueira, L. Rittner, and R. Lotufo, “Rapidly deploying a covid-19 decision support system in one of the largest brazilian hospitals,” _Health Informatics Journal_ , vol. 27, no. 3, p. 14604582211033017, 2021\.
  * [7] R. Gauriau, R. Cuingnet, D. Lesage, and I. Bloch, “Multi-organ localization with cascaded global-to-local regression and shape prior,” _Medical image analysis_ , vol. 23, no. 1, pp. 70–83, 2015.
  * [8] J. M. Noothout, B. D. De Vos, J. M. Wolterink, E. M. Postma, P. A. Smeets, R. A. Takx, T. Leiner, M. A. Viergever, and I. Išgum, “Deep learning-based regression and classification for automatic landmark localization in medical images,” _IEEE transactions on medical imaging_ , vol. 39, no. 12, pp. 4011–4022, 2020.
  * [9] Y. Beauferris, J. Teuwen, D. Karkalousos, N. Moriakov, M. Caan, G. Yiasemis, L. Rodrigues, A. Lopes, H. Pedrini, L. Rittner _et al._ , “Multi-coil mri reconstruction challenge—assessing brain mri reconstruction models and their generalizability to varying coil configurations,” _Frontiers in Neuroscience_ , vol. 16, p. 919186, 2022.
  * [10] G. R. Pinheiro, R. Voltoline, M. Bento, and L. Rittner, “V-net and u-net for ischemic stroke lesion segmentation in a small dataset of perfusion data,” in _International MICCAI Brainlesion Workshop_. Springer, 2018, pp. 301–309.
  * [11] D. Carmo, I. Campiotti, I. Fantini, L. Rodrigues, L. Rittner, and R. Lotufo, “Multitasking segmentation of lung and covid-19 findings in ct scans using modified efficientdet, unet and mobilenetv3 models,” in _17th International Symposium on Medical Information Processing and Analysis_ , vol. 12088\. SPIE, 2021, pp. 65–74.
  * [12] A. G. Roy, S. Conjeti, N. Navab, C. Wachinger, A. D. N. Initiative _et al._ , “QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy,” _NeuroImage_ , vol. 186, pp. 713–727, 2019.
  * [13] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [14] D. Carmo, B. Silva, C. Yasuda, L. Rittner, R. Lotufo, A. D. N. Initiative _et al._ , “Hippocampus segmentation on epilepsy and alzheimer’s disease studies with multiple convolutional neural networks,” _Heliyon_ , vol. 7, no. 2, p. e06226, 2021.
  * [15] L. Rodrigues, T. Rezende, A. Zanesco, A. L. Hernandez, M. Franca, and L. Rittner, “Hypothalamus fully automatic segmentation from mr images using a u-net based architecture,” in _15th International Symposium on Medical Information Processing and Analysis_ , vol. 11330. SPIE, 2020, pp. 144–150.
  * [16] M. M. Rahman, L. Duerselen, and A. M. Seitz, “Automatic segmentation of knee menisci - A systematic review,” _Artificial Intelligence in Medicine_ , vol. 105, p. 101849, 2020.
  * [17] W. Tang, D. Zou, S. Yang, J. Shi, J. Dan, and G. Song, “A two-stage approach for automatic liver segmentation with faster R-CNN and DeepLab,” _Neural Computing and Applications_ , vol. 32, no. 11, pp. 6769–6778, 2020\.
  * [18] T. M. Buzug, “Computed tomography,” in _Springer handbook of medical technology_. Springer, 2011, pp. 311–342.
  * [19] K. Suzuki, “Overview of deep learning in medical imaging,” _Radiological physics and technology_ , vol. 10, no. 3, pp. 257–273, 2017.
  * [20] I. D. Kilsdonk, M. P. Wattjes, A. Lopez-Soriano, J. Kuijer, M. C. de Jong, W. L. de Graaf, M. Conijn, C. H. Polman, P. R. Luijten, J. J. Geurts _et al._ , “Improved differentiation between MS and vascular brain lesions using FLAIR* at 7 tesla,” _European radiology_ , vol. 24, no. 4, pp. 841–849, 2014.
  * [21] P. Basser and C. Pierpaoli, “Diffusion tensor MRI: A new tool for elucidating tissue microstructure and organization,” _ASME-PUBLICATIONS-BED_ , vol. 29, pp. 367–367, 1995.
  * [22] S. A. Huettel, A. W. Song, G. McCarthy _et al._ , _Functional magnetic resonance imaging_. Sinauer Associates Sunderland, 2004, vol. 1.
  * [23] R. K. Harris, _Nuclear magnetic resonance spectroscopy_. John Wiley and Sons Inc., New York, NY, 1986.
  * [24] D. Carmo, J. Ribeiro, S. Dertkigil, S. Appenzeller, R. Lotufo, and L. Rittner, “A systematic review of automated segmentation methods and public datasets for the lung and its lobes and findings on computed tomography images,” _Yearbook of Medical Informatics_ , vol. 31, no. 01, pp. 277–295, 2022.
  * [25] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” _nature_ , vol. 521, no. 7553, pp. 436–444, 2015.
  * [26] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. Jorge Cardoso, “Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2017, pp. 240–248.
  * [27] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _ICLR (Poster)_ , 2015. [Online]. Available: http://arxiv.org/abs/1412.6980
  * [28] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _CoRR_ , vol. abs/1502.03167, 2015. [Online]. Available: http://arxiv.org/abs/1502.03167
  * [29] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” _arXiv preprint arXiv:1505.00853_ , 2015.
  * [30] F. Shamshad, S. Khan, S. W. Zamir, M. H. Khan, M. Hayat, F. S. Khan, and H. Fu, “Transformers in medical imaging: A survey,” _arXiv preprint arXiv:2201.09873_ , 2022.
  * [31] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [32] M. Tan and Q. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in _International conference on machine learning_. PMLR, 2019, pp. 6105–6114.
  * [33] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A convnet for the 2020s,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 11 976–11 986.
  * [34] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [35] A. Abdollahi, B. Pradhan, and A. Alamri, “Vnet: An end-to-end fully convolutional neural network for road extraction from high-resolution remote sensing data,” _IEEE Access_ , vol. 8, pp. 179 424–179 436, 2020.
  * [36] R. A. Karwoski, B. Bartholmai, V. A. Zavaletta, D. Holmes, and R. A. Robb, “Processing of ct images for analysis of diffuse lung disease in the lung tissue research consortium,” in _Medical imaging 2008: physiology, function, and structure from medical images_ , vol. 6916. SPIE, 2008, pp. 356–364.
  * [37] S. Armato, “rd, mclennan g, bidaut l, mcnitt-gray mf, meyer cr, reeves ap, et al. the lung image database consortium (lidc) and image database resource initiative (idri): A completed reference database of lung nodules on ct scans,” _Med Phys_ , vol. 38, no. 2, pp. 915–31, 2011.
  * [38] “LOLA11,” https://lola11.grand-challenge.org/.
  * [39] “LUNA16,” https://luna16.grand-challenge.org/.
  * [40] T. Yan, “Ccap,” _IEEE Dataport_ , 2020.
  * [41] “MSL,” http://medicalsegmentation.com/covid19/,https://www.medseg.ai/.
  * [42] S. P. Morozov, A. Andreychenko, N. Pavlov, A. Vladzymyrskyy, N. Ledikhova, V. Gombolevskiy, I. A. Blokhin, P. Gelezhe, A. Gonchar, and V. Y. Chernina, “Mosmeddata: Chest ct scans with covid-19 related findings dataset,” _arXiv preprint arXiv:2005.06465_ , 2020.
  * [43] M. Jun, G. Cheng, W. Yixin, A. Xingle, G. Jiantao, Y. Ziqi, Z. Minqing, L. Xin, D. Xueyuan, C. Shucheng _et al._ , “Covid-19 ct lung and infection segmentation dataset. 2020,” 2020.
  * [44] A. Kumar, S. Agarwala, A. K. Dhara, D. Nandi, S. B. Thakur, A. K. Bhadra, and A. Sadhu, “Segmentation of lung field in hrct images using u-net based fully convolutional networks,” in _Annual Conference on Medical Image Understanding and Analysis_. Springer, 2018, pp. 84–93.
  * [45] K. Murphy, B. Van Ginneken, J. M. Reinhardt, S. Kabus, K. Ding, X. Deng, K. Cao, K. Du, G. E. Christensen, V. Garcia _et al._ , “Evaluation of registration methods on thoracic ct: the empire10 challenge,” _IEEE transactions on medical imaging_ , vol. 30, no. 11, pp. 1901–1920, 2011.
  * [46] “VESSEL12,” https://vessel12.grand-challenge.org.
  * [47] G. Langs, A. Hanbury, B. Menze, and H. Müller, “Visceral: towards large data in medical imaging—challenges and directions,” in _MICCAI international workshop on medical content-based retrieval for clinical decision support_. Springer, 2012, pp. 92–98.
  * [48] “DSB,” https://www.kaggle.com/c/data-science-bowl-2017.
  * [49] “Kaggle,” https://www.kaggle.com/kmader/finding-lungs-in-ct-data.
  * [50] J. Wang and H. Guo, “Automatic approach for lung segmentation with juxta-pleural nodules from thoracic ct based on contour tracing and correction,” _Computational and mathematical methods in medicine_ , vol. 2016, 2016.
  * [51] P. Lo, B. Van Ginneken, J. M. Reinhardt, T. Yavarna, P. A. De Jong, B. Irving, C. Fetita, M. Ortner, R. Pinho, J. Sijbers _et al._ , “Extraction of airways from ct (exact’09),” _IEEE Transactions on Medical Imaging_ , vol. 31, no. 11, pp. 2093–2107, 2012.
  * [52] Y. D. Cid, O. A. J. Del Toro, A. Depeursinge, and H. Müller, “Efficient and fully automatic segmentation of the lungs in ct volumes.” in _VISCERAL Challenge@ ISBI_ , 2015, pp. 31–35.
  * [53] E. Soares, P. Angelov, S. Biaso, M. H. Froes, and D. K. Abe, “Explainable-by-design approach for covid-19 classification via ct-scan,” 2020\.
  * [54] “TCIA,” https://wiki.cancerimagingarchive.net/display/Public/CT+Images+in+COVID-19.
  * [55] C. R. Jack Jr, M. A. Bernstein, N. C. Fox, P. Thompson, G. Alexander, D. Harvey, B. Borowski, P. J. Britson, J. L. Whitwell, C. Ward _et al._ , “The alzheimer’s disease neuroimaging initiative (adni): Mri methods,” _Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine_ , vol. 27, no. 4, pp. 685–691, 2008.
  * [56] P. J. LaMontagne, T. L. Benzinger, J. C. Morris, S. Keefe, R. Hornbeck, C. Xiong, E. Grant, J. Hassenstab, K. Moulder, A. Vlassenko _et al._ , “OASIS-3: longitudinal neuroimaging, clinical, and cognitive dataset for normal aging and Alzheimer disease,” _MedRxiv_ , 2019.
  * [57] D. C. Van Essen, K. Ugurbil, E. Auerbach, D. Barch, T. E. Behrens, R. Bucholz, A. Chang, L. Chen, M. Corbetta, S. W. Curtiss _et al._ , “The human connectome project: a data acquisition perspective,” _Neuroimage_ , vol. 62, no. 4, pp. 2222–2231, 2012.
  * [58] “IXI Dataset,” https://brain-development.org/ixi-dataset/, accessed: 2021-06-12.
  * [59] A. Di Martino, C.-G. Yan, Q. Li, E. Denio, F. X. Castellanos, K. Alaerts, J. S. Anderson, M. Assaf, S. Y. Bookheimer, M. Dapretto _et al._ , “The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism,” _Molecular psychiatry_ , vol. 19, no. 6, pp. 659–667, 2014.
  * [60] B. A. Landman, A. J. Huang, A. Gifford, D. S. Vikram, I. A. L. Lim, J. A. Farrell, J. A. Bogovic, J. Hua, M. Chen, S. Jarso _et al._ , “Multi-parametric neuroimaging reproducibility: a 3-T resource study,” _NeuroImage_ , vol. 54, no. 4, pp. 2854–2866, 2011.
  * [61] G. B. Frisoni, C. R. Jack Jr, M. Bocchetta, C. Bauer, K. S. Frederiksen, Y. Liu, G. Preboske, T. Swihart, M. Blair, E. Cavedo _et al._ , “The eadc-adni harmonized protocol for manual hippocampal segmentation on magnetic resonance: Evidence of validity,” _Alzheimer’s & Dementia_, vol. 11, no. 2, pp. 111–125, 2015.
  * [62] R. Souza, O. Lucena, J. Garrafa, D. Gobbi, M. Saluzzi, S. Appenzeller, L. Rittner, R. Frayne, and R. Lotufo, “An open, multi-vendor, multi-field-strength brain mr dataset and analysis of publicly available skull stripping methods agreement,” _NeuroImage_ , vol. 170, pp. 482–494, 2018.
  * [63] B. H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Burren, N. Porz, J. Slotboom, R. Wiest _et al._ , “The multimodal brain tumor image segmentation benchmark (brats),” _IEEE transactions on medical imaging_ , vol. 34, no. 10, pp. 1993–2024, 2014.
  * [64] L. Rodrigues, T. Rezende, G. Wertheimer, Y. Santos, M. Franca, and L. Rittner, “A benchmark for hypothalamus segmentation on t1-weighted mr images,” _NeuroImage_ , p. 119741, 2022.
  * [65] D. W. Shattuck, M. Mirza, V. Adisetiyo, C. Hojatkashani, G. Salamon, K. L. Narr, R. A. Poldrack, R. M. Bilder, and A. W. Toga, “Construction of a 3d probabilistic atlas of human cortical structures,” _Neuroimage_ , vol. 39, no. 3, pp. 1064–1080, 2008.
  * [66] B. Puccio, J. P. Pooley, J. S. Pellman, E. C. Taverna, and R. C. Craddock, “The preprocessed connectomes project repository of manually corrected skull-stripped t1-weighted anatomical mri data,” _Gigascience_ , vol. 5, no. 1, pp. s13 742–016, 2016.
  * [67] J. Frazier, V. Caviness, D. Kennedy, A. Worth, C. Haselgrove, D. Caplan, and N. Makris, “Internet brain segmentation repository (ibsr) 1.5 mm dataset,” _Collections_ , vol. 10, no. C60W3M, p. C6RC85, 1910.
  * [68] M. Buda, A. Saha, and M. A. Mazurowski, “Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm,” _Computers in biology and medicine_ , vol. 109, pp. 218–225, 2019.
  * [69] J. Sivaswamy, A. J. Thottupattu, R. Mehta, R. Sheelakumari, C. Kesavadas _et al._ , “Sub-cortical structure segmentation database for young population,” _arXiv preprint arXiv:2111.01561_ , 2021.
  * [70] V. Thambawita, P. Salehi, S. A. Sheshkal, S. A. Hicks, H. L. Hammer, S. Parasa, T. d. Lange, P. Halvorsen, and M. A. Riegler, “Singan-seg: Synthetic training data generation for medical image segmentation,” _PloS one_ , vol. 17, no. 5, p. e0267976, 2022.
  * [71] B. Billot, D. N. Greve, O. Puonti, A. Thielscher, K. Van Leemput, B. Fischl, A. V. Dalca, and J. E. Iglesias, “Synthseg: Domain Randomisation for Segmentation of Brain MRI Scans of any Contrast and Resolution,” _arXiv:2107.09559 [cs]_ , 2021.
  * [72] B. Fischl, “Freesurfer,” _Neuroimage_ , vol. 62, no. 2, pp. 774–781, 2012\.
  * [73] M. Jenkinson, C. F. Beckmann, T. E. Behrens, M. W. Woolrich, and S. M. Smith, “Fsl,” _Neuroimage_ , vol. 62, no. 2, pp. 782–790, 2012.
  * [74] D. W. Shattuck and R. M. Leahy, “Brainsuite: an automated cortical surface identification tool,” _Medical image analysis_ , vol. 6, no. 2, pp. 129–142, 2002.
  * [75] J. V. Manjón and P. Coupé, “volbrain: an online mri brain volumetry system,” _Frontiers in neuroinformatics_ , vol. 10, p. 30, 2016.
  * [76] Y. Huo, Z. Xu, Y. Xiong, K. Aboud, P. Parvathaneni, S. Bao, C. Bermudez, S. M. Resnick, L. E. Cutting, and B. A. Landman, “3d whole brain segmentation using spatially localized atlas network tiles,” _NeuroImage_ , vol. 194, pp. 105–119, 2019.
  * [77] B. Billot, D. N. Greve, O. Puonti, A. Thielscher, K. Van Leemput, B. Fischl, A. V. Dalca, and J. E. Iglesias, “Synthseg: Domain randomisation for segmentation of brain mri scans of any contrast and resolution,” _arXiv preprint arXiv:2107.09559_ , 2021.
  * [78] L. Henschel, S. Conjeti, S. Estrada, K. Diers, B. Fischl, and M. Reuter, “Fastsurfer-a fast and accurate deep learning based neuroimaging pipeline,” _NeuroImage_ , vol. 219, p. 117012, 2020.
  * [79] A. G. Roy, S. Conjeti, N. Navab, C. Wachinger, A. D. N. Initiative _et al._ , “Quicknat: A fully convolutional network for quick and accurate segmentation of neuroanatomy,” _NeuroImage_ , vol. 186, pp. 713–727, 2019.
  * [80] C. Wachinger, M. Reuter, and T. Klein, “Deepnat: Deep convolutional neural network for segmenting neuroanatomy,” _NeuroImage_ , vol. 170, pp. 434–445, 2018.
  * [81] J. Hofmanninger, F. Prayer, J. Pan, S. Röhrich, H. Prosch, and G. Langs, “Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem,” _European Radiology Experimental_ , vol. 4, no. 1, pp. 1–13, 2020.
  * [82] P. A. Yushkevich, J. Piven, H. Cody Hazlett, R. Gimpel Smith, S. Ho, J. C. Gee, and G. Gerig, “User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability,” _Neuroimage_ , vol. 31, no. 3, pp. 1116–1128, 2006.
  * [83] R. Kikinis, S. D. Pieper, and K. G. Vosburgh, “3d slicer: a platform for subject-specific image analysis, visualization, and clinical support,” in _Intraoperative imaging and image-guided therapy_. Springer, 2013, pp. 277–289.
  * [84] F.-C. Yeh, “Shape analysis of the human association pathways,” _Neuroimage_ , vol. 223, p. 117329, 2020.
  * [85] M. Brett, C. J. Markiewicz, M. Hanke, M.-A. Côté, B. Cipollini, P. McCarthy, D. Jarecka, C. P. Cheng, Y. O. Halchenko, M. Cottaar, E. Larson, S. Ghosh, D. Wassermann, S. Gerhard, G. R. Lee, H.-T. Wang, E. Kastman, J. Kaczmarzyk, R. Guidotti, J. Daniel, O. Duek, A. Rokem, C. Madison, D. Papadopoulos Orfanos, A. Sólon, B. Moloney, F. C. Morency, M. Goncalves, Z. Baratz, R. Markello, C. Riddell, C. Burns, J. Millman, A. Gramfort, J. Leppäkangas, J. J. van den Bosch, R. D. Vincent, H. Braun, K. Subramaniam, A. Van, K. J. Gorgolewski, P. R. Raamana, J. Klug, B. N. Nichols, E. M. Baker, S. Hayashi, B. Pinsard, C. Haselgrove, M. Hymers, O. Esteban, S. Koudoro, F. Pérez-García, J. Dockès, N. N. Oosterhof, B. Amirbekian, I. Nimmo-Smith, L. Nguyen, S. Reddigari, S. St-Jean, E. Panfilov, E. Garyfallidis, G. Varoquaux, J. H. Legarreta, K. S. Hahn, L. Waller, O. P. Hinds, B. Fauber, J. Roberts, J.-B. Poline, J. Stutters, K. Jordan, M. Cieslak, M. E. Moreno, T. Hrnčiar, V. Haenel, Y. Schwartz, B. C. Darwin, B. Thirion, C. Gauthier, I. Solovey, I. Gonzalez, J. Palasubramaniam, J. Lecher, K. Leinweber, K. Raktivan, M. Calábková, P. Fischer, P. Gervais, S. Gadde, T. Ballinger, T. Roos, V. R. Reddam, and freec84, “nipy/nibabel: 5.0.1,” Feb. 2023. [Online]. Available: https://doi.org/10.5281/zenodo.7633628
  * [86] B. C. Lowekamp, D. T. Chen, L. Ibáñez, and D. Blezek, “The design of simpleitk,” _Frontiers in neuroinformatics_ , vol. 7, p. 45, 2013.
  * [87] O. Diaz, K. Kushibar, R. Osuala, A. Linardos, L. Garrucho, L. Igual, P. Radeva, F. Prior, P. Gkontra, and K. Lekadir, “Data preparation for artificial intelligence in medical imaging: A comprehensive guide to open-access platforms and tools,” _Physica medica_ , vol. 83, pp. 25–37, 2021.
  * [88] I. Despotovic, E. Vansteenkiste, and W. Philips, “Brain volume segmentation in newborn infants using multi-modal mri with a low inter-slice resolution,” in _2010 Annual International Conference of the IEEE Engineering in Medicine and Biology_. IEEE, 2010, pp. 5038–5041.
  * [89] J. G. Sled, A. P. Zijdenbos, and A. C. Evans, “A nonparametric method for automatic correction of intensity nonuniformity in mri data,” _IEEE transactions on medical imaging_ , vol. 17, no. 1, pp. 87–97, 1998.
  * [90] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 Fourth International Conference on 3D Vision (3DV)_ , 2016, pp. 565–571.
  * [91] C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi, “Inception-v4, inception-resnet and the impact of residual connections on learning,” in _Thirty-first AAAI conference on artificial intelligence_ , 2017.
  * [92] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [93] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [94] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” in _OpenReview_ , 2017.
  * [95] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [96] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” _arXiv preprint arXiv:1711.05101_ , 2017.
  * [97] A. A. Taha and A. Hanbury, “Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool,” _BMC medical imaging_ , vol. 15, no. 1, pp. 1–28, 2015.
  * [98] N. M. Razali, Y. B. Wah _et al._ , “Power comparisons of shapiro-wilk, kolmogorov-smirnov, lilliefors and anderson-darling tests,” _Journal of statistical modeling and analytics_ , vol. 2, no. 1, pp. 21–33, 2011.
  * [99] J. L. Gastwirth, Y. R. Gel, and W. Miao, “The impact of levene’s test of equality of variances on statistical theory and practice,” _Statistical Science_ , vol. 24, no. 3, pp. 343–360, 2009.
  * [100] N. Nachar _et al._ , “The mann-whitney u: A test for assessing whether two independent samples come from the same distribution,” _Tutorials in quantitative Methods for Psychology_ , vol. 4, no. 1, pp. 13–20, 2008.
  * [101] A. Hart, “Mann-whitney test is not just a test of medians: differences in spread can be important,” _Bmj_ , vol. 323, no. 7309, pp. 391–393, 2001\.
  * [102] B. M. King, P. J. Rosopa, and E. W. Minium, _Statistical reasoning in the behavioral sciences_. John Wiley & Sons, 2018.
  * [103] M. Antonelli, A. Reinke, S. Bakas, K. Farahani, A. Kopp-Schneider, B. A. Landman, G. Litjens, B. Menze, O. Ronneberger, R. M. Summers _et al._ , “The medical segmentation decathlon,” _Nature Communications_ , vol. 13, no. 1, pp. 1–13, 2022.
  * [104] M. Jun, G. Cheng, W. Yixin, A. Xingle, G. Jiantao, Y. Ziqi, Z. Minqing, L. Xin, D. Xueyuan, C. Shucheng, W. Hao, M. Sen, Y. Xiaoyu, N. Ziwei, L. Chen, T. Lu, Z. Yuntao, Z. Qiongjie, D. Guoqiang, and H. Jian, “COVID-19 CT Lung and Infection Segmentation Dataset,” Apr. 2020. [Online]. Available: https://doi.org/10.5281/zenodo.3757476