# Medical Image Segmentation Review:
The Success of U-Net

Reza Azad , Ehsan Khodapanah Aghdam , Amelie Rauland, Yiwei Jia , Atlas
Haddadi Avval , Afshin Bozorgpour , Sanaz Karimijafarbigloo , Joseph Paul
Cohen , Ehsan Adeli , and Dorit Merhof
R. Azad and E. Khodapanah Aghdam contributed equally to this work.R. Azad, A.
Rauland, Y. Jia, and S. Karimijafarbigloo are with the Institute of Imaging
and Computer Vision, RWTH Aachen University, 52074 Aachen, Germany.E.
Khodapanah Aghdam is with the Department of Electrical Engineering, Shahid
Beheshti University, Tehran 1983969411, Iran.A. Haddadi Avval is with the
School of Medicine, Mashhad University of Medical Sciences, Mashhad
9177899191, Iran.A. Bozorgpour is with the Department of Computer Engineering,
Sharif University of Technology, Tehran 1458889694, Iran.J.P. Cohen is with
the Center for Artificial Intelligence in Medicine & Imaging, Stanford
University, Palo Alto, California 94304, USA, and this work was done prior to
joining Amazon.E. Adeli is with Stanford University, Stanford, California
94305, USA.D. Merhof is with the Faculty of Informatics and Data Science,
University of Regensburg, 93053 Regensburg, Germany, and also with the
Fraunhofer Institute for Digital Medicine MEVIS, 28359 Bremen,
Germany.(Corresponding author: D. Merhof. E-mail: dorit.merhof@ur.de.)This
work has been submitted to the IEEE for possible publication. Copyright may be
transferred without notice, after which this version may no longer be
accessible. 0000-0002-4772-2161 0000-0002-2849-1070 0000-0002-5824-8821
0000-0002-3896-7810 0000-0003-1857-1058 0000-0002-6632-6121
0000-0002-1334-3059 0000-0002-0579-7763 0000-0002-1672-2185

###### Abstract

Automatic medical image segmentation is a crucial topic in the medical domain
and successively a critical counterpart in the computer-aided diagnosis
paradigm. U-Net is the most widespread image segmentation architecture due to
its flexibility, optimized modular design, and success in all medical image
modalities. Over the years, the U-Net model achieved tremendous attention from
academic and industrial researchers. Several extensions of this network have
been proposed to address the scale and complexity created by medical tasks.
Addressing the deficiency of the naive U-Net model is the foremost step for
vendors to utilize the proper U-Net variant model for their business. Having a
compendium of different variants in one place makes it easier for builders to
identify the relevant research. Also, for ML researchers it will help them
understand the challenges of the biological tasks that challenge the model. To
address this, we discuss the practical aspects of the U-Net model and suggest
a taxonomy to categorize each network variant. Moreover, to measure the
performance of these strategies in a clinical application, we propose fair
evaluations of some unique and famous designs on well-known datasets. We
provide a comprehensive implementation library with trained models for future
research. In addition, for ease of future studies, we created an online list
of U-Net papers with their possible official implementation. All information
is gathered in https://github.com/NITR098/Awesome-U-Net repository.

###### Index Terms:

Medical Image Segmentation, Deep Learning, U-Net, Convolutional Neural
Network, Transformer.

##  1 Introduction

Image segmentation, defined as the partition of the entire image into a set of
regions, plays a vital role in a wide range of applications. Medical image
segmentation is a crucial example of this domain and offers numerous benefits
for clinical use. Automated segmentation facilitates the data processing time
and guides clinicians by providing task-specific visualizations and
measurements. In almost all clinical applications the visualization algorithm
not only provides insight into the abnormal regions in human tissue but also
guides the practitioners to monitor cancer progression. Semantic segmentation
as a preparatory step in automatic image processing technique can further
enhance the visualization quality by modeling to detect specific regions which
are more relevant to the task on hand (e.g., heart segmentation) [1].

Image segmentation tasks can be classified into two categories: semantic
segmentation and instance segmentation [2, 3]. Semantic segmentation is a
pixel-level classification that assigns corresponding categories to all the
pixels in an image, whereas instance segmentation also needs to identify
different objects within the same category based on semantic segmentation.
Designing segmentation methods to distinguish organ or lesion pixels requires
task-specific image data to provide the appropriate critical details. Common
medical imaging modalities for acquiring data are X-ray, Positron Emission
Tomography (PET), Computed Tomography (CT), Magnetic Resonance Imaging (MRI),
and Ultrasound (US) [4]. Early traditional approaches to medical image
segmentation mainly focused on edge detection, template matching techniques,
region growing, graph cuts, active contour lines, machine learning, and other
mathematical methods. In recent years, deep learning has matured in diverse
fields for solving many edge cases specific to the medical domain.
Convolutional neural networks (CNNs) have successfully implemented feature
representation extraction for images, thus eliminating the need for hand-
crafted features in image segmentation, and their superior performance and
accuracy make them the main choice in this field.

An initial attempt to model the semantic segmentation using a deep neural
network was proposed in [5]. This approach passes the input images through the
convolutional encoder to produce the latent representation. Then on top of the
generated feature maps the fully connected layers are included to produce a
pixel-level prediction. The main limitation of this architecture was the use
of fully connected layers, which depleted the spatial information and
consequently degraded the overall performance. Long et al. [6] proposed Fully
Convolutional Networks (FCNs) to address this limitation. The FCN structure
applies several convolutional blocks consisting of the convolution,
activation, and pooling layers on the encoder path to capture semantic
representation, and similarly uses the convolutional layer along with the up-
sampling operation in the decoding path to provide a pixel-level prediction.
The main motivation underlying the successive up-sampling process on the
decoding path was to gradually increase the spatial dimension for a fine-
grained segmentation result.

Inspired by the architecture of FCNs and the encoder-decoder models,
Ronneberger et al. develop the U-Net [7] model for biomedical image
segmentation. It is tailored to practical use in medical image analysis and
can be applied in a variety of modalities, including CT [8, 9, 10, 11, 12],
MRI [13, 14, 15, 16, 17], US [18, 19, 20], X-ray [21, 22], Optical Coherence
Tomography (OCT) [23, 24], and PET [25, 26].

FCN networks, specifically the U-Net, can efficiently exploit a limited number
of annotated datasets by leveraging data augmentation (e.g., random elastic
deformation) to extract detailed features of images without the need for new
training data, resulting in good segmentation performance [27]. This
superiority has made it a great success and has led to the extensive use of
U-Net model in the field of medical segmentation. The U-Net network is
composed of two parts. The first part is the contracting path that employs the
downsampling module consisting of several convolutional blocks to extract
semantic and contextual features. And in the second part, the expansive path
applies a set of convolutional blocks equipped with the upsampling operation
to gradually increase the spatial resolutions of the feature maps, usually by
a factor of two, while reducing the feature dimensions to produce the pixel-
wise classification score. The most significant and important part of U-Net is
the skip connections which copy the outputs of each stage within the
contracting path to the corresponding stages in the expansive path. This novel
design propagates essential high-resolution contextual information along the
network, which encourages the network to re-use the low-level representation
along with the high-context representation for accurate localization. This
novel structure becomes the backbone in the field of medical image
segmentation since 2015, and several variants of the model have been derived
to progress the state of the art based on it.

The auto-encoder design of U-Net makes it a unique tool for breaching its
structure in significant applications, e.g., image synthesis [28, 29, 30],
image denoising [31, 32, 33], image reconstruction [34, 35], and image super-
resolution [36]. To provide more insight into the importance of the U-Net
model in the medical domain, we provide Figure 1, statistical information
regarding the methods utilized U-Net model in their pipeline to address
medical image analysis challenges. From Figure 1, it is evident that U-Net
influenced most of the diverse segmentation tasks in the medical image
analysis domain with the extreme growth in publication numbers during the past
decade and being bespeak for future remedies.

Figure 1: The number of research works published in the past decade using the
U-Net model as their baseline to address various medical image analysis
challenges. The visualization shows sumptuous attention from the
research/industry community for this architecture, particularly the
segmentation task which is the main objective of this review paper.

Our review covers the most recent U-Net-based medical image segmentation
literature and discusses more than a hundred methods proposed until September
2022. We deliver a broad review and perspicuity on different aspects of these
methods, including network architecture enhancements concerning vanilla U-Net,
medical image data modalities, loss functions, evaluation metrics, and their
critical contributions. According to the rapid developments in U-Net and its
variants, we propose a summary of highly cited approaches in our taxonomy. we
group the U-Net variants into the following categories:

  1. 1.

Skip Connection Enhancements

  2. 2.

Backbone Design Enhancements

  3. 3.

Bottleneck Enhancements

  4. 4.

Transformers

  5. 5.

Rich Representation Enhancements

  6. 6.

Probabilistic Design

Some of the key contributions of this review paper can be outlined as follows:

  * •

This review covers the most recent literature on U-Net and its variants for
medical image segmentation problems and overviews more than 100 segmentation
algorithms proposed till September 2022, grouped into six categories.

  * •

We provide a comprehensive review and insightful analysis of different aspects
of U-Net-based algorithms, including the refinement of base U-Net
architectures, training data modality, loss functions, evaluation metrics, and
their critical contributions.

  * •

We provide comparative experiments of some reviewed methods on popular
datasets and offer codes and pre-trained weights on GitHub.

As a result, the remainder of the paper is organized as follows: Section 2
includes the taxonomy of review methods. Section 2.1 and Section 2.2 provide a
a detailed insight into the basic 2D U-Net and 3D U-Net architectures,
respectively. In Section 3 we will cover U-Net extensions, overview at least
five top-cited methods in each taxonomical branch, and highlights their key
contribution. Section 4 provides a comprehensive practical information such as
the experimental datasets, training process, the loss functions, evaluation
metrics, comparative results, and ablation studies. Section 5 discusses the
current challenges in the literature and future directions. Eventually, the
last chapter provides the conclusion.

##  2 Taxonomy

This section suggests a taxonomy that organizes different approaches presented
in the literature to modify U-Net architecture for medical image segmentation.
Due to the modular design of U-Net, we proposed our taxonomy to cope with the
inheritance design of U-net rather than the conceptual taxonomies offered in
[37]. Furthermore, this property makes it difficult to fit each study into
only one group so that a method may belong to several groups of divisions.
Figure 2 depicts our structure for taxonomy, and we think this taxonomy helps
the field be organized and even motivational for future research. In Section
3, we will go through each concept of taxonomy. In the remainder of this
section, we will first explain the naive 2D U-Net, and following that, we will
introduce the 3D U-Net. Eventually, we will elaborate on the importance of the
U-Net model from a clinical perspective.

Figure 2: The proposed U-Net taxonomy categorizes different extensions of the
U-Net model based on their underlying design idea. More specifically, our
taxonomy takes into account the modular design of the U-Net model and shows
where the improvement happens (e.g., skip connection). Due to the
clarification and unity in the studies’ denomination, we may utilize some
brevities. In this case, each prefix number denotes 1. [38], 2. [39], 3. [40],
4. [41], 5. [42], 6. [43], 7. [44], 8. [45], 9. [46], 10. [47], 11. [48], 12.
[49], 13. [50], 14. [51], 15. [52], 16. [53], 17. [54], 18. [18], 19. [55],
20. [56], 21. [57], 22. [58], 23. [59], 24. [60], 25. [61], 26. [62], 27.
[15], 28. [63], 29. [64], 30. [65], 31. [13], 32. [66], 33. [32], 34. [67],
35. [68], 36. [69], 37. [70], 38. [71], 39. [72], 40. [73], 41. [74], 42.
[75], 43. [76], 44. [77], 45. [78], 46. [79], 47. [80], 48. [81], 49. [82],
50. [83].

###  2.1  2D U-Net

Before recapitulating the U-Net structure in more detail, we will first
consider the path that brings us to the U-Net architecture. The story begins
with the EM segmentation challenge in 2012, where Ciresean et al. [5] were the
first researchers who outperform the previous biomedical imaging segmentation
methods using convolutional layers. The key factor that made them able to win
the challenge was the availability of huge annotated data (CNN can learn
comparatively better than the classical machine learning approach on large
datasets [84, 85]). However, access to the high amount of annotated data in
biomedical tasks is always inherently challenging due to privacy concerns, the
complexity of the annotation process, expert skill requirements, and the high
price of taking images with biomedical imaging systems. The first step toward
alleviating the need for large annotated data was proposed in [5]. This method
used an image patching technique to not only increase the number of samples
but also model the data distribution with small patches. Using this technique
the CNN network learns the visual concept by simply deploying a sliding
window. However, the sliding window usually brings more computation burden
than its performance increase. Hence, there is always a trade-off between
performance and computational complexity.

In 2015, Ronnebreger et al. [7] proposed a new architecture with respect to
Long et al.’s [6] FCN framework in conjunction with ISBI cell tracking
challenge, where they won the competition by a large margin. Figure 3 shows
the structure of the U-Net model. Their proposed method is a cornerstone in a
few attitudes those days. First, it is based on a fully convolutional network
in an encoder-decoder design with insufficient data than the DNNs instinct
with some intuitive data augmentation techniques. Second, their model was
reasonably fast and outperformed other methods in the challenge. The model
architecture can be divided into two parts: The first part is the contracting
path, also known as the encoder path, where its purpose is to capture
contextual information. This path consists of repeated blocks, where each
block contains two successive $3\times 3$ convolutions, followed by a ReLU
activation function and max-pooling layers. The max pooling layer is also
included to gradually increase the receptive field of the network without
imposing an additional computational burden.

The second part is expanding the path, also called the decoder path, where it
aims to gradually up-sample feature maps to the desired resolution. This path
consists of one $2\times 2$ transposed convolution layer (up-sampling),
followed by two consecutive $3\times 3$ convolutions and a ReLU activation.
The connection path between encoder and decoder paths (also known as a
bottleneck) includes two successive $3\times 3$ convolutions followed by a
ReLU activation. The successive convolutional operations included in the U-Net
model enables the network’s receptive field size to be increased linearly.
This process makes a network gradually learn coarse contextual and semantic
representation in deep layers compared to shallow layers. Learning high-level
semantic features makes the network slowly lose localization of extracted
features, where this aspect is essential to reconstruct segmentation results.
Ronnebreger et al. presented skip connections from the encoder path to the
decoder path on the same scales to overcome this challenge. The existential
reason for these skip connections is to impose localization information of
extracted semantic features at the same stage from the encoder. To this end,
the connection module concatenates low-level features coming from the encoder
path with high-level representation derived from the decoding path to enrich
localization information. Eventually, the network uses a $1\times 1$
convolution to map the final representation to the desired number of classes.
To mitigate the loss of contextual information in the missing image’s border
pixels, the U-Net model uses an overlap tile strategy. In addition, to deal
with insufficient training data a typical data augmentation technique such as
rotation, and gray-level intensities invariance, elastic deformation is
utilized. It should be noted that elastic deformation is a common strategy to
make the model resistant to deformations, a common variation in tissues. From
a practical perspective, the original U-Net model outperformed a sliding-
window convolutional network [5] in warping error terminology in the EM
segmentation challenge dataset [7]. This network also became a new state-of-
the-art on two other cell segmentation datasets, PhC-U373 and DIC-Hela cells,
by a large margin of approximately $9\%$ and $31\%$ from the previous best
methods in the ISBI Cell Tracking Challenge 2015 by reporting Intersection
over Union (IoU) metric [7].

Figure 3: The initial 2D U-Net architecture that is designed to cope with
semantic segmentation challenge. Figure from [86].

Figure 4:  Sample of the 3D medical dataset and a single selected 2D frame,
where the target area (e.g., organ) is highlighted using the annotation mask.
c.1) Cervical spine [87], c.2) Lung [88], c.3) Fourteen abdominal organs [89],
c.4) Brain [90, 91], c.5) Heart [90], c.6) Hepatic vessel [90], c.7) Liver
[90], c.8) Pancreas [90], c.9) Prostate [90].

###  2.2 3D U-Net

Due to the abundance and representation power of volumetric data, most medical
image modalities are three-dimensional. So, Çiçek et al. [92] proposed a 3D
volumetric-based U-Net not only to pay attention to this need but also to
overcome the time-consuming slice-by-slice annotation process for data. As it
is noticeable that neighboring slices share the same information, there is no
need for this much data redundancy. In [92], they replaced all 2D operations
in U-Net architecture with the equivalent 3D companions and embedded a batch
normalization layer for faster convergence after each 3D convolution layer. 3D
U-Net was successfully applied to sparsely annotated three samples of Xenopus
kidney embryos with reporting comparison results of IoU between 2D U-Net and
3D U-Net. To further support this we find the top 9/10 participants of the
Kidney Tumor Segmentation (KiTS) 2021 challenge hosted by the Medical Image
Computing and Computer Assisted Intervention (MICCAI) 2021 society [93, 94]
challenge utilized a 3D U-Net and 1/10 utilized a 2D one
111https://kits21.kits-challenge.org/public/html/kits21_results.html#.

Figure 4 shows samples of 2D and 3D medical image segmentation challenges
designed for different tasks. It can be seen that the 3D data provides more
comprehensive information regarding the tissue and tumors, however, compared
to the 2D data it has more computational cost.

###  2.3 Clinical Importance and Effect of U-Net

During the start of the COVID-19 pandemic and the inevitable loss of
healthcare and staff, the importance of utilizing artificial intelligence in
images and test analysis was prompted. According to WHO, between January 2020
till May 2021, almost 80,000 to 180,000 healthcare and staff could have died
from COVID-19 infection worldwide [95]. Compensating for these skilled
workforce losses, each country would incur a significant economic cost, and
also transferring experiences among medical staff is a time-consuming process.
In this direction, Michael et al. [96] applied a Large scale segmentation
network to count specific cells in pathological images. They explicitly
indicate that detecting cancerous cells from histopathological images is a
challenging task that relies on the experiences of the expert pathologist.
However, workflow efficiency can be increased with automatic system. Indeed
the recent success of deep-learning-based segmentation methods, expansion of
medical datasets and their easy accessibility, and facilitated access to
modern and efficient GPUs, their applicability to specific image analysis
problems of end-to-end users are eased. Semantic segmentation transforms a
plain biomedical image modality into a meaningful and spatially well-organized
pattern that could aid medical discoveries and diagnoses [97, 98] and
sometimes is beneficial to patients too as they may be able to avoid an
invasive medical procedure [99]. Medical image segmentation is a vital
component and a cornerstone in most clinical applications, including
diagnostic support systems [100, 101], therapy planning support [102, 103],
intra-operative assistance [104], tumor growth monitoring [105, 106], and
image-guided clinical surgery [107]. Figure 5 shows a general pipeline where
the U-Net can be utilized in a clinical application to reduce experts’ burden
and accelerate the disease detection process. The entire end-to-end paradigm
for using deep learning-based methods, especially U-Net, is an empirical
struggle to fit this concept into everyday life [98]. Computer-Aided Diagnosis
(CAD) can build from four main counterparts: Input, Network, Output, and
Application. The input block could leverage different analyses on various
available data like documented transcripts, diverse human body signals
(EEG/ECG), and medical images. The multi-modal fusion of different data types
could boost the performance of a pipeline for higher accuracy diagnosis. Based
on specific criteria like Image modality, and data distribution, the network
module could make decisions to choose one of the U-Net extensions which fit
more to the setting. The output is a task-specific counterpart that the
ultimate application block’s decision could decide.

On the other hand, international image analysis competitions have a high
demand for automatic segmentation methods, accounting for $70\%$ [108] in the
biomedical section, which universities of medical sciences primarily host or
collaborate with them. One of the advantages of deep learning competitions
over conventional hypothesis-driven research is innate distinctions in the
approach to problem-solving [109]. Data competitions, by nature, encourage
multiple individuals or groups to address a specific problem independently or
collaboratively. According to Maier-Hein et al. [108] of the 150 medical
segmentation competitions before 2016 the majority used U-Net based models

Based on the points above and across-the-board of U-Net-based architectures,
medical and clinical facilities could utilize these in real-world and
commercial settings where nnU-Net [98] is one of these successful end-to-end
designs.

Figure 5: A detailed overview of the U-Nets core involvement in medical image
analysis and clinical use. An illustration of how U-Nets are involved in
clinical decisions is discussed in research papers. The first block deals with
image acquisition, preparation, and pre-processing steps to provide the data
in a common format for the deep neural network. The second step uses a neural
architecture search algorithm to find an efficient architecture for the task
at hand while the third block is designed to perform post operations to
further refine the network output. Finally, the application block uses the
software output to assist specialists with a certain action (e.g., tumor
growth monitoring).

##  3 U-Net Extensions

U-Net is a ubiquitous network according to its approximately 48 thousand
citations during its first release in 2015. This is evidence that it can
handle diverse image modalities in broad domains and not only in medical
fields. From our sight, the core advantage of U-Net is its modular and
symmetric design, which makes it a suitable choice for broad modification and
collaboration with diverse plug-and-play modules to increase performance.
Therefore, by pursuing this cue, we infringe the Ronneberger et al. [7]
network to modular improvable counterparts besides solid auxiliary
modification for achieving SOTA or par with segmentation performances. In this
respect, we offer our taxonomy (Figure 2) and divide the diverse variants of
U-Net modifications into systematic categories as follows:

  1. 1.

Skip Connection Enhancements

  2. 2.

Backbone Design Enhancements

  3. 3.

Bottleneck Enhancements

  4. 4.

Transformers

  5. 5.

Rich Representation Enhancements

  6. 6.

Probabilistic Design

This taxonomy aims to provide comprehensive and practical information for both
vendors and researchers. In the following parts of this section, each category
will be extensively discussed along with relevant papers.

###  3.1 Skip Connection Enhancements

Skip connections are an essential part of the U-Net architecture as they
combine the semantic information of a deep, low-resolution layer with the
local information of a shallow, high-resolution layer. This section provides a
definition of skip connections and explains their role in the U-Net
architecture before introducing extensions and variants of the classic skip
connection used in the original U-Net. Skip connections are defined as
connections in a neural network that do not connect two following layers but
instead skip over at least one layer. Considering a two-layer network a skip
connection would connect the input directly to the output, skipping over the
hidden layer [110]. In image segmentation, skip connections were first used by
Long et al. in [6]. At the time, the most common use of convolutional networks
was for image classification tasks which only have a single label as output.
In a segmentation task, however, a label should be assigned to each pixel in
the image adding a localization task to the classification task.

Long et al. [6] added additional layers to a usual contracting network using
upsampling instead of pooling layers to increase the resolution of the output
and obtain a label for every pixel. Since local, high-resolution information
gets lost in the contracting part of the network it cannot be completely
recovered when upsampling these volumes. To combine the deep, coarse semantic
information with the shallow fine appearance information they add skip
connections that connect up-sampled lower layers with finer stride with the
final prediction layer.

In the original U-Net architecture by Ronneberger et al. [7] each level in the
encoder path is connected to the corresponding same-resolution level in the
decoder path by a skip connection to combine the global information describing
what with the local information resolving where. The difference to the above
approach is not only the higher number of skip connections but also the way in
which the features are combined. Long et al. [6] up-sampled feature maps from
earlier layers to the output resolution and added them to the output of the
final layer. Ronneberger et al. [7] concatenated the features of the
corresponding encoder and decoder level and process them together by passing
them through two convolutional layers and an up-sampling layer together.

Li et al. [62] conducted an ablation study on skip connections by training a
dense U-Net with and without skip connections. The results clearly show that
the network with the skip connections generalize better than the network
without skip connections.

Over the following years, many variants and extensions of the original U-Net
architecture were developed concerning the skip connections [111, 112].
Different types of extensions dealing with processing the encoder feature maps
passed through the skip connections, combining the two sets of feature maps,
and extending the number of skip connections will be presented in the
following sections.

####  3.1.1 Increasing the Number of Skip Connections

In 2020 Zhou et al. [67] introduced the U-Net++ in which they redesign skip
connections to be more flexible and therefore exploit multiscale features more
effectively. Instead of restricting skip connections to only aggregate
features that have the same scale in the encoder and decoder path, they
redesign them in such a way that features of different semantic scales can be
aggregated [67].

They argue that there has been no proof so far that encoder and decoder
feature maps at the same scale are the best match for feature fusion and
therefore design a more flexible setup.

Figure 6: Architecture of the U-Net++ [67]. The U-Net++ uses a nested
structure of convolutional layers $X^{i,j}$ (indicate $j$-th convolution layer
in the $i$-th scale) connected through the skip connection paths to model
multi-scale representation.

In their approach, they tackle two problems simultaneously. Since the optimal
depth of a U-Net is unknown apriori and usually has to be determined through
an exhaustive search, they incorporate U-Nets of different depths into one
architecture. As can be seen in Figure 6 all the U-Nets share the same encoder
but have their own decoder. Instead of only passing the same-scale encoder
feature maps through the skip connections, each node in the decoder is also
presented with the feature maps of the same-level decoders of the U-Nets with
a lower depth. It can then be learned during training, which of the presented
feature maps should ideally be used for the segmentation.

Huang et al. [68] take the dense skip connections introduced in the U-Net++
one step further by introducing full-scale skip connections in their
architecture the U-Net3+. They argue that both the original U-Net with plain
skip connections between same-level encoder and decoder nodes and the U-Net++
with the dense and nested skip connections do not sufficiently explore
features from full scales making it challenging for the network to learn the
position and boundary of an organ explicitly.

Figure 7: Architecture showing the full-scale skip connections of the UNet3+
[68]. $X_{Ee}^{i}$ and $X_{De}^{i}$ denote the encoder and decoder path
feature maps at $i$-th scale, respectively. In addition, each subscript under
each feature map symbol represents the number of feature maps to the
corresponding block.

To overcome this limitation they connect each decoder level with all encoder
levels and all preceding decoder levels as can be seen in Figure 7. Since not
all feature maps arriving at a decoder node through skip connections have the
same scale, higher-resolution encoder feature maps will be downscaled using a
max-pooling operation and lower-resolution feature maps coming from intra-
decoder skip connections will be upsampled using bilinear upsampling.
Additionally, apart from the up- or down-sampling operation, each skip
connection is equipped with a $3\times 3$ convolutional layer calculating 64
output maps. The 64 feature maps arriving through each skip connection are
stacked and the stack of feature maps is passed through another convolutional
layer, followed by batch normalization and a ReLU activation before being
further processed in the respective decoder node.

Instead of increasing the number of forward skip connections, Xiang et al.
[69] add additional backward skip connections: Their Bi-directional O-Shape
network (BiO-Net) is a U-Net architecture with bi-directional skip
connections. This means that there are two types of skip connections:

  1. 1.

The forward skip connections are known from the original U-Net architecture,
combining encoder and decoder layers at the same level. These skip connections
preserve the low-level visual features from the encoder and combine them with
the semantic decoder information.

  2. 2.

The backward skip connections pass decoded high-level features from the
decoder back to the same level encoder. The encoder can then combine the
semantic decoder features with its original input and flexibly aggregate the
two types of features.

Together these two types of skip connections build an O-shaped recursive
architecture that can be traversed multiple times to receive improved
performance (See Figure 8).

Figure 8: BiO-Net Architecture. Figure from [113].

The recursive output of the encoder and decoder can be defined as follows:

$\displaystyle\mathbf{x}_{out}^{i}$
$\displaystyle=\text{DOWN}(\text{ENC}([\text{DEC}([\mathbf{f}_{enc}^{i-1},\mathbf{\hat{x}}_{in}^{i-1}]),\mathbf{x}_{in}^{i}])),$
$\displaystyle\mathbf{\hat{x}}_{out}^{i}$
$\displaystyle=\text{UP}(\text{DEC}([\text{ENC}([\mathbf{f}_{dec}^{i},\mathbf{x}_{in}^{i}]),\mathbf{\hat{x}}_{in}^{i}])),$
(1)

Here, $i$ represents the current inference iteration, UP stands for an
upsampling operation, DOWN for a downsampling operation, DEC and ENC stand for
a decoder and encoder level, respectively. An additional improvement was
achieved when collecting decoded features from all iterations and feeding them
to the last decoder stage together to calculate the final output. Although the
recurrent training scheme might increase training time, this extension of the
U-Net has the advantage that it does not introduce any additional parameters
as claimed by the authors.

####  3.1.2 Processing Feature Maps within the Skip Connections

In the attention U-Net established by Oktay et al. [70], attention gates (AGs)
are added to the skip connections to implicitly learn to suppress irrelevant
regions in the input image while highlighting the regions of interest for the
segmentation task at hand.

Figure 9: Attention U-Net Architecture. Figure from [70].

In biomedical imaging, when organs to be segmented show high inter-patient
variation in terms of shape and size, a common approach is to use a cascaded
network. The first network extracts a rough region of interest (ROI) including
the organ to be segmented and the second network predicts the exact organ
segmentation in this ROI. These approaches, however, suffer from redundant
model parameters and high computational resources. Adding attention gates to
the skip connections maintains a high prediction accuracy without the need for
an external organ localization model. It is therefore trainable from scratch
and introduces no significant computational overload and only a few additional
model parameters. The output of an AG is the elementwise multiplication of the
input feature maps with attention coefficients $\alpha_{i}\in[0,1]$ as
$\mathbf{\hat{x}}_{i,c}^{l}=\mathbf{x}_{i,c}^{l}\cdot\alpha_{i}^{l}$. For the
computation of the attention coefficients both the input feature maps $x$,
that have been passed through the skip connection from the encoder and the
gating signal $g$ are analyzed. Here, the gating signal is collected from a
coarser scale as can be seen in Figure 9 for adding contextual information.
The applied additive attention is formulated as follows:

$\displaystyle q_{\text{att}}^{l}$
$\displaystyle=\psi^{T}(\sigma_{1}(W_{x}^{T}\mathbf{x}_{i}^{l}+W_{g}^{T}g_{i}+b_{g}))+b_{\psi}$
$\displaystyle\alpha_{i}^{l}$
$\displaystyle=\sigma_{2}(q_{\text{att}}^{l}(\mathbf{x_{i}^{l}},g_{i};\Theta_{\text{att}})),$
(2)

where $\sigma_{1}$ and $\sigma_{2}$ are ReLU and sigmoid activations
respectively, $W_{x}\in\mathbb{R}^{F_{l}\times F_{int}}$,
$W_{g}\in\mathbb{R}^{F_{g}\times F_{int}}$ and
$\psi\in\mathbb{R}^{F_{int}\times 1}$ are linear transforms and $b_{g}$ and
$b_{\psi}$ are bias terms. Adding an AG to a skip connection, therefore,
highlights the ROIs in the feature maps from the encoder path before they are
concatenated with the feature maps of the decoder path. So in addition to
adding higher resolution information, additional information on the location
of the object(s) to be segmented is added, eliminating the need for cascaded
multi-network approaches.

The attention U-Net++ by Li et al. combines the attention U-Net with the
U-Net++ [72]. Attention gates as described in [70] are added to all the skip
connections of the U-Net++ with its nested U-Nets and dense skip connections.
With similar motivation, Jin et al. [71] introduced a 3D U-Net with attention
residual modules in the skip connections, called the RA-UNet. The network was
developed for the task of segmenting tumors in the liver. The main
difficulties of this task lie in the large spatial and structural variability,
low contrast between liver and tumor, and similarity to nearby organs. The
added attention residual learning mechanism in the skip connections improve
the performance by focusing on specific parts of the image as claimed by the
authors. The output of the attention module ($\mathbf{OA}$) in the RA-UNet
structure is formulated as:

$\mathbf{OA}(\mathbf{x})=(1+\mathbf{S}(\mathbf{x}))\mathbf{F}(\mathbf{x}),$
(3)

where $\mathbf{S}(\mathbf{x})$ originates from the soft mask branch and has
values in [0,1] to highlight important features and suppress noise and
redundant features in the original feature maps $\mathbf{F}(\mathbf{x})$
passed through the trunk branch. The soft mask branch itself uses a residual
encoder-decoder architecture to calculate its output.

To improve performance on the difficult task of the ovary and follicle
segmentation from ultrasound images, Li et al. [75] added spatial recurrent
neural networks (RNNs) to the skip connections of a U-Net. Since there are
usually many small follicles in an image, it is very likely that the
neighboring follicles are spatially correlated. In addition, there might be a
possible spatial correlation between the follicles and the ovary. As the max-
pooling operation in the original U-Net brings a loss of spatially relative
information the spatial RNNs should improve the segmentation results by
learning multi-scale and long-range spatial contexts.

Li et al. [75] built the spatial RNNs from plain RNNs with a ReLU activation.
Each spatial RNN module takes feature maps as input and produces spatial RNN
features as output. It uses four independent data translations to integrate
local spatial information in up, down, left, and right directions. The maps
from each direction are concatenated and passed through a $1\times 1$
convolutional layer to produce feature maps where each point contains
information from all four directions. The process is then repeated to extend
the local spatial information to global contextual information. As can be seen
in Figure 10, the final feature maps passed through the skip connection are a
combination of the original encoder feature maps and the RNN features
extracted from these maps. The authors claim that the architecture is
especially strong at avoiding the segmentation of false positives and
detecting and segmenting very small follicles. A limitation of the RNN modules
is that they make training more difficult and computationally expensive. To
compensate for this, Li et al. added deep supervision.

Figure 10: The spatial RNN module used in the skip connections of the CR-Unet
by Li et al. in [75].

While most medical applications demand segmentations to be in the same
dimension as the input image, there are also medical protocols that require
segmentation of the image projection, e.g., Liefers et al. [114] studied the
retinal vessel segmentation as a $2$D $\rightarrow$ $1$D retinal OCT
segmentation task. This adds the problem of dimensionality reduction to the
segmentation. Lachinov et al. [73] introduced a U-Net with projective skip
connections to handle $N$D $\rightarrow$ $M$D segmentations, where $M<N$.

Figure 11: Architecture of U-Net with 3D input and 2D output segmentation.
Figure from [115].

The encoder is a classic U-Net encoder with residual blocks. The decoder
however only restores the input resolution for the $M$ dimensions of the
segmentation. The remaining reducible dimensions $M<d\leq N$ are left
compressed. This means that the sizes of the encoder and decoder feature maps
no longer match which is why Lachinov et al. [73] introduce the projective
skip connections. The encoder feature maps passed along the projective skip
connections are processed by an average pooling layer with varying kernel size
so that the dimensions which are not present in the segmentation are reduced
to the size they have in the bottleneck. This way they can be concatenated
with the corresponding decoder feature maps. Global Average Pooling (GAP) and
a convolutional layer are added after the last decoder level to calculate the
final $M$D segmentation. The overall architecture for $N=3$ and $M=2$ can be
seen in Figure 11. The third dimension is not upsampled to its original
resolution in the decoder path and is finally reduced to one by the GAP.

####  3.1.3 Combination of Encoder and Decoder Feature Maps

Another extension of the classic skip connections is introduced in the BCDU-
Net by Azad et al. [74] where a bi-directional convolutional long-term-short-
term-memory (LSTM) module is added to the skip connections. Azad et al. argue
that a simple concatenation of the high-resolution feature maps from the
encoder and the feature maps extracted from the previous up-convolutional
layer containing more semantic information might not lead to the most precise
segmentation output. Instead, they combine the two sets of feature maps with
non-linear functions in the bi-directional convolutional LSTM module. Ideally,
this leads to a set of feature maps rich in both local and semantic
information. The architecture of the bi-directional convolutional LSTM module
used to combine the feature maps at the end of the skip connection can be seen
in Figure 12.

Figure 12: Bi-directional convolutional LSTM module for the combination of
encoder feature maps passed through the skip connection and decoder feature
maps from the previous up-convolutional layer. $\mathcal{X}_{e}$ and
$\mathcal{X}_{d}$ represent the set of feature maps copied from the encoding
part and the output of the previous scale’s convolutional layer, respectively.
$\mathbf{Y}_{i}$ indicates the output of the BConvLSTM module in a $i$-th
block of a single skip connection. Figure from [116].

It uses two ConvLSTMs, processing the input data in two directions in the
forward and backward paths. The output will be determined by taking into
consideration the data dependencies in both directions. In contrast to the
approach by Li et al. [75], where only the encoder feature maps are processed
by the RNN and then concatenated with the decoder features, this approach
processes both sets of feature maps with the RNN.

###  3.2 Backbone Design Enhancements

Apart from adapting the skip connections of a U-Net it is also common to use
different types of backbones in newer U-Net extensions. The backbone defines
how the layers in the encoder are arranged and its counterpart is therefore
used to describe the decoder architecture.

In the original U-Net by Ronneberger et al. [7] each level in the encoder
consists of two $3\times 3$ convolutional layers with ReLU activation followed
by a max pooling operation. The number of feature maps doubles at each level.
Any 2D or 3D CNN image classifier can be used as an encoder in a U-Net adding
its mirrored counterpart as the decoder. Dozens of studies modified the
vanilla U-Net main blocks to broaden the receptive fields of convolution
operations and extract rich, and fine-grained semantic representations for
challenging multi-class problems, e.g., [64, 117, 118, 119, 120]. This section
presents several prominent backbones used in the U-Net architecture and
explains their benefits and downsides.

####  3.2.1 Residual Backbone

A very common backbone for the U-Net architecture is the ResNet initially
developed by He et al. [121]. Residual networks enable deeper network
architectures by tackling the vanishing gradient problem that often occurs
when stacking several layers in deep neural networks as well as a degradation
problem that leads to first saturating and then degrading accuracy when adding
more and more layers to a network. Residual building blocks, explicitly fit a
residual mapping by adding skip connections and performing an identity mapping
that is added to the output of the stacked layers.

In their implementation of a residual U-Net, Drozdzal et al. [60] refer to the
standard skip connections in the U-Net as long skip connections and the
residual skip connections as short skip connections, as they only skip ahead
over two convolutional layers. Using residual blocks as the backbone in a
U-Net, Drozdzal et al. [60] can build deeper architectures and find that the
network training converges faster compared to the original U-Net. Milletari et
al. [61] report the same findings in their 3D U-Net architecture using 3d
residual blocks as the backbone.

Figure 13: Schematic representation of V-Net structure for volumetric
biomedical image segmentation that comprises residual connections in each
scale. In addition, V-Net utilizes the strided convolutional kernels rather
than max-pooling layers to lessen the memory footprint in the training stage.
Figure from [122].

A prominent adaption of the backbone is to exchange all 2D convolutions with
3D convolutions to process an entire image volume as can often be found in
medical applications. When processing a 3D image in a slice-wise fashion using
2D convolutions, the contexts on the z-axis can not be captured and learned by
the network. Using fully convolutional architecture with 3D convolutions
elevates this drawback and can fully leverage the spatial information along
all three dimensions.

A drawback of using 3D convolutional layers as the backbone in a u-net is the
high computational cost and GPU memory consumption which limits the depth of
the network and the filter’s size i.e. its field-of-view. Milletari et al.
[61] fully convolutional volumetric, V-Net architecture uses 3D residual
blocks (Figure 13) as a backbone, thereby enabling fast and accurate
segmentation in 3D images. The H-DenseUNet by Li et al. [62] uses two U-Nets,
one with 2D-dense-blocks as the backbone and the other with 3D-dense-blocks as
the backbone. This enables them to first extract deep intra-slice features and
then learn inter-slice features in shallower volumetric architecture with a
lower computational burden.

####  3.2.2 Multi-Resolution blocks

To tackle the difficulty of analyzing objects at different scales, Ibtehaz et
al. introduce the MultiResUNet with inception-like blocks as a backbone [15].
Inception blocks, introduced by Szegedy et al. [123], use convolutional layers
with different kernel sizes in parallel on the same input and combine the
perceptions from different scales before passing them deeper into the network.
The two following convolutions with $3\times 3$ kernels in the classical U-Net
resemble one convolution with a $5\times 5$ kernel. For incorporating a multi-
resolution analysis into the network, $3\times 3$ and $7\times 7$ convolutions
should be added in parallel to the $5\times 5$ convolution. This can be
achieved by replacing the convolutional layers with inception-like blocks.
Adding the additional convolutional layers increases the memory requirement
and computational burden. Ibtehaz et al., therefore, formulate the more
expensive $5\times 5$ and $7\times 7$ convolutions as consecutive $3\times 3$
convolutions. The final MultiRes block is created by adding a residual
connection. The evolution from the original inception block to the MultiRes
block can be seen in Figure 14. Instead of keeping an equal number of filters
for all consecutive convolutions, the number of filters is gradually increased
to further reduce the memory requirements.

(a)

(b)

(c)

Figure 14: The MultiRes Block (c) is developed from the original inception
Block (a) with three parallel convolutions with $3\times 3$, $5\times 5$, and
$7\times 7$ kernels by expressing the $5\times 5$ and $7\times 7$ convolutions
as two and three consecutive $3\times 3$ convolutions as can be seen in (b)
and adding a residual skip connection. Figure from [124].

In the final architecture, the two consecutive $3\times 3$ convolutions from
the original U-Net are replaced by one MultiRes block, leading to faster
convergence, improved delineation of faint boundaries, and higher robustness
against outliers and perturbations.

Another well-known backbone for U-Net extensions is the DenseNet introduced by
Huang et al. in [125]. Similarly to residual networks, the DenseNet also aims
at fighting the vanishing gradient problem by creating skip connections from
early layers to later layers. The DenseNet maximizes the information flow by
connecting all layers with the same feature map size with each other. This
means that every layer obtains concatenated inputs from all preceding layers.
Contrary to what one might expect, a dense net actually requires fewer
parameters compared to a traditional CNN because it does not have to relearn
redundant feature maps and can therefore work with very narrow layers with
e.g. only 12 filters and can learn multi-resolution features. The direct
connection from each layer to the loss function implements implicit deep
supervision which helps train deeper network architectures without vanishing
gradients.

Karaali et al. [63] utilized Dense Residual blocks in the U-Net-like
representation for retinal vessel segmentation. To this end, they were
inspired by DenseNet [125], and ResNet [121] to design a Residual Dense-Net
(RDN) block. In their architecture the first sub-block comprises successive
batch Normalization, ReLu, Convolution, and Dropout counterparts, which
employs the dense connectivity pattern as in [125]. The following sub-block
applies a residual connectivity pattern. Using a DenseNet-like backbone helps
the U-Net architecture learn more relevant features using fewer parameters.
The residual connectivity smooths the information flow across the layers to
facilitate the optimization step.

####  3.2.3 Re-considering Convolution

This direction aims to reduce the computational burden of the naive
convolution operation by re-considering the alternative convolutional
operations. Jin et al. [65] exchange each $3\times 3$ convolutional layer in
the original U-Net with a deformable convolutional block for the accurate
segmentation of retinal vessels. Their architecture is named DUNet. The
deformable convolutional blocks are inspired by the work on deformable
convolutional networks by Dai et al. [126] and should adapt the receptive
fields to adjust optimally to different shapes and scales of complicated
vessel structures in the input features. In deformable convolutions, offsets
are learned and added to the grid sampling locations normally used in the
standard convolution. One exemplary illustration of adjusted sampling
locations for a $5\times 5$ kernel can be seen in Figure 15.

Figure 15: One exemplary deformable convolutions with their respective normal
convolution with a $5\times 5$ kernel.

In a classic convolution the kernel sampling grid $G$ would be defined as:

$G={(-2,-2),(-2,-1),...,(2,1),(2,2)}.$ (4)

Considering this grid, every pixel $m_{0}$ in the output feature map
$\mathbf{y}$ can be calculated as:

$\mathbf{y}(m_{0})=\sum_{m_{i}\in
G}\mathbf{(}w)(m_{i})\cdot\mathbf{x}(m_{0}+m_{i})$ (5)

from the input $\mathbf{x}$. In the deformable convolution, an offset $\Delta
m_{i}$ is added to the grid locations.

$\mathbf{y}(m_{0})=\sum_{m_{i}\in
G}\mathbf{(}w)(m_{i})\cdot\mathbf{x}(m_{0}+m_{i}+\Delta m_{i})$ (6)

Every deformable convolutional block consists of a convolutional layer, to
learn the ideal offsets from the input. A deformable convolution layer
applying the convolution with the adapted sampling points followed by batch
normalization and ReLU activation. Since the calculated offset $\Delta m_{i}$
is usually not an integer, the input value at the sampling point is determined
using bilinear interpolation. Exchanging the simple convolutions with
deformable convolutions helps the network adapt to different shapes, scales,
and orientations but comes at a higher computational burden because an
additional convolutional layer per block is needed to determine the offsets of
the sampling grid.

When segmenting from 3D images it is important to make use of the full spatial
information from the volumetric data. However, this is not possible with 2D
convolutions and 3D convolutions are computationally very expensive. To
address this problem, Chen et al. [13] used separable 3D convolutions as the
backbone of the U-Net. Each 3D convolutional block in the original U-Net is
replaced by an S3D block which can be seen in Figure 16.

Figure 16: Architecture of one separable 3D convolutional block [13]. Using
parallel design the architecture performs the 3D convolution in three paths
with less computation burden.

The 3D convolution is divided into three branches where each branch represents
a different orthogonal view so that the input is processed in axial, sagittal,
and coronal views. Additionally, a residual skip connection is added to the
separated 3D convolution. Using separable 3D convolutions as the backbone of
the U-Net, Chen et al. [13] can take into consideration the full spatial
information from the volumetric data in the U-Net architecture without the
extremely high computational burden of standard 3D convolution.

####  3.2.4 Recurrent Architecture

Recurrent neural networks (RNN) are used frequently to process sequential data
such as in speech recognition. Liang et al. [127] were among the first groups
to design a recurrent convolutional neural network (RCNN) for images
recognition. Although the input image, in contrast to sequential data, is
static, the activity of each unit is modulated by the activities of its
neighboring units because the activities of RCNNs evolve over time. By
unfolding the RCNN through time, they can obtain arbitrarily deep networks
with a fixed number of parameters.

Using these RCNN blocks as the backbone of the U-Net architecture enhances the
ability of the model to integrate contextual information. Alom et al. [128]
used RCNN blocks as a backbone in their RU-Net architecture, ensuring better
feature representation for segmentation tasks.

][c]0.48

(a) Architecture of double recurrent convolutional block.

][c]0.50

(b) A single unfolded recurrent convolutional block for $t=2$.

Figure 17: Recurrent Convolutional Mechanism introduced by Alom et al. [128],
namely RU-Net.

Figure 17(a) shows a recurrent convolutional unit, they used as a backbone.
Figure 17(b) shows one of the two sub-blocks in Figure 17(a) unfolded for
$t=2$, which is also the unfolding parameter chosen in their experiments.
Adding additional residual connections to the separate recurrent convolutional
blocks enables deeper networks and results in their R2U-Net architecture.

###  3.3 Bottleneck Enhancements

The U-Net architecture can be separated into three main parts: the encoder
(contracting path), the decoder (expanding path), and the bottleneck which
lies between the encoder and decoder. The bottleneck is used to force the
model to learn a compressed representation of the input data which should only
contain the important and useful information needed to restore the input in
the decoder. To this end, various modules are designed in multiple studies
[129, 79] to recalibrate and highlight the most discriminant features. In the
original U-Net, the bottleneck consists of two $3\times 3$ convolutional
layers with ReLU activation. More recent approaches however have extended the
classic bottleneck architecture to improve performance.

####  3.3.1 Attention Modules

Several works apply attention modules in the bottleneck of their U-Net
architecture. Fan et al. used a position-wise attention block (PAB) in their
MA-Net to model spatial dependencies between pixels in the bottleneck feature
maps with self-attention [76]. The architecture of the PAB can be seen in
Figure 18.

Figure 18: Architecture of the position-wise attention block introduced in
[76].

The feature maps passed into the bottleneck at the end of the encoder path are
first processed by a $3\times 3$ convolutional layer. The resulting outputs
are then processed by three individual $1\times 1$ convolutional layers
producing $A$, $B$, and $C$. $A$ and $B$ are reshaped to form two vectors. A
matrix multiplication of these two vectors passed through a softmax function
yields the spatial feature attention map $P\in\mathbb{R}^{N\times N}$ in which
the positions $p_{i,j}$ encode the influence of the $i^{th}$ position on the
$j^{th}$ position in the feature map. Subsequently, a matrix multiplication is
performed between the reshaped $C$ and the spatial feature attention map $P$,
and the resulting feature maps are multiplied with the input $I^{\prime}$
before being passed through a final $3\times 3$ convolutional layer. The final
output $O$ is therefore defined as follows:

$O_{i}=\alpha\sum_{i=1}^{N}(P_{ji}C_{i})+I^{\prime}_{j}$ (7)

$\alpha$ is set to zero at the beginning of training and it is learned to
assign more weight during the training process. Considering that the final
output is the weighted sum of the feature maps across all positions and the
original feature maps, it has a global contextual view and can selectively
aggregate rich contextual information. Intra-class correlation and semantic
consistency are improved because the PAB can consider long-range spatial
dependency between features in a global view.

Guo et al. also add a spatial attention module to the bottleneck of their SA-
UNet architecture [77]. The spatial attention module should enhance relevant
features and compress unimportant features in the bottleneck. In their
approach the input feature maps are passed through an average pooling and a
max pooling layer in parallel. Both pooling operations are applied along the
channel dimension to produce efficient feature descriptors. The outputs are
then concatenated and passed through a $7\times 7$ convolutional layer and
sigmoid activation to obtain a spatial attention map. By multiplying the
spatial attention map with the original input features, the inputs can be
weighted based on their importance for the segmentation task at hand. The
attention module only adds 98 parameters to the original U-Net and is
therefore computationally very lightweight.

In another work, Azad et al. [80] utilized the idea of a texture/style
matching mechanism in the U-Net bottleneck for brain tumor segmentation. In
their design, an attention agent is designed to distill the informative
information from a full modality (four MRI modalities, T1, T2, Flair and T1c)
into a missing-modality network (only Flair). Further information regarding
the missing-modality task can be found in [27]. A deep frequency attention
module is proposed in [79] to perform a frequency recalibration process on the
U-Net bottleneck. This attention block aims to recalibrate the feature
representation based on the structure and shape information rather than
texture representation to alleviate the texture bias in object recognition.

####  3.3.2 Multi-Scale Representation

The aim of this direction is to enhance the bottleneck design by including
multi-scale feature representation, e.g. atrous convolution. The atrous
convolutions are performed like standard convolutions, but with convolutional
kernels with inserted holes in them. The holes are defined by setting the
weight of the convolutional kernel to zero at the corresponding locations and
the pattern for doing so is defined by the atrous sampling rate $r$.
Considering a sampling rate $r$, this introduces $r-1$ zeros between
consecutive filter values. A $k\times k$ convolutional kernel is thereby
enlarged to a $k+(k-1)*(r-1)\times k+(k-1)*(r-1)$ filter. This way the
receptive field of the layer is expanded without introducing any additional
network parameters to be learned.
Figure 19 shows a $3\times 3$ kernel with atrous sampling rates $r$ of $r=1$,
$r=2$ and $r=4$.

][t]0.22

(a)

][t]0.42

(b)

][t]0.32

(c)

Figure 19: $3\times 3$ convolutional kernel with different atrous sampling
rates $r$ of (a) $r=1$, (b) $r=2$ and (c) $r=4$.

When the objects to be segmented are of very different sizes it is important
for the network to extract multiscale information. Combining the ideas of
spatial pyramid pooling and atrous convolutions, the feature maps in the
bottleneck of the U-Net can be resampled in parallel by atrous convolutions
with different sampling rates and then combined to obtain rich multiscale
features.

Hai et al. [81] use atrous spatial pyramid pooling (ASPP) in the bottleneck of
a U-Net architecture for the segmentation of breast lesions. The final feature
maps of the encoder are passed in parallel through a $1\times 1$ convolutional
layer and three atrous $3\times 3$ convolutional layers with atrous sampling
rates of 6, 12 and 18 respectively. These four processed groups of feature
maps are concatenated together with the original feature maps passed to the
bottleneck and processed by a final $1\times 1$ convolution before being
passed to the decoder.

Wang et al. make use of ASPP in the bottleneck as well in their COPLE-Net for
the segmentation of pneumonia lesions from CT scans of COVID-19 patients [82].
Here, four atrous convolutional layers with dilation rates of 1, 2, 4, and 6
respectively are used to process the bottleneck feature maps to capture multi-
scale features for the segmentation of small and large lesions.

Similarly, Wu et al. [83] proposed a multi-task learning paradigm, JCS, for
COVID-19 CT image classification and segmentation. JCS [83] is a two branches
architecture, which utilizes a Group Atrous (GA) module, in its segmentation
branches bottleneck for feature modification. GA first applies $1\times 1$
convolution operation to expand the channels of the feature map. Then the
feature map is divided into four equal sets. Utilizing the atrous convolutions
with different rates on these sets results in more global feature maps with
diverse receptive fields. To fully extract more discriminant features from the
final feature map, JCS adopts a squeeze and Excitation (SE) [130] block as an
attention mechanism for recalibrating channel-wise convolution features.

][c]0.48

(a) Overview of preliminary Vision Transformer (ViT) structure proposed by
Dosovitskiy et al. [131]. ViT splits an image into fixed-size patches with a
non-overlapping regime and then linearly embeds each of them to a 1D vector
space, afterward adds position embeddings, and feeds the resulting sequence of
vectors to a standard Transformer encoder. In order to perform classification,
naive ViT uses the standard approach of adding an extra learnable
classification token to the sequence. However, this token is not practical in
the segmentation literature, although the MLP head (from the dashed line in
the above figure) is omitted in segmentation tasks.

][c]0.48

(b) Up: Transformer Encoder block visualization, down-left: Multi-Head Self-
Attention (MHSA) block, down-right: Attention mechanism block feed by the
query (Q), key (K), and value (V) representations learned from the input
embedded patches. LN and MLP denote the Layer Normalization [132] and Multi-
Layer Perceptron operations, respectively.

Figure 20: The above figure portrays the ViT [131] pipeline with all of its
detailed counterparts from a top-down view.

###  3.4 Transformers

Inspired by the recent success of the Transformer models in Natural Language
Processing (NLP), these models were further extended to perform vision
recognition tasks. More specifically, the Vision Transformer model was
introduced by Dosovitskiy et al. [131] to alleviate the deficiency of CNNs in
capturing the long-range semantic dependencies. Before going deeper into
transformer-based methods, it might be practical to review the concept of
vision transformers and the mechanism of self-attention utilized in these
networks.

Contrary to the Transformers in NLP tasks [133], the computer vision tasks
usually contains more than one dimensional data (e.g., 2D image, 3D video)
which needs to be prepared for the transformer model. Hence, ViT’s pipeline
starts with image sequentialization (see Figure 20(a)) process to prepare the
tokenized sequence for the encoder module. From now on, the words patch and
token will be used interchangeably.

If $x\in\mathbb{R}^{H\times W\times D\times C}$ is a volumetric 3D image with
a $\left(H,W,D\right)$ spatial resolutions and $C$ input channels, first the
$x$ is dividing into $N=\frac{H\times W\times D}{P^{3}}$ flattened uniform,
non-overlapping patches $x_{p}^{i}\in\mathbb{R}^{N\times(P^{3}.C)}$ ,
$i\in\\{1,\cdots,N\\}$ with $(P,P,P)$ spatial resolution for each patch,
therefore each patch is representing by a 1D sequence with a length of
$1\times(P^{3}.C)$. Afterward, a linear layer applies on top of the sequence
to map them to a $K$ dimensional embedding space. In order to retain the
positional information of patches, a 1D learnable positional encoding
$\mathbf{E}_{\text{pos}}\in\mathbb{R}^{N\times K}$ adds to patch embedding as
follows:

$\displaystyle\mathbf{z}_{0}=\left[x_{\text{class}};x_{p}^{1}\mathbf{E};x_{p}^{2}\mathbf{E};\cdots;x_{p}^{N}\mathbf{E}\right]+\mathbf{E}_{\text{pos}},$
(8)

where $\mathbf{E}$ denotes patch embedding operation and the class
token,$x_{\text{class}}$, omitable in segmentation tasks. In the next step,
the embedded patch feed to the stack of Transformer encoder blocks ($L\times$)
containing the Multi-Head Self-Attention (MSHA), Multi-Layer Perceptron (MLP),
and Layer Normalization [132] sub-blocks to generate the latent
representation. The following formulations show the mathematical process in
Transformer encoder:

$\displaystyle\hat{\mathbf{z}}^{l}$
$\displaystyle=\text{MSA}(\text{LN}(\mathbf{z}^{l-1}))+\mathbf{z}^{l-1}$
$\displaystyle\mathbf{z}^{l}$
$\displaystyle=\text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l}))+\hat{\mathbf{z}}^{l},$
(9)

where $l\in\\{1,\cdots,L\\}$, $\hat{\mathbf{z}}^{l}$, and $\mathbf{z}^{l}$
denote output of MHSA operation and MLP function, respectively.

From Figure 20(b) the MHSA block comprises $h$ parallel Self-Attention sub-
blocks that perform the attention (Scaled Dot-Product Attention) $h$ times
with different Qeury ($Q$), Key ($K$), and Value ($V$) matrices from the input
1D sequence, $\mathbf{z}^{l}\in\mathbb{R}^{N\times K}$. The attention function
is a mapping operation between query and key-value pairs to an output that
measures the similarity between two components in $\mathbf{z}$ as:

$\displaystyle\text{SA}(\mathbf{z})=\text{SoftMax}(\frac{QK^{T}}{\sqrt{K_{h}}})V,$
(10)

where $\sqrt{K_{h}}$ denotes a normalization factor to preserve the attention
matrix (Equation 10) from the possible gradient vanishing or exploding through
the training. Furthermore, the output of MHSA derives from the concatenation
of multiple heads:

$\displaystyle\text{MHSA}=\text{Linear}([\text{SA}_{1}(\mathbf{z});\text{SA}_{2}(\mathbf{z});\cdots;\text{SA}_{h}(\mathbf{z})]).$
(11)

So far, we have briefly introduced the ViT pipeline and the related
mathematics. In the next sections, we will discuss the integration of the
Transformer into the U-Net structure in medical segmentation. We categorized
the presence of Transformers in U-shaped networks into two sub-categories: (a)
Transformer as a complement to CNN-based U-Net-like structures and (b)
U-shaped standalone Transformer architectures.

Figure 21: The overview architecture of the TransUNet. The Transformer layers
are employed in the encoder part. The schematic of the Transformer is shown on
the left. Figure from [38].

####  3.4.1 Transformer Complement to CNN-based U-Net

The success of convolutional neural networks (CNN) in diverse dense prediction
tasks in the vision domain, e.g., segmentation, is noticeable. Their
performance is underlined in their multi-scale representation and ability to
capture local semantic and texture information. However, the local
representation derving from the CNN architecture might not be robust enough to
capture geometrical and structural information existing in the medical data.
Therefore, there is a need for a mechanism to capture inter-pixel long
relations to extend the performances of the existing CNN-based U-Net variants
suffering from the limited receptive field of convolutional operations. Chen
et al. [38] proposed one of the first studies that utilized the Vision
Transformer (ViT) in the U-Net structure to compensate for the U-Net’s
disability in long-range modeling dependencies, namely TransUNet (See Figure
21). The stacked Transformers in the encoder path feed with the tokenized
paths from abstract features extracted within the primal input to extract
global contexts. The decoder path upsamples the encoded features combined with
the high-resolution CNN feature maps to enable precise localization. Chen et
al. clarified that the naive Transformer is not fit for downstream tasks like
segmentation well due to its 1D functionality for capturing the interaction of
the tokenized information. Therefore, they proposed this complementary
Transformer design with U-Net, which they conducted several ablation studies
to prove their superiority within the conventional attention collaborated
networks such as Attention U-Net [70] on Synapse [134] and ACDC [101]
datasets.

TransUNet is a 2D network that processes the volumetric 3D medical image
slice-by-slice, and due to its seminal ViT adaptation for its building blocks,
it relies on pre-trained ViT models on large-scale image datasets. These
restrictions made Wang et al. [39] point them out and propose TransBTS as a
U-Net-like architecture, modeling local and global information in spatial and
slice/depth dimensions. While Transformer’s computational complexity is
quadratic and the volumetric 3D data is large, on the other hand, ViT’s fixed-
size tokenization process [131] discards the local structural information,
TransBTS utilizes the 3D CNN backbone for its encoder and decoder path to
capture local representation across spatial and depth dimensions and unleash
from the high computational burden for the Transformer counterpart in overall.
The essential key points in the amalgamation of the Transformer with the
encoded low-resolution with high-level representation flow come from CNN
blocks, are linear projection and feature mapping blocks, where the
input/output signals reshape and downsample to be compatible for their usage.
This hybrid network captures the local and global information from 3D data and
demonstrates the improved performance within two Brain Tumor Segmentation
(BraTS) 2019-2020 [91, 135, 136] datasets over the previous CNN U-Net
structures.

Li et al. [40] proposed the GT U-Net structure to address the low performance
of previous segmentation methods in fuzzy boundaries while keeping the
computational complexity low within the hybrid structure of CNN and the
Transformer in a U-Net-like paradigm. Their method was applied to the private
orthodontist tooth X-ray images and DRIVE dataset [137]. All the main
counterparts of U-Net are based on Group Transformer (GT) to dispense the
quadratic computational complexity within these successive parallel
convolution, Multi-Head Self-Attention (MHSA), convolution modules in each
stage to gradually increase receptive filed and extracting long local-
dependencies. So far, the presence of a Transformer in the segmentation tasks
is crucial because if a network wants to provide an efficient prediction mask,
it should be able to minimize the miss-classifying of the background and
foreground pixels that leads to a reduction in False Positives (FP). Therefore
learning long-range contextual features is as essential as fuzzy boundaries
resulting from object overlappings or variation in exposure of the medical
imaging devices. To mitigate the occurrence of this miss predicting in
boundary levels, GT U-Net utilizes a Fourier descriptor loss term within
binary cross entropy to impose the prior shape knowledge.

Xie et al. [41] addressed the computational complexity that restrains the
multi-scale functionality of conventional Self-Attention (SA) and proposed the
hybrid CoTr architecture for volumetric medical image segmentation. The whole
network is a U-Net-like structure with CNN-based 3D residual blocks for
encoder and decoder paths with the amalgamation of Deformable Transformer
(DeTrans) for multi-scale fusion, besides the conventional skip connections
from the encoder to the decoder for better localization information and faster
convergence. TransUNet suffers from parameters overload within MHSA, which
treats all image tokenization positions equally. Therefore, CoTr instantiates
the deformation concept from [126, 138] into the deformable self-attention
mechanism in Transformer to decrease the computation complexity and prepare
the ground for using Transformer to process multi-scale and high-resolution
feature maps. MS-DMSA layer is a deformable transformer instead of MHSA that
focuses on only a small set of key sampling locations around a reference path.
CoTr demonstrates the competitive results in a score-parameter trade-off on
The Multi-Atlas Labeling Beyond the Cranial Vault (BCV) [89] dataset.

UNETR [139] is a 3D segmentation network that directly utilizes volumetric
data incorporating ViT solely at the encoder stage to capture global multi-
scale contextual information in a 3D volumetric style which is usually of
paramount importance in medical image segmentation domain. The architecture
follows the U-shaped structure of [7] with skip connections carrying
successive 3D convolution operations to the 3D CNN-based decoder. Using a CNN-
based decoder is since transformers can not capture spatial localization
information well despite their excellent capability of learning global
information. Analogous to U-Net, Hatamizadeh et al. [139] uses the different
stages of Transformer in the encoder to pass the flow from the contracting
path to extracting path, and the multi-resolution contextual information
(after reshaping the embedded sequence to a proper tensor shape and applying
convolution operations) merges with CNN-based decoder to improve the
segmentation mask prediction. UNETR produces uniform, non-overlapping patches
from volumetric data and applies a linear projection to project patches into a
constant embedding dimensional space throughout the Transformer layers. Their
ablation studies depict that they outperformed the TranUNet [38], TransBTS
[39], and CoTr [41] on BCV [89], and MSD [90] datasets on an average of 1%
margin in the dice score metric.

In computer vision tasks, neighboring information of a specific region tends
to be more correlated than far regions. To this end, Wang et al. [43] proposed
the MT-UNet network utilized with the Mixed Transformer Module (MTM) to
capture long-range dependencies wisely concerning the most neighboring
contextual information. Another critical point is that the ViT with Self
Attention (SA) calculates the intra-tokens affinities, ignoring the inter-
tokens connections dispensed through the other dimensions, especially in
medical images. Therefore, MTM consists of an External Attention (EA)
counterpart in itself to address this concern. MTM is used in conjunction with
a U-Net-like structure accompanied by CNN blocks. CNN blocks are used to not
only reduce the computational overhead by downsampling the input feature maps
but also introduce a structure prior to the model in the case of small medical
datasets. MT-UNet performs well on Synapse [89] and ACDC [101] datasets in
comparison with TranUNet [38].

Azad et al. [44] proposed a contextual attention network, namely TMU, for
adaptively synthesizing the U-Net produced local feature with the ViT’s global
information for enhanced overlap boundary areas in medical images. TMU is two
branches pipeline, wherein the first stream utilizes a U-Net-like block
without a segmentation head (Resnet backbone [121]) to extract high semantic
features and object-level boundary heatmap interaction representation. In the
next branch, the ViT-based Transformer module applies to non-overlap input
images to extract long-range dependencies. Whereas the objective of
segmentation differs from one subject to another data, as mentioned before,
TMU aims to merge the local and global information adaptively. To do so, Azad
et al. proposed a contextual attention mechanism to produce image-level
contextual information and highlight the most discriminative regions within
importance coefficients delivered by attention weights from Transformer. This
paradigm not only revealed the efficiency of the boundary information as a
prior and adaptive collaboration of local and long-range dependencies but also
outperforms the conventional hybrid and solely CNN-based methods on SegPC
challenge dataset [140, 141, 142] and skin lesion segmentation datasets [143,
144, 145].

Figure 22: Overview of the UCTransNet architecture. The original skip
connections of U-Net architecture are replaced with the proposed CTrans Module
including Channel-wise Cross fusion Transformer (CCT) and Channel-wise Cross
Attention (CCA). CCT with the Multi-head Cross-Attention module is illustrated
on the left. Figure from [146].

Skip connections in the U-Net-based model are used to transfer high spatial
information from the encoder to the decoder for accurate localization, while
the successive downsampling operations suffer from the loss of spatial
information. However, Wang et al. [45] studied the effectiveness of the
preliminary U-Net skip connections and stated that the naive skip connections
suffer from the highly semantic gap such as semantic gaps among multi-scale
encoder features and between the encode-decoder stages. They proposed
UCTransNet [45] that alleviates these mentioned issues from the channel
perspective with an attention mechanism, namely Channel Transformer (CTrans).
CTrans is a modification for skip connections in a U-Net-based pipeline and
consists of two sub-counterparts Channel Cross fusion with Transformer (CCT)
and Channel-wise Cross-Attention (CCA), for aggregating multi-scale features
adaptively and guiding the fused multi-scale channel-wise features to decoder
effectively, respectively (see Figure 22). CCT aims to fuse multi-scale
encoder features to adaptively compensate for the semantic gap between
different scales with the advantage of long-range dependency modeling in the
Transformer. CCT tokenized feature maps at each stage within patch sizes from
a multiple of $\frac{1}{2}$, preserving the channel dimensions. From Figure
22, the proposed CCT module accompanies tokenized feature maps as a query and
concatenated four tokens come from stages as key and value matrixes. With the
use of instance normalization [147] operation for gradient smoothing through
the process, the primary distinction between the CCT and Self Attention (SA)
is that the attention operation applies on the channel axis rather than the
patch axis. Afterward, to rectify the gap between the encoder and decoder’s
inconsistent feature representation, the output tokens of CCT pass through CCA
to apply a better fusion step and lessen the ambiguity with the decoder
feature. The UCTransNet network performs SOTA Dice results on the GlaS [148],
MoNuSeg [149, 150] and Synapse [89] datasets in comparison with TransUNet
[38].

Figure 23: Overview of the ScaleFormer [46] architecture. The input image is
first passed through the CNN Block to extract the local details. The output
features of the last several layers in the CNN block are then fed into the
Intra-Scale Transformers to model the global information for each scale.
Spatial-Aware Inter-Scale Transformer fuses the outputs of Inter-scale
Transformer modules to enable the interaction among different scales. Finally,
the decoder block performs upsampling and concatenation with features of
corresponding scales to produce the segmentation prediction.

Similar to UCTransNet, Huang et al. [46] addressed the inconsistency between
local and global features in inter and intra-scales in conventional
architectures (hybrid / standalone) [38, 48, 49, 151] and proposed a
ScaleFormer backbone based on a U-Net-like structure which during this study
is the SOTA method in 2D modality. Their innovations through their design are
to couple CNN-based features within long-range contextual features in each
scale effectively within lightweight Dual-Axis MSA captures attention in a
row/column-wise manner. In addition, ScaleFormer [46] make a bridge with a
spatial-aware inter-scale Transformer to interact with the target regions’
multi-scales features to surpass the shape, location, and variability of
organs’ limitation. ScaleFormer utilizes ResNet [121] variant backbone, basic
ResNet-34 blocks for CNN-feature extractor, and in each stage, scale-wise
intra-scale transformer (Dual-Axis MSA) couples with the local features to
highlight both detailed local-spatial and long-spatial affinities in each
scale. To alleviate the deficiencies of previous methods in capturing
sufficient information from multiple scales by a hierarchical encoder,
spatial-aware inter-scale Transformers merge these features adaptively to
strengthen the ScalFormer in effectively segmenting various-scale organs. From
Figure 23, the inter-scale Transformer is a computation-efficient design by
applying successive point-wise convolutions followed by average-pooling on
row/column-wise query and key matrices of the Transformer. This pipeline
embraces the whole operations in a single block rather than multi blocks for
row and column on input data [152]. The spatial-aware inter-scale Transformer
is a conventional Transformer with a difference in interaction calculation of
tokens cue. To be more precise, each input token to this Transformer first
reshapes to its 2D representation. Every 2D representation of specific tokens
in each scale concatenates with their successive 2D patch feature map in the
following scales and then flattens to its 1D representation by producing a
master token for that specific token and applying the standard Transformer
calculation to it. Afterward, the enhanced representations are aggregated in
the decoder path with local-level CNN features on the same stage in each skip
connection to each decoder block. ScaleFormer proves its functionality through
multiple datasets [89, 149, 150, 101] by outperforming TransUNet [38], Swin-
Unet[48], MISSFormer [49] and AFTer-UNet [151].

Figures 21 and 23 show sample CNN-Transformer U-shaped structures that
Transformer is an add-on to a U-Net-like network to model the long-range
contextual information in diverse stages of U-Net from encoder-decoder to skip
connection and bottleneck. The Swin UNETR [42] is a modification to the
original UNETR [139] that the 3D Vision Transformer replaced by the Swin
Transformer in encoder path. There are still other studies such as [153, 154]
noteworthy to review, but due to the limitation of the paper, we excluded
them.

][c]0.48

(a) Overview of the MedT architecture. The network uses the LoGo strategy for
training. The upper global branch utilizes the first fewer blocks of the
transformer layers to encode the long-range dependency of the original image.
In the local branch, the images are converted into small patches and then fed
into the network to model the local details within each patch. The output of
the local branch is re-sampled relying on the location information. Finally, a
$1\times 1$ convolution layer fuses the output feature maps from the two
branches to generate the final segmentation mask. Figure from [155].

][c]0.48

(b) The architecture of the Swin-Unet follows the U-Shape structure. It
contains the encoder, the bottleneck, and the decoder part which are built
based on the Swin transformer block. The encoder and the decoder are connected
with skip connections. Figure from [48].

][c]0.48

(c) Overview of the MISSFormer architecture. The network is composed of a
hierarchical encoder, a decoder, and an Enhanced Transformer Context Bridge.
The encoder and decoder are constructed based on the enhanced transformer
blocks and modules for patch processing. The outputs of each stage within the
encoder are fused and passed through the bridge to model the local and global
dependencies of different scales. Figure from [49].

][c]0.48

(d) Overview of the D-Former architecture. One dynamic position encoding block
(DPE), multiple local scope modules (LSMs), and global scope modules (GSMs)
constitute each D-Former block. Figure from [50].

Figure 24: Overview of architectures mentioned in Section 3.4.2

][c]0.3

(a) The calculations through the Swin block are:
$\hat{\mathbf{z}}^{l}=\text{W-MSA}(\text{LN}(\mathbf{z}^{l-1}))+\mathbf{z}^{l-1}$,
$\mathbf{z}^{l}=\text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l}))+\hat{\mathbf{z}}^{l}$,
$\hat{\mathbf{z}}^{l+1}=\text{SW-
MSA}(\text{LN}(\mathbf{z}^{l}))+\mathbf{z}^{l}$,
$\mathbf{z}^{l+1}=\text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l+1}))+\hat{\mathbf{z}}^{l+1}$,
$\hat{\mathbf{z}}^{l}$ and $\hat{\mathbf{z}}^{l+1}$ denote the output features
of the W-MSA and SW-MSA modules, respectively.

][c]0.3

(b) The Efficient Mix-FFN block applies the following operations:
$y_{1}=\text{LN}(\text{Conv}_{3\times
3}(\text{FC}(x_{in}))+\text{FC}(x_{in}))$,
$x_{out}=\text{FC}(\text{GELU}(y_{1}))+x_{in}$.

][c]0.3

(c) The LSM and GSM modules (main counterparts of the D-Former block) apply
the following formulations:
$\hat{\mathbf{z}}^{l}=\text{LS-
MSA}(\text{LN}(\mathbf{z}^{l-1}))+\mathbf{z}^{l-1}$,
$\mathbf{z}^{l}=\text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l}))+\hat{\mathbf{z}}^{l}$,
$\hat{\mathbf{z}}^{l+1}=\text{GS-
MSA}(\text{LN}(\mathbf{z}^{l}))+\mathbf{z}^{l}$,
$\mathbf{z}^{l+1}=\text{MLP}(\text{LN}(\hat{\mathbf{z}}^{l+1}))+\hat{\mathbf{z}}^{l+1}$,
$\hat{\mathbf{z}}^{l}$ and $\hat{\mathbf{z}}^{l+1}$ denote the output features
of the LS-MSA and GS-MSA modules, respectively.

Figure 25: Overview of main Transformer counterpart blocks mentioned in
Section 3.4.2. (a) Swin Transformer block [156, 48], (b) MISSFormer Enhanced
Transformer block [49], (c) D-Former Transformer block [50]. MLP, FC, and LN
operations indicate the Multi-Layer Perceptron, Fully Connected, and Layer
Normalization [132], respectively.

####  3.4.2 Standalone Transformer Backbone for U-Net Designs

So far, multiple studies incorporating the Transformer concept and
conventional CNN modules have been reviewed in Section 3.4.1. In this section,
we investigate the usage of Transformer as a standalone main counterpart for
designing backbones for U-Net-like structures. One of the pioneering
structures in this domain was proposed by Valanarasu et al. [47], namely MedT.
Like most of the other networks, MedT plans to contribute to capturing long-
range spatial context with pure Transformer rather than the CNN-based methods
that partially broaden the hindered-receptive field of CNN, e.g., D-UNet [65]
with deformable convolution operations [126], ASPP-FC-DenseNet [81] with
atrous convolution operations [157], and H-DenseUNet [62] with successive
convolution operations. However, Transformer’s performance (also ViTs) has a
strong bond with the fed data scale to the Transformer module [131], which in
the medical scale, could be degraded more, and a high amount of data could not
be available. This lack of data is a considerable corner point in learning
positional encoding as one of the preliminary steps of Transformers, which
have shown their capacity to model images’ spatial structure. Therefore MedT
[47] proposes a gated axial-attention mechanism to control the information
flow by positional embeddings to query, key, and value [158] in a multi-axis
attention operation [152]. In [158] the accurate relative positional encoding
learned on large-scale datasets rather than small-scale datasets improves the
performance, therefore MedT introduces a gating parameter to control the
amount of positional bias in capturing non-local information in hindering non-
accurate positional embedding. In addition, to effectively extract
information, MedT utilizes a Local-Global (LoGo) training strategy to
compensate for the Transformer’s patch-wise technique weakness in capturing
inter-patch pixel dependencies. To do so, MedT investigates two branches in
its network diagram (see Figure 24(a)), one as a global branch to work on the
original resolution of the image and the local branch that operates on the
patches of the image. Overall, MedT demonstrated vanguard results on Brain US
[159, 160], Glas [148], and MoNuSeg [149, 150] datasets in Dice and IoU
metrics.

Transformers are well capable of capturing long-range dependencies through
data, however, they suffer from severe and inevitable handicaps that impede
them from their versatile use in vision tasks. These shortages commonly are
correlated with chains to each other, e.g., Transformers computational
complexity is quadratic [131, 161] and this restrains its usability in dense
vision tasks such as segmentation and detection, which needs the neighboring
pixel dependencies in the multi-scale and hierarchical pattern. Due to the
fixed size non-overlapping tokenization step in the naive Transformer rather
than the pixel-by-pixel calculation of attention to diminishing the mentioned
computational burden, the Transformer is unworthy of extracting the local
contextual dependencies in intra-path pixels. These constraints make the
interest to provide efficient Transformers, Linear Transformers, with a
significant amount of reduction in parameters and computational complexity
[162]. In the vision tasks, Swin Transformer [156] plays a critical role as an
efficient and linear Transformer with the capability of supporting
hierarchical architectures. A key design counterpart of the Swin Transformer
is its shifted windowing scheme that makes the Transformer calculate the
affinities for patches in the same window. Afterward, the window swipes on the
patches, and the attention calculates among the patches in the same window.
This successive shifting operation and capturing local contextual information
within patches in windows can stack multiple times. Ultimately a patch merging
layer is introduced to build CNN-like hierarchical feature maps by merging
image patches in deeper layers. This intuition and the U-Net-like structure
success emerged Swin-Unet [48] structure in the medical segmentation field.
From Figure 24(b), Cao et al. [48] used the Swin Transformer block as the main
counterpart of their U-shaped network. 2D medical images split into non-
overlapping patches, and each patch fed into the encoder path comprised of
Swin blocks. The contextual features from the bottleneck output upsample in
the decoder path with patch expanding layer (contrary to path merging layer)
end couples with the multi-stage features from the encoder via skip
connections to restore the spatial information. Swin-Unet presented SOTA
results over the CNN-Transformer hybrid structures like TransUNet [38] and
demonstrated the robust generalization ability with the help of two multi-
organ (Synapse) [89] and cardiac (ACDC) [101] segmentation datasets.

Due to some intrinsic features of medical images or, more specifically, the
properties of human body organs, e.g., multi-scale and deformations, [7], the
necessity to capture long-range dependencies for accurate segmentation
boundaries [47] and inherence of generalizability of models even with no pre-
training strategies and applicable for low-data regime [108, 109, 163], Huang
et al. [49] proposed MISSFormer a pure U-shaped Transformer network to address
these apprehensions. Figure 24(c) displays the MISSFormer network with
enhanced Transformer block as a primary entity in the network. One of the
Transformer’s drawbacks mentioned earlier is their un-suitability for
capturing local context [164, 165], which comes with the solution for
lessening the computational complexity with patching operation. However, the
local contextual information plays a pivotal role in high-resolution vision
tasks, therefore some studies in the vision Transfomer domain tackle this
problem by embedding convolution operations in their attention module, e.g.,
PVTv1 [166], PVTv2 [167], and Uformer [168]. Huang et al. [49] argues this
methodology and brings up the point that direct usage of convolution layers in
Transformer blocks limits the discrimination of features. For an input image,
MISSFormer applies a $4\times 4$ convolutions with the stride size instead of
$4$ (overlapping windows) for preserving local continuity in building the
patches stage. The encoder path molds hierarchical representation (see Figure
24(c)) with the help of Enhanced Transformer Block, which accompanies the
Figure 25(b) Transformer module. Enhanced Transformer block comprises an
Efficient Self-Attention module to decrease the traditional attention
calculation by downsampling the corresponding query, key, and value matrixes.
Afterward, the Enhanced Mix-Feed Forward Network (FFN) sub-module (modified
clone of Mix FFN from [169]) aligns features and makes discriminant
representation with $3\times 3$ depth-wise convolutions for capturing the
local context efficiently. Analogous to [45], MISSFormer rethinks the skip
connection design, utilizes the Enhanced Transformer Context Bridge module for
multi-scale information fusion, and hinders the gap between encoder and
decoder feature maps. This module captures the local and global correlations
between different scale features. For a brief review of this module, the
context bridge (Figure 24(c)) flattens the output attention matrixes from
different scales in the spatial dimension and reshapes it in a way to have a
consistent channel dimension and concatenates the representations in flattened
spatial dimension and feed to enhanced Transformer block to produce long-range
dependencies and local context correlations. Ultimately the output of Enhanced
Mix-FFN is split and restores to its original shape to get the discriminant
fused hierarchical multi-scale representation. MISSFormer plotted high
performances compared to TransUNet [38] and Swin-Unet [48] over Synapse [89]
and ACDC [101] datasets.

Since most U-Net-shaped standalone Transformer utilized models incorporate 2D
inputs, Wu et al. [50] proposed a Dilated Transformer (D-Former) for 3D
medical image segmentation with consideration of degrading the computational
complexity. D-Former [50], Figure 24(d), is a U-shaped hierarchical
architecture with specialized D-former blocks which consist of Local Scope
Modules (LSMs) and Global Scope Modules (GSMs) in an alternate order to
capture local and global contextual information. Each D-Former block could
repeat the successive LSM and GSM counterparts, however the original D-Former
[50] uses three successive sequences of LSM-GSM modules in the third and sixth
D-Former blocks. LSM captures local self-attention by dividing a 3D feature
map into non-overlapping 3D-volumetric units that consist of 3D patches and
calculating the self-attention within these 3D units. As a complement to the
locally fine-grained attention, the GSM module attains interactions through
different units in a dilated manner to enlarge the attention’s performance
range. This intuition captures local and global interactions with fixed-size
patches within sets of 3D volumetric units without any computation overflow.
Also, D-Former takes advantage of positional encoding within Transformers with
Dynamic Positional Encoding (DPE), which can be a vital cue in dense
prediction tasks such as segmentation. Due to the 3D functionality and 3D
contextual information, D-Former is a SOTA method on various 3D datasets such
as Synapse [89] and ACDC [101] datasets with large margins in Dice score over
CNN-hybrid or standalone Transformer designs.

Still, numerous studies out there, e.g., DS-TransUNet [170] utilize the Swin
Transformer in the dual-path U-shaped structure for modeling the multi-scale
patch size to lessen the deficiency of Transformers in capturing the local
context. We reviewed multiple studies that utilized the Transformer in their
U-Net pipeline in various methods. With such evidence and stunning growth in
the Transformer field, we still hear about the outstanding collaboration
between Transformers and U-Net-like networks so often.

###  3.5 Rich Representation Enhancements

To obtain a rich representation, the common approaches applied to medical
image segmentation are multi-scale and multi-modal methods, e.g. [171, 172,
173]. The key objective is to enhance the performance of the trained models by
utilizing all available information from multi-modal or multi-scale images
while retaining the most desirable and relevant features.

The multi-scale method, also referred to as the pyramid method originated from
the Laplace pyramid method proposed by Burt et al. [174]. The approach
converts the source input image by resizing it into a series of images with
decreasing spatial resolutions. This scheme allows encoders of models to
directly access the features of the enhanced images of different sizes and
thus learn the respective features.

The study of the organs of interest requires their specific imaging modality
to provide targeted information. However, each imaging technique has its
limitations and can only reveal partial details about the organ, which may
lead to inaccurate clinical analysis. Therefore, a fusion of images from
various imaging modalities can be conducted to supplement each other’s
information by integrating complementary information retrieved from several
input images.

The powerful structural design of the UNet network with the encoder and
decoder allows the network to mine salient features at multiple input levels
and enables effective feature fusion of different modalities.

Lachinov et. al. evaluate the performance of the Cascaded U-Net[58] with
multiple encoders processing each modality respectively to demonstrate the
improvement due to the extraction multi-modal representation. The results
indicate that the architecture taking the multiple modalities into account
outperforms the network only relying on one single modality.

The following classifications will illustrate the modality fusion proposed to
learn richer representations.

####  3.5.1 Multi-scale Fusion

Image pyramid input or side output layers are aggregated into U-Net structures
to fuse the multi-scale information in the encoder or decoder stage.

Abraham et al. [18] propose the Focal Tversky Attention U-Net with a
generalized focal loss function that modulates the Tversky index [175] to
address the issue of data imbalance and improve precision and recall balance
in medical image segmentation. Furthermore, they incorporate multi-scale image
inputs into the attention U-Net model with deep supervision output layers
[70]. The novel architecture facilitates the extraction of richer feature
representations and results in 3% dice score improvement on multi-class CT
abdominal segmentation task. Compared to the commonly used Dice loss, the
Tversky similarity index introduces a specific weight for each class, which is
inversely proportionate to the label occurrences. This index shown as follows
alleviates precision and recall imbalance due to equal weighting of false
positive (FP) and false negative (FN).

$\displaystyle TI_{c}$
$\displaystyle=\frac{\sum^{N}_{i=1}p_{ic}g_{ic}+\epsilon}{\sum^{N}_{i=1}p_{ic}g_{ic}+\alpha\mathbf{A}+\beta\mathbf{B}+\epsilon}$
(12) $\displaystyle\mathbf{A}$
$\displaystyle=\sum^{N}_{i=1}p_{i\bar{c}}g_{ic},\qquad\mathbf{B}=\sum^{N}_{i=1}p_{ic}g_{i\bar{c}}$

where, $\mathbf{A}$ and $\mathbf{B}$ refer to FN and FP respectively. In the
terms $\mathbf{A}$ and $\mathbf{B}$, $p_{ic}$ denotes the probability that the
pixel $i$ is classified to the lession class $c$ and $p_{i\bar{c}}$ is the
probability pixel $i$ belongs to the non-lesion class $\bar{c}$. $g_{ic}$ and
$g_{i\bar{c}}$ are of the same form for ground truth pixels. Adjusting the
weights $\alpha$ and $\beta$ to fine-tune the emphasis can enhance recall in
case of significant class imbalance.

The authors further develop a focal Tversky loss function (FTL) for better
small regions-of-interest (ROIs) segmentation by forcing the function to shift
the focus on less accurate and misclassified predictions.

$FTL_{c}=\sum(1-TI_{c})^{1/\gamma}$ (13)

where $\gamma$ is set in the interval $[1,3]$ to enable the loss function to
concentrate more on incorrectly classified predictions that are less accurate.
The reason is that when $\gamma>1$, the FTL is almost unaffected if a pixel is
misclassified with a high Tversky index. And if a pixel is incorrectly
classified with a small Tversky index, the FTL will be high and forces the
model to focus on hard samples.

Figure 26: Architecture of the Focal Tversky Attention U-Net with multi-scale
input and deep supervised output layers. AGs indicate soft attention gates
which combine spatial information from low-level feature maps with coarser
contextual information from skip connections. Figure from [176].

As shown in Figure 26, they use Soft Attention Gates (AGs) to prune features
and propagate only relevant spatial information to the decoding layers to
enhance the balance between precision and recall at a structural level. In
addition, an input image pyramid injected into each of the max pooling layers
in the encoder and deep supervision module enriches feature learning at
different scales.

To better segment the Optic Disc (OD) and Optic Cup (OC) for accurate
diagnosis of glaucoma from fundus images, Fu et al. [55] introduce the polar
transformation into the U-shape convolutional network with multi-scale input
layers to build the Polar Transformation M-Net, aims to extract the richer
context representation of the original image in the polar coordinate system.
Compared to prior work, which treats OD and OC individually while ignoring
their interdependency and overlap, the proposed Polar transformation M-Net
considers both OD and OC simultaneously and formulates the segmentation as a
multi-label task. Moreover, a novel loss function built on the Dice
coefficient is employed to address the data imbalance between OD and OC in the
fundus images. The model aggregates the side-output layers serving as early
classifiers that generate prediction maps at different scales.

Figure 27: Illustration of the Polar Transformation M-Net method. The upper
part of the figure shows the overall pipeline. The lower part demonstrates the
architecture of M-Net. The method first localizes the disc center to extract
the ROI, then applies the polar transformation to the detected patches of ROI.
The transferred results are passed through the M-Net to obtain multi-label
maps. The inverse polar transformation restored the final segmentation
prediction to the Cartesian coordinate. The M-Net architecture is composed of
an input image pyramid, U-shape convolutional network, and side-output layers.
Figure from [177].

Figure 27 demonstrates the overall structure of M-Net. It consists of a multi-
scale input layer, U-shape convolutional network, side-output layer, and
multi-label loss function. The initial multi-scale layer constructs the image
input with descending resolutions for hierarchical representation learning.
The processed image pyramid is passed through a U-shape network similar to the
original U-Net architecture [7] whose encoder path and decoder path are
concatenated by the skip connections. The output of each stage within the
decoder path is fed to a side-output layer that produces a local output map.
The superiority of the side-output layer is that the issue of gradient
vanishing could be mitigated by the backpropagation of the side-output loss
together with the final layer loss. Since the disc region overlays the cup
pixels, the authors evaluate segmentation performance by the proposed multi-
label loss function based on the Dice coefficient, which is defined as:

$L_{s}=1-\sum^{K}_{k}\frac{2w_{k}\sum^{N}_{n}p_{(k,i)}g_{k,i}}{\sum^{N}_{i}p^{2}_{(k,i)}+\sum^{N}_{i}g^{2}_{(k,i)}}$
(14)

where $K$ is the total number of classes and $N$ denotes the number of pixels.
$p_{(k,i)}$ and $g_{(k,i)}$ are the predicted probability and binary ground
truth respectively. Class weights $w_{k}$ control the contribution of each
class to the final results. Another contribution of the network is the pixel-
wise polar transformation operated on the fundus image plane. The polar
transformation can map the radial relationship that OC should be within the OD
to the layer-like spatial structure, which makes the features easier to
identify. Additionally, the interpolation during the polar transformation
based on the OD center could expand the cup region. It helps relieve the
heavily biased distribution of OC/background pixels. The experiments on the
ORIGA dataset have demonstrated an increase in segmentation performance.

The original U-Net may not fully exploit all contributions of the semantic
strength since it only generates output segmentation maps from the final layer
of the decoder path. Moreover, the output of the layers in different steps
cannot be connected to one another, which blocks feature sharing and leads to
redundant parameters. To address the mentioned issue, Moradi et al. proposed
MFP-UNet which allows the output of all the blocks in different stages to be
fed to the last layer [178]. Their architecture is composed of two pathways,
the “bottom-up pathway” and the “top-down pathway”. The encoder of the UNet
with dilated convolution filter serves as the ”bottom-up pathway”. The dilated
convolutional kernel can increase the receptive fields of the module by the
dilation factor. Besides, the expansion path of U-Net acts as the FPN top-down
pathway. Each step of the top-down pathway provides prediction maps where
lower-resolution semantically stronger features can be processed for transfer
to higher resolutions. Additional convolution layers are included for
processing the feature maps at different scales to one fixed resolution
compared to the decoder path of the original U-Net, which can boost the
accuracy and improve the resolution of each stage. According to the experiment
results, the novel model provides a robust and powerful architecture regarding
the capabilities of feature representation in a pyramid, which shows
robustness to large and rich training sets.

####  3.5.2 Multi-modality Fusion

In this section, we summarize the U-Net variant models with multimodal fusion
modules, where a single encoder of U-Net is extended to multiple encoders to
receive medical images in different modalities. The branches of encoders are
connected by their respective strategies of aggregation, thus sharing
information in different modalities, extracting richer representations, and
complementing each other.

The novel architecture Dolz et al. [57] propose in the Dense Multi-path U-Net
enhances traditional U-Net models regarding rich representation learning on
two key aspects: modality fusion and inception module extension. Two typical
strategies are employed to deal with multi-modal image segmentation tasks. The
early fusion merges the low-level features of inputs of multiple imaging
modalities at the very early stage. As for the late fusion strategy, the CNN
outputs of different modalities are fused at a later point. Nevertheless,
these previous strategies cannot thoroughly model the highly complex relation
of the image information across different paths of modalities. To alleviate
the limitation, the proposed HyperDenseNet adopts the strategy where each
stream receives inputs of image data of one modality, and the layers in the
same and different paths are densely connected.

Figure 28: The architecture of network for segmenting ischemic stroke lesions
across various imaging modalities. In the encoding path, each imaging modality
is input into a single stream. All layers are directly connected to one
another in a single stream, which facilitates the information flow of the
network. The dashed lines depict the dense connections between the layer
outputs in the streams of different modalities. Figure from [179].

As shown in Figure 28, the encoding path contains N streams, each responsible
for one imaging modality. The Dense Multi-path U-Net supports Hyper-dense
connections both within a single path and between several paths. In a densely-
connected network, the output of the $l^{th}$ layer is produced by the mapping
$H_{l}$ of the concatenation of all the feature layers.

$x_{l}=H_{l}([x_{l-1},x_{l-2},...,x_{0}])$ (15)

HyperDenseNet integrates the outputs among different paths based on the
densely-connected network to obtain richer feature representation from
combined modalities. In addition, the permuting and interleaving operations
are applied to the concatenation to improve performance. Considering the case
of two modalities with streams 1 and 2 denoted as $x_{l}^{1}$ and $x_{l}^{2}$
respectively, the output of the $l^{th}$ layer of HyperDenseNet can be
expressed as follows:

$x_{l}^{s}=H_{l}^{s}(\pi_{l}^{s}([x_{l-1}^{1},x_{l-1}^{2},x_{l-2}^{1},x_{l-2}^{2},...,x_{0}^{1},x_{0}^{2}])),$
(16)

where $\pi^{s}_{l}$ represents the shuffling function acting on the feature
maps.

Inspired by the Inception module in [123] which employs convolutions with
multiple kernel sizes on the same level to capture both local and global
information, the authors further expand the convolutional module of Inception
with two additional convolutional blocks to facilitate the learning of multi-
scale features. The two blocks exploit different dilation rates to enable
multiple receptive fields larger than the original inception module. The
$n\times n$ convolutions are replaced with the consequent $1\times n$ and
$n\times 1$ convolutions with the aim to be more efficient.

Lachinov et al. [58] propose a deep cascaded variant of U-Net, Cascaded Unet,
to process multi-modal input for better performance regarding brain tumor
segmentation. Despite the feasibility of the original U-Net to handle multi-
modal MRI image input, it fuses the feature information of all the modalities
which is processed in an identical manner. Based on the original U-Net, the
proposed Cascaded Unet employs multiple encoders in parallel for better
exploiting feature representations for each specific modality.

Figure 29: Architecture of Cascaded Unet with multiple encoders. T1, T2, T1CE,
and FLAIR indicate several MRI modalities. N denotes the current number of
filters. K is a number of filters in the context feature map produced from
lower-scale models. At each stage of the encoding path across modalities, the
resulting features are calculated as a maximum of the feature maps processed
by encoders. In the decoder of the model, the output at each scale is obtained
from the skip connections from the encoder, the decoder output from the
previous stage, and the context connections. Figure from [180].

The overall architecture of the Cascaded Unet model can be seen in Figure 29.
The encoder path contains separate subpaths where every subpath utilizes a
convolution group to process one input modality and generate feature maps.
Then elementwise maximum operation acts on the multiple feature maps per stage
to obtain the resulting features. The output of the feature map is afterward
joined with the corresponding feature map of the larger-scale block, which
boosts the information flow between the feature maps at different scales. The
decoder of Cascaded UNet produces output at each level depending on the output
at the same scale and the output of the decoder block at the previous stage.
This strategy encourages the model to iteratively improve the results from
earlier iterations.

####  3.5.3 Leveraging Depth Information

Some methods modify the U-Net into a 3D model and design modules to extract
the information across channels in order to fully exploit the structural
information of the third-dimension medical images. For improving automatic
brain tumor prognosis, Islam et al. adapt the U-Net architecture to a 3D model
and integrate the 3D attention strategy to perform image segmentation [59].
Compared to only skip connections, the introduced 3D attention model is
aggregated into the decoder part of U-Net that includes channel and spatial
attention in parallel with skip connections. The additional 3D attention
layers encourage the module to encode richer spatial features from the
original images.

Figure 30: Architecture of 3D Attention U-Net with a channel and a spatial
attention parallel to skip connections. Figure from [181].

As shown in Figure 30, the 3D attention U-Net is composed of a 3D encoder, the
decoder, and skip connections combined with the channel and spatial attention
mechanism. In the path for 3D spatial attention, the authors perform $1\times
1\times C$ convolution on the input feature maps to obtain the result of the
$H\times W\times 1$ dimension. In parallel, the input feature maps are passed
through an average pooling and then fed to the fully-connected layer to get
the $1\times 1\times C$ sequential channel correlation. Since the two paths
capture features parallelly, the inconsistency and sparsity caused by the two
excitations can be alleviated by fusing skip connections. Furthermore, the
integration of skip connections can enhance the performance of segmentation
prediction which can be inferred from the experiments on the BraTS 2019
dataset.

###  3.6 Probabilistic Design

Another type of U-Net extension combines the classic U-Net with different
types of probabilistic extensions. Depending on the task that should be
achieved or the process that should be enhanced, different types of extensions
from bayesian skip connections, over variational auto-encoders to Markov
random fields are used, which are introduced in the following.

####  3.6.1 Variational Auto Encoder (VAE) Regularization

In medical image segmentation tasks, different graders often produce different
segmentations. Most of these different segmentations are plausible as many
medical images contain ambiguities that can not be resolved considering only
the image at hand. Taking this into consideration, Kohl et al. learn a
distribution over segmentations from an ambiguous input to produce an
unlimited number of possible segmentations instead of just providing the most
likely hypothesis [54]. In their approach, they combine a U-Net, for producing
reliable segmentations, with a conditional variational autoencoder (CVAE),
which can model complex distributions and encodes the segmentation variants in
a low-dimensional latent space. The best results were obtained for a latent
space of dimension 6.

Figure 31: (a) Illustration of the sampling process of M segmentations with
probabilistic U-Net. (b) Illustration of the training process of the
probabilistic U-Net for one training sample. The green arrows represent loss
functions. Figure from [182].

Figure 31 (a) shows the sampling process given a trained prior net and u-net
as well as a low-dimensional latent space. Each position in the latent space
encodes a different segmentation variant. Passing the input image through the
prior net, it will determine the probability of the encoded variants for the
given input image. For each possible segmentation to be predicted the network
is applied to the same input image. A random sample from the prior probability
distribution is drawn and broadcast to an N-channel feature map with the same
shape as the segmentation map. It will then be concatenated with the final
feature maps of the u-net and processed with successive $1\times 1$
convolutions to produce the segmentation map corresponding to the drawn point
from the latent space. Only the combination needs to be recalculated in each
iteration, as the last feature maps of the U-Net and the output of the prior
net can be reused for each hypothesis.
Figure 31 (b) shows the training procedure of the probabilistic U-Net.

Apart from the standard training procedures for conditional VAEs and
deterministic segmentation models, it has to be learned how to embed the
segmentation variants in the latent space in a useful way. This is solved by
the posterior net. It learns to recognize a segmentation variant and map it to
a certain position in the latent space. A sample from its output posterior
distribution combined with the activation map of the u-net must result in a
segmentation identical to the ground truth segmentation. From this, it follows
that the training data set must include a set of different but plausible
segmentations for each input image.

Myronenko [53] adds a VAE branch to a 3D U-Net architecture to address the
problem of limited training data for brain tumor segmentation. In their
architecture, the U-Net is used for the segmentation of the tumor, and the VAE
is used for the reconstruction of the image sharing the same encoder. For the
VAE, the output of the encoder is reduced to a lower dimensional space and a
sample is drawn from the Gaussian distribution with the given mean and
standard derivation (std). The sample is then reconstructed to the input image
using an architecture similar to that of the U-Net decoder but without any
skip connections. The total loss to be minimized during training is made up of
three terms:

$\mathbf{L}=\mathbf{L}_{\text{dice}}+0.1\cdot\mathbf{L}_{\text{L2}}+0.1\cdot\mathbf{L}_{\text{KL}}$
(17)

$\mathbf{L}_{\text{dice}}$ is a soft dice loss between the predicted
segmentation of the u-net and the GT segmentation.
$\mathbf{L}_{\text{L2}}$ and $\mathbf{L}_{\text{KL}}$ are the losses for the
VAE where $\mathbf{L}_{\text{L2}}$ describes how well the reconstructed image
matches the input image and $\mathbf{L}_{\text{KL}}$ is the Kullback-Leibler
(KL) divergence between the estimates normal distribution and a prior
distribution $\mathcal{N}(0,1)$. Using the VAE brach helps to better cluster
the features at the end of the encoder. This helps to guide and regularize the
shared encoder for small training set sizes. Adding the additional VAE branch,
therefore, improved the performance and led to stable results for different
random initializations of the network.

####  3.6.2 Graphical Model Algorithm

While the classic U-Net performs well on data from the same distribution as
the training data, its accuracy decreases on out-of-distribution data.

To address this problem, Brudfors et al. [51] combine a U-Net with Markov
random fields (MRFs) to form the MRF-Unet. The low-parameter, first-order MRFs
are better at generalization because they encode simpler distributions which
is an important quality to fit out of distribution data. The very accurate
U-Net predictions make up for the fact that the MRFs are less flexible. The
architecture of the proposed model can be seen in Figure 32. As the
combination of the U-Net and MRF distribution is intractable by calculating
the product of the two, an iterative mean-field approach is used to estimate
the closest factorized distribution under the Kullback-Leibler divergence. A
detailed mathematical derivation of the process can be found in the work by
Brudfors et al. [51]. Experiments showed that the combination of MRF and U-Net
improved performance on in- and out-of-distribution data. The lightweight MRF
component, which does not add any additional parameters to the architecture,
serves as a simple prior and therefore learns abstract label-specific
features.

Figure 32: Schematic illustration of the MRF-UNet product. Figure from [183].

Klug et al. [52] use a Bayesian skip connection in an attention-gated 3D U-Net
to allow a prior to bypass most of the network and be reintegrated at the
final layer in their work to segment stroke lesions in perfusion CT images.
The skip connection provides the prior to the final network layer and should
reduce false-positive rates for small and patchy segmentations of varying
shapes. As a prior, the segmentation of the ischemic core obtained by a
standard thresholding method is used. Klug et al. [52] evaluated two ways to
combine the prior and the output of the U-Net to calculate the final output
segmentation: Addition and convolution of the two maps. Superior results were
achieved by using convolution for combination in all experiments.

The input to the U-Net is the concatenation of the 3D perfusion CT image and
the prior. When comparing a 3D attention gate U-Net to the same architecture
with the bayesian skip connection additionally reintegrating the prior at the
end of the network, the latter achieves a better performance in terms of dice
score with faster convergence. It is worth mentioning that we have found
excessive papers in which probabilistic design is integrated into the U-Net in
applications such as for brain tumor [184], and skin lesion segmentation
[185].

Figure 33: The timeline of prominent U-Net-based methods proposed in medical
semantic segmentation literature, from 2015 to 2022. The superscripts in
ascending order denote the 1. [7], 2. [92], 3. [61], 4. [60], 5. [186], 6.
[187], 7. [70], 8. [11], 9. [13], 10. [55], 11. [57], 12. [58], 13. [53], 14.
[54], 15. [78], 16. [74], 17. [188], 18. [65], 19. [81], 20. [18], 21. [56],
22. [59], 23. [68], 24. [82], 25. [72], 26. [71], 27. [15], 28. [76], 29.
[52], 30. [73], 31. [83], 32. [77], 33. [38], 34. [47], 35. [48], 36. [39],
37. [51], 38. [69], 39. [139], 40. [44], 41. [45], 42. [46], 43. [49], 44.
[50], 45. [43], respectively. TABLE I: The review of U-Net-like models for
medical image segmentation based on the proposed taxonomy, Figure 2. {SCE,
BDE, BE, T, RRE and PD} stands for Skip Connection Enhancement, Backbone
Design Enhancement, Bottleneck Enhancement, Transformer, Rich Representation
Enhancement and Probabilistic Design, respectively.

Strategy | Networks |  Core Ideas | Practical Use Cases
---|---|---|---
SCE | Attention U-Net [70] |  |
UNet++ [11, 67] |  |  |
RA-Unet [71] |  |  |
BCDU-Net [74] |  |  |
U-Net3+ [68] |  |  |
BiO-Net [69] | Skip connections are defined as connections in a neural network that do not connect two following layers but instead skip over at least one layer. This strategy initially aimed to encourage feature reusability and compensate for gradient vanishing in a deeper network. This modification introduces the feasibility of transferring high spatial localization features from the encoder to the decoder for better segmentation maps. In addition, some use cases used skip connections as hierarchical multi-scale fusion paths for feature enrichment in diverse U-Net stages. Furthermore, skip connection could efficiently decrease the semantic feature gaps between different layers and scales. |  $\bullet$ Exploit multiscale features [67] |
$\bullet$ Robust boundary representation [68] |  |  |
$\bullet$ Bi-directional feature representation [69] |  |  |
$\bullet$ Feature recalibration [70] |  |  |
$\bullet$ Suppress irrelevant regions besides feature reusability [71] |  |  |
$\bullet$ Enrich semantic representation [74] |  |  |
BDE | Residual U-Net [121, 60, 61] |  |
Multi-Res U-Net [15, 123] |  |  |
Dense U-Net [125, 188] |  |  |
H-DenseUNet [62] |  |  |
DUNet [65, 126] |  |  |
S3D U-Net [13] | The backbone defines how the layers in the encoder are arranged and its counterpart is therefore used to describe the decoder architecture. Ideally, the strong backbone design (e.g., inception model) with pre-trained weight can further improve the model generalization capability. |  $\bullet$ Converges to lower loss [62] |
$\bullet$ Addressing gradient vanishing [121] |  |  |
$\bullet$ Faster convergence rate [60] |  |  |
$\bullet$ Feature reusability [61] |  |  |
$\bullet$ Multi-scale encoding [15] |  |  |
$\bullet$ Better boundary representation [123] |  |  |
$\bullet$ Fine-grained feature set [125] |  |  |
$\bullet$ Cross-modality representation [62] |  |  |
$\bullet$ Reducing computation burden of multi-scale representation [65] |  |  |
$\bullet$ Efficient multi-scale computation [13] |  |  |
BE | ASPP [81] |  |
MA-Net [76] |  |  |
COPLE-Net [82] |  |  |
SA-UNet [77] |  |  |
FRCU-Net [79] |  |  |
JCS [83] |  |  |
MS-Net [129] | The network bottleneck contains the compressed representation of the input data and provides necessary information (e.g., semantic, texture, shape features) to reconstruct the segmentation map. Any improvement in the bottleneck design can further improve the prediction result. |  $\bullet$ Frequency recalibration [79] |
$\bullet$ Spatial attention [77] |  |  |
$\bullet$ Feature pyramid [81, 82] |  |  |
$\bullet$ Imposing attention mechanism [83] |  |  |
T | TransUNet [38] |  |
TransBST [39] |  |  |
Swin-Unet [48] |  |  |
UNETR [139] |  |  |
TMU [44] |  |  |
UCTransNet [45] | Transformers’ critical point is to compensate for CNN’s limited receptive field. Extracting long-range contextual information with intuition to look at the whole image at once is a promotional function of Transformers. Due to the 1D sequence mapping functionality of Transformers, they can use as a play-and-plug module at different parts of U-Net-like structures. However, due to the quadratic computational complexity nature, utilizing efficient Transformers’ design even from the NLP field within vision architectures is beneficial. Since Transformers calculate the affinities between different parts of input data adaptively, utilizing them is a wise solution for multi-scale feature amalgamation. |  $\bullet$ Improving CNN bottleneck’s feature discriminancy [38] |
$\bullet$ Capture inter-slice affinities from 3D data [39] |  |  |
$\bullet$ Hierarchical Efficient Transformer-based design [48] |  |  |
$\bullet$ Modeling 3D volumetric data with global multi-scale information [139] |  |  |
$\bullet$ Feature re-calibration / degrading the boundary maps erroneous [44] |  |  |
$\bullet$ Decreasing the gap between multi-scale semantic features [45] |  |  |
RRE | Focal Tversky Attention U-Net [18] |  |
PT M-Net [55] |  |  |
Dense Multi-path U-Net [57] |  |  |
MFP-Unet [56] |  |  |
Cascaded Unet [58] | The key objective is to enhance the performance of the trained models by utilizing all available information from multi-modal or multi-scale images while retaining the most desirable and relevant features. Some methods also operate directly on volumetric images to take full advantage of depth information. |  $\bullet$ Improved precision and recall balance [18] |
$\bullet$ Hierarchical representation learning [55] |  |  |