# Modality specific U-Net variants for biomedical image segmentation: A survey

Narinder Singh Punn and Sonali Agarwal N. S. Punn, S. Agarwal, Indian
Institute of Information Technology Allahabad, Jhalwa, Prayagraj, Uttar
Pradesh, India; emails: {pse2017002<EMAIL_ADDRESS>
###### Abstract

With the advent of advancements in deep learning approaches, such as deep
convolution neural network, residual neural network, adversarial network;
U-Net architectures are most widely utilized in biomedical image segmentation
to address the automation in identification and detection of the target
regions or sub-regions. In recent studies, U-Net based approaches have
illustrated state-of-the-art performance in different applications for the
development of computer-aided diagnosis systems for early diagnosis and
treatment of diseases such as brain tumor, lung cancer, alzheimer, breast
cancer, etc. This article contributes to present the success of these
approaches by describing the U-Net framework, followed by the comprehensive
analysis of the U-Net variants for different medical imaging or modalities
such as magnetic resonance imaging, X-ray, computerized
tomography/computerized axial tomography, ultrasound, positron emission
tomography, etc. Besides, this article also highlights the contribution of
U-Net based frameworks in the on-going pandemic, severe acute respiratory
syndrome coronavirus 2 (SARS-CoV-2) also known as COVID-19.

Biomedical image segmentation, Deep learning, U-Net.

##  1 Introduction

The evolving medical imaging acquisition system [1] has brought the
consideration of the research community towards the non-invasive practice of
disease diagnosis. Every diagnostic procedure involves the careful and
critical examination of medical scans which represents the complex interior
structure within the body, illustrating the functioning of various organs.

With a wide variety of medical imaging such as the magnetic resonance imaging
(MRI), X-ray, computerized tomography/computerized axial tomography (CT/CAT),
ultrasound (US), positron emission tomography (PET), etc., the medical domain
has experienced exponential growth in the diagnosis practices. Each of these
scans varies in the imaging procedure, usecases and its average diagnosis
duration [2, 3], as shown in Table 1. For any radiologist, analyzing such
complex scans is tedious and time consuming, thereby to fill this void of
complexity, deep learning approaches are well explored to address the
automated assistance in diagnosis procedure, resulting into faster and better
practices for monitor, cure and treatment of the diseases [4, 5, 6, 7].

Segmentation [8] is one such automation task that helps to identify and detect
the desired regions or objects of interest for the concerned issue. Depending
on the depth of identifying the classes of objects, segmentation is divided
into two levels as semantic and instance. The semantic segmentation [9]
segregates the objects belonging to different classes, whereas instance
segmentation [10] goes deeper to also segregate the objects within the common
class. With the exhaustive analysis [8, 11], it is observed that among the
latest advancements to perform segmentation, mostly U-Net [12] based
frameworks are adopted to achieve state-of-the-art segmentation performance
which follows from its symmetrical encoder-decoder structure to extract and
reconstruct the feature maps.

Table 1: Medical imaging approaches for diagnosis.

Imaging type | Approach | Usecase | Duration
---|---|---|---
(in min.)
MRI |  Magnetic fields and radio waves |  Multiple sclerosis, stroke, tumors, spinal cord disorders, etc. | 45 – 60
X-ray |  Ionizing radiation |  Fractures, arthritis, osteoporosis, breast cancer, etc. | 10 – 15
CT/CAT |  Ionizing radiation |  Trauma injuries, tumors and cancers, vascular and heart diseases, etc. | 10 – 15
US |  Sound waves |  Gallbladder illness, breast lumps, genital disorder, joint problems, etc. | 30 – 60
PET |  Radioactive tracer |  Alzheimer, epilepsy, seizures, parkinsons’ disease, etc. | 90 – 120

###  1.1 Motivation and contribution

Though there are a lot of review articles on biomedical image segmentation
(BIS) using deep learning; however, none of the articles is focused on the
variants of U-Net architectures which brought the breakthrough in the
biomedical image segmentation (to the best of our knowledge). The
understanding of the available methods is critical for developing the
computer-aided diagnosis systems; however, to contribute to this domain as a
researcher, one needs to understand the underlying mechanics of the methods
that make those systems achieve promising results. For instance, the work of
Haque et al. [11] reviewed the standard deep learning approaches for BIS using
different modalities, whereas, Zhou et al. [13] explored the comprehensive
analysis focused on multi-modality fusion approaches. Following from this
context, the proposed article is intended to contribute for an exhaustive
analysis of the state-of-the-art U-Net based approaches to make the
researchers or readers reap the most benefits from the current advancements in
U-Net and aid in further contributions towards the research in BIS.

Table 2: Search strings to acquire research papers and analyse research trend using GoogleScholar. No. | Search string | Queried date | Year | No. of papers
---|---|---|---|---
SS1 | (U-Net segmentation CT OR X-ray OR PET OR US OR MRI) | February 22, 2021 | 2015-20 | 25,500
SS2 | (biomedical image segmentation) | February 22, 2021 | 2015-20 | 142,500
SS3 | (biomedical image segmentation ”U-Net”) | February 22, 2021 | 2015-20 | 28,929
Figure 1: Research trend in biomedical image segmentation per year.

###  1.2 Review process

The basis of including a research article in this survey is that the article
describes the research on U-Net based biomedical image segmentation. The
articles confirming vivid architectures or frameworks are only included if the
authors claimed certain advancement or novel contribution, whereas articles
with pure discussions are excluded; fortunately, such articles are limited and
hence will not affect the outcome of this survey.

The search for the articles is performed on Google Scholar, which is one of
the best academic search engine [14], where relevant articles are identified
using the search string, SS1 as shown in Table 2. Among the acquired papers,
the high quality journals or conferences are confirmed by analysing its impact
factor (high), h-index (high), peer-review process (transparent), indexing
(MEDLINE, Elsevier Scopus and EMBASE, Clarivate Analytics Web of Science,
Science Citation Index, etc.) and scientific rigor. These reputed journals are
identified from the ranked list, CORE [15]. However, some articles are also
included from popular preprint servers such as arXiv. With such a huge pool of
acquired articles, the most relevant articles are filtered with a thorough
examination (journal or conference quality, cite score and contribution) to
include in this survey.

###  1.3 Research trend in BIS

A comparative literature exploration is conducted on the Google Scholar search
engine using the search strings, SS1 and SS2, as shown in Table 2. The number
of BIS approaches without U-Net are acquired by subtracting the number of BIS
U-Net articles from the pool of BIS articles, to understand the latest trend
of research. Fig. 1 illustrates that the latest approaches are developed by
employing the U-Net framework while experiencing exponential growth every
year. In order to analyse such trend, this article aims to provide the
exhaustive review of the variants of U-Net architectural design developed for
segmentation. It is evident that the U-Net model incorporates the huge
potential for further advancements due to its mutable and modular structure
that would result in the state-of-the-art diagnosis system.

###  1.4 Article structure

The remaining portion of the article is divided into several sections, where
sections 2 presents the overview of biomedical image analysis and in sections
3, 4 and 5 the comprehensive analysis of U-Net variants is presented that
covers implementation strategies and advancements. Later, section 6 presents
the observations concerned with the current advancements in U-Net based
approaches, followed by the scope and challenges in section 7 and concluding
remarks in the final section.

##  2 Biomedical image analysis

The success of deep learning in image analysis has encouraged the biomedical
imaging researchers to investigate its potential in analyzing various medical
modalities to aid clinicians in faster diagnosis and treatment of diseases or
infections like the on-going pandemic of SARS-CoV-2 (COVID-19). Following the
deep learning usecases, the implication of classification can ascertain the
presence or absence of disease in some modality e.g. the ground glass
opacification (GGO) in the lungs via CT imaging. Furthermore, in localization,
normal anatomy can be identified e.g. lungs in the CT or X-ray imaging, and
later segmentation can generate refined boundaries around the GGOs to
understand its impact on the anatomical structures for further analysis.
Since, segmentation is an extension to classification, localization or
detection, it offers very rich information about the disease and infected
regions. With this interest, many architectures have been proposed for the
segmentation of the targeted regions from vivid modalities [11]. In addition,
segmentation is the most widely researched application of deep learning in
biomedical image analysis [13], where U-Net based segmentation architectures
have gained significant popularity to develop computer-aided diagnosis (CAD)
systems.

###  2.1 Rise of segmentation architectures

Despite the advancements in deep learning, segmentation is still one of the
challenging tasks due to the varying dimensions, shape and locale of the
target tissues. Traditionally, the segmentation process was carried manually
by expert clinicians to illuminate the regions of interest in the whole volume
of samples, thereby it is ideal to automate this process for faster diagnosis
and treatment. In recent years, various deep learning models are developed for
BIS that are categorized into manual, semi-automatic and fully automatic
approaches [11]. Fig. 2 presents the schematic representation of the pipeline
of the recent deep learning based segmentation frameworks for biomedical
images, which is divided into data preprocessing [16], deep learning model
[8], and post-processing [17, 18]. In the data preprocessing stage, the data
undergoes a certain set of operations like resize and normalization to reduce
the intensity variation in the image samples, augmentation to generate more
training samples for avoiding the class biasness and overfitting problem,
removal of irrelevant artefacts or noise from the data samples, etc. The pre-
processed data is then fed to train the deep neural segmentation network,
where mostly U-Net based architectures are deployed. The output of the network
undergoes post-processing with techniques such as morphological and
conditional random field based feature extraction to refine the final
segmentation results.

Figure 2: Schematic representation of deep learning based segmentation
architectures.

Initiated from the sliding window approach of Ciresan et al. [19] in 2012 to
classify each pixel while also localizing the regions using patch based input,
the model outperformed in the ISBI 2012 challenge, however, the training was
slow because of a large number of overlapping patches and also lacked the
balance of context and localization accuracy. Long et al. [20] proposed fully
convolutional neural network (FCN) for semantic segmentation, defined on the
state-of-the-art classification networks like Alex-Net, VGG-Net and Google-
Net. This model achieved the state-of-the-art segmentation results on PASCAL
VOC, NYUDv2, and SIFT flow datasets. Later, the U-Net model proposed by
Ronnerberger et al. [12], consists of FCN along with the contraction-expansion
paths. The contraction phase tends to extract high and low level features,
whereas expansion phase follows from the features learned in corresponding
contraction phase (skip connections) to reconstruct the image into the desired
dimensions with the help of transposed convolutions or upsampling operations.
The U-Net model won the ISBI 2015 challenge and outperformed its predecessors.
Later, a similar approach is proposed by Cicek et al. [21] in the three
dimensional feature space to perform volumetric segmentation of Xenopus kidney
and achieved the promising results. Following from the state-of-the-art
potential of the U-Net model, many variants have been proposed based on the
variation in the convolution and pooling operations, skip connections, the
arrangement of the components in each layer and hybrid approaches that make
use of the state-of-the-art deep learning models, to tackle the challenges
associated with different applications.

####  2.1.1 U-Net

With the sense of segmentation being a classification task where every pixel
is classified as being part of the target region or background, Ronneberger et
al. [12] proposed a U-Net model to distinguish every pixel, where input is
encoded and decoded to produce output with the same resolution as input. As
shown in Fig. 3, the symmetrical arrangement of encoder-decoder blocks
efficiently extracts and concatenates multi-scale feature maps, where encoded
features are propagated to decoder blocks via skip connections and a
bottleneck layer.

Figure 3: U-Net archiecture.

The encoder block (contraction path) consists of a series of operations
involving valid $3\times 3$ convolution followed by a ReLU activation function
(as shown in Fig. 4(a)), where a 1-pixel border is lost to enable processing
of the large images in individual tiles. The obtained feature maps from the
combination of convolution and ReLU are downsampled with the help of max
pooling operation, as illustrated in Fig. 4(b). Later, the number of feature
channels are increased by a factor of 2, following each layer of convolution,
activation and max pooling, while resulting into spatial contraction of the
feature maps. The extracted feature maps are propagated to decoder block via
bottleneck layer that uses cascaded convolution layers. The decoder block
(expansion path) consists of sequences of up-convolutions (shown in Fig. 4(c))
and concatenation with high-resolution features from the corresponding encoded
layer. The up-convolution operation uses the kernel to map each feature vector
to the $2\times 2$ pixel output window followed by a ReLU activation function.
Finally, the output layer generates segmentation mask with two channels
comprising background and foreground classes. In addition, the authors
addressed the challenge to segregate the touching or overlapping regions by
inserting the background pixels between the objects and assigning an
individual loss weight to each pixel. This energy function is represented as a
pixel-wise weighted cross entropy function as shown in Eq. 1. The authors
established the state-of-the-art results by winning the ISBI 2015 challenge.

Figure 4: Summary of operations in U-Net. (a) $3\times 3$ convolution + ReLU,
(b) $2\times 2$ max-pooling and (c) $2\times 2$ up-convolution operation.

$\displaystyle
E=\sum_{{x}\in\Omega}\left(w_{c}({x})+w_{0}\cdot\exp\left(-\frac{(d_{1}({x})+d_{2}({x}))^{2}}{2\sigma^{2}}\right)\right)$
$\displaystyle\log({p}_{\ell({x})}({x}))$ (1)

where softmax,
${p}_{k}({x})=\exp({a_{k}({x})})/\left(\sum_{k^{\prime}=1}^{K}\exp(a_{k^{\prime}}({x}))\right)$
with activation, $a_{k}({x})$ for channel $k$ and pixel $x\in\Omega$ with
$\Omega\in\mathbb{Z}^{2}$, $w_{c}$ is the weight map, $d_{1}$ and $d_{2}$ are
the distances to the nearest and the second nearest boundary pixels, and
$w_{o}$ and $\sigma$ are constants.

Table 3: Summary of loss functions for biomedical image segmentation with respect to the predicted mask ($\mathcal{P}$) and ground truth mask ($\mathcal{G}$), $\alpha$ and $\gamma$ as constants, $h$ is Hausdorff distance and $d$ is the operator for Euclidean distance. Type | Objective functions | Usecase
---|---|---
Distribution | $\mathcal{L}_{BCE}=-(glog(p)+(1-g)log(1-p))$ | Balanced distribution of data
$\mathcal{L}_{WCE}=-(\alpha.glog(p)+(1-g)log(1-p))$ | Skewed dataset
$\mathcal{L}_{BaCE}=-(\alpha glog(p)+(1-\alpha)(1-g)log(1-p))$ | Skewed dataset
$\mathcal{L}_{Focal}=\left\\{\begin{aligned} &-\alpha(1-p)^{\gamma}log(p),&&\text{if}\ g=1\\\ &-(1-\alpha)(p)^{\gamma}log(1-p),&&\text{otherwise}\end{aligned}\right.$ | Focuses on hard samples
Region | $\mathcal{L}_{DSC}=1-\frac{2gp+1}{g+p+1}$ | Widely used for segmentation
$\mathcal{L}_{IoU}=1-\frac{gp}{g+p-gp}$ | Widely used for segmentation
$\mathcal{L}_{SS}=\alpha*\mathrm{sensitivity}+(1-\alpha)*\mathrm{specificity}$ | Focuses to improve true positive rate
$\mathcal{L}_{Tversky}=1-\frac{1+gp}{1+gp+\alpha(1-g)p+(1-\alpha)g(1-p)}$ | Introduces weights for false predictions
Boundary | $\mathcal{L}_{HD}=\frac{1}{N}\sum_{i=0}^{N}\Big{(}(p_{i}-g_{i})^{2}.(h_{pi}^{2}+h_{gi}^{2})\Big{)}$ | Widely used for segmentation
$\mathcal{L}_{SA}=-\sum_{i}CE(p_{i},g_{i})-\sum_{i}{\alpha_{i}}d(\mathcal{P},\mathcal{G})CE(p_{i},g_{i})$ | Focuses to segment boundaries of the regions
Compound | $\mathcal{L}_{Combo}=\alpha\mathcal{L}_{BaCE}(g,p)-(1-\alpha)\mathcal{L}_{DSC}(g,p)$ | Leverages the features of $BaCE$ and $DSC$ for skewed data
$\mathcal{L}_{EL}=\alpha_{DSC}e^{(-ln(\mathcal{L}_{DSC})^{\gamma})}+\alpha_{CE}\,e^{(-ln(\mathcal{L}_{CE})^{\gamma})}$ | Focuses on less accurate predictions
$BCE$ \- binary cross-entropy, $WCE$ \- weighted cross-entropy, $BaCE$ \-
balanced cross-entropy, $DSC$ \- dice similarity coefficient, $IoU$ \-
intersection-over-union, $SS$ \- sensitivity-specificity, $HD$ \- Hausdorff
distance, $SA$ \- shape-aware, $EL$ \- exponential-logarithmic.

####  2.1.2 Implementation strategies

The implementation strategies of segmentation architectures can be divided
into two categories: a) training from scratch and b) training using a pre-
trained model (also known as transfer learning). In first approach (shown in
Fig. 5(a)), an entire model is trained in which training parameters are
initialized with Xavier initialization [22] or Kaming initialization [23]. Due
to which this approach requires a large number of labelled data samples to
optimize the training parameters and learn the desired task. Hence, this
approach requires intensive time and efforts to develop and train the model.
In the transfer learning paradigm, as simulated in Fig. 6, a pre-trained model
(models trained on benchmark datasets such as ImageNet) is utilized as a
backbone model to train on different data involving similar or different tasks
such as object detection and image segmentation. As shown in Fig. 5(b) and
Fig. 5(c), the transfer learning or domain adaptation can be applied in two
schemes, either freezing the base model (using the frozen pre-trained model)
and training the later layers for prediction, or semi-freezing the base model,
where few high level layers are retrained along with the prediction layers.
The transfer learning approach typically produces better results than the
random initialization of the training parameters [24].

Figure 5: Typical approaches for model training. Figure 6: Illustration of transfer learning approach to adapt to new task. Table 4: Summary of performance metrics for BIS in terms of number of true positive (TP), true negative (TN), false positive (FP) and false negative (FN), predicted mask ($\mathcal{P}$) and ground truth ($\mathcal{G}$), $\mathcal{H}(X,Y)$ is the mean of directed $AHD$ from X to Y and Y to X with $d$ as euclidean distance, $\mathcal{V}_{p}$ and $\mathcal{V}_{g}$ refer to the volumes of generated and reference segmentation. Metric | Expression
---|---
Accuracy | $A=\frac{(TP+TN)}{(TP+TN+FP+FN)}$
Precision | $P=\frac{TP}{(TP+FP)}$
Recall | $R=\frac{(TP)}{(TP+FN)}$
F1-score | $F1=2\times\frac{(P\times R)}{(P+R)}$
Specificity | $S=\frac{TN}{(TN+FP)}$
Dice similarity coefficient | $DSC=\frac{2\times|\mathcal{P}\cap\mathcal{G}|}{|\mathcal{P}|+|\mathcal{G}|}=\frac{2TP}{2TP+FP+FN}$
Intersection-over-union | $IoU=\frac{\mathcal{P}\cap\mathcal{G}}{\mathcal{P}\cup\mathcal{G}}=\frac{TP}{TP+FP+FN}$
Average Hausdo-rff distance | $AHD=\frac{1}{2}\left(\frac{\mathcal{H}(\mathcal{P},\mathcal{G})}{\mathcal{P}}+\frac{\mathcal{H}(\mathcal{G},\mathcal{P})}{\mathcal{G}}\right)$
| $\mathcal{H}(X,Y)=\frac{1}{2}\left(\frac{1}{X}\sum_{x\in X}\min_{y\in
Y}d(x,y)\right.$
| $\left.\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;+\frac{1}{Y}\sum_{y\in Y}\min_{x\in
X}d(x,y)\right)$
Absolute Volume Difference | $AVD=\frac{|\mathcal{V}_{p}-\mathcal{V}_{g}|}{\mathcal{V}_{g}}\times 100$

####  2.1.3 Loss functions

The loss functions or objective functions drive the training procedure of the
deep learning models. For BIS task, loss functions are tuned to alleviate the
above discussed class imbalance problem by refining the distributions of the
training data. With each dataset introducing its complexities and challenges,
the loss functions are grouped into four categories based on the distribution,
region, boundary and hybrid [25], as shown in Table 3. For ease in
representation, the loss functions are summarized for the semantic
segmentation scenario, where the number of classes are limited to two
(background and target region).

####  2.1.4 Performance metrics

The performance metrics are the key factors to evaluate and compare the
segmentation performance of the models. Due to unavailability of the standard
metrics, each system requires an appropriate and different selection of
metrics that can quantify time, computational and memory space requirements
and overall performance [26]. Table 4 presents the most popular evaluation
metrics that are utilized to analyse the performance in BIS models. In BIS,
mostly the datasets are imbalanced i.e. the number of pixels/voxels concerning
the target region (region of interest) are relatively less than the dark
pixels/voxels (background region), due to which the metrics such as accuracy,
which are best suited for a balanced distribution of data samples, are not
recommended for BIS evaluation of the models. Among the discussed metrics
intersection-over-union (IoU or Jaccard index) and dice similarity coefficient
are the most widely used evaluation metrics in BIS for various modalities.
More details can be found in the recent review articles [11, 8].

##  3 U-Net variants for medical imaging

The numerous development in medical imaging acquisition system has resulted in
the rise of usage frequency of modalities. Smith-Bindman et al. [27] observed
the dramatic increase in the utilization of diagnostic imaging in the USA over
the period from 1996-2010, where CT, MRI and PET imaging utilization increased
by 7.8%, 10% and 57% respectively. Similarly, Dovales et al. [28] analysed the
trends and patterns for the period of 2004-2014 associated with applications
of the diagnostic imaging in the public healthcare system of Brazil. The
authors observed the noticeable increase in the utilization of diagnostic
imaging, especially for CT and MRI which increased by 12% and 19% per year
respectively.

Despite vanilla U-Net being super-efficient in the ISBI cell tracking
challenge, there is still a void to fill with improvements in certain aspects.
The most apparent problem in the vanilla U-Net is that the learning may slow
down in deeper layers of the U-Net model which increases the possibility of
the network ignoring the layers representing abstract features of the target
structure. This slack in the learning process is due to the generation of
diluted gradients in the deeper layers. Following this context, various U-Net
variants are proposed to improve the segmentation performance. These
improvements are observed in the form of integration of certain mechanism with
U-Net model such as 1) Network design (ND) - pre-trained, fusion, dense,
multi-task, residual, cascaded, parallel, nested, deep supervision and
attention, 2) Operation design (OD) - Convolution (dilated or atrous and
depthwise separable), pooling (spectral and spatial), activation and training
loss, and 3) Ensemble design (ED) - combines the multiple design aspects into
one model. Most of the U-Net variants falls in the category of ensemble
design.

Hence, for faster and efficient computer-aided diagnosis practices, the
following sections present wide varieties of U-Net based approaches for
biomedical image segmentation using various modalities. Table 5 summarizes the
various U-Net variants reviewed in the following sections.

Table 5: Summary of popular U-Net variants for BIS. Author | Year |  U-Net variant |  Modality |  TL |  SL |  Pr |  Po |  Dataset |  Design |  Description
---|---|---|---|---|---|---|---|---|---|---
Dong et al. [29] | 2017 |  Modified U-Net |  MRI |  - |  ✓ |  ✓ |  - |  [30] |  ND |  FCN based U-Net
Rashid et al. [31] | 2018 |  Modified U-Net |  X-ray |  - |  ✓ |  ✓ |  ✓ |  [32, 33], LD |  ND |  FCN based U-Net
Frid-Adar et al. [34] | 2018 |  Modified U-Net |  X-ray |  ✓ |  - |  ✓ |  - |  [32] |  ND |  U-Net with pre-trained VGG-16 encoder
Que et al. [35] | 2018 |  CardioXNet framework |  X-ray |  - |  ✓ |  ✓ |  ✓ |  [36] |  ED |  Two parallel U-Net models with binary contours
Oktay et al. [37] | 2018 |  Attention U-Net |  CT |  - |  ✓ |  ✓ |  - |  [38], LD |  ND |  Attention skip-connections
Kohl et al. [39] | 2018 |  Probabilistic U-Net |  CT |  - |  ✓ |  - |  - |  [40, 41] |  ED |  U-Net with conditional variational autoencoder
Tong et al. [42] | 2018 |  Improved U-Net |  CT |  - |  ✓ |  ✓ |  - |  [43] |  ED |  Mini-residual connections within encoder-decoder phases
Janssens et al. [44] | 2018 |  Two stage U-Net model |  CT |  - |  ✓ |  ✓ |  - |  [45] |  ED |  3D FCN LocalizationNet followed by SegmentationNet
Kumar et al. [46] | 2018 |  U-SegNet |  MRI |  ✓ |  - |  ✓ |  - |  [47] |  ED |  Integration of skip connections with SegNet
Kermi et al. [48] | 2018 |  Residual U-Net |  MRI |  - |  ✓ |  ✓ |  - |  [49] |  ND |  Residual blocks between two convolution layers
Chen et al. [50] | 2018 |  S3DU-Net |  MRI |  - |  ✓ |  ✓ |  - |  [49] |  OD |  U-Net with spatiotemporal separable convolution
Durand et al. [51] | 2018 |  Vanilla 3D U-Net |  PET |  - |  ✓ |  ✓ |  ✓ |  LD |  ND |  CNN based 3D U-Net
Zhao et al. [52] | 2018 |  3D FCN |  PET |  - |  ✓ |  ✓ |  - |  LD |  ED |  3D FCN multi-modal fusion network
Almajalid et al. [53] | 2018 |  U-Net + SRAD |  US |  - |  ✓ |  ✓ |  ✓ |  LD |  ED |  Base U-Net with speckle reducing anisotropic diffusion
Wang et al. [54] | 2018 |  cU-Net |  US |  - |  ✓ |  ✓ |  - |  [55], LD |  ND |  Classification and segmentation U-Net
Alom et al. [56] | 2018 |  R2U-Net |  Multi-modality |  - |  ✓ |  ✓ |  ✓ |  [57, 58, 59, 43] |  ED |  Recurrent Residual convolutional neural network based on U-Net (R2U-Net)
Isensee et al. [60] | 2018 |  nnU-Net |  Multi-modality |  - |  ✓ |  ✓ |  ✓ |  [61] |  ED |  Self-adapting no-newU-Net Framework
Zhou et al. [62] | 2018 |  UNet++ |  Multi-modality |  - |  ✓ |  ✓ |  - |  [63, 64, 65, 40] |  ND |  Nested U-Net model
Subramanian et al. [66] | 2019 |  CVC framework |  X-ray |  ✓ |  - |  - |  - |  [36] |  ED |  Two parallel U-Net models with spatial priors and pre-trained NN-RF
Li et al. [67] | 2019 |  U-Net based framework |  X-ray |  ✓ |  - |  ✓ |  ✓ |  [68] |  ED |  SE and residual based attention CNN
Dong et al. [69] | 2019 |  U-Net-GAN |  CT |  - |  ✓ |  ✓ |  - |  [70] |  ED |  U-Net act as a generator and FCN as discriminator network
Liu et al. [71] | 2019 |  GIU-Net |  CT |  - |  ✓ |  ✓ |  ✓ |  [65] |  ED |  Deeper U-Net model with graph cut algorithm
Man et al. [72] | 2019 |  GAU-Net |  CT |  ✓ |  - |  ✓ |  - |  [73] |  ED |  Deformable geometry-aware U-Net with deep Q learning
Seo et al. [74] | 2019 |  mU-Net |  CT |  - |  ✓ |  - |  - |  [65] |  ED |  Object dependent filters in skip connections
Hiasa et al. [75] | 2019 |  Bayesian U-Net |  CT |  - |  ✓ |  ✓ |  ✓ |  [76], LD |  ED |  Cascaded U-Net and Bayesian U-Net models
Song et al. [77] | 2019 |  U-NeXt |  CT |  - |  ✓ |  ✓ |  - |  LD |  ED |  U-Net model loaded with attention blocks, SkipSPP and dense convolutions
Rundo et al. [78] | 2019 |  USE-Net |  MRI |  - |  ✓ |  ✓ |  ✓ |  LD |  ED |  U-Net model with the squeeze-and-excitation blocks
Wang et al. [79] | 2019 |  MSU-Net |  MRI |  - |  ✓ |  ✓ |  - |  [80] |  ED |  Multiscale statistical U-Net
Dong et al. [81] | 2019 |  DAU-Net |  MRI |  - |  ✓ |  - |  - |  LD |  ED |  Deep attention U-Net with deep supervision
Wang et al. [82] | 2019 |  3D DSD-FCN |  MRI |  - |  ✓ |  ✓ |  ✓ |  [83] |  ED |  3D FCN with deep supervision and group dilation
Guo et al. [84] | 2019 |  3D Dense U-Net |  PET |  - |  ✓ |  ✓ |  - |  [85] |  ND |  3D U-Net with dense convolution blocks
continue to the next page |  |  |  |  |  |  |  |  |
Yang et al. [86] | 2019 |  DPU-Net |  US |  - |  ✓ |  ✓ |  - |  [87] |  ED |  Dual path U-Net with parallel multi-branch encoding and decoding
---|---|---|---|---|---|---|---|---|---|---
Li et al. [88] | 2019 |  DU-Net |  US |  - |  ✓ |  ✓ |  ✓ |  LD |  ND |  Dense convolution U-Net
Lin et al. [89] | 2019 |  SSU-Net |  US |  - |  ✓ |  ✓ |  - |  LD |  ED |  Semantic-embedding and shape-aware U-net
Azad et al. [90] | 2019 |  BCDU-Net |  Multi-modality |  - |  ✓ |  ✓ |  - |  [57, 91, 43] |  ED |  Bi-directional ConvLSTM U-Net with densley connected convolutions
Gu et al. [92] | 2019 |  CE-Net |  Multi-modality |  ✓ |  - |  ✓ |  - |  [93, 94, 95] |  ND |  U-Net based context encoder network
Abedalla et al. [96] | 2020 |  2STU-Net |  X-ray |  ✓ |  - |  ✓ |  ✓ |  [97] |  ED |  U-Net with pre-trained ResNet-34 model
Zhang et al. [98] | 2020 |  DEFU-Net |  X-ray |  - |  ✓ |  ✓ |  - |  [99] |  ED |  U-Net with encoder fusion of dense and inception CNN
Wang et al. [100] | 2020 |  MDU-Net |  X-ray |  ✓ |  - |  ✓ |  - |  LD |  ED |  Multi-task dense connection U-Net
Park et al. [101] | 2020 |  3D U-Net |  CT |  - |  ✓ |  ✓ |  ✓ |  LD |  ED |  3D U-Net with segmentation error correction
Fan et al. [102] | 2020 |  MA-Net |  CT |  - |  ✓ |  ✓ |  - |  [65] |  ED |  U-Net based multi-scale attention model
Dong et al. [103] | 2020 |  DeU-Net |  MRI |  - |  ✓ |  ✓ |  - |  [80] |  ED |  3D deformable attention U-Net
Punn et al. [104] | 2020 |  3D inception U-Net |  MRI |  - |  ✓ |  ✓ |  - |  [105, 49] |  ED |  3D inception U-Net with modality fusion
Lu et al. [106] | 2020 |  Modified U-Net |  PET |  ✓ |  - |  - |  ✓ |  ED |  LD |  U-Net with pre-trained VGG-19 encoder
Leung et al. [107] | 2020 |  Modified U-Net |  PET |  ✓ |  - |  ✓ |  - |  LD |  ED |  Physics guided minimal U-Net with dropout regularization
Dunnhofer et al. [108] | 2020 |  Siam-U-Net |  US |  - |  ✓ |  ✓ |  - |  LD |  ED |  U-Net with siamese tracking framework
Zhang et al. [109] | 2020 |  AU-Net |  US |  - |  ✓ |  ✓ |  - |  LD |  ED |  Attention guided U-Net with total variation regularization
Byra et al. [110] | 2020 |  SKU-Net |  US |  - |  ✓ |  ✓ |  - |  LD |  ED |  Attention based selective kernel U-Net
Punn et al. [111] | 2020 |  IU-Net |  Histopathol-ogical |  - |  ✓ |  ✓ |  - |  [64] |  ED |  Inception U-Net model
Ibtehaz et al. [112] | 2020 |  MR-UNet |  Multi-modality |  - |  ✓ |  ✓ |  - |  [91, 113, 114, 105] |  ED |  MultiResUNet
TL - Transfer learning, SL - Scratch learning, Pr - Pre-processing, Po - Post-
processing, LD - Local dataset, ND - Network design, OD - Operation design, ED
- Ensemble design

###  3.1 X-ray

In radiology, X-ray imaging is utilized as a diagnostic procedure of the human
bones and tissues. X-ray possesses the properties of penetrability,
photographic effect and fluorescence effect. Human body tissues vary in
density and thickness due to which X-rays are absorbed with different degrees,
resulting in black and white contrast images [115]. The wide and easy
availability of X-ray imaging has encouraged research community to contribute
towards smart diagnosis systems.

The segmentation of lungs from chest X-ray (CXR) imaging is a crucial step for
any CAD system. Following this, Rashid et al. [31] exploits the potential of
U-Net model to generate the segmentation masks of the lungs from CXR images,
where the produced masks are iteratively refined with post-processing
techniques such as flood fill algorithm and morphological operations. The
authors conducted exhaustive trials on three datasets, JSRT, MC and PUMHS, and
achieved promising results with $DSC$ values of 0.95, 0.95 and 0.88
respectively, while also outperforming other approaches with significant
improvement. In another work, Frid-Adar et al. [34] employed a pre-trained
VGG-16 model in the encoder phase, where decoder or the expansion phase uses
upsampling and standard convolution operations sequentially for multi-class
segmentation involving anatomical structures like lungs field, heart and
clavicles in chest X-ray samples. While training, the pre-trained weights are
fine-tuned to better extract or encode the desired features of the target
classes. This model with transfer learning achieved promising results on JSRT
database. Besides, the authors also analysed the proposed model with multiple
loss functions like $DSC$, $IoU$, Tversky and $BCE$, where the use of $DSC$
produced the best results.

With cardiomegaly being one of the most common inherited cardiovascular
disease, Que et al. [35] proposed a CardioXNet framework to identify and
localize the cardiomegaly present in the chest X-ray images. CardioXNet is
equipped with two parallel U-Net models to generate the segmentation masks for
cardiac and thorax respectively, that follows typical CNN architecture in
contraction and expansion paths. Due to the possibility of the presence of
noise in the output masks, the post-processing is applied to keep the binary
contours that represent the largest area. Later, the processed output mask is
utilized to compute the cardiothoracic ratio defined as $CTR=(L+R)/(T)$, where
$L$ and $R$ indicates the maximum distances from the center to the left and
right farthest boundaries of the heart region, and $T$ is the maximum
horizontal distance between the lungs boundaries. The $CTR$ value is then
utilized to determine the cardiomegaly from the generated masks, where normal
value ranges between 0.39 to 0.50 and above 0.50 indicates a high probability
of the presence of cardiomegaly. In another approach, Subramanian et al. [66]
proposed an automated system involving two U-Net models, where the output
features are exploited to identify the type of central venous catheters (CVC)
as peripherally inserted central catheters (PICC), internal jugular (IJ),
subclavian and Swan-Ganz catheters. The first U-Net model is utilized for CVC
segmentation by using the exponential logarithmic loss to address the class
imbalance problem, whereas the other U-Net model tends to extract the
anatomical structures to distinguish the ambiguous classes such as PICC and
subclavian lines. Clinicians manually annotated the CVCs to obtain the
signature spatial priors which undergo pixel-wise multiplication with the
segmentation output. Later, the produced output is fed to the pre-trained
neural network random forest (NN-RF) classifier to distinguish the type of
CVC. This hybrid combination of segmentation and classification achieved
promising results on the test set of NIH database.

Motivated by the success of squeeze-and-excitation network (SENet) [116] to
suppress the irrelevant features, Li et al. [67] proposed an attention guided
deep learning framework divided into three components: preprocessing, region
of interest (ROI) segmentation with transfer learning followed by pneumonia
detection model. In the preprocessing stage, apart from the trivial processes
like resizing, the authors synthesized the adversarial samples to gain
attention of the model towards pneumonia. The pneumonia infected area is
erased by replacing with an average pixel value of the image and then labelled
as non-pneumonia, which helped to distinguish between noise and relevant data.
To further suppress the background interference, authors adopted the approach
proposed by Rashid et al. [31] to perform the lungs segmentation followed by
post-processing with conditional random fields. The segmented, original and
synthesized images together form the training and validation set for the
pneumonia segmentation network. The network follows SENet design in which SE-
ResNet34 is utilized as a backbone architecture. The proposed framework tends
to learn the pneumonia features effectively and achieves a significant
reduction in the false positive predictions with FPR value of 0.19, in
contrast to mask R-CNN [117] and RetinaNet [118] on RSNA challenge.

In another work, Abedalla et al. [96] proposed a deep learning framework 2STU-
Net to perform segmentation of pneumothorax (collapsed lung) in the CXR
samples. It comprises state-of-the-art residual network (ResNet-34) that are
pre-trained on the ImageNet dataset (transfer learning) and arranged in the
U-Net topology. Similar to the work by Frid-Adar et al. [34], the encoder is
built with ResNet-34 [119] by removing the last layers, whereas the decoder
follows standard blocks of CNN with upsampling. Initially, the data is pre-
processed which follows converting images from grayscale channel to RGB
channel and resizing to 256$\times$ 256 and 512$\times$ 512 pixels for 2 stage
training scheme. The ResNet34U-Net is first trained with lower resolution
images and later the same model is fine-tuned (keeping previous learned
weights as initial weights) to adapt high resolution images. The significance
of 2 stage training is justified with the faster convergence and better
results. Besides, stochastic weight averaging (SWA) and test-time augmentation
(TTA) techniques are employed to improve the test results. The authors
achieved 0.84 of $DSC$ value that lead them among the top 9% proposed
approaches for the 2019 SIIMACR pneumothorax segmentation challenge.

In another U-Net variant, Zhang et al. [98] proposed a DEFU-Net model that
uses the fusion of dual encoder models to better extract the spatial features,
and a standard decoder network with upsampling. The dual encoder network is
equipped with a densely connected recurrent convolutional (DCRC) neural
network (inspired from DenseNet [120] and R2U-Net [56]) and dilated inception
convolution neural network (inspired from GoogleNet [121]), where the output
from each layer is merged by addition operation which is later concatenated
with the corresponding decoder layer. The DCRC aids in extracting high level
features, whereas the inception block facilitates to increase the network
width and improve the horizontal feature representation using various
receptive fields with dilated convolutions. The advantage of using dilated
convolutions is that it tends to increase the receptive field without changing
the number of training parameters [122]. The authors achieved significant
improvements over several U-Net variants such as residual U-Net [123], BCDU-
Net [90], R2U-Net and attention R2U-Net [56], etc. with dice score of 0.97 on
chest X-ray dataset. Wang et al. [100] synthesized a CXR dataset annotated
with clavicles, anterior ribs, posterior ribs and bones, on which a multitask
dense connection U-Net (MDU-Net) is trained for multi-class segmentation. A
feature separation network is introduced for multilabel segmentation where a
pixel value is associated with more than one label e.g. the pixels in the
overlapped regions of anterior and posterior ribs have multiple tags. The
encoder-decoder network uses a pre-trained DenseNet201 [120] model, where skip
connections are loaded with feature adaptation layers to adapt relevant
channels for fusion with the decoder network. For every CXR image, multiple
masks are generated concerning different annotations, thereby multiple
networks are trained to generate the corresponding mask. The implication of
increased training time is addressed with the help of transfer learning. The
authors compared the outcome with various deep learning models and achieved
improvements with $DSC$ values of 0.93, 0.81, 0.89, and 0.88 in segmentation
of clavicle, anterior rib, posterior rib, and bones respectively.

###  3.2 Computed tomography

Computed tomography imaging is based on the principle of utilizing the series
of the system of rotating X-rays to develop cross-sectional images or series
of slices of bones, blood vessels and soft tissues of the body [115]. In
contrast to plain X-ray imaging, CT scans provide rich information with high
quality images. This is generally utilized to examine people with serious
injuries or diseases like trauma, tumors, pneumonia, etc., and also to plan
medical, surgical or radiation treatment. Hence, various deep learning based
approaches are developed for faster diagnosis and treatment using CT imaging.

When the target is the segmentation of the internal organs, then models
adopting the attention mechanism help to focus the network on regions of
interest. Oktay et al. [37] proposed a novel attention gate based U-Net
framework to focus on pancreas regions and generate the corresponding
segmentation masks. The attention approach tends to suppress irrelevant
features and highlight the prominent features corresponding to the target
regions. The authors utilized the FCN with U-Net connectivity, where the skip
connections are loaded with these attention filters. Inspired from the work of
Shen et al. [124], each pixel is associated with a gating vector to determine
the regions to focus. The incorporation of this attention mechanism allowed
the authors to achieve significant improvements in the segmentation results
over other approaches on CT-150 and CT-82 datasets. In the real world
scenario, modalities may suffer from inherent ambiguities that coagulate the
actual nature of the disease. Following this, Kohl et al. [39] introduced a
probabilistic U-Net framework that combines the standard U-Net model with
conditional variational autoencoder (CVAE). For a sample image, CVAE generates
diverse plausible hypotheses from a low-dimensional latent space which are fed
to U-Net to generate the corresponding segmentation mask. It is shown that the
model can generate diverse segmentation samples, given the ground-truth
delineation from multiple experts. The trained model is evaluated on LIDC-IDRI
and Cityscapes datasets which outperformed other approaches in reproducing the
segmentation probabilities and masks. Inspired from this work many other
variants have been developed to capture the uncertainties, e.g. [125, 126,
127].

Tong et al. [42] proposed a U-Net framework for lung nodule segmentation,
where mini residual connections are introduced within the encoder and decoder
phases. The algorithm initiates with the process of generating the
segmentation of lung parenchyma with morphological operations and removal of
irrelevant features. The segmented lung parenchyma images are divided into
64$\times$64 slices along with the input images. Finally, the improved U-Net
model is trained and validated against the preprocessed dataset for segmenting
the pulmonary nodules. The authors evaluated the approach on LUNA2016 dataset
against various models and achieved promising results with a dice score of
0.74, however, the samples of pulmonary nodules were very limited and the
approach also lacked the 3D volumetric analysis. Later, Janssens et al. [44]
proposed a cascaded 3D FCN based deep learning model consisting of
“LocalizationNet” and “SegmentationNet” to estimate the bounding box (RoI) and
generate volumetric segmentation masks of lumbar vertebrae respectively. The
LocalizationNet comprises a 3D FCN regression model which is trained to
regress the displacement vectors associated with a voxel, representing
diagonal corners of the rectangular box. The localized information is fed to
SegmentationNet comprising a FCN 3D U-Net model to produce segmentation mask
for lumbar vertebrae. This two stage approach exhibited significant
improvement over the existing approaches but with the overhead computations of
two dedicated models. Recently, Park et al. [101] utilized a 3D U-Net model to
segment the lung lobe regions while also addressing the miss-detection of the
lobar fissure. Initially, the volumetric CT scans are preprocessed with
thresholding to identify lungs parenchyma, and region growing techniques [128]
to separate overlapping left and right lung regions. Later, these lobe
segmentations are generated with the help of 3D U-Net model, where the
segmentation results are further refined with the upsampling and segmentation
error correction. The authors utilized CT volumes from multiple centres
(hospitals) to evaluate the model performance, while achieving significant
improvements.

Motivated by the success of adversarial techniques, Dong et al. [69] proposed
a U-Net-GAN framework in which set of U-Nets is trained as a generator to
produce organs-at-risk (OARs) segmentation and FCN as a discriminator to
distinguish segmented masks from the ground-truth masks. The generator and
discriminator networks followed adversarial training, where each network
competes to achieve optimal segmentation masks of OARs. In another work, Liu
et al. [71] proposed a liver CT image segmentation framework named GIU-Net,
inspired by the supervised interactive segmentation approach named, graph cut
[129]. The improved U-Net (IU) model is designed with increased depth to
better extract semantic features that are trained to generate the segmentation
mask of the liver regions. Later, to further refine the segmentation results,
a slice covering maximum liver region is used as an initial slice to generate
graph cut energy function followed by maximum flow minimum cut algorithm. The
process is then repeated for all the slices to generate a complete sequence of
precise and stable segmentation masks with smoother boundaries.

In another work, a deep Q network (DQN) [130] driven approach is proposed by
Man et al. [72] that uses deformable U-Net to efficiently generate the
segmentation mask of the pancreas from CT scans with the extraction of its
contextual information and anisotropic features. Initially, the 3D volumes are
split into axial, coronal and sagittal 2D slices for each of which, DQN-based
deep reinforcement learning (DRL) agents tend to localize the pancreas to form
RoI slices. These slices are fed to the deformable U-Net models and finally,
based on the majority voting scheme 3D segmentation mask is generated. The
deformable U-Net [131] follows standard encoder-decoder architecture, where
convolution operations are replaced with deformable convolutions (DC). In DC,
regular convolution operation is accompanied by another convolution layer to
learn 2D offset for each pixel. It leverages the deep network’s ability to
learn the required receptive field rather than being fixed for segmenting the
regions having varying geometrical structure. This can also be understood as a
learnable dilated convolution.

Seo et al. [74] proposed a modified U-Net (mU-Net) framework that addressed
the classical problems associated with the standard U-Net model concerning
skip connection [132] and pooling operation (loss of spatial information). In
the mU-Net, the standard skip connections are replaced by object-dependent
filters to dynamically filter the feature maps based on the object size, where
features concerning the small objects are preserved by blocking the
deconvolution path and in case of large objects, feature maps indicating
boundary information is propagated to avoid duplication. The authors verified
the effectiveness of adaptive filters to preserve the features using the
permeation rate, while achieving the $DSC$ values of 0.98 and 0.89 on the
liver and liver-tumor segmentation respectively. In another application, a
Bayesian CNN with U-Net model and Monte Carlo (MC) dropout is introduced by
Hiasa et al. [75] for automated muscle segmentation from CT imaging for
musculoskeletal modelling. The design comprises two cascaded U-Net models,
where first is standard U-Net that localizes the skin surface and later
individual muscles (21 muscles) are segmented with Bayesian U-Net [133] that
uses MC dropout based on the structure-wise uncertainty, predictive structure-
wise variance (PSV) and predictive dice coefficient (PDC). Besides, authors
employed an active learning method to produce segmentation and uncertainty
from the unlabeled data, where the high uncertain data are relabeled manually
by experts while other data is directly used as training data. The authors
achieved a $DSC$ score of 0.89 on the popular TCIA dataset.

In another work, Song et al. [77] proposed a U-NeXt model to segment CT images
of gallstones, which is one of the common and frequently occurring diseases
worldwide. The U-NeXt model is equipped with the attention up-sampling blocks,
spatial pyramid pooling [134] of skip connections (SkipSPP), and multi-scale
feature extraction with the series of convolution layers along with the dense
connections. The overall architecture design is similar to U-Net++ model [135]
with slight variation in connections, convolution and pooling operations. The
authors trained and evaluated the model on the proposed dataset with 5,350
images and reported that with U-NeXt, IoU improved by 7% over baseline
biomedical image segmentation models. Unlike other U-Net variants that applies
multi-scale feature fusion, Fan et al. [102] recently proposed a multi-scale
attention U-Net model that uses a self-attention scheme for adaptive feature
extraction. The self-attention design comprises position-wise attention block
(PAB - installed on bottleneck layer) and multi-scale fusion attention block
(MFAB - installed on every stage of encoder path), where PAB captures feature
interdependencies in spatial dimension and MFAB captures the channel
dependencies for any feature map. The MA-Net is trained and evaluated on the
2017 LiTS challenge and achieved a $DSC$ score of 0.96 and 0.75 for liver and
liver-tumor segmentation respectively. However the results are not as
promising as achieved using mU-Net model [74].

###  3.3 Magnetic resonance imaging

Magnetic resonance imaging is synthesized by using the principles of nuclear
magnetic resonance (NMR) [136]. It is utilized in radiology to visualize the
anatomy and physiological process of the body organs. It uses a large magnetic
field and radio waves to create detailed images of organs and tissues within
the body. Based on the different attenuation values of the tissues e.g.
T1-weighted (T1), fluid attenuation inversion recovery (FLAIR), Dixon, etc.,
the electromagnetic waves emitted from the gradient magnetic field is detected
using the applied strong magnetic field by which the position and type of the
nucleus can be drawn inside the object. Unlike the X-rays, CT-scans and PET
scans; MRI scans do not involve the usage of ionizing radiations.

In recent years, MRI is utilized in computer-aided diagnosis systems involving
brain tumor segmentation. Inspired from the BraTS 2015 challenge, Dong et al.
[29] analysed the potential of FCN based U-Net model for brain tumor
segmentation via MRI sequences, where the authors achieved significant
improvement over the traditional segmentation approaches. SegNet is another
model that is most widely used for semantic segmentation [137]. Following
this, Kumar et al. [46] proposed a hybrid approach, U-SegNet, by integrating
skip connections into the base SegNet model. This enabled the model to capture
multiscale information and efficiently identify the tissue boundaries
concerning the white matter (WM), gray matter (GM), and cerebro-spinal fluid
(CSF). The authors achieved significant improvement over the base SegNet and
U-Net models with $DSC$ value of 0.90 on IBSR-18 dataset. In another
application, skull stripping is an essential step to study brain imaging,
where Hwang et al. [138] proposed to utilize a standard 3D U-Net model to
automate the process of skull stripping (brain extraction) from T1 MRI scans
for faster diagnosis and treatment. The training is carried with dice loss and
adam optimizer on neurofeedback skull-stripped (NFBS) dataset. The authors
achieved a dice value of 0.99, however, the comparative study is limited to
brain surface extractor (BSE) and robust brain extraction (ROBEX) algorithms.
The prostate cancer diagnosis is another challenging tasks for which Rundo et
al. [78] proposed an automated approach, USE-Net, that uses the U-Net model by
incorporating the squeeze-and-excitation (SE) blocks [116] in skip connections
to perform multi-class segmentation. Similar to the attention scheme [37], the
SE blocks tend to calibrate the channel-wise correlation while improving the
generalization capability of the model across multi-institutional datasets.
The USE-Net model outperformed its competitors when trained and evaluated on
all datasets combined, where for other scenarios (individual dataset and mixed
datasets), USE-Net struggled to achieve better results.

In recent years, to improve the biomedical image segmentation results, multi-
modality fusion (MMF) [139] approaches are utilized. The fused scans are rich
in information and offer multi-dimensional features. In this context, Kermi et
al. [48] proposed a modified U-Net model to segment the whole tumor and intra
tumor regions like enhancing tumor, edema and necrosis affected with high
grade glioma (HGG) and lower grade glioma (LGG) following from the BraTS 2018
challenge. The authors fused the T1, T2, T1c and FLAIR modalities and resized
to form the input feature map with rich tumor information. In the modified
model, residual blocks [123] are added between two convolution blocks and the
max-pooling operation is replaced with the strided convolutions [140]. The
model is trained and evaluated with fused modalities to obtain the multi-class
segmentation masks. Though the authors achieved good results, but lacked the
3D volumetric analysis. Later, Chen et al. [50] improved the performance of
the vanilla 3D U-Net model by adding spatiotemporal-separable 3D convolutions
[141] to form S3DU-Net model. The S3D convolution involves two convolution
layers i.e. 2D convolution operation to extract spatial features and then
additional 1D convolution to learn temporal features, furthermore, inception
[121] and residual connections [119] are added to better learn the complex
patterns. The S3DU-Net model is trained with dice loss and evaluated on dice
coefficient and Hausdorff distance metrics. The authors achieved average dice
scores of 0.69, 0.84 and 0.78, for enhancing tumor, whole tumor and tumor core
respectively on BraTS 2018 challenge. Recently, Punn et al. [104] proposed a
3D U-Net based framework for volumetric brain tumor segmentation. The proposed
architecture is divided into three components: multi-modalities fusion \- to
merge the MRI sequences with deep encoded fusion, tumor extractor \- to learn
the tumor patterns with 3D inception U-Net model using fused modalities, and
tumor segmenter \- to decode the multi-scale extracted features into multi-
class tumor regions. With such dedicated components trained using weighted
average of dice and IoU loss functions, the authors achieved significant
improvement over the existing approaches for BraTS 2017 and BraTS 2018
datasets.

For real-time applications, Wang et al. [79] proposed a multiscale statistical
U-Net (MSU-Net) to segment cardiac regions in MRI. The MSU-Net incorporates
statistical CNN (SCNN) [142] to fully exploit the temporal and contextual
information present in various channels of an input image or feature map along
with the multiscale parallelized data sampling approach. For multi-scale data
sampling, independent component analysis (ICA) [142] is applied over the
patches of data to form clusters of canonical form distributions which
represent spatio-temporal correlations at coarser scales. This data sampling
parallelization tends to speed up the performance significantly by 26.8% as
compared to the standard U-Net model and achieved an increased dice score by
1.6% on ACDC MICCAI 2017 challenge, while also improving significantly over
state-of-the-art GridNet [143] model. With the introduction of modality
transformations, Dong et al. [81] proposed a deep attention U-Net (DAU-Net)
model to automate the process of multi-organ segmentation for prostate cancer
diagnosis via synthetic MRI, that is generated by processing the computed
tomography scans using a cyclic generative adversarial network (CycleGAN)
[144]. Initially, the CycleGAN model is trained to learn CT to MRI
transformation which tends to add additional soft-tissue information without
additional data acquisition technique to produce sMRI data. Later, the sMRI
data is used to train 3D DAU-Net model which incorporates conventional
attention scheme [37] and deep supervision [82] with the U-Net model. The
approach is trained and evaluated with 140 datasets from prostate patients to
achieve $DSC$ value of 0.95, 0.87 and 0.89 for segmentation of bladder,
prostate and rectum respectively, while also showing improvement over using
raw CT images.

In another work, Wang et al. [82] proposed a 3D FCN model with deep
supervision and group dilation (DSD-FCN model) to address various challenges
concerning the automated MRI prostate segmentation like inhomogeneous
intensity distribution, varying prostate anatomy, etc., which makes it hard
for manual intervention. The proposed architecture follows vanilla U-Net
topology in which deep supervision is adopted to learn discriminative
features, whereas group dilated convolutions tend to acquire multi-scale
contextual information. The model is trained with the objective function
defined as the weighted average of cosine similarity and cross entropy using
the manually annotated institutional dataset and MICCAI PROMISE12 dataset,
where authors achieved the $DSC$ values of 0.86 and 0.88 respectively.
Recently, Dong et al. [103] integrated 3D U-Net model with deformable
convolutions [131] for cardiac MRI segmentation. The deformable U-Net (DeU-
Net) includes a temporal deformable aggregation module (TDAM) to generate
fused feature maps using offset prediction network. The fused feature maps are
then fed to deformable global position attention (DGPA) network to map the
multi-dimensional contextual information into generalized and localized
features. The proposed approach outperformed other models to generate
efficient segmentation masks involving subtle structures.

###  3.4 Positron emission tomography

The positron emission tomography [145] is a widely used imaging in various
clinical applications like oncology, brain, heart, etc., that helps in
visualizing the biochemical and physiological reaction processes within the
human body. The PET images are obtained by injecting a full dose of
radioactive tracer or inhalation of gas to meet the clinical requirements.
However, for minimal harm to human health, low-dose PET imaging is adopted to
produce high quality imaging [146].

With the huge success of U-Net in biomedical image segmentation, Durand et al.
[51] demonstrated the potential of 3D U-Net model in 18F-fluoro-ethyl-tyrosine
(18F-FET) PET lesion detection and segmentation. F-FET PET/CT scans were
acquired using a dynamic protocol from 37 patients, where the ground-truth
segmentation masks were generated using manual delineation and binary
thresholding. The 3D U-Net model comprises three stages of encoder and decoder
paths with standard convolutions and pooling operations. The authors achieved
a $DSC$ value of 0.79 on training and validation sets. However, the results
could further be improved by increasing the data size with GAN based data
augmentation techniques [147] and other U-Net based approaches.

The integration of PET and CT modalities offer metabolic and anatomical
information simultaneously. In this context, Zhao et al. [52] proposed to
utilize the multi-modalities (PET and CT) for computer-aided cancer diagnosis
and treatment with the help of 3D FCN based V-Net [148] model, which is an
extension of U-Net model for volumetric segmentation. A feature or
intermediate level fusion approach is adopted, where two independent sub-
segmentation networks are constructed to extract dedicated feature maps from
each modality and are later fused with the cascaded convolution blocks that
follow the V-Net model scheme to finally compute the tumor segmentation mask.
The proposed framework is trained and validated on a clinical dataset of 84
patients suffering from lung cancer that consists of PET and CT imaging, where
a dice value of 0.85 is achieved while outperforming other traditional models
that use unary modality. In similar approach, Guo et al. [84] adopted the
fusion of PET and CT modalities to segment head and neck cancer (HNC) labelled
as gross tumor volume (GTV). The authors utilized the modified 3D U-Net model
in which the convolution blocks in encoder and decoder paths are replaced by
dense convolution blocks [120]. The authors trained and evaluated the model on
TCIA-HNC dataset, while achieving the $DSC$ value of 0.73 on the dedicated
test set.

Recently, Lu et al. [106] proposed U-Net based automatic tumor segmentation
approach in PET scans. The authors employed a transfer learning approach,
where pre-trained VGG-19 blocks are added in the encoder phase to address the
challenge of limited data availability. The authors adopted the DropBlock as a
replacement for dropout to effectively regularize the convolution blocks. The
model is fine-tuned using the Jaccard distance (IoU) as the loss function and
the performance is validated with 1,309 PET images provided by the Shanghai
Xinhua hospital (XH), that displayed improvements over the vanilla U-Net
model. To address the need of reliable and robust PET based tumor segmentation
model, Leung et al. [107] proposed a novel physics guided deep learning based
framework comprising three dedicated modules that segment each slice of PET
volume to generate a complete mask. The first module tends to extract the
realistic tumors with the available ground-truth boundaries via stochastic
kernel-density estimation and physics based approach to generate simulated
images. These images are fed to improved U-Net model in the second module,
that has minimal convolution and pooling blocks accompanied by dropout layers
to aid in learning the complex features and generate efficient masks. Later,
in the third module, the network is fine-tuned with delineation provided by
the radiologist as surrogate masks to improve the learned features. The
proposed framework achieved dice scores of 0.87 and 0.73 to segment primary
tumors on simulated and patient images and outperformed several semi-automated
approaches.

###  3.5 Ultrasound

Ultrasound is acoustic energy in the form of waves having a frequency beyond
the human hearing range. These are generated with the help of piezoelectric
crystals which deform under the influence of electric field and generate
compression waves when an alternating voltage is applied. Ultrasonography
[149] is an ultrasound based diagnostic imaging technique used for visualizing
the internal body organs by processing the reflected signals. The deep
learning technologies aid in diagnosing US imaging to segments regions of
interest like breast mass, pelvic floor levator muscle discontinuity, etc.

In consideration of breast cancer being the deadliest cancers among women,
Almajalid et al. [53] proposed an automatic breast ultrasound (BUS) image
segmentation system to aid in its diagnosis and treatment. The authors
utilized the vanilla U-Net model on the preprocessed BUS images. The images
are preprocessed using the contrast enhancement with histogram equalization
and noise reduction with speckle reducing anisotropic diffusion (SRAD) [150]
techniques to improve the image quality. Finally, with the assumption of the
presence of a single tumor region the authors filtered the false positive
regions to remove the noisy regions. Later, Wang et al. [54] proposed a multi-
feature guided CNN model for classification and segmentation of the bone
surfaces in the US scans. The US images are initially processed with pre-
enhancing (PE) net to synthesize a US scan that highlights the bone surface,
by using B-mode US scan and three filtered image features, including local
phase tensor image (LPT), local phase bone image (LB) and bone shadow enhanced
image (BPE). The feature enriched images are then used by a classification
U-Net model (cU-Net) to produce the segmentation mask and identify the type of
the bone surface. This multi-task deep learning framework achieved promising
segmentation and classification results with F1-score of 0.96 and 0.90 on
SonixTouch and Clarius C3 datasets respectively.

In another approach, Yang et al. [86] proposed a dual path U-Net model for
segmentation of lumen and media-adventitia from the IntraVascular UltraSound
(IVUS) scans to aid in cardiovascular diseases diagnosis. Due to the limited
availability of the data samples, the DPU-Net is trained with the real-time
augmentor that generates and integrates three types of artefacts: bifurcation,
side vessel, and shadow, and other common augmentation operations with
training images. In contrast to vanilla U-Net, DPU-Net involves multi-branch
parallel encoding and decoding operations, where feature maps are extracted
and reconstructed with different kernel sizes at the same hierarchical level.
With this network-in-network architecture and real-time augmentation approach,
the authors achieved Jaccard measure (IoU) of 0.87 and 0.90 on 40 MHz and 20
MHz frames respectively from IVUS dataset. Li et al. [88] incorporate dense
connections in the U-Net model (DenseU-Net) to efficiently segment levator
hiatus from ultrasound images. The implication of dense connections enabled
feature reuse and reduction in the trainable parameters. The DenseU-Net model
is trained to generate the binary segmentation mask which is post-processed
with binary thresholding, and localized regions are generated with active
contour model [151]. In another application of eyeball segmentation, Lin et
al. [89] proposed a semantic-embedding and shape-aware U-Net model (SSU-Net),
where the authors employed a signed distance field (SDF) instead of a binary
mask as the label to learn the shape information. In addition, the model is
equipped with a semantic embedding module (SEM) to fuse the semantic
information at coarser levels of the SSU-Net model. The SEM block draws
features from two low-level stages and one corresponding stage, where lower
level features are convolved and bilinear interpolation is applied to restore
the resolution at the same scale. The authors achieved better segmentation
performance with $DSC$ value of 0.96 on a dataset with 668 US images collected
from Beijing Tongren hospital.

In another application area, Dunnhofer et al. [108] emphasized on the tracking
of knee femoral condyle cartilage during ultrasound guided invasive
procedures. The Siam-U-Net model combines the potential of the U-Net model
with siamese framework [152] for tracking the cartilage in the real-time
ultrasound sequences. In Siam-U-Net two encoder blocks are adopted which are
fed with resized-cropped US sequences named as, searching area and target
cartilage. After five blocks of encoding layers, the acquired feature maps of
two inputs are cross-correlated using convolution operation applied to
searching area feature maps with target embedding as a filter, which results
in localizing the implicit position of the cartilage in the searching area
slice. Later, the slice is reconstructed in the decoder phase to generate the
segmentation mask of the cartilage. The Siam-U-Net model achieved an average
dice score of 0.70 with significant improvement over other approaches.
However, the results could further be improved by expanding the dimension of
the model into 3D space for considering the neighbouring voxels correlation.

Due to the low signal to noise ratio (SNR) in US imaging, real-time analysis
is still a challenging task. Recently, Zhang et al. [109] proposed a U-Net
based deep learning approach to realize the multi-needle segmentation in the
3D transrectal US (TRUS) images of high dose rate (HDR) prostate
brachytherapy. The U-Net model is loaded with the attention scheme in the skip
connections to address the challenge of identifying the smaller needles, while
spatial continuity of the needles is maintained with total variation
regularization. The model is trained with a deep supervision approach, where
patches of needle masks are generated to compute the cross entropy loss and
accordingly optimize the training weights. With the proposed framework, the
authors achieved adequate performance gain on multi-needle segmentation for
prostate brachytherapy. Byra et al. [110] proposed a selective kernel U-Net
(SKU-Net) model for breast mass segmentation in the US imaging while also
addressing the challenge of variable breast mass size and image properties. In
SKU-Net, each convolution layer of the U-Net model is replaced by SK block,
that tends to dynamically adapt the receptive field. Similar to the concept of
dual path U-Net [86], the SK module [153] is designed using two branches,
where one uses dilated convolutions and other is without dilation to generate
feature maps. Later, these features are merged and global average pooling,
followed by FC layer and sigmoid activation is applied to construct attention
coefficients for each channel in the feature map. With this approach authors
achieved significant improvement over vanilla U-Net model across multiple
datasets.

##  4 Other U-Net variants and imaging

In this section, various U-Net variants are presented that are introduced as
the biomedical image segmentation networks, where each model acts as a generic
architecture that is trained and evaluated on multiple modalities. In the
growing phase of biomedical image segmentation, Alom et al. [56] integrated
the potential of multiple state-of-the-art deep learning models such as
recurrent CNN [154], residual CNN [119] and U-Net to form RU-Net and R2U-Net
for BIS. In the RU-Net model, the standard convolution and up-convolution
units are improved by incorporating recurrent convolutional layers (RCL),
whereas in R2U-Net both RCL and residual units are added. These models are
trained and evaluated on three different modalities such as retina blood
vessel segmentation (DRIVE, STARE, and CHASH-DB1 datasets), skin cancer
segmentation (ISIC 2017 Challenge), and lung segmentation (KDSB 2017
challenge). With the immense application of U-Net model in the medical domain,
Isensee et al. [60] proposed a self-adapting framework, no-newU-Net (nnU-Net)
to establish the generalized architecture and training mechanism for vivid
modalities, inspired from the medical segmentation decathlon (MSD) challenge.
The nnU-Net framework comprises an ensemble of 2D U-Net, 3D U-Net and 3D U-Net
cascade, along with an automated pipeline to adapt the requirements of the
dataset such as preprocessing, data augmentation and post-processing. The
model achieved state-of-the-art segmentation results without manual
intervention for different modalities in medical segmentation decathlon
challenge.

Zhou et al. [62] proposed a nested U-Net architecture, U-Net++, to narrow down
the gap between the encoded and decoded feature maps. In contrast to the U-Net
model, U-Net++ model follows convolutions on dense and nested skip connections
to effectively capture the coarser details. Furthermore, a deep supervision
approach is adopted to prune the model based on the loss (combined binary
cross entropy and dice coefficient) estimated at different semantic levels.
The performance of the model is validated with multiple datasets involving
KDSB18, ASU-Mayo, MICCAI 2018 LiTS Challenge and LIDC-IDRI, while
outperforming other models. Azad et al. [90] proposed another extension of
U-Net, where bi-directional ConvLSTM (BConvLSTM) with densely connected
convolutions (BCDU-Net) is introduced for BIS. The skip connections are
equipped with BConvLSTM [155] to concatenate the feature maps between the
encoded layer and the corresponding decoded layer. Furthermore, the dense
connections are added at the bottleneck layer to extract and propagate
features with minimal parameters. The authors achieved promising results
across DRIVE, ISIC 2018 and LUNA datasets. Gu et al. [92] addressed the loss
of spatial information while using the strided convolutions and pooling in
U-Net with context-encoder network (CE-Net) to capture and preserve the
information flow for BIS. In CE-Net the encoder unit is loaded with pre-
trained ResNet blocks, the bottleneck layer (context extractor) includes dense
atrous convolution (DAC) and residual multi-kernel pooling (RMP) blocks, and
decoder block follows consecutive convolution and deconvolution blocks. The
DAC module combines the design of Inception-ResNet-V2 model and atrous or
dilated convolution, whereas RMP generates stacked feature maps followed from
the pooling operations with varying window sizes.

For histopathological image segmentation, Punn et al. [111] proposed an
inception U-Net model where standard convolution layers are replaced by
inception blocks that consists of parallel convolutions of varying filter
sizes and a hybrid pooling operation. The hybrid pooling operation draws the
potential feature maps from the spectral domain via Hartley transform [156] to
preserve more spatial information, and spatial domain with the help of max
pooling to aim for sharp features, by using the $1\times 1$ convolution. The
model is trained using the segmentation loss function described as the average
of binary cross entropy, dice coefficient and Jaccard index losses to address
the class imbalance problem in KDSB18 dataset. The authors achieved
significant improvement over other models with less number of parameters.
Ibtehaz et al. [112] proposed another extension of the U-Net model as
MultiResU-Net, where the convolution operations are replaced with MultiRes
blocks in encoder-decoder paths, and Res path is added in the bottleneck
layer. Inspired from the inception and residual model, the MultiRes blocks are
built using stacked convolutions with a succession of $3\times 3$ filters, and
a residual $1\times 1$ convolution connection is added. The Res path tends to
propagate the feature maps from the encoder phase to decoder phase with the
series of residual convolution blocks. The model is evaluated on different
datasets covering fluorescence images, ISBI-2012, ISIC-2017, CVC-ClinicDB and
BraTS17.

Table 6: Summary of popular BIS datasets. Dataset |  Description |  Availability
---|---|---
ISBI 2012 |  Electron microscopy cell slides for cell segmentation |  http://brainiac2.mit.edu/isbi_challenge/
ISBI |  2D and 3D videos of moving cells for cell tracking |  http://celltrackingchallenge.net/
KDSB 2018 |  Histopathological cell images for nuclei segmentation |  https://www.kaggle.com/c/data-science-bowl-2018
PanNuke |  Histopathological slides for nuclei segmentation |  https://jgamper.github.io/PanNukeDataset/
DRIVE |  Retinal fundus images for vessel extraction |  https://drive.grand-challenge.org/
STARE |  Retinal fundus imaging for blood vessel segmentation |  http://cecas.clemson.edu/%7Eahoover/stare/
CHASE_DB1 |  Retinal fundus imaging for blood vessel segmentation |  https://blogs.kingston.ac.uk/retinal/chasedb1/
LiTS |  Liver CT scans for tumor segmentation |  https://competitions.codalab.org/competitions/17094
LIDC-IDRI |  Lung CT scans for cancer segmentation |  https://wiki.cancerimagingarchive.net/display/Public/LIDC-IDRI
LUNA 2016 |  CT scans for lung nodule segmentation |  https://luna16.grand-challenge.org/
xVertSeg |  CT spine images for vertebra segmentation |  http://lit.fe.uni-lj.si/xVertSeg/
SIIM-ACR |  Chest X-rays for pneumothorax segmentation |  https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation/data
ISIC |  Dermoscopy images for skin lesion segmentation |  https://www.isic-archive.com/
BraTS 2012 – 2020 |  MRI modalities (T1, T2, FLAIR) for brain tumor segmentation. |  http://braintumorsegmentation.org/
ISLES |  MRI scans for stroke lesion segmentation |  http://www.isles-challenge.org/
ICCVB |  Prostate MRI and retinal fundus imaging |  http://i2cvb.github.io/
IBSR |  Repository of MRI imaging |  https://www.nitrc.org/projects/ibsr
ACDC 2017 |  MRI imaging for cardiac diagnosis and segmentation |  https://www.creatis.insa-lyon.fr/Challenge/acdc/index.html
PROMIS 2012 |  Prostate MRI image segmentation |  https://promise12.grand-challenge.org/
Medical Segmentation Decathlon |  MRI and CT modalities for tumor segmentation in various organs like liver, brain, lung, etc. |  http://medicaldecathlon.com/
OASIS |  MRI and PET images for aging analysis and segmentation |  https://www.oasis-brains.org/
Head-Neck-PET-CT |  PET and CT imaging for tumor segmentation |  https://wiki.cancerimagingarchive.net/display/Public/Head-Neck-PET-CT
BUSIS |  Ultrasound imaging for breast tumor segmentation |  http://cvprip.cs.usu.edu/busbench/
BUSI |  Breast ultrasound scans for tumor segmentation |  https://scholar.cu.edu.eg/?q=afahmy/pages/dataset

##  5 U-Net in COVID-19 diagnosis

The on-going pandemic of the severe acute respiratory syndrome - coronavirus
(SARS-CoV-2) also known as COVID-19 has brought the worldwide crisis along
with the rampant loss of lives. This contagious virus initiated from Wuhan,
the People’s Republic of China in December 2019 and till April 3, 2021, have
caused 130,771,176 infections and 2,846,263 deaths worldwide [157]. Currently,
the most reliable COVID-19 diagnosis approach follows reverse-transcriptase
polymerase chain reaction (RT-PCR) testing, however, it is time consuming and
less sensitive to identify the virus at the early stages.

With the advancements in the technology and data acquisition systems [158,
159], deep learning based approaches are developed to assist in the COVID-19
diagnosis with the help of CT and X-ray modalities [160] to control the
exponential growing trend [161] of the spread. Wu et al. [162] proposed a JCS
framework (similar to cU-Net) for joint classification and segmentation of
COVID-19 from chest CT scans using the U-Net model. In another U-Net based
implementation, a feature variation block is introduced in the COVID-SegNet
model [163] to better segment the COVID-19 infected regions by highlighting
the boundaries and diverse infected regions. The lung infection segmentation
deep network (Inf-Net) [164] followed U-Net topology with diverse
modifications including reverse attention and parallel partial decoder. The
authors validated the performance in the supervised and semi-supervised mode
to address the challenge of limited availability of the labelled data.
Recently, Punn et al. [165] introduced a hierarchical segmentation approach,
CHS-Net that involves two cascaded residual attention inception U-Net (RAIU-
Net) models, where first generates lungs contour, which are fed to the second
model to identify COVID-19 infected regions using CT images. The RAIU-Net
model is designed with residual inception U-Net model and spectral-spatial-
depth attention blocks. The authors achieved promising results in generating
the infected segmentation masks.

Furthermore, similar approaches are also developed for X-ray imaging for the
screening of COVID-19 [166]. Alom et al. [167] proposed a robust
classification and segmentation framework of coronavirus infected X-ray and CT
images, where classification is performed using inception residual recurrent
convolutional neural network (IRRCNN) with transfer learning and NABLA-N model
is used for localizing the infected regions. In addition, other deep learning
based application areas are also explored to control the spread of virus such
as automated social distancing monitoring [168], mask detection [169], etc.
Furthermore, the survey of deep learning based approaches for COVID-19
diagnosis [159] reveals the significant impact of U-Net for CAD systems.
Following these developments, it is believed that these artificial intelligent
approaches will continue to evolve and contribute towards the faster and
efficient diagnosis of the coronavirus.

##  6 Analysis

Over the years, the advancements in deep learning and computer vision
techniques have attracted many researchers to contribute to the healthcare
domain with a variety of tasks e.g. classification, detection, segmentation,
etc. With segmentation being a critical task that drives the diagnosis process
[170], researchers have developed a keen interest to develop a computer-aided
diagnosis system to speed up the treatment process.

Among the published approaches or frameworks, U-Net appears to be the
prominent choice [8] to develop novel architectures to adapt multiple
modalities with optimal segmentation performance. Following such high utility
of the model, this article presented the recent developments in U-Net based
approaches for biomedical image segmentation. Due to the high mutability and
modularity design, U-Net topology can easily be integrated with other state-
of-the-art deep learning models such as AlexNet [171], VGGNet [172], ResNet
[119], GoogLeNet [121], MobileNet [173], DenseNet [120], etc., to produce the
desired results depending on the application. This ease of integration opens a
wide spectrum of application for U-Net with endless possibilities of novel
architecture designs. Considering the implementation strategies mostly authors
applied an end-to-end training-from-scratch approach with minimal pre-
processing i.e. resizing and normalization. For the training phase, most
models employed a hybrid loss function that combines the binary cross entropy
loss with dice similarity coefficient loss or with Jaccard loss, which tends
to better penalize the false positive and false negative predictions.

From the reviewed articles it is observed that some of the segmentation
approaches utilize the local dataset (datasets that are not publicly
accessible), which tend to limit their reusability and reachability. In order
to develop a widely acceptable solution, we provide the summary of publicly
available datasets for BIS (shown in Table 6). These benchmark datasets aid
the research community to validate the existing performance and propose
further improvements. Among the reviewed articles, CT and MRI modalities cover
a wide range of U-Net variants for biomedical image segmentation. Moreover,
for PET scan and ultrasound imaging most of the proposed approaches are
validated on the local dataset, where for X-rays the approaches aim to
localize the target structure with the bounding boxes. Despite such variants,
it is difficult to conduct an effective comparative analysis of the results
because each approach is evaluated with different evaluation metrics such as
accuracy, F1-score, Jaccard index, etc. However, among these metrics, the dice
similarity coefficient is most widely utilized to quantize the segmentation
performance.

Considering the present survey it is also observed that each modality requires
a different approach to address the corresponding challenges. Though there are
segmentation approaches that are validated on multiple modalities to form
generic architectures like nn-UNet, U-Net++, MR-Unet, etc. but it is difficult
to achieve optimal performance in all segmentation tasks. The main reason is
due to the diverse variation in the features corresponding to the target
structures involving lungs nodule, brain tumor, skin lesions, retina blood
vessels, nuclei cells, etc. and hence require different mechanism (dense,
residual, inception, attention, fusion, etc.) to integrate with U-Net model to
effectively learn the complex target patterns. Moreover, the presence of noise
or artefacts in different modalities adds another factor to propose different
segmentation methods.

##  7 Scope and challenges

The deep learning technologies have played a vital role in advancements
towards medical diagnosis and applications. Generally, the deep learning based
technologies such as U-Net aims to develop CAD systems to achieve the desired
results with minimal error. Despite U-Net being superefficient for biomedical
image segmentation, it certainly has its limits and challenges. One such major
challenge is concerned with the computational power requirement which tends to
limit the feasibility of the approach. Following this many cloud based high
performance computing environments are developed for mobile, efficient and
faster computations. Although progress is also made towards the model
compression and acceleration techniques [174] with great achievements,
however, it is still required to establish the concrete benchmark results for
real-time applications. Recently, Tan et al. [175] proposed an EfficientNet
framework that uses compound coefficients for uniform scaling in all
dimensions. This could make U-Net design streamline for complex segmentation
tasks with minimal change in the parameters.

Furthermore, these powerful deep learning approaches are data-hungry i.e. the
amount of data available directly affects the model performance towards
achieving the robust results. However, the expense of data acquisition and
delineation, and data security, results in the limited availability of the
data which bottlenecks the development of real-world systems. In this context,
various data augmentation strategies [176] are proposed that tend to alleviate
the performance of the model while drawing the advantages of big data.
Generally, the image augmentation strategies involve geometric
transformations, color space augmentations, kernel filters, mixing images,
random erasing, feature space augmentation, adversarial training, generative
adversarial networks, neural style transfer, and meta-learning. However, the
diversity of augmented data is limited by the available data which could
result in overfitting. In another approach, U-Net models utilize transfer
learning approaches [177] to optimize the pre-trained model to adapt the
targeted task while having insufficient training data. These deep transfer
learning techniques are categories under four broad areas: instances based,
mapping based, network based and adversarial based [178]. These approaches are
generally adopted in combinations for practical situations. The potential of
this approach attracts many researchers to advance the U-Net based BIS
approaches.

In general, the decision made in the rule-based applications can be traced
back to its origin, however, deep CNN models lack transparency in the decision
making process, where the input and output are well-presented but the
processing in the hidden layers is difficult to interpret and understand, and
hence these are also termed as black-box models. To better interpret these
models various visualization based approaches are proposed such as local
interpretable model-agnostic explanations (LIME) [179], shapley additive
explanation (SHAP) [180], partial dependence plots (PDP) [181], anchor [182],
etc. Currently, these approaches are applied to explain and interpret the
obtained results from deep learning models, but still a concrete benchmark
scheme is required to be established.

It is evident that the above discussed challenges are the most crucial to
address for developing the real-world implications of the deep learning
models. With regular advancements in deep learning, these challenges are
tackled with hardware and software oriented approaches which consequently
attracts researchers to develop novel architectures and frameworks for
biomedical image segmentation.

##  8 Conclusion

The deep learning approaches especially U-Net has great potential to influence
the clinical applications involving automated biomedical imaging segmentation.
With U-Net being a breakthrough development, it sets up the foundation for the
development of novel architectures concerning identification and localization
of the target regions or sub-regions. Following from this context, in this
article, various U-Net variants are presented, covering current advancement
and developments in the area of biomedical image segmentation serving various
modalities. Each U-Net variant features unique developments over the
challenges incurred due to different modalities such as noise, overlap, narrow
regions, etc. With such high utility and potential of the U-Net models, it is
believed that U-Net based models will be widely applied to address various
challenging problems incurred in the biomedical image segmentation for
developing the real world computer-aided diagnosis systems.

## Acknowledgment

We thank our institute, Indian Institute of Information Technology Allahabad
(IIITA), India and Big Data Analytics (BDA) lab for allocating the necessary
resources to perform this research. We extend our thanks to our colleagues for
their valuable guidance and suggestions.

## References

  * [1] A. Alexander, M. McGill, A. Tarasova, C. Ferreira, and D. Zurkiya, “Scanning the future of medical imaging,” _Journal of the American College of Radiology_ , vol. 16, no. 4, pp. 501–507, 2019.
  * [2] Z. Hughes, “Medical imaging types and modalities,” https://www.ausmed.com/cpd/articles/medical-imaging-types-and-modalities, 2019, [Online; accessed November 25, 2020].
  * [3] TMI, “Types of medical imaging,” https://www.doc.ic.ac.uk/~jce317/types-medical-imaging.html, 2019, [Online; accessed November 25, 2020].
  * [4] A. Elnakib, G. Gimel’farb, J. S. Suri, and A. El-Baz, “Medical image segmentation: a brief survey,” in _Multi Modality State-of-the-Art Medical Image Segmentation and Registration Methodologies_. Springer, 2011, pp. 1–39.
  * [5] S. Masood, M. Sharif, A. Masood, M. Yasmin, and M. Raza, “A survey on medical image segmentation,” _Current Medical Imaging_ , vol. 11, no. 1, pp. 3–14, 2015.
  * [6] S. Deepa, B. A. Devi _et al._ , “A survey on artificial intelligence approaches for medical image classification,” _Indian Journal of Science and Technology_ , vol. 4, no. 11, pp. 1583–1595, 2011.
  * [7] J. A. Maintz and M. A. Viergever, “A survey of medical image registration,” _Medical image analysis_ , vol. 2, no. 1, pp. 1–36, 1998.
  * [8] S. Minaee, Y. Boykov, F. Porikli, A. Plaza, N. Kehtarnavaz, and D. Terzopoulos, “Image segmentation using deep learning: A survey,” _arXiv preprint arXiv:2001.05566_ , 2020.
  * [9] X. Liu, Z. Deng, and Y. Yang, “Recent progress in semantic image segmentation,” _Artificial Intelligence Review_ , vol. 52, no. 2, pp. 1089–1106, 2019.
  * [10] L. Chen, M. Strauch, and D. Merhof, “Instance segmentation of biomedical images with an object-aware embedding learned with local constraints,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 451–459.
  * [11] I. R. I. Haque and J. Neubert, “Deep learning approaches to biomedical image segmentation,” _Informatics in Medicine Unlocked_ , vol. 18, p. 100297, 2020\.
  * [12] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [13] T. Zhou, S. Ruan, and S. Canu, “A review: Deep learning for medical image segmentation using multi-modality fusion,” _Array_ , vol. 3, p. 100004, 2019\.
  * [14] S. engines, “The top list of academic search engines,” https://paperpile.com/g/academic-search-engines/, 2020, [Online; accessed December 06, 2020].
  * [15] CORE, “Computing research and education association of australasia,” https://www.core.edu.au/, 2020, [Online; accessed December 06, 2020].
  * [16] S. Bhattacharyya, “A brief survey of color image preprocessing and segmentation techniques,” _Journal of Pattern Recognition Research_ , vol. 1, no. 1, pp. 120–129, 2011.
  * [17] J. Zhou, Q. Zhang, B. Zhang, and X. Chen, “Tonguenet: A precise and fast tongue segmentation system using u-net with a morphological processing layer,” _Applied Sciences_ , vol. 9, no. 15, p. 3128, 2019.
  * [18] P. F. Christ, M. E. A. Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, M. Armbruster, F. Hofmann, M. D’Anastasi _et al._ , “Automatic liver and lesion segmentation in ct using cascaded fully convolutional neural networks and 3d conditional random fields,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 415–423.
  * [19] D. Ciresan, A. Giusti, L. M. Gambardella, and J. Schmidhuber, “Deep neural networks segment neuronal membranes in electron microscopy images,” in _Advances in neural information processing systems_ , 2012, pp. 2843–2851.
  * [20] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [21] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3d u-net: learning dense volumetric segmentation from sparse annotation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2016, pp. 424–432.
  * [22] X. Glorot and Y. Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , 2010, pp. 249–256.
  * [23] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 1026–1034.
  * [24] A. Garcia-Garcia, S. Orts-Escolano, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Garcia-Rodriguez, “A survey on deep learning techniques for image and video semantic segmentation,” _Applied Soft Computing_ , vol. 70, pp. 41–65, 2018.
  * [25] J. Ma, “Segmentation loss odyssey,” _arXiv preprint arXiv:2005.13449_ , 2020\.
  * [26] A. Fenster and B. Chiu, “Evaluation of segmentation algorithms for medical imaging,” in _2005 IEEE Engineering in Medicine and Biology 27th Annual Conference_. IEEE, 2006, pp. 7186–7189.
  * [27] R. Smith-Bindman, D. L. Miglioretti, E. Johnson, C. Lee, H. S. Feigelson, M. Flynn, R. T. Greenlee, R. L. Kruger, M. C. Hornbrook, D. Roblin _et al._ , “Use of diagnostic imaging studies and associated radiation exposure for patients enrolled in large integrated health care systems, 1996-2010,” _Jama_ , vol. 307, no. 22, pp. 2400–2409, 2012.
  * [28] A. C. Dovales, R. W. Harbron, A. A. de Souza, L. A. da Rosa, A. B. de González, M. S. Pearce, and L. H. Veiga, “Patterns and trends in outpatient diagnostic imaging studies of the brazilian public healthcare system, 2002–2014,” _Health Policy and Technology_ , vol. 8, no. 3, pp. 254–260, 2019.
  * [29] H. Dong, G. Yang, F. Liu, Y. Mo, and Y. Guo, “Automatic brain tumor detection and segmentation using u-net based fully convolutional networks,” in _annual conference on medical image understanding and analysis_. Springer, 2017, pp. 506–517.
  * [30] MICCAI, “Brats 2015 - multimodal brain tumor segmentation,” https://www.smir.ch/BRATS/Start2015, 2015, [Online; accessed December 15, 2020].
  * [31] R. Rashid, M. U. Akram, and T. Hassan, “Fully convolutional neural network for lungs segmentation from chest x-rays,” in _International Conference Image Analysis and Recognition_. Springer, 2018, pp. 71–80.
  * [32] J. Shiraishi, S. Katsuragawa, J. Ikezoe, T. Matsumoto, T. Kobayashi, K.-i. Komatsu, M. Matsui, H. Fujita, Y. Kodera, and K. Doi, “Development of a digital image database for chest radiographs with and without a lung nodule: receiver operating characteristic analysis of radiologists’ detection of pulmonary nodules,” _American Journal of Roentgenology_ , vol. 174, no. 1, pp. 71–74, 2000.
  * [33] NIH, “Tuberculosis chest x-ray image data sets,” https://lhncbc.nlm.nih.gov/publication/pub9931, 2019, [Online; accessed January 25, 2021].
  * [34] M. Frid-Adar, A. Ben-Cohen, R. Amer, and H. Greenspan, “Improving the segmentation of anatomical structures in chest radiographs using u-net with an imagenet pre-trained encoder,” in _Image Analysis for Moving Organ, Breast, and Thoracic Images_. Springer, 2018, pp. 159–168.
  * [35] Q. Que, Z. Tang, R. Wang, Z. Zeng, J. Wang, M. Chua, T. S. Gee, X. Yang, and B. Veeravalli, “Cardioxnet: Automated detection for cardiomegaly based on deep learning,” in _2018 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2018, pp. 612–615.
  * [36] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2097–2106.
  * [37] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz, B. Glocker, and D. Rueckert, “Attention u-net: Learning where to look for the pancreas,” 2018.
  * [38] H. R. Roth, A. Farag, E. Turkbey, L. Lu, J. Liu, and R. M. Summers, “Data from pancreas-ct. the cancer imaging archive,” 2016.
  * [39] S. Kohl, B. Romera-Paredes, C. Meyer, J. De Fauw, J. R. Ledsam, K. Maier-Hein, S. A. Eslami, D. J. Rezende, and O. Ronneberger, “A probabilistic u-net for segmentation of ambiguous images,” in _Advances in Neural Information Processing Systems_ , 2018, pp. 6965–6975.
  * [40] S. G. Armato III, G. McLennan, L. Bidaut, M. McNitt-Gray, C. Meyer, A. Reeves, and L. Clarke, “Data from lidc-idri. the cancer imaging archive,” 2015.
  * [41] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [42] G. Tong, Y. Li, H. Chen, Q. Zhang, and H. Jiang, “Improved u-net network for pulmonary nodules segmentation,” _Optik_ , vol. 174, pp. 460–469, 2018.
  * [43] LUNA16, “Luna - lung nodule analysis,” https://luna16.grand-challenge.org/, 2016, [Online; accessed December 11, 2020].
  * [44] R. Janssens, G. Zeng, and G. Zheng, “Fully automatic segmentation of lumbar vertebrae from ct images using cascaded 3d fully convolutional networks,” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 893–897.
  * [45] MICCAI, “xvertseg challenge,” http://lit.fe.uni-lj.si/xVertSeg/, 2016, [Online; accessed December 11, 2020].
  * [46] P. Kumar, P. Nagar, C. Arora, and A. Gupta, “U-segnet: fully convolutional neural network based automated brain tissue segmentation tool,” in _2018 25th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2018, pp. 3503–3507.
  * [47] MICCAI, “Ibsr - internet brain segmentation repository,” https://www.nitrc.org/projects/ibsr, 2018, [Online; accessed December 15, 2020].
  * [48] A. Kermi, I. Mahmoudi, and M. T. Khadir, “Deep convolutional neural networks using u-net for automatic brain tumor segmentation in multimodal mri volumes,” in _International MICCAI Brainlesion Workshop_. Springer, 2018, pp. 37–48.
  * [49] MICCAI, “Brats 2018 - multimodal brain tumor segmentation,” https://www.med.upenn.edu/sbia/brats2018/data.html, 2018, [Online; accessed December 15, 2020].
  * [50] W. Chen, B. Liu, S. Peng, J. Sun, and X. Qiao, “S3d-unet: separable 3d u-net for brain tumor segmentation,” in _International MICCAI Brainlesion Workshop_. Springer, 2018, pp. 358–368.
  * [51] P. Blanc-Durand, A. Van Der Gucht, N. Schaefer, E. Itti, and J. O. Prior, “Automatic lesion detection and segmentation of 18f-fet pet in gliomas: a full 3d u-net convolutional neural network study,” _PLoS One_ , vol. 13, no. 4, p. e0195798, 2018.
  * [52] X. Zhao, L. Li, W. Lu, and S. Tan, “Tumor co-segmentation in pet/ct using multi-modality fully convolutional neural network,” _Physics in Medicine & Biology_, vol. 64, no. 1, p. 015011, 2018.
  * [53] R. Almajalid, J. Shan, Y. Du, and M. Zhang, “Development of a deep-learning-based method for breast ultrasound image segmentation,” in _2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA)_. IEEE, 2018, pp. 1103–1108.
  * [54] P. Wang, V. M. Patel, and I. Hacihaliloglu, “Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided cnn,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2018, pp. 134–142.
  * [55] H. Piotrzkowska-Wróblewska, K. Dobruch-Sobczak, M. Byra, and A. Nowicki, “Open access database of raw ultrasonic signals acquired from malignant and benign breast lesions,” _Medical physics_ , vol. 44, no. 11, pp. 6105–6109, 2017.
  * [56] M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, and V. K. Asari, “Recurrent residual convolutional neural network based on u-net (r2u-net) for medical image segmentation,” _arXiv preprint arXiv:1802.06955_ , 2018.
  * [57] DRIVE, “Digital retinal images for vessel extraction,” https://drive.grand-challenge.org/, 2017, [Online; accessed January 21, 2021].
  * [58] STARE, “Structured analysis of the retina,” https://cecas.clemson.edu/~ahoover/stare/, 2017, [Online; accessed January 21, 2021].
  * [59] ISIC, “Challenge,” https://challenge.isic-archive.com/landing/2017, 2017, [Online; accessed January 21, 2021].
  * [60] F. Isensee, J. Petersen, A. Klein, D. Zimmerer, P. F. Jaeger, S. Kohl, J. Wasserthal, G. Koehler, T. Norajitra, S. Wirkert _et al._ , “nnu-net: Self-adapting framework for u-net-based medical image segmentation,” _arXiv preprint arXiv:1809.10486_ , 2018.
  * [61] MSD, “Medical segmentation decathlon,” http://medicaldecathlon.com/, 2018, [Online; accessed January 21, 2021].
  * [62] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 2018, pp. 3–11.
  * [63] N. Tajbakhsh, J. Y. Shin, S. R. Gurudu, R. T. Hurst, C. B. Kendall, M. B. Gotway, and J. Liang, “Convolutional neural networks for medical image analysis: Full training or fine tuning?” _IEEE Transactions on Medical Imaging_ , vol. 35, no. 5, pp. 1299–1312, 2016.
  * [64] KDSB, “Data science bowl,” https://www.kaggle.com/c/data-science-bowl-2018, 2018, [Online; accessed January 21, 2021].
  * [65] ISBI, “Lits - liver tumor segmentation challenge,” https://competitions.codalab.org/competitions/17094, 2017, [Online; accessed December 11, 2020].
  * [66] V. Subramanian, H. Wang, J. T. Wu, K. C. Wong, A. Sharma, and T. Syeda-Mahmood, “Automated detection and type classification of central venous catheters in chest x-rays,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 522–530.
  * [67] B. Li, G. Kang, K. Cheng, and N. Zhang, “Attention-guided convolutional neural network for detecting pneumonia on chest x-rays,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 4851–4854.
  * [68] Kaggle, “Rsna pneumonia detection challenge,” https://www.kaggle.com/c/rsna-pneumonia-detection-challenge, 2018, [Online; accessed January 25, 2021].
  * [69] X. Dong, Y. Lei, T. Wang, M. Thomas, L. Tang, W. J. Curran, T. Liu, and X. Yang, “Automatic multiorgan segmentation in thorax ct images using u-net-gan,” _Medical physics_ , vol. 46, no. 5, pp. 2157–2168, 2019.
  * [70] J. Yang, H. Veeraraghavan, S. G. Armato III, K. Farahani, J. S. Kirby, J. Kalpathy-Kramer, W. van Elmpt, A. Dekker, X. Han, X. Feng _et al._ , “Autosegmentation for thoracic radiation treatment planning: A grand challenge at aapm 2017,” _Medical physics_ , vol. 45, no. 10, pp. 4568–4581, 2018.
  * [71] Z. Liu, Y.-Q. Song, V. S. Sheng, L. Wang, R. Jiang, X. Zhang, and D. Yuan, “Liver ct sequence segmentation based with improved u-net and graph cut,” _Expert Systems with Applications_ , vol. 126, pp. 54–63, 2019.
  * [72] Y. Man, Y. Huang, J. Feng, X. Li, and F. Wu, “Deep q learning driven ct pancreas segmentation with geometry-aware u-net,” _IEEE transactions on medical imaging_ , vol. 38, no. 8, pp. 1971–1980, 2019.
  * [73] H. R. Roth, L. Lu, A. Farag, H.-C. Shin, J. Liu, E. B. Turkbey, and R. M. Summers, “Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation,” in _International conference on medical image computing and computer-assisted intervention_. Springer, 2015, pp. 556–564.
  * [74] H. Seo, C. Huang, M. Bassenne, R. Xiao, and L. Xing, “Modified u-net (mu-net) with incorporation of object-dependent high level features for improved liver and liver-tumor segmentation in ct images,” _IEEE transactions on medical imaging_ , vol. 39, no. 5, pp. 1316–1325, 2019.
  * [75] Y. Hiasa, Y. Otake, M. Takao, T. Ogawa, N. Sugano, and Y. Sato, “Automated muscle segmentation from clinical ct using bayesian u-net for personalized musculoskeletal modeling,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 4, pp. 1030–1040, 2019.
  * [76] TCIA, “Sts soft-tissue-sarcoma,” https://wiki.cancerimagingarchive.net/display/Public/Soft-tissue-Sarcoma, 2020, [Online; accessed December 15, 2020].
  * [77] T. Song, F. Meng, A. Rodriguez-Paton, P. Li, P. Zheng, and X. Wang, “U-next: A novel convolution neural network with an aggregation u-net architecture for gallstone segmentation in ct images,” _IEEE Access_ , vol. 7, pp. 166 823–166 832, 2019.
  * [78] L. Rundo, C. Han, Y. Nagano, J. Zhang, R. Hataya, C. Militello, A. Tangherloni, M. S. Nobile, C. Ferretti, D. Besozzi _et al._ , “Use-net: Incorporating squeeze-and-excitation blocks into u-net for prostate zonal segmentation of multi-institutional mri datasets,” _Neurocomputing_ , vol. 365, pp. 31–43, 2019.
  * [79] T. Wang, J. Xiong, X. Xu, M. Jiang, H. Yuan, M. Huang, J. Zhuang, and Y. Shi, “Msu-net: Multiscale statistical u-net for real-time 3d cardiac mri video segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 614–622.
  * [80] MICCAI, “Acdc - automated cardiac diagnosis challenge,” https://www.creatis.insa-lyon.fr/Challenge/acdc/databases.html, 2017, [Online; accessed December 15, 2020].
  * [81] X. Dong, Y. Lei, S. Tian, T. Wang, P. Patel, W. J. Curran, A. B. Jani, T. Liu, and X. Yang, “Synthetic mri-aided multi-organ segmentation on male pelvic ct using cycle consistent deep attention network,” _Radiotherapy and Oncology_ , vol. 141, pp. 192–199, 2019.
  * [82] B. Wang, Y. Lei, S. Tian, T. Wang, Y. Liu, P. Patel, A. B. Jani, H. Mao, W. J. Curran, T. Liu _et al._ , “Deeply supervised 3d fully convolutional networks with group dilated convolution for automatic mri prostate segmentation,” _Medical physics_ , vol. 46, no. 4, pp. 1707–1718, 2019.
  * [83] MICCAI, “Promis - prostate mr image segmentation,” https://promise12.grand-challenge.org/, 2012, [Online; accessed December 15, 2020].
  * [84] Z. Guo, N. Guo, K. Gong, Q. Li _et al._ , “Gross tumor volume segmentation for head and neck cancer radiotherapy using deep dense multi-modality network,” _Physics in Medicine & Biology_, vol. 64, no. 20, p. 205015, 2019\.
  * [85] V. Martin, K.-R. Emily, J. P. Léo, L. Xavier, F. Christophe, K. Nader, N.-T. P. Félix, W. Chang-Shu, and S. Khalil, “Data from head-neck-pet-ct,” _The Cancer Imaging Archive_ , 2017.
  * [86] J. Yang, M. Faraji, and A. Basu, “Robust segmentation of arterial walls in intravascular ultrasound images using dual path u-net,” _Ultrasonics_ , vol. 96, pp. 24–33, 2019.
  * [87] S. Balocco, C. Gatta, F. Ciompi, A. Wahle, P. Radeva, S. Carlier, G. Unal, E. Sanidas, J. Mauri, X. Carillo _et al._ , “Standardized evaluation methodology and reference database for evaluating ivus image segmentation,” _Computerized medical imaging and graphics_ , vol. 38, no. 2, pp. 70–90, 2014\.
  * [88] X. Li, Y. Hong, D. Kong, and X. Zhang, “Automatic segmentation of levator hiatus from ultrasound images using u-net with dense connections,” _Physics in Medicine & Biology_, vol. 64, no. 7, p. 075015, 2019.
  * [89] F. Lin, C. Liu, H. Xie, Z.-J. Zha, and Y. Zhang, “Semantic-embedding and shape-aware u-net for ultrasound eyeball segmentation,” in _2019 IEEE International Conference on Multimedia and Expo (ICME)_. IEEE, 2019, pp. 892–897.
  * [90] R. Azad, M. Asadi-Aghbolaghi, M. Fathy, and S. Escalera, “Bi-directional convlstm u-net with densely connected convolutions,” in _Proceedings of the IEEE International Conference on Computer Vision Workshops_ , 2019, pp. 0–0.
  * [91] ISIC, “Challenge,” https://challenge2018.isic-archive.com/, 2018, [Online; accessed January 21, 2021].
  * [92] Z. Gu, J. Cheng, H. Fu, K. Zhou, H. Hao, Y. Zhao, T. Zhang, S. Gao, and J. Liu, “Ce-net: Context encoder network for 2d medical image segmentation,” _IEEE transactions on medical imaging_ , vol. 38, no. 10, pp. 2281–2292, 2019\.
  * [93] Z. Zhang, F. S. Yin, J. Liu, W. K. Wong, N. M. Tan, B. H. Lee, J. Cheng, and T. Y. Wong, “Origa-light: An online retinal fundus image database for glaucoma analysis and research,” in _2010 Annual International Conference of the IEEE Engineering in Medicine and Biology_. IEEE, 2010, pp. 3065–3068.
  * [94] E. Decencière, X. Zhang, G. Cazuguel, B. Lay, B. Cochener, C. Trone, P. Gain, R. Ordonez, P. Massin, A. Erginay _et al._ , “Feedback on a publicly distributed image database: the messidor database,” _Image Analysis & Stereology_, vol. 33, no. 3, pp. 231–234, 2014.
  * [95] F. Fumero, S. Alayón, J. L. Sanchez, J. Sigut, and M. Gonzalez-Hernandez, “Rim-one: An open retinal image database for optic nerve evaluation,” in _2011 24th international symposium on computer-based medical systems (CBMS)_. IEEE, 2011, pp. 1–6.
  * [96] A. Abedalla, M. Abdullah, M. Al-Ayyoub, and E. Benkhelifa, “2st-unet: 2-stage training model using u-net for pneumothorax segmentation in chest x-rays,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–6.
  * [97] Kaggle, “Siim-acr pneumothorax segmentation,” https://www.kaggle.com/c/siim-acr-pneumothorax-segmentation, 2019, [Online; accessed December 11, 2020].
  * [98] L. Zhang, A. Liu, J. Xiao, and P. Taylor, “Dual encoder fusion u-net (defu-net) for cross-manufacturer chest x-ray segmentation,” 2020.
  * [99] S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and G. Thoma, “Two public chest x-ray datasets for computer-aided screening of pulmonary diseases,” _Quantitative imaging in medicine and surgery_ , vol. 4, no. 6, p. 475, 2014.
  * [100] W. Wang, H. Feng, Q. Bu, L. Cui, Y. Xie, A. Zhang, J. Feng, Z. Zhu, and Z. Chen, “Mdu-net: A convolutional network for clavicle and rib segmentation from a chest radiograph,” _Journal of Healthcare Engineering_ , vol. 2020, 2020.
  * [101] J. Park, J. Yun, N. Kim, B. Park, Y. Cho, H. J. Park, M. Song, M. Lee, and J. B. Seo, “Fully automated lung lobe segmentation in volumetric chest ct with 3d u-net: validation with intra-and extra-datasets,” _Journal of Digital Imaging_ , vol. 33, no. 1, pp. 221–230, 2020.
  * [102] T. Fan, G. Wang, Y. Li, and H. Wang, “Ma-net: A multi-scale attention network for liver and tumor segmentation,” _IEEE Access_ , vol. 8, pp. 179 656–179 665, 2020.
  * [103] S. Dong, J. Zhao, M. Zhang, Z. Shi, J. Deng, Y. Shi, M. Tian, and C. Zhuo, “Deu-net: Deformable u-net for 3d cardiac mri video segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 98–107.
  * [104] N. S. Punn and S. Agarwal, “Multi-modality encoded fusion with 3d inception u-net and decoder model for brain tumor segmentation,” _Multimedia Tools and Applications_ , pp. 1–16, 2020.
  * [105] MICCAI, “Brats 2017 - multimodal brain tumor segmentation,” https://www.med.upenn.edu/sbia/brats2017/data.html, 2017, [Online; accessed December 15, 2020].
  * [106] Y. Lu, J. Lin, S. Chen, H. He, and Y. Cai, “Automatic tumor segmentation by means of deep convolutional u-net with pre-trained encoder in pet images,” _IEEE Access_ , vol. 8, pp. 113 636–113 648, 2020.
  * [107] K. H. Leung, W. Marashdeh, R. Wray, S. Ashrafinia, M. G. Pomper, A. Rahmim, and A. K. Jha, “A physics-guided modular deep-learning based automated framework for tumor segmentation in pet,” _Physics in Medicine & Biology_, 2020.
  * [108] M. Dunnhofer, M. Antico, F. Sasazawa, Y. Takeda, S. Camps, N. Martinel, C. Micheloni, G. Carneiro, and D. Fontanarosa, “Siam-u-net: encoder-decoder siamese network for knee cartilage tracking in ultrasound images,” _Medical Image Analysis_ , vol. 60, p. 101631, 2020.
  * [109] Y. Zhang, Y. Lei, R. L. Qiu, T. Wang, H. Wang, A. B. Jani, W. J. Curran, P. Patel, T. Liu, and X. Yang, “Multi-needle localization with attention u-net in us-guided hdr prostate brachytherapy,” _Medical Physics_ , 2020\.
  * [110] M. Byra, P. Jarosik, A. Szubert, M. Galperin, H. Ojeda-Fournier, L. Olson, M. O’Boyle, C. Comstock, and M. Andre, “Breast mass segmentation in ultrasound with selective kernel u-net convolutional neural network,” _Biomedical Signal Processing and Control_ , vol. 61, p. 102027, 2020.
  * [111] N. S. Punn and S. Agarwal, “Inception u-net architecture for semantic segmentation to identify nuclei in microscopy cell images,” _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , vol. 16, no. 1, pp. 1–15, 2020.
  * [112] N. Ibtehaz and M. S. Rahman, “Multiresunet: Rethinking the u-net architecture for multimodal biomedical image segmentation,” _Neural Networks_ , vol. 121, pp. 74–87, 2020.
  * [113] ISBI, “Segmentation of neuronal structures in em stacks,” http://brainiac2.mit.edu/isbi_challenge/home, 2012, [Online; accessed January 21, 2021].
  * [114] CVC-ClinicDB, “Endsocopy,” https://polyp.grand-challenge.org/CVCClinicDB/, 2015, [Online; accessed January 21, 2021].
  * [115] E. Bercovich and M. C. Javitt, “Medical imaging: from roentgen to the digital revolution, and beyond,” _Rambam Maimonides medical journal_ , vol. 9, no. 4, 2018.
  * [116] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [117] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [118] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2980–2988.
  * [119] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [120] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [121] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1–9.
  * [122] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” 2016.
  * [123] K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” in _European conference on computer vision_. Springer, 2016, pp. 630–645.
  * [124] T. Shen, T. Zhou, G. Long, J. Jiang, S. Pan, and C. Zhang, “Disan: Directional self-attention network for rnn/cnn-free language understanding,” _arXiv preprint arXiv:1709.04696_ , 2017.
  * [125] M. Raghu, K. Blumer, R. Sayres, Z. Obermeyer, B. Kleinberg, S. Mullainathan, and J. Kleinberg, “Direct uncertainty prediction for medical second opinions,” in _International Conference on Machine Learning_ , 2019, pp. 5281–5290.
  * [126] C. F. Baumgartner, K. C. Tezcan, K. Chaitanya, A. M. Hötker, U. J. Muehlematter, K. Schawkat, A. S. Becker, O. Donati, and E. Konukoglu, “Phiseg: Capturing uncertainty in medical image segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2019, pp. 119–127.
  * [127] R. Tanno, A. Saeedi, S. Sankaranarayanan, D. C. Alexander, and N. Silberman, “Learning from noisy labels by regularized estimation of annotator confusion,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 244–11 253.
  * [128] J. K. Leader, B. Zheng, R. M. Rogers, F. C. Sciurba, A. Perez, B. E. Chapman, S. Patel, C. R. Fuhrman, and D. Gur, “Automated lung segmentation in x-ray computed tomography: development and evaluation of a heuristic threshold-based scheme,” _Academic radiology_ , vol. 10, no. 11, pp. 1224–1236, 2003.
  * [129] Y. Boykov and G. Funka-Lea, “Graph cuts and efficient nd image segmentation,” _International journal of computer vision_ , vol. 70, no. 2, pp. 109–131, 2006.
  * [130] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [131] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei, “Deformable convolutional networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 764–773.
  * [132] Y. Han and J. C. Ye, “Framing u-net via deep convolutional framelets: Application to sparse-view ct,” _IEEE transactions on medical imaging_ , vol. 37, no. 6, pp. 1418–1429, 2018.
  * [133] A. Kendall, V. Badrinarayanan, and R. Cipolla, “Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding,” _arXiv preprint arXiv:1511.02680_ , 2015.
  * [134] K. He, X. Zhang, S. Ren, and J. Sun, “Spatial pyramid pooling in deep convolutional networks for visual recognition,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 9, pp. 1904–1916, 2015\.
  * [135] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 2018, pp. 3–11.
  * [136] S. A. Morris and T. C. Slesnick, “Magnetic resonance imaging,” _Visual Guide to Neonatal Cardiology_ , pp. 104–108, 2018.
  * [137] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [138] H. Hwang, H. Z. U. Rehman, and S. Lee, “3d u-net for skull stripping in brain mri,” _Applied Sciences_ , vol. 9, no. 3, p. 569, 2019.
  * [139] A. P. James and B. V. Dasarathy, “Medical image fusion: A survey of the state of the art,” _Information fusion_ , vol. 19, pp. 4–19, 2014.
  * [140] R. Ayachi, M. Afif, Y. Said, and M. Atri, “Strided convolution instead of max pooling for memory efficiency of convolutional neural networks,” in _International conference on the Sciences of Electronics, Technologies of Information and Telecommunications_. Springer, 2018, pp. 234–243.
  * [141] S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 305–321.
  * [142] T. Wang, J. Xiong, X. Xu, and Y. Shi, “Scnn: A general distribution based statistical convolutional neural network with application to video object detection,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 5321–5328.
  * [143] C. Zotti, Z. Luo, A. Lalande, and P.-M. Jodoin, “Convolutional neural network with shape prior applied to cardiac mri segmentation,” _IEEE journal of biomedical and health informatics_ , vol. 23, no. 3, pp. 1119–1128, 2018.
  * [144] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2223–2232.
  * [145] J. M. Ollinger and J. A. Fessler, “Positron-emission tomography,” _IEEE Signal Processing Magazine_ , vol. 14, no. 1, pp. 43–55, 1997.
  * [146] Y. Wang, B. Yu, L. Wang, C. Zu, D. S. Lalush, W. Lin, X. Wu, J. Zhou, D. Shen, and L. Zhou, “3d conditional generative adversarial networks for high-quality pet image estimation at low dose,” _Neuroimage_ , vol. 174, pp. 550–562, 2018.
  * [147] M. Frid-Adar, I. Diamant, E. Klang, M. Amitai, J. Goldberger, and H. Greenspan, “Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification,” _Neurocomputing_ , vol. 321, pp. 321–331, 2018.
  * [148] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _2016 fourth international conference on 3D vision (3DV)_. IEEE, 2016, pp. 565–571.
  * [149] C. L. Moore and J. A. Copel, “Point-of-care ultrasonography,” _New England Journal of Medicine_ , vol. 364, no. 8, pp. 749–757, 2011.
  * [150] Y. Yu and S. T. Acton, “Speckle reducing anisotropic diffusion,” _IEEE Transactions on image processing_ , vol. 11, no. 11, pp. 1260–1270, 2002.
  * [151] X. Li, C. Li, A. Fedorov, T. Kapur, and X. Yang, “Segmentation of prostate from ultrasound images using level sets on active band and intensity variation across edges,” _Medical physics_ , vol. 43, no. 6Part1, pp. 3090–3103, 2016.
  * [152] A. Gomariz, W. Li, E. Ozkan, C. Tanner, and O. Goksel, “Siamese networks with location prior for landmark tracking in liver ultrasound sequences,” in _2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019)_. IEEE, 2019, pp. 1757–1760.
  * [153] X. Li, W. Wang, X. Hu, and J. Yang, “Selective kernel networks,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * [154] T. Mikolov, S. Kombrink, L. Burget, J. Černockỳ, and S. Khudanpur, “Extensions of recurrent neural network language model,” in _2011 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2011, pp. 5528–5531.
  * [155] H. Song, W. Wang, S. Zhao, J. Shen, and K.-M. Lam, “Pyramid dilated deeper convlstm for video salient object detection,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 715–731.
  * [156] H. Zhang and J. Ma, “Hartley spectral pooling for deep learning,” _arXiv preprint arXiv:1810.04028_ , 2018.
  * [157] G. repository., “2019 novel coronavirus covid-19 (2019-ncov) data repository by johns hopkins csse,” https://github.com/CSSEGISandData/COVID-19, 2020, [Online; accessed April 3, 2021].
  * [158] S. Agarwal, N. S. Punn, S. K. Sonbhadra, P. Nagabhushan, K. Pandian, and P. Saxena, “Unleashing the power of disruptive and emerging technologies amid covid 2019: A detailed review,” _arXiv preprint arXiv:2005.11507_ , 2020\.
  * [159] F. Shi, J. Wang, J. Shi, Z. Wu, Q. Wang, Z. Tang, K. He, Y. Shi, and D. Shen, “Review of artificial intelligence techniques in imaging data acquisition, segmentation and diagnosis for covid-19,” _IEEE reviews in biomedical engineering_ , 2020.
  * [160] F. Huazhu, F. Deng-Ping, C. Geng, and Z. Tao, “Covid-19 imaging-based ai research collection,” https://git.io/JYAtL, 2020, [Online; accessed January 11, 2021].
  * [161] N. S. Punn, S. K. Sonbhadra, and S. Agarwal, “Covid-19 epidemic analysis using machine learning and deep learning algorithms,” _medRxiv_ , 2020.
  * [162] Y.-H. Wu, S.-H. Gao, J. Mei, J. Xu, D.-P. Fan, C.-W. Zhao, and M.-M. Cheng, “Jcs: An explainable covid-19 diagnosis system by joint classification and segmentation,” _arXiv preprint arXiv:2004.07054_ , 2020.
  * [163] Q. Yan, B. Wang, D. Gong, C. Luo, W. Zhao, J. Shen, Q. Shi, S. Jin, L. Zhang, and Z. You, “Covid-19 chest ct image segmentation–a deep convolutional neural network solution,” _arXiv preprint arXiv:2004.10987_ , 2020.
  * [164] D.-P. Fan, T. Zhou, G.-P. Ji, Y. Zhou, G. Chen, H. Fu, J. Shen, and L. Shao, “Inf-net: Automatic covid-19 lung infection segmentation from ct images,” _IEEE Transactions on Medical Imaging_ , 2020.
  * [165] N. S. Punn and S. Agarwal, “Chs-net: A deep learning approach for hierarchical segmentation of covid-19 infected ct images,” _arXiv preprint arXiv:2012.07079_ , 2020.
  * [166] N. Punn and S. Agarwal, “Automated diagnosis of covid-19 with limited posteroanterior chest x-ray images using fine-tuned deep neural networks,” _Applied Intelligence_ , 2020.
  * [167] M. Zahangir Alom, M. Shaifur Rahman, M. Shamima Nasrin, T. M. Taha, and V. K. Asari, “Covid_mtnet: Covid-19 detection with multi-task deep learning approaches,” _arXiv_ , pp. arXiv–2004, 2020.
  * [168] N. S. Punn, S. K. Sonbhadra, and S. Agarwal, “Monitoring covid-19 social distancing with person detection and tracking via fine-tuned yolo v3 and deepsort techniques,” _arXiv preprint arXiv:2005.01385_ , 2020.
  * [169] G. J. Chowdary, N. S. Punn, S. K. Sonbhadra, and S. Agarwal, “Face mask detection using transfer learning of inceptionv3,” _arXiv preprint arXiv:2009.08369_ , 2020.
  * [170] M. H. Hesamian, W. Jia, X. He, and P. Kennedy, “Deep learning techniques for medical image segmentation: Achievements and challenges,” _Journal of digital imaging_ , vol. 32, no. 4, pp. 582–596, 2019.
  * [171] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [172] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [173] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [174] Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and acceleration for deep neural networks,” _arXiv preprint arXiv:1710.09282_ , 2017.
  * [175] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” _arXiv preprint arXiv:1905.11946_ , 2019.
  * [176] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmentation for deep learning,” _Journal of Big Data_ , vol. 6, no. 1, p. 60, 2019.