# FANet: A Feedback Attention Network for Improved Biomedical Image
Segmentation

Nikhil Kumar Tomar, Debesh Jha, , Michael A. Riegler, , Håvard D. Johansen, ,
Dag Johansen, , Jens Rittscher, , Pål Halvorsen, , and Sharib Ali N. K. Tomar
was with SimulaMet, Oslo, Norway and now with NAAMII, Kathmandu, Nepal.D. Jha
is with SimulaMet, Oslo, Norway and UiT The Arctic University of Norway,
Tromsø, Norway (corresponding email: debesh@simula.no).M. A. Riegler is with
SimulaMet, Oslo, Norway.H.D. Johansen and D. Johansen are with UiT The Arctic
University of Norway, Tromsø, Norway.J. Rittscher is with the Department of
Engineering Science, University of Oxford, Oxford, UK.P. Halvorsen is with
SimulaMet and Oslo Metropolitan University, Oslo, Norway.S. Ali is with the
Department of Engineering Science, University of Oxford, and Oxford NIHR
Biomedical Research Centre, Oxford, UK (corresponding email:
sharib.ali@eng.ox.ac.uk).

###### Abstract

The increase of available large clinical and experimental datasets has
contributed to a substantial amount of important contributions in the area of
biomedical image analysis. Image segmentation, which is crucial for any
quantitative analysis, has especially attracted attention. Recent hardware
advancement has led to the success of deep learning approaches. However,
although deep learning models are being trained on large datasets, existing
methods do not use the information from different learning epochs effectively.
In this work, we leverage the information of each training epoch to prune the
prediction maps of the subsequent epochs. We propose a novel architecture
called feedback attention network (FANet) that unifies the previous epoch mask
with the feature map of the current training epoch. The previous epoch mask is
then used to provide a hard attention to the learned feature maps at different
convolutional layers. The network also allows to rectify the predictions in an
iterative fashion during the test time. We show that our proposed feedback
attention model provides a substantial improvement on most segmentation
metrics tested on seven publicly available biomedical imaging datasets
demonstrating the effectiveness of FANet.

###### Index Terms:

Medical image segmentation, deep learning, feedback attention, colon polyps,
skin lesion, retinal vessels, cell nuclei, lung segmentation

##  I Introduction

IMAGE segmentation is one of the most studied problems in computer vision,
where the main goal is to classify each pixel of an image to a specific class
instance. This can either be a pixel of arbitrary objects such as cars or
humans and pixels of healthy and sick medical findings. Substantial progresses
has been made in imaging modalities such as X-ray, Computerized Tomography
(CT), Magnetic Resonance Imaging (MRI), endoscopy imaging, fundus imaging,
Electron Microscopy (EM), and histology imaging. While Machine Learning (ML)
methods provide improved performance over the traditional computer vision
methods, most of them require ground truth labels from domain experts, which
comes with some challenges. For example, due to the poor image quality of most
biomedical imaging data, expert interpretations can be affected. Moreover,
complex and irregular shapes of the object-of-interest in these imaging data,
such as cell nuclei in EM, can pose challenges for model training. Although it
is difficult to address these biases and provide sample-specific priors,
designing a model that is capable of pruning and self-rectification for each
sample during training can help to improve model performance.

Current developments of Convolutional Neural Networks, Recurrent Neural
Networks, and attention modules have improved automated methods in biomedical
image analysis. While CNNs help to overcome the limitations of handcrafted
features by learning features in a supervised end-to-end manner, they require
the estimation of millions of parameters. As a result, a large and diverse
training dataset is required to avoid overfitting. RNNs can be used to
preserve the model compactness and can be effectively used in resource-
constrained settings [1]. However, RNNs are known for their memory inefficient
memory-bandwidth-bound computation and model complexity as they hold their
internal state to process the data [2]. Additionally, in many applications,
applying visual attention mechanism in deep learning has shown promising
results and improvements in terms of model usability [3, 4]. Attention allows
networks to focus on a concrete class instance, thereby penalizing non-
specific regions. Therefore, it is required to adhere with the knowledge
transfer that integrates the past information, similar to recurrent networks,
and use them to provide stronger acceptance of learned features belonging to
the region-of-interest as an attention mechanism [5, 6].

While most deep learning-based segmentation methods are trained on large
datasets to avoid overfitting, existing methods do not incorporate any sample-
specific information learned in the previous training epochs. In current deep
learning approaches, only aggregated average weights are used for information
propagation through gradients. A mask-guided contrastive attention model was
used by Song et al. [7] to deal with the background clutter. Unlike classical
training mechanisms and motivated by the work of Song et al. [7], we propose
to propagate the sample-specific mask output from the previous epoch to the
successive epoch in a recursive fashion. Such a feedback mechanism can provide
prior information that can help to learn sample variability, thereby enabling
to train effectively on diverse and small or large datasets. Here, iterative
prediction can be used to prune the predicted masks during the inference (see
Figure 1). This allows to both locally and globally learn to rectify the mask
output from the learnt weights. Unlike Test-Time Augmentation (TTA) [8], where
different transforms are utilized to mimic the sample representations that are
better learnt by the network, we interactively rectify masks during training.
To our knowledge, Feedback Attention Network (FANet) is the first deep
learning model that has the ability to self-rectify its predictions without
requiring transformations, ensemble strategies, and prior sample-specific
knowledge. We use a single end-to-end trainable network that allows
information propagation during both train and test time.

Including a feedback mechanism as a part of training is central to our novel
FANet approach for semantic segmentation. The predicted map of each sample
from the previous epoch is used to provide attention unified with the current
state feature map. We call this architecture, FANet, as it uses an attention
mechanism to different feature scales in the network, allowing it to capture
variability in image samples. Additionally, our residual block with Squeeze
and Excitation (SE) layer allows us to improve channel interdependencies,
which can be critical to tackle image quality issues.

Figure 1: Semantic segmentation using our FANet architecture. Otsu
thresholding is used for generating the initial mask used during $0^{th}$
iteration. Then the predictions are iteratively updated with the predicted
mask. It can be observed that already at the 2nd iteration, the results
converge. The corresponding feature maps before and after feedback attention
at the last decoder layer of our FANet are shown as color images on the right.

The main contributions made in this work can be summarized as follows:

  1. 1.

Feedback attention learning —  A novel mechanism to utilize the variability
present in each training sample. The mask outputs are propagated from one to
subsequent epochs to repress the unwanted feature clutter.

  2. 2.

Iterative refining of prediction masks —  Using feedback information helps in
refining the predicted masks in training as well as inference. During testing,
we iterate over the input image and keep updating the input mask with the
predicted mask.

  3. 3.

Embedded run-length encoding strategy —  Binary mask outputs of each samples
are efficiently compressed before being propagated to the next epoch. This
provides a memory efficient mechanism for passing sample specific masks.

  4. 4.

Systematic evaluation —  Experiments on seven vastly different biomedical
datasets suggest that FANet outperforms other state-of-the-art (SOTA)
algorithms.

  5. 5.

Efficient training —  FANet achieves near SOTA performance with far fewer
training epochs.

##  II Related work

In this section, we summarize the most relevant advances in medical image
segmentation, highlight advances in feedback attention networks, and summarise
the recent contributions to iterative refinement methods for image
segmentation.

###  II-A Biomedical image segmentation

The basis of most modern CNN-based semantic segmentation architectures are
either Fully Convolutional Network (FCN) [9] or an encoder-decoder
architecture such as U-Net [10] originally designed for cell segmentation
[10]. Various modifications of these networks have been proposed both for
semantic segmentation of natural images [11, 12, 13] and biomedical image
segmentation [14, 15, 16, 17, 18, 19, 20, 21, 22]. In general, in the encoder,
the image content is encoded using multiple convolutions to capture from low-
level to high-level features, whereas in the decoder part of the network the
prediction masks are obtained by multiple upsampling mechanism and
deconvolution. Methods like PSPNet [12] and DeepLab [23] incorporate
convolutional feature maps of varying resolutions to effectively segment both
small and large sized objects. While PSPNet used a pyramid pooling module,
DeepLab used Atrous Spatial Pyramidal Pooling (ASPP) for encoding the multi-
scale contextual information. Both PSPNet and DeepLab based architectures have
been used widely in the medical imaging community [24, 25, 17].

Recently, Cheng et al. [26] proposed the Panoptic-DeepLap method with a class-
agnostic regression, for instance, segmentation that uses a dual-ASPP and
dual-decoder structure. Wang et al. [27] proposed an attention-based panoptic
instance segmentation showing improved performance than the former. High-
Resolution Network (HRNet) [13] employed exchange of information across
different resolutions to resolve the issue of loss of high-resolution feature
representations. Application of these architectures in the biomedical imaging
community is, however, yet to be explored.

###  II-B Feedback attention networks

Visual attention has been widely used in computer vision for object detection
[28], image segmentation [29, 30] and pose estimation [31]. Attention
mechanisms have also been utilized for posing explicit focus on the target
region in medical imaging [6, 32]. Attention U-Net [21] used a gated operation
in the U-Net architecture to focus on the target abdominal regions of CT
datasets. Feedback mechanism for attention using two U-Net architectures with
shared weights was used for cell segmentation [33, 34]. The latter used a
standard U-Net architecture with the second U-Net incorporating ConvLSTM [35]
to store the feature map (input-to-state) from the first U-Net network.
However, feedback is only applied to the same epoch with state-to-state
transitions. On the contrary, our approach utilizes a feedback mechanism that
propagates information flow from the previous epoch to the current epoch in an
attention mechanism setting. We employ the predicted masks from the previous
epoch as hard attention to prune the segmentation output.

###  II-C Iterative refinement for segmentation

Inherent variation in object shape and changes in scale pose significant
challenges in biomedical imaging. An iterative refinement of the segmentation
mask by feeding the input image and the predicted segmentation mask to a
modified U-Net architecture was done by Mosinska et al. [36]. Similarly,
iterative update of latent space and minimization of the Structure Similarity
Index Measure (SSIM) loss was used to refine the predicted segmentation maps
during test time in [37].

Recently, iterative refinement strategies have also been used for pose
estimation [38, 39, 40] that used consecutive modules for refinement of the
predictions with a loss function for the evaluation of output in each module.
These iterative refinement processes show improved predictions and are able to
handle domain shifts or object shape variability without requiring very deep
networks [37]. A major bottleneck in these methods is the number of iterations
required for convergence. Unlike these methods, our proposed FANet provides
attention to the specific region-of-interest and can prune the segmentation
predictions in less than ten iterations without requiring any optimization
scheme, i.e., simple weight updates are used.

##  III Method

This section introduces the main components required to build the proposed
architecture. We present the overall architecture used along with the proposed
feedback attention learning mechanism (see Figure 2).

Figure 2: FANet with squeeze and excite residual (SE-Residual) block and MixPool block. (a) SE-Residual block integrated with squeeze and excite layer uses $1\times 1$ convolution to concatenate the high-resolution feature representation with the the encoded feature vector. (b) MixPool block represents attention mechanism in our network. The input mask is downscaled to the corresponding layer feature map size $M_{l}$ which is fused with the masked feature map representation $M_{l}^{{}^{\prime}}$ for hard attention of input feature in that layer $F_{l}$. Finally, the attenuated feature map $F_{l}^{M^{{}^{\prime}}}$ and the feature maps $F_{l}^{{{}^{\prime}}}$ are both concatenated. c) Proposed FANet showing the complete network architecture. Encoder-decoder architecture with skip-connections (in dotted arrows) from SE-Residual blocks to preserve high- and intermediate resolution feature representations and MixPool block connections (with solid arrows) that allow to feedback the previous mask predictions. TABLE I: Details of the biomedical datasets used in our experiments. “Train”, “train after augmentation (aug.)” and “test” denote the number of training image sample, number of training sample after image augmentation, and number of test sample used in the study. Dataset | Images | Size | Train | Train after aug. | Test | Application
---|---|---|---|---|---|---
Kvasir-SEG [41] | 1000 | Variable | 880 | 16720 | 120 | Colonoscopy
CVC-ClinicDB [42] | 612 | $384\times 288$ | 490 | 14210 | 61 | Colonoscopy
2018 Data Science Bowl [43] | 670 | $256\times 256$ | 335 | 10720 | 134 | Nuclie
ISIC 2018 (Lesion Boundary Segmentation) [44, 45] | 2596 | Variable | 1815 | 39930 | 259 | Dermoscopy
EM dataset [46] | 30 | $512\times 512$ | 24 | 384 | 3 | Cell
DRIVE Database [47] | 40 | $584\times 565$ | 20 | 640 | 20 | Retina Vessel
CHASE-DB1 [48] | 28 | $584\times 565$ | 20 | 640 | 8 | Retina Vessel

###  III-A SE-Residual block

Deeper networks improve the performance of the model significantly, but,
increase in depth can cause either a vanishing or exploding gradients problem
[49]. To deal with this, we take advantage of shortcut connections between
layers in the residual learning paradigm. Our SE-Residual block uses two
3$\times$3 convolutions and an identity mapping, where each convolution layer
is followed by a Batch Normalization (BN) layer and a Rectified Linear Unit
(ReLU) non-linear activation function. The identity mapping is used to connect
the input and the output of the convolution layer (Figure 2 a).

Similar to the work by Hu et al. [50], we add a SE layer in the residual
network. The SE layer acts as a content-aware mechanism that re-weights each
channel accordingly to create robust representations. Hence, it allows the
network to become more sensitive to significant features while suppressing
irrelevant features. This goal is accomplished in two steps. First, the
feature maps are squeezed by using the global average pooling to get a global
understanding of each channel. The squeeze operation results in a feature
vector of size $\mathit{n}$, where $\mathit{n}$ refers to the number of
channels. In the second step: excitation, this feature vector is feed through
a two-layered feed-forward neural network, where the number of features is
first reduced and then expanded to the original size $\mathit{n}$. Now, this
$\mathit{n}$ sized vector represents the weight of the original feature maps,
which is used to scale each channel.

###  III-B MixPool block

The proposed MixPool block shown in Figure 2 (b) is used in multiple layers of
our FANet architecture. This block facilitates the flow of sample-wise
feedback information between consecutive epochs providing a hard attention to
the learnt features from SE-Residual block. The layer provides focus to the
relevant features in both contraction path and expansion path layers. The
‘hard’ attention map consists of the values 0 and 1, i.e., attention to a
specific region only unlike soft attention where the probability map is
estimated. The advantage of hard attention it that it allows to keep only the
important features and ignore irrelevant features. During the element-wise
multiplication, the values from the input feature map, if multiplied by 0,
becomes 0, leaving the essential features for further operations. Another
advantage of such methods is their computational speed, scalability, and ease
of interpretation [3, 51]. The input mask used during training is compressed
using the run-length encoding (RLE) technique to save the memory footprint.

As in Figure 2 (b-c), fist feature maps from the SE-Residual blocks $F_{l}$ in
each layer is passed through a $3\times 3$ convolution followed by a BN and a
ReLU activation function. Then, we apply a $1\times 1$ convolution and a
sigmoid activation function $\sigma(\cdot)$ with a threshold of 0.5 to obtain
the binary mask $M_{l}^{{}^{\prime}}$ to contribute to the spatial attention
map generation given by:

$M_{l}^{{}^{\prime}}=\sigma\left(\mathit{conv}(F_{l})\right)=\begin{cases}1,&\text{if
}\sigma(\cdot)\geq 0.5\\\ 0,&\text{otherwise}.\end{cases}\\\ $ (1)

Secondly, we apply appropriate max-pooling on the input mask (from the
previous epoch) and resize it to the size of the spatial attention map
$M_{l}^{{}^{\prime}}$. A union operation is then applied between the resized
mask and the spatial attention map. This confirms that we obtain the feature
from both the feedback and the spatial attention maps to further create a new
unified spatial attention map. Next, an element-wise multiplication operation
is applied between the unified mask and the original feature map that
suppresses the irrelevant features and enhances the important ones. The
enhanced and the original feature maps are then followed by a $3\times 3$
convolution, BN, and a ReLU. These operations are used to improve the
network’s ability to learn non-linearity in the model prediction.

Finally, we concatenate the output of both activation functions, which
constitutes the output of our MixPool block given by:

$\displaystyle\text{Output}_{\text{MixPool}}=F_{l}^{{}^{\prime}\frown}\left(F_{l}\otimes(M_{l}\cup
M_{l}^{{}^{\prime}})\right)^{{}^{\prime}},$ (2)

where $\frown$ denotes the concatenation operator, $\otimes$ is element-wise
multiplication and $\cup$ represents the union operation.

###  III-C Proposed FANet architecture

The block diagram of FANet is illustrated in Figure 2 (c). It uses an encoder-
decoder design common to many semantic segmentation architectures. We combine
the strength of a residual network enhanced with SE as SE-Residual block,
MixPool block that facilitates the attention and propagation of information
flow from the current learning paradigm and that of the previous epoch. We
implement a recurrent learning mechanism in both encoder and decoder layers
that allows to achieve efficient segmentation. The MixPool block uses the
previous segmentation map (as an input mask through RLE encoding), which
contains the information from prior training and uses it to improve the
semantic representation of the feature maps.

We first use the Otsu thresholding [52] to generate an initial input mask for
training the proposed architectural model. The variability in the input mask
is refined over the training epochs and the model learns over time to prune
input or previous epoch masks with learnt semantically meaningful features
together. To achieve this, we use the novel MixPool block that uses the input
mask and applies hard attention over the subsequent input feature maps. The
hard attention enables the network to highlight semantically meaningful
features for the target region-of-interest in the entire network. The network
thus not only learns to predict features maps but also strengthens a joint
pruning mechanism that is dependent on the input mask. As a result, the
devised network is able to rectify the predicted segmentation maps in an
iterative fashion unlike conventional methods which do not have such pruning
ability. This provides a strong rational behind our work that is applicable
beyond single step inference prediction with capability of refining prediction
maps.

The proposed network architecture is a Fully Convolutional Neural Network
(FCNN) consisting of four encoder and four decoder blocks. The encoder takes
the input image, downsamples it gradually, and encodes it in a compact
representation. Then, the decoder takes this compact representation and tries
to reconstruct the semantic representation by gradually upsampling it and
combining the features from the encoder. Finally, we receive a pixel-wise
categorization of the input image. Both the encoder and the decoder are built
using the SE-Residual block, and an additional concatenation of the original
resolution feature representation in the encoder is added at each resolution
scale. This mechanism minimizes the loss of feature representations during
downscaling and upscaling processes.

Each encoder network starts with two SE-Residual blocks, which consist of two
$3\times 3$ convolutions and a shortcut connection known as identity mapping,
connecting the input and output of the two convolution layers. Each
convolution is followed by a BN and a ReLU activation function. The output of
the second SE-Residual block acts as skip connection for the corresponding
decoder block. After that, it is followed by the MixPool block, which has the
previous epoch segmentation mask and provides a hard-attention over the
incoming feature maps. This process is repeated for each of the downscaled
layers.

Each decoder network starts with a $4\times 4$ transpose convolution that
doubles the spatial dimensions of the incoming feature maps. These feature
maps are concatenated with feature maps from the corresponding encoder block
through skip connections. The skip connections help to propagate the
information from the upper layers, which are sometimes lost due to the depth
of the network. The skip connections are followed by two SE-Residual blocks,
which help to eliminate the problem of vanishing gradient. The MixPool block
that utilizes the segmentation mask from the previous epoch is then applied
creating a hard-attention over the learnt feature maps. Next, we concatenate
the feature maps from the last decoder block and the segmentation mask from
the previous epoch. Finally, we apply a $1\times 1$ convolution with the
sigmoid activation function. The output of this is used to both minimize the
training loss, using a combined binary cross-entropy and dice loss, and to
generate segmentation masks that are stored as a run-length encoded (RLE)
compression for each sample and propagated during the next epoch. The RLE is
updated after each epoch. Similarly, the network learns to adapt the weights
in iterative training, this mechanism is also utilized during the test time.
As shown in Figure 1, test results are pruned in a few iterations during the
test time. Unlike many methods in literature [36, 37], we utilize the same
network without any complementary loss function optimization.

##  IV Experiments

###  IV-A Setup

####  IV-A1 Dataset and Evaluation Metrics

To evaluate the proposed architecture, we have selected seven datasets that
capture different segmentation tasks in biomedical imaging. The details of
each dataset can be found in Table I. The dataset images contain the images of
organs and lesions acquired under different imaging protocols. For the retina
vessel segmentation task, we use DRIVE and CHASE-DB1 datasets. These two
datasets are aimed at various diseases related to diseases of retina vessels,
such as hypertensive retinopathy, retinal vein occlusion, and retinal artery
occlusion. The ISIC 2018 dataset, which is a dermoscopy dataset that is useful
in the diagnosis of skin cancer, is the third dataset focused on medical
imaging data. This dataset contains a wide variety of skin cancer images of
different sizes and shapes, which helps in a better understanding of the
disease. We have further included Kvasir-SEG and CVC-ClinicDB colonoscopy
datasets. These datasets contain the image frames extracted from different
colonoscopy interventions and are focused on colorectal polyps that are one of
the cancer precursors in the colon and rectum. It highly increases the chance
of avoiding lethal cancer by early detection. In addition, we have included
two datasets acquired from biological imaging aimed at understanding of the
cellular processes. These include the 2018 Data Science Bowl and the EM
datasets. The 2018 Data Science Bowl dataset contains images with a large
number of variable shaped nuclei acquired from different cell types,
magnification, and imaging modalities. This dataset is designed for automated
nuclei segmentation. Similarly, the EM dataset contains the transmission
electron microscopy images of the neural structures of the Drosophila nerve
cord. This dataset is aimed at the automated segmentation of the neural
structures. All experiments on these datasets are conducted on the same train,
validation and test splits as provided by the previously published works
reported in this paper.

To evaluate SOTA deep learning methods and our proposed FANet, we have used
standard evaluation metrics that includes Dice Coefficient (DSC) (a.k.a. F1),
mean Intersection over Union (mIoU), precision, and recall. We have
additionally calculated specificity for those datasets where this metric was
previously used for benchmarking.

####  IV-A2 Implementation details

All the training are performed on a Volta 100 GPU and an NVIDIA DGX-2 system
using the PyTorch 1.6. framework. For test inference we have used NVIDIA GTX
1050 Ti GPU for our method and all SOTA methods used in the paper as they are
widely available. Our model is trained for 100 epochs (empirically set) using
an Adam optimizer with a learning rate of $1e^{-4}$ for all the experiments
except for the Digital Retinal Images for Vessel Extraction (DRIVE) and the
CHASE-DB1 dataset where the learning rate was adjusted to $1e^{-3}$ due to the
small size of the training dataset. Datasets were chosen such that the
efficiency of our model could be compared to the SOTA methods. A combination
of binary cross-entropy and dice loss has been used as the loss function.
ReduceLROnPlateau callback was used to monitor the learning rate and adjust it
to obtain optimal training performance. All the images used in the study were
resized to $512\times 512$ except for the 2018 Data Science Bowl and the CVC-
ClinicDB dataset, where images were resized to $256\times 256$. Data
augmentation, such as random crop, flipping, rotation, elastic transformation,
grid distortion, optical distortion, grayscale conversion, random brightness,
contrast, channel, and course dropout were used.

####  IV-A3 Ablation study

In order to evaluate the strength of our proposed FANet architecture, we
perform a thorough ablation study. For this, we have used all seven datasets
and evaluated on several metrics for baseline (FANet without MixPool),
baseline with MixPool, and the combination of baseline, MixPool, and feedback
(proposed).

###  IV-B Results

Below we present quantitative results on seven different biomedical imaging
datasets and compare with corresponding SOTA methods.

####  IV-B1 Results on Kvasir-SEG

TABLE II: Results on the Kvasir-SEG [41] Method | Backbone | F1 | mIoU | Recall | Prec.
---|---|---|---|---|---
U-Net [10] | - | 0.5969 | 0.4713 | 0.6171 | 0.6722
ResUNet [53] | - | 0.6902 | 0.5721 | 0.7248 | 0.7454
ResUNet++ [17] | - | 0.7143 | 0.6126 | 0.7419 | 0.7836
FCN8 [9] | VGG 16 | 0.8310 | 0.7365 | 0.8346 | 0.8817
HRNet [13] | - | 0.8446 | 0.7592 | 0.8588 | 0.8778
DoubleU-Net [18] | VGG 19 | 0.8129 | 0.7332 | 0.8402 | 0.8611
PSPNet [12] | ResNet50 | 0.8406 | 0.7444 | 0.8357 | 0.8901
DeepLabv3+ [54] | MobileNet | 0.8425 | 0.7575 | 0.8377 | 0.9014
DeepLabv3+ [54] | ResNet50 | 0.8572 | 0.7759 | 0.8616 | 0.8907
DeepLabv3+ [54] | ResNet101 | 0.8643 | 0.7862 | 0.8592 | 0.9064
U-Net [10] | VGG19 | 0.7535 | 0.6571 | 0.7364 | 0.8565
U-Net++ [55] | - | 0.8002 | 0.7000 | 0.8716 | 0.7992
$R{2,2}$Attention U-Net [21] | - | 0.7944 | 0.6959 | 0.8383 | 0.8287
FANet | - | 0.8803 | 0.8153 | 0.9058 | 0.9005

Kvasir-SEG [41] is a publicly available polyp segmentation dataset acquired
from clinical colonoscopy (endoscopy) procedures. This dataset has been widely
used for algorithm benchmarking. We have trained our model and compared it
with recent SOTA methods on Kvasir-SEG. A comparison with widely accepted
segmentation methods with different backbones (see Table II) shows that our
approach is improved performance compared to the SOTA methods (on the same
train-test split). Our FANet outperforms all the SOTA methods on almost all
metrics. While outperforming most U-Net and its variants, it can be observed
that FANet achieved an F1 score of 0.8803, which is 1.6% and 3.57% better than
the most accurate DeepLabv3+ with ResNet101 backbone and the recent HRNet.

####  IV-B2 Results on CVC-ClinicDB dataset

TABLE III: Results on the CVC-ClinicDB [42] Method | F1 | mIoU | Recall | Precision
---|---|---|---|---
U-Net (MICCAI’15) [10] | 0.8230 | 0.7550 | - | -
ResUNet-mod [53] | 0.7788 | 0.4545 | 0.6683 | 0.8877
ResUNet++ [17] | 0.7955 | 0.7962 | 0.7022 | 0.8785
SFA (MICCAI’19) [56] | 0.7000 | 0.6070 | - | -
PraNet [14] | 0.8990 | 0.8490 | - | -
U-Net++ [55] | 0.9377 | 0.8890 | 0.9405 | 0.9432
Attention U-Net [21] | 0.9325 | 0.8856 | 0.9276 | 0.9546
FANet | 0.9355 | 0.8937 | 0.9339 | 0.9401

CVC-ClinicDB is another commonly used dataset for colonoscopy image analysis.
FANet architecture outperforms all the SOTA methods on this dataset by a large
margin with F1 of 0.9355, mIoU of 0.8937, recall of 0.9339, and precision of
0.9401 (see Table III). FANet achieves the best trade-off between recall and
precision compared to the ResUNet-based architectures [53, 17]. The strength
of the FANet can be observed by the large improvement of 23.17% in the recall
and 5.24% in the precision over the SOTA ResUNet++ [17]. The recall suggests
that our method is more clinically preferable than the SOTA. A higher recall
is desired in the systems used for clinical diagnosis [57].

####  IV-B3 Results on 2018 Data Science Bowl

TABLE IV: Results on the 2018 Data Science Bowl [43] Method | Backbone | F1 | mIoU | Recall | Prec.
---|---|---|---|---|---
U-Net [10] | ResNet101 | 0.7573 | 0.9103 | - | -
DoubleU-Net [18] | VGG19 | 0.7683 | 0.8407 | 0.6407 | 0.9596
U-Net++ [55] | - | 0.9117 | 0.8477 | 0.9203 | 0.9107
Attention U-Net [21] | - | 0.9179 | 0.8570 | 0.9183 | 0.9235
FANet | None | 0.9176 | 0.8569 | 0.9222 | 0.9194

Cell nuclei segmentation in microscopy imaging is a common task in the
biological image analysis [43]. We used the publicly available 2018 Data
Science Bowl (DSB) challenge dataset and compared our results with the SOTA
methods. Table IV shows that FANet produces an F1 of 0.9176, mIoU of 0.8569,
and recall of 0.9222 with an improvement of 2.02% in F1 with respect to SOTA
UNet++ [16] and 28.15% improvement in recall compared to the best performing
DoubleU-Net [18]. In general, FANet achieves the best trade-off between
precision and recall compared to the SOTA methods resulting in the highest F1
score (0.9176). The qualitative results with 2018 DSB also show that the
predicted FANet produces high-quality segmentation masks for cell nuclei with
respect to the ground truth (see Figure 3).

####  IV-B4 Results on ISIC 2018 dataset

TABLE V: Results on the ISIC 2018 (Skin Caner Segmentation) [44, 45] Method | F1 | mIoU | Recall | Spec. | Prec.
---|---|---|---|---|---
U-Net [10] | 0.6740 | 0.5490 | 0.7080 | 0.9640 | -
R2U-Net [58] | 0.6790 | 0.5810 | 0.7920 | 0.9280 | -
Attention R2U-Net [58] | 0.6910 | 0.5920 | 0.7260 | 0.9710 | -
BCDU-Net (d=1) [59] | 0.8470 | - | 0.7830 | 0.9800 | -
BCDU-Net (d=3) [59] | 0.8510 | - | 0.7850 | 0.9820 | -
U-Net++ [55] | 0.8088 | 0.7319 | 0.8450 | 0.9110 | 0.8648
Attention U-Net [21] | 0.8205 | 0.7346 | 0.8516 | 0.9135 | 0.8645
FANet | 0.8731 | 0.8023 | 0.8650 | 0.9611 | 0.9235

Skin cancer is one of the most commonly diagnosed cancers in the US. Early
detection of melanoma can improve the five-year survival rate and help prevent
it in 99% of the cases [60]. Table V shows the results on the publicly
available International Skin Imaging Collaboration (ISIC) 2018 dataset. FANet
outperformed all the methods on almost all evaluation metrics (F1, mIoU, and
recall). FANet achieved 0.8731 on F1 and recall of 0.8650 with an improvement
of 2.21% and 8.00%, respectively, over the most accurate SOTA BCDU-Net (d=3)
method. A competitive specificity and precision were also recorded. From the
qualitative results in Figure 3, we can see that the input mask produced by
Otsu thresholding shows under segmentation, which is improved significantly
using FANet. The masks produced by FANet have smooth boundaries.

####  IV-B5 Results on DRIVE dataset

TABLE VI: Results on the DRIVE dataset [47] Method | F1 | mIoU | Recall | Spec. | Prec.
---|---|---|---|---|---
U-Net [10] | 0.8174 | - | 0.7822 | 0.9808 | -
Residual U-Net [58] | 0.8149 | - | 0.7726 | 0.9820 | -
Recurrent U-Net [58] | 0.8155 | - | 0.7751 | 0.9816 | -
R2U-Net [58] | 0.8171 | - | 0.7792 | 0.9813 | -
DenseBlock-UNet 111https://github.com/DeepTrial/Retina-VesselNet | 0.8146 | - | 0.7928 | 0.9776 | -
DUNet [61] | 0.8190 | - | 0.7863 | 0.9805 | -
IterNet [62] | 0.8218 | - | 0.7791 | 0.9831 | -
IterNet(Patched) [62] | 0.8205 | - | 0.7235 | 0.9838 | -
U-Net++ [55] | 0.7960 | 0.6615 | 0.7903 | 0.9818 | 0.8070
Attention U-Net [21] | 0.7984 | 0.6648 | 0.7877 | 0.9827 | 0.8146
FANet | 0.8183 | 0.6927 | 0.8215 | 0.9826 | 0.8189

The automated segmentation of vessels in fundus images can assist in the
diagnosis and treatment of diabetic retinopathy. The quantitative result on
the publicly available DRIVE dataset is presented in Table VI. We can observe
that the proposed FANet achieves an F1 score of 0.8183, mIoU of 0.6927, recall
of 0.8215, and precision of 0.8189. The proposed method achieves an
improvement of 4.24% in the recall over SOTA IterNet [62]. Although the F1 of
the IterNet is 0.35% higher than FANet, the recall is relatively lower, and
other metrics such as mIoU and precision are not presented. For our proposed
FANet, the precision of 0.8189 is well balanced with the obtained recall. The
higher recall produced by FANet shows that our method is more clinically
relevant. The quality of the segmentation masks in Figure3 demonstrates the
efficiency of FANet.

####  IV-B6 Results on CHASE-DB1 dataset

TABLE VII: Results on the CHASE-DB1 dataset [48] Method | F1 | mIoU | Recall | Spec. | Prec.
---|---|---|---|---|---
U-Net [10] | 0.7993 | - | 0.7840 | 0.9880 | -
DenseBlock-UNet 222https://github.com/DeepTrial/Retina-VesselNet | 0.8005 | - | 0.8177 | 0.9848 | -
DUNet [61] | 0.8000 | - | 0.7858 | 0.9880 | -
IterNet [62] | 0.8072 | - | 0.7969 | 0.9881 | -
U-Net++ [55] | 0.7954 | 0.6606 | 0.8114 | 0.9847 | 0.7818
Attention U-Net [21] | 0.7941 | 0.6589 | 0.8049 | 0.9852 | 0.7852
FANet | 0.8108 | 0.6820 | 0.8544 | 0.9830 | 0.7722

CHASE-DB1 is the second retinal image segmentation dataset used to evaluate
our method. For this dataset, there is no official training and test split. We
have used 20 images to train our model and 8 images to test as reported in the
work of Li et al. [62]. From Table VII, we can observe that our method
achieved the highest F1 of 0.8108, mIoU of 0.6820, and the highest recall of
0.8544. FANet achieved an improvement of 3.67% in the recall compared to the
SOTA DenseBlock-UNet.

Figure 3: Qualitative results of FANet on seven biomedical image segmentation
datasets. The initial “input mask” is an output from Otsu thresholding for
which a threshold of 0.5 was applied. The final “output mask” is the predicted
segmentation mask from FANet model.

####  IV-B7 Results on EM dataset

TABLE VIII: Results on the EM dataset [46] Method | F1 | mIoU | Recall | Specificity | Prec.
---|---|---|---|---|---
U-Net [10] | - | 0.8830 | - | - | -
Wide U-Net [15] | - | 0.8837 | - | - | -
U-Net++ [55] | 0.9495 | 0.9038 | 0.9520 | 0.7875 | 0.9474
Attention U-Net [21] | 0.9492 | 0.9033 | 0.9502 | 0.7912 | 0.9484
FANet | 0.9547 | 0.9134 | 0.9568 | 0.8096 | 0.9529
TABLE IX: Detailed ablation study of the FANet architecture. Flop is calculated in terms of GMac. “Rec” stands for Recall, “Prec” stands for precision, “Spec” stands for Specificity, “Acc” stands for Accuracy, and “Param” stands for total number parameters. B1 – B4 denotes different network configurations.  Method |  mIoU |  F1 |  Rec |  Prec |  Spec |  Acc |  F2 |  Param |  Flops |  FPS |  Image Size
---|---|---|---|---|---|---|---|---|---|---|---
Dataset: Kvasir-SEG
Baseline (FANet without MixPool, B1) |  0.7732 |  0.8516 |  0.8835 |  0.8710 |  0.9783 |  0.9563 |  0.8614 |  5.76M |  70.38 |  104.20 |  $512\times 512$
Baseline + MixPool (B2) |  0.6378 |  0.7302 |  0.6982 |  0.9098 |  0.9815 |  0.9412 |  0.7039 |  7.72M |  94.75 |  66.75 |  $512\times 512$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.7688 |  0.8460 |  0.9047 |  0.8479 |  0.9576 |  0.9474 |  0.8699 |  5.78M |  76.53 |  101.10 |  $512\times 512$
Baseline + MixPool + feedback (B4) |  0.8153 |  0.8803 |  0.9058 |  0.9005 |  0.9794 |  0.9667 |  0.8872 |  7.72M |  94.75 |  68.18 |  $512\times 512$
Dataset: CVC-ClinicDB
Baseline (FANet without MixPool, B1) |  0.8619 |  0.9166 |  0.9310 |  0.9247 |  0.9934 |  0.9877 |  0.9194 |  5.76M |  70.38 |  103.46 |  $256\times 256$
Baseline + MixPool (B2) |  0.8541 |  0.9108 |  0.9026 |  0.9296 |  0.9943 |  0.9864 |  0.9048 |  7.72M |  94.75 |  67.490 |  $256\times 256$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.8729 |  0.9162 |  0.9052 |  0.9462 |  0.9941 |  0.9889 |  0.9093 |  5.78M |  76.53 |  99.03 |  $256\times 256$
Baseline + MixPool + feedback (B4) |  0.8937 |  0.9355 |  0.9339 |  0.9401 |  0.9948 |  0.9916 |  0.9342 |  7.72M |  94.75 |  67.910 |  $256\times 256$
Dataset: 2018 Data Science Bowl
Baseline (FANet without MixPool, B1) |  0.8495 |  0.9121 |  0.9047 |  0.9283 |  0.9871 |  0.9800 |  0.9068 |  5.76M |  70.38 |  114.82 |  $256\times 256$
Baseline + MixPool (B2) |  0.8158 |  0.8893 |  0.8665 |  0.9289 |  0.9887 |  0.9751 |  0.8733 |  7.72M |  94.75 |  69.64 |  $256\times 256$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.8552 |  0.9165 |  0.9189 |  0.9199 |  0.9863 |  0.9802 |  0.9173 |  5.78M |  76.53 |  100.27 |  $256\times 256$
Baseline + MixPool + feedback (B4) |  0.8569 |  0.9176 |  0.9222 |  0.9194 |  0.9860 |  0.9800 |  0.9195 |  7.72M |  94.75 |  69.22 |  $256\times 256$
Dataset: ISIC 2018
Baseline (FANet without MixPool, B1) |  0.7908 |  0.8647 |  0.9033 |  0.8780 |  0.9151 |  0.9151 |  0.8778 |  5.76M |  70.38 |  111.95 |  $512\times 512$
Baseline + MixPool (B2) |  0.7486 |  0.8303 |  0.8049 |  0.9214 |  0.9617 |  0.9211 |  0.8081 |  7.72M |  94.75 |  65.91 |  $512\times 512$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.8078 |  0.8780 |  0.8746 |  0.9252 |  0.9614 |  0.9374 |  0.8719 |  5.78M |  76.53 |  99.06 |  $512\times 512$
Baseline + MixPool + feedback (B4) |  0.8023 |  0.8731 |  0.8650 |  0.9235 |  0.9611 |  0.9351 |  0.8630 |  7.72M |  94.75 |  71.02 |  $512\times 512$
Dataset: DRIVE
Baseline (FANet without MixPool, B1) |  0.6912 |  0.8172 |  0.8048 |  0.8339 |  0.9846 |  0.9687 |  0.8093 |  5.76M |  70.38 |  103.68 |  $512\times 512$
Baseline + MixPool (B2) |  0.6895 |  0.8161 |  0.8219 |  0.8145 |  0.9820 |  0.9678 |  0.8190 |  7.72M |  94.75 |  68.47 |  $512\times 512$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.6928 |  0.8183 |  0.8124 |  0.8280 |  0.9839 |  0.9687 |  0.8142 |  5.78M |  76.53 |  95.30 |  $512\times 512$
Baseline + MixPool + feedback (B4) |  0.6927 |  0.8183 |  0.8215 |  0.8189 |  0.9826 |  0.9683 |  0.8197 |  7.72M |  94.75 |  70.66 |  $512\times 512$
Dataset: CHASE-DB1
Baseline (FANet without MixPool, B1) |  0.6419 |  0.7816 |  0.7876 |  0.7768 |  0.9848 |  0.9723 |  0.7850 |  5.76M |  70.38 |  95.77 |  $512\times 512$
Baseline + MixPool (B2) |  0.5419 |  0.7009 |  0.8116 |  0.6209 |  0.9664 |  0.9565 |  0.7625 |  7.72M |  94.75 |  65.03 |  $512\times 512$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.6877 |  0.8147 |  0.8372 |  0.7948 |  0.9855 |  0.9760 |  0.8279 |  5.78M |  76.53 |  99.00 |  $512\times 512$
Baseline + MixPool + feedback (B4) |  0.6820 |  0.8108 |  0.8544 |  0.7722 |  0.9830 |  0.9749 |  0.8363 |  7.72M |  94.75 |  71.67 |  $512\times 512$
Dataset: EM
Baseline (FANet without MixPool, B1) |  0.9128 |  0.9544 |  0.9597 |  0.9495 |  0.7946 |  0.9263 |  0.9575 |  5.76M |  70.38 |  79.59 |  $512\times 512$
Baseline + MixPool (B2) |  0.9121 |  0.9540 |  0.9596 |  0.9488 |  0.7918 |  0.9257 |  0.9573 |  7.72M |  94.75 |  59.15 |  $512\times 512$
Baseline + MixPool(E1, D4) + feedback (B3) |  0.9042 |  0.9497 |  0.9404 |  0.9594 |  0.8378 |  0.9198 |  0.9441 |  5.78M |  76.53 |  90.62 |  $512\times 512$
Baseline + MixPool + feedback (B4) |  0.9134 |  0.9547 |  0.9568 |  0.9529 |  0.8096 |  0.9271 |  0.9559 |  7.72M |  94.75 |  70.70 |  $512\times 512$

The EM dataset aims to develop an automatic ML algorithm for the segmentation
of the neural structures so that difficulties due to manual labeling can be
resolved. Table VIII shows the quantitative results on the EM dataset. The
proposed FANet also obtains F1 of 0.9547, mIoU of 0.9134, and a recall of
0.9568. The presented results demonstrate that FANet produces SOTA results,
surpassing other recent methods in terms of mIoU metric that was used by other
methods for comparison.

###  IV-C Qualitative results

The qualitative results on all seven datasets are presented in Figure 3. It
can be observed that for colonoscopy datasets (Kvasir-SEG and CVC-ClinicDB),
even though the initial input mask covers the entirety of the image, our model
is able to prune and provide accurate masks. The same can be observed for the
two retina vessel segmentation datasets, DRIVE and CHASE-DB1. It can be
observed that our model is able to segment the challenging retinal vessels,
including small retinal vessel bifurcations, and it well resembles the ground
truth mask. For the 2018 DSB, ISIC-2018, and EM cell data, again, the input
masks are finely rectified, achieving close to ground truth results by the
proposed FANet model.

###  IV-D Ablation study

In this section, we ablate our model architecture and present extensive
experimental results related to the effectiveness of proposed FANet. To
evaluate the contribution of the MixPool block and the feedback, we created
the following configurations:

  1. 1.

Baseline (B1): It refers to the FANet without the MixPool block, which means
“no feedback mechanism” or “iterative pruning”. We require the MixPool block
to provide feedback as it unifies the attention from the network feature map
and input mask (refer to Figure 2 (b)).

  2. 2.

Baseline + MixPool (B2): We integrate the MixPool block in all the encoder
blocks and decoder blocks. During the inference, we directly apply the trained
model weights with the Ostu thresholding (initial input mask) only once, i.e.,
no iterative pruning is used.

  3. 3.

Baseline + MixPool(E1, D4) + Feedback (B3): Here, we integrate the MixPool
block in the first encoder block and the last decoder block. Feedback
(iterative pruning) is used during the inference.

  4. 4.

Baseline + MixPool + Feedback (B4): This is the final FANet architecture, with
MixPool block in all encoder and decoder blocks and the feedback (iterative
pruning) mechanism is used during the inference.

Table IX presents the ablation results on these four configurations performed
on all seven datasets. Below we provide detailed analyses of the use of
different model architectural settings and validate them with the above
described four network configurations (B1-B4):

####  IV-D1 Effectiveness of MixPool block

The MixPool block is an essential part of the proposed FANet architecture. It
uses the previously predicted mask as the attention to improve the
semantically meaningful features and allows higher-level abstractions. The
effectiveness of the MixPool block can be evaluated by comparing the network
configurations B1 and B4.

From the experiments in Table IX, we can conclude that the B4 outperforms the
B1 on all the datasets. On the F1 metric, B4 shows an improvement of 2.87% on
the Kvasir-SEG dataset, 1.89% improvement on the CVC-ClinicDB, 0.55%
improvement on the 2018 Data Science Bowl dataset, 0.84% improvement on the
ISIC 2018 dataset, 0.11% improvement on the DRIVE dataset, 2.92% improvement
on the CHASE-DB1 dataset, and a 0.03% improvement on the EM dataset. These
performance gains are significant and thus demonstrate the effectiveness of
the use of MixPool block in the proposed FANet.

####  IV-D2 Optimum position of MixPool block in FANet architecture

The positioning of the MixPool is an important factor determining the
performance of the model. In the FANet (B4), we integrate the MixPool block in
all the encoder blocks and the decoder blocks. In B3, we integrate MixPool
block in the first encoder block and the last decoder block only. To evaluate
the effectiveness of the integrating MixPool block, we compare B3 with B4 in
Table IX. It can be observed that out of the seven datasets, on three
datasets, i.e., Kvasir-SEG, CVC-ClinicDB, and 2018 Data Science Bowl, a
significant improvement in B4 is observed as compared to the B3. On the F1
metric, we can observe that B4 achieves an improvement of 3.43 % on Kvasir-
SEG, 1.93 % on CVC-ClinicDB, 0.11 % on the 2018 Data Science Bowl, and 0.5 %
on the EM dataset.

####  IV-D3 Significance of feedback during evaluation

The proposed architecture uses the feedback information (input mask) while
training. This feedback mechanism is also used during the evaluation for
iterative pruning the predicted the mask. To evaluate the effectiveness of the
feedback mechanism, we compare the B2 (FANet without feedback) with the B4
(FANet with feedback) in Table IX. On all the datasets, we used feedback
during inference and compared its performance with the model without feedback.
We can observe that the majority of performance gains in mIoU and F1. For
Kvasir-SEG, B4 shows a 17.75 % improvement in the mIoU, 15.01 % improvement in
the F1, and a 20.76 % improvement in the Recall. Likewise, on the CVC-
ClinicDB, we can see that B4 has 3.96 % improvement in mIoU and 2.27 %
improvement in the F1.

Figure 4: Comparison of the intermediate feature map of the different networks
on the Kvasir-SEG dataset. Figure 5: Iterative pruning on different dataset
images.

###  IV-E Algorithm efficiency

TABLE X: Algorithm complexity of our proposed FANet and other SOTA methods. Method |  Params. (million) |  Flops (GMac) |  Inf. Time (in ms.) |  Image size (pixels)
---|---|---|---|---
U-Net [10] | 31.04 | 219.01 | 3.14 | $512\times 512$
ResU-Net[53] | 8.22 | 181.68 | 2.93 | $512\times 512$
U-Net++[55] | 9.16 | 138.6 | 4.07 | $512\times 512$
Attention U-Net [21] | 34.88 | 266.54 | 4.47 | $512\times 512$
FANet | 7.72 | 94.75 | 8.25 | $512\times 512$
Params.: parameters; Inf.: inference ; ms: milliseconds

We have analyzed the algorithm efficiency in terms of the number of
parameters, flops, and inference time for SOTA methods and FANet (see Table
X). During the architectural design, we limit the number of trainable
parameters in order to minimize the computational cost of our model. The
proposed FANet has only 7.72 M parameters and 94.75 GMac flops, i.e. FANet has
the least number of parameters and flops as compared to other deeper
architectures. However, our inference time is higher than the other baseline
networks which is due to the introduction of novel MixPool block in the FANet
that incorporates additional operations such as element-wise multiplication
from the readout of the RLE encoded mask that resulted in larger computational
time. However, in terms of FPS per iteration this is still above 60 (see Table
IX). In the FANet, the MixPool block facilitates attention and propagation of
information flow from the current learning paradigm and that of the previous
epoch, which helps to achieve a performance boost (refer Table IX). To verify
the efficiency of the MixPool block, we have compared our network with and
without the MixPool block in Table IX. It is also evident that removing the
MixPool block reduces the overall performance in all datasets.

###  IV-F Extended ablation study

TABLE XI: Extended ablation study demonstrating effectiveness of our proposed FANet architecture on the Kvasir-SEG dataset. Here, we ablate our network using different configuration that includes: (i)$R_{2,3}$ removing Fl of MixPool block in Figure 2 (b), (ii)$R_{2,4}$ removing and adding SE-Residual networks in FANet Figure 2 (c), and (iii)$R_{3,2}$series concatenation of FANet in contrast to iterative mechanism. Method | F1 | mIoU | Recall | Prec.
---|---|---|---|---
$R{2,3}$MixPool block (w/o Fl) | 0.6527 | 0.5523 | 0.7175 | 0.7255
$R{2,4}$One SE block only | 0.7641 | 0.6810 | 0.9531 | 0.7133
$R{2,4}$Three SE blocks | 0.8570 | 0.7908 | 0.8974 | 0.8768
$R{2,4}$Four SE blocks | 0.8158 | 0.7417 | 0.8528 | 0.8605
$R{3,2}$Four FANet (in series) | 0.8682 | 0.8004 | 0.8994 | 0.8913
Two SE in FANet (Ours) | 0.8803 | 0.8153 | 0.9058 | 0.9005

We have performed an extended ablation study to demonstrate the architectural
effectiveness of the proposed FANet. Here, we begin with the experimental
verification of the MixPool block by removing its certain components. From the
Table XI, we can observe a performance drop, F1 drops by $22.76$% and mIoU
drops by $26.3$% when feature map $F_{l}$ is not used during feature
concatenation. In order to justify the use of two SE-Residual block in our
proposed FANet, we conducted studies by removing and adding SE-Residual blocks
from the FANet. We first began by removing a SE-Residual block for which we
observed an $11.62$% drop in F1 and $13.43$% drop in mIoU.

Further, we modified the FANet architecture by adding three SE-Residual blocks
and we observed again a decrease in the performance. For this case, F1 drops
by $2.33$% and mIoU by $2.45$%. Next, we added one more SE-Residual block and
a severe performance drop can be observed. The F1 dropped by $6.45$% and the
mIoU drops by $7.36$%. In our proposed architecture we used an iterative
pruning. However, we experimented an alternative strategy by concatenating the
four FANet together in a series. From this experiment we observed a drop of F1
by $1.21$% and $1.49$% drop in mIoU with nearly four times increase in the
number of trainable parameters.

##  V Discussion

While deep learning semantic segmentation has been widely implemented, to the
best of our knowledge, only direct inference strategies have been published
till date. In this work, we utilize a segmentation map pruning mechanism that
demonstrates a clear advantage over the current state of the art models due to
its ability to self-rectify the predicted mask during the evaluation process
(see Table II-Table VIII). The process of self-rectification or iterative
pruning helps to improve the performance of the proposed FANet architecture.
This improvement is due to the feedback provided by the input mask in the
MixPool block which is further validated from our two ablation studies (Table
IX and Table XI). Furthermore, a joint configuration together with mask and
the feature embeddings allow learning to achieve better feature representation
of target regions and learning to adjust weights dependent on the input mask.
This establishes an effective pruning mechanism of the network enabling input
mask to be steered in the direction of the relevant learnt features of the
network. Additionally, it can capture the variability in datasets (e.g., shape
distributions, surface morphology etc.), allowing the network to rectify the
predicted/input masks.

Table IX shows the complete ablation study of the MixPool block in the FANet
architecture. In this ablation, we provide experimental results with (proposed
network, B4) and without the MixPool block (B1). Here, B1 refers to the “no
feedback mechanism” as no MixPool block is applied. In the proposed FANet, we
require the MixPool block to provide feedback through a unified attention
mechanism taking into account the network feature map and the input mask from
the previous epoch (refer to Figure 2 (b)). However, for the MixPool block
without feedback (i.e., B2), we provide the attention from the generated
feature map and the input mask but we do not perform the iterative pruning
during the evaluation. Thus, even though B2 and B4 networks have the same
number of parameters (7.72M parameters), the removal of the feedback mechanism
affects the algorithm performance (see Table IX). SE-Residual blocks that
serve as a self-attention mechanisms on feature channels by performing global
average pooling followed by multi layer perceptron that allows to explicitly
model the interdependencies between feature channels. Further, in our network
we introduce spatial attention mechanisms. Multiple SE-residual blocks allow
to learn complex non-linear feature interdependencies (also see Table XI for
different combinations of SE-Residual blocks). Further, other ablation
experiments such as series concatenation of FANet and removal of $F_{l}$ layer
in MixPool block in Table XI showed that the proposed FANet achieves the
highest performance. This justifies the importance of different components
integrated in the proposed FANet architecture.

With the introduction of the iterative pruning in our FANet architecture, we
introduce a new hyperparameter, i.e., the number of iterations during the
evaluation. $R_{3,1}$The optimal number of iterations is 10, which was
empirically established across datasets. The computed number of iterations is
the same for all datasets. For this, we have plotted a graph (Figure 5)
showing the iterative pruning on different dataset images. From the graph, it
is observed that there is a significant improvement from iteration 1 to 5.
However, from 5 to 10 iterations, there is a minor to negligible improvement.
Thus, we considered the highest of 10 iterations during the evaluation. The
iterative pruning over the input image increases the inference time. However,
this process allows us to refine the predicted segmentation masks, unlike most
current methods. For obtaining a better trade-off between efficiency and
accuracy, we advise using a lesser number of iterations. We plot the F1 score
for different dataset images for five iterations during evaluation. Figure 5
shows that our proposed FANet benefits with just two iterations. Additionally,
we have used the NVIDIA GTX 1050 Ti (released in 2016) for inference, and thus
using a more recent GPU with higher performance can provide better inference
time.

##  VI Conclusion

With the FANet architecture, we proposed a novel approach for biomedical image
segmentation that can self-rectify the predicted masks. By introducing a
feedback mechanism, we achieved an improvement on seven publicly available
biomedical datasets when compared with existing SOTA methods. Our approach
requires far fewer epochs for training and is well-suited to diverse
biomedical imaging datasets. The feedback mechanism integrated in FANet design
effectively acts as hard attention that is used with the existing feature maps
to boost the strength of feature representations. The experimental results
demonstrate that the proposed architecture achieves accurate and consistent
segmentation results across several biomedical imaging datasets despite its
simple and straightforward network architecture. The ablation study also
reveals that FANet requires less training time to achieve near SOTA
performance. In the future, we will use a contrastive learning approach to
improve the performance of FANet further and test it on additional multimodal
biomedical images.

## Acknowledgment

D. Jha is funded by the PRIVATON project from the Research Council of Norway,
project number 263248 (Privaton). S. Ali is supported by the National
Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC).
The computations in this paper are performed on equipment provided by the
Experimental Infrastructure for Exploration of Exascale Computing (eX3), which
is financially supported by the Research Council of Norway under contract
270053. The views expressed are those of the author(s) and not necessarily
those of the NHS, the NIHR or the Department of Health.

## References

  * [1] W. Wang, K. Yu, J. Hugonot, P. Fua, and M. Salzmann, “Recurrent u-net for resource-constrained segmentation,” in _Proc. ICCV_ , 2019, pp. 2142–2151.
  * [2] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling.” _CoRR_ , vol. abs/1803.01271, 2018.
  * [3] K. Xu _et al._ , “Show, attend and tell: Neural image caption generation with visual attention,” in _Proc. ICML_ , 2015, pp. 2048–2057.
  * [4] A. Vaswani _et al._ , “Attention is all you need,” in _Proc. NeurIPS_ , 2017, pp. 5998–6008.
  * [5] O. Tilk and T. Alumäe, “Bidirectional recurrent neural network with attention mechanism for punctuation restoration,” in _Proc. INTERSPEECH_ , 2016.
  * [6] J. Schlemper _et al._ , “Attention gated networks: Learning to leverage salient regions in medical images,” _Med Image Anal_ , vol. 53, pp. 197 – 207, 2019.
  * [7] C. Song, Y. Huang, W. Ouyang, and L. Wang, “Mask-guided contrastive attention model for person re-identification,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018.
  * [8] G. Wang _et al._ , “Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks,” _Neurocomputing_ , vol. 338, pp. 34 – 45, 2019.
  * [9] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2015, pp. 3431–3440.
  * [10] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: convolutional networks for biomedical image segmentation,” in _Proc. MICCAI_ , 2015, pp. 234–241.
  * [11] Z. Zhong _et al._ , “Squeeze-and-attention networks for semantic segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2020, pp. 13 065–13 074.
  * [12] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 2881–2890.
  * [13] J. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _IEEE Trans. Pattern Anal. Mach. Intell._ , 2020.
  * [14] D.-P. Fan _et al._ , “PraNet: parallel reverse attention network for polyp segmentation,” in _Proc. MICCAI_ , 2020, pp. 263–273.
  * [15] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: A nested U-Net architecture for medical image segmentation,” in _Proc. DLMIAMLCDS_ , 2018, pp. 3–11.
  * [16] Z. Zhou, M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “UNet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _IEEE Trans. Med. Imag._ , vol. 39, pp. 1856 – 1867, 2019.
  * [17] D. Jha _et al._ , “ResUnet++: An advanced architecture for medical image segmentation,” in _Proc. ISM_ , 2019, pp. 225–230.
  * [18] D. Jha, M. A. Riegler, D. Johansen, P. Halvorsen, and H. D. Johansen, “DoubleU-Net: a deep convolutional neural network for medical image segmentation,” in _Proc. CBMS_ , 2020.
  * [19] C. Guo, M. Szemenyei, Y. Yi, W. Wang, B. Chen, and C. Fan, “SA-UNet: spatial attention u-net for retinal vessel segmentation,” _arXiv preprint arXiv:2004.03696_ , 2020.
  * [20] N. Ibtehaz and M. S. Rahman, “Multiresunet: Rethinking the U-Net architecture for multimodal biomedical image segmentation,” _Neural Netw._ , vol. 121, pp. 74–87, 2020.
  * [21] O. Oktay _et al._ , “Attention U-Net: Learning where to look for the pancreas,” _arXiv preprint arXiv:1804.03999_ , 2018.
  * [22] R. Wang, S. Chen, C. Ji, J. Fan, and Y. Li, “Boundary-aware context neural network for medical image segmentation,” _arXiv preprint arXiv:2005.00966_ , 2020.
  * [23] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 40, no. 4, pp. 834–848, 2017.
  * [24] T. Hassan, M. Usman Akram, and N. Werghi, “Evaluation of Deep Segmentation Models for the Extraction of Retinal Lesions from Multi-modal Retinal Images,” _arXiv e-prints_ , 2020.
  * [25] X. Sun, P. Zhang, D. Wang, Y. Cao, and B. Liu, “Colorectal polyp segmentation by u-net with dilation convolution,” in _Proc. ICMLA_ , 2019, pp. 851–858.
  * [26] B. Cheng _et al._ , “Panoptic-DeepLab: A simple, strong, and fast baseline for bottom-up panoptic segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2020, pp. 12 472–12 482.
  * [27] H. Wang _et al._ , “Axial-deeplab: Stand-alone axial-attention for panoptic segmentation,” in _Proc. ECCV_ , 2020, pp. 108–126.
  * [28] L. Chen _et al._ , “SCA-CNN: Spatial and channel-wise attention in convolutional networks for image captioning,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017.
  * [29] L.-C. Chen, Y. Yang, J. Wang, W. Xu, and A. L. Yuille, “Attention to scale: Scale-aware semantic image segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016.
  * [30] L. Ye, M. Rochan, Z. Liu, and Y. Wang, “Cross-modal self-attention network for referring image segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2019, pp. 10 502–10 511.
  * [31] X. Chu _et al._ , “Multi-context attention for human pose estimation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 5669–5678.
  * [32] A. S. Lundervold and A. Lundervold, “An overview of deep learning in medical imaging focusing on MRI,” _Zeitschrift für Medizinische Physik_ , vol. 29, no. 2, pp. 102 – 127, 2019.
  * [33] H. Tsuda, E. Shibuya, and K. Hotta, “Feedback attention for cell image segmentation,” in _Proc. ECCVW_ , 2020.
  * [34] E. Shibuya and K. Hotta, “Feedback U-Net for cell image segmentation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Worksh._ , 2020, pp. 974–975.
  * [35] X. SHI _et al._ , “Convolutional LSTM Network: A machine learning approach for precipitation Nowcasting,” in _Proc. NeurIPS_ , vol. 28, 2015, pp. 802–810.
  * [36] A. Mosinska, P. Marquez-Neila, M. Koziński, and P. Fua, “Beyond the pixel-wise loss for topology-aware delineation,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2018, pp. 3136–3145.
  * [37] P. Pandey _et al._ , “Unsupervised Domain Adaptation for Semantic Segmentation of NIR Images Through Generative Latent Search,” in _Proc. ECCV_ , 2020, pp. 413–429.
  * [38] A. Newell, K. Yang, and J. Deng, “Stacked hourglass networks for human pose estimation,” in _Proc. ECCV_ , 2016, pp. 483–499.
  * [39] V. Ramakrishna, D. Munoz, M. Hebert, J. A. Bagnell, and Y. Sheikh, “Pose machines: Articulated pose estimation via inference machines,” in _Proc. ECCV_ , 2014, pp. 33–47.
  * [40] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 4724–4732.
  * [41] D. Jha _et al._ , “Kvasir-SEG: A segmented polyp dataset,” in _Proc. MMM_ , 2020, pp. 451–462.
  * [42] J. Bernal _et al._ , “Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians,” _Comput Med Imaging Graph_ , vol. 43, pp. 99–111, 2015.
  * [43] J. C. Caicedo _et al._ , “Nucleus segmentation across imaging experiments: the 2018 data science bowl,” _Nat. Methods_ , vol. 16, no. 12, pp. 1247–1253, 2019.
  * [44] N. C. Codella _et al._ , “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (ISBI),” in _Proc. ISBI_ , 2018, pp. 168–172.
  * [45] P. Tschandl, C. Rosendahl, and H. Kittler, “The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Sci. Data_ , vol. 5, p. 180161, 2018.
  * [46] A. Cardona _et al._ , “An integrated micro-and macroarchitectural analysis of the Drosophila brain by computer-assisted serial section electron microscopy,” _PLoS Biol._ , vol. 8, no. 10, p. e1000502, 2010.
  * [47] J. Staal, M. D. Abràmoff, M. Niemeijer, M. A. Viergever, and B. Van Ginneken, “Ridge-based vessel segmentation in color images of the retina,” _IEEE Trans. Med. Imag._ , vol. 23, no. 4, pp. 501–509, 2004.
  * [48] C. G. Owen _et al._ , “Measuring retinal vessel tortuosity in 10-year-old children: validation of the computer-assisted image analysis of the retina (CAIAR) program,” _ARVO_ , vol. 50, no. 5, pp. 2004–2010, 2009.
  * [49] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , 2016, pp. 770–778.
  * [50] J. Hu, L. Shen, and G. Sun, “Squeeze-and-Excitation Networks,” in _Proc. IEEE Conf. Comput. Vis. Pattern Recognit._ , June 2018.
  * [51] M. Malinowski, C. Doersch, A. Santoro, and P. Battaglia, “Learning visual question answering by bootstrapping hard attention,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 3–20.
  * [52] N. Otsu, “A threshold selection method from gray-level histograms,” _IEEE transactions on systems, man, and cybernetics_ , vol. 9, no. 1, pp. 62–66, 1979.
  * [53] Z. Zhang, Q. Liu, and Y. Wang, “Road extraction by deep residual U-Net,” _IEEE Geosci. Remote. Sens. Lett._ , vol. 15, no. 5, pp. 749–753, 2018.
  * [54] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proc. ECCV_ , 2018, pp. 801–818.
  * [55] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_ , 2018, pp. 3–11.
  * [56] Y. Fang, C. Chen, Y. Yuan, and K.-y. Tong, “Selective feature aggregation network with area-boundary constraints for polyp segmentation,” in _Proc. MICCAI_ , 2019, pp. 302–310.
  * [57] C. Gilvary, N. Madhukar, J. Elkhader, and O. Elemento, “The missing pieces of artificial intelligence in medicine,” _Trends Pharmacol. Sci._ , vol. 40, no. 8, pp. 555–564, 2019.
  * [58] M. Z. Alom, M. Hasan, C. Yakopcic, T. M. Taha, and V. K. Asari, “Recurrent residual convolutional neural network based on U-Net (r2u-net) for medical image segmentation,” _arXiv preprint arXiv:1802.06955_ , 2018.
  * [59] R. Azad, M. Asadi-Aghbolaghi, M. Fathy, and S. Escalera, “Bi-directional ConvLSTM U-Net with densely connected convolutions,” in _Proc. ICCVW_ , 2019, pp. 0–0.
  * [60] A. C. Society, “Cancer facts & figures 2018,” 2018.
  * [61] Q. Jin _et al._ , “DUNet: a deformable network for retinal vessel segmentation,” _Knowl.-Based Syst._ , vol. 178, pp. 149–162, 2019.
  * [62] L. Li, M. Verma, Y. Nakashima, H. Nagahara, and R. Kawasaki, “IterNet: retinal image segmentation utilizing structural redundancy in vessel networks,” in _Proc. WCACV_ , 2020, pp. 3656–3665.

  *[ML]: Machine Learning
  *[CNNs]: Convolutional Neural Network
  *[RNNs]: Recurrent Neural Network
  *[TTA]: Test-Time Augmentation
  *[SE]: Squeeze and Excitation
  *[SOTA]: state-of-the-art
  *[CNN]: Convolutional Neural Network
  *[FCN]: Fully Convolutional Network
  *[ASPP]: Atrous Spatial Pyramidal Pooling
  *[BN]: Batch Normalization
  *[ReLU]: Rectified Linear Unit
  *[FCNN]: Fully Convolutional Neural Network
  *[DSC]: Dice Coefficient
  *[mIoU]: mean Intersection over Union
  *[DRIVE]: Digital Retinal Images for Vessel Extraction