# Improved Abdominal Multi-Organ Segmentation via 3D Boundary-Constrained Deep
Neural Networks

Samra Irshad<EMAIL_ADDRESS>Douglas P.S. Gomes  Seong Tae Kim
Swinburne University of Technology, Hawthorn, Australia  Victoria University,
Melbourne, Australia  Kyung Hee University, Yongin-si, Gyeonggi-do, South
Korea

###### Abstract

Background and Objective: Quantitative assessment of the abdominal region from
clinically acquired CT scans requires the simultaneous segmentation of
abdominal organs. Therefore, for the past two decades, automatic abdominal
image segmentation has been the subject of intensive research to facilitate
the health professionals easing the clinical workflow. Thanks to the
availability of high-performance and powerful computational resources, deep
learning-based methods have resulted in state-of-the-art performance for the
segmentation of 3D abdominal CT scans. However, the complex characterization
of organs with fuzzy and weak boundaries prevents the deep learning methods
from accurately segmenting these anatomical organs. Specifically, the voxels
on the boundary of organs are more vulnerable to misprediction due to the
highly-varying intensity of inter-organ boundaries, and the misprediction of
these voxels is detrimental to overall segmentation performance. This paper
investigates the possibility of improving the abdominal image segmentation
performance of the existing 3D encoder-decoder networks by leveraging organ-
boundary prediction as a complementary task.

Method: To address the problem of abdominal multi-organ segmentation, we train
the 3D encoder-decoder network to simultaneously segment the abdominal organs
and their corresponding boundaries in CT scans via multi-task learning. The
network is trained end-to-end using a loss function that combines two task-
specific losses, i.e., complete organ segmentation loss and boundary
prediction loss. We explore two different network topologies based on the
extent of weights shared between the two tasks within a unified multi-task
framework. In the first topology, the whole-organ prediction task and the
boundary detection task share all the layers in the encoder-decoder network
except for the last task-specific prediction layers. In contrast, the second
topology employs a single shared encoder but two separate task-specific
decoders. To evaluate the utilization of complementary boundary prediction
task in improving the abdominal multi-organ segmentation, we use three state-
of-the-art encoder-decoder networks: 3D UNet, 3D UNet++, and 3D Attention-
UNet.

Results: The effectiveness of utilizing the organs’ boundary information for
abdominal multi-organ segmentation is evaluated on two publically available
abdominal CT datasets: Pancreas-CT and the BTCV dataset. The improvements
shown in segmentation results (evaluated via Dice Score, Average Hausdorff
Distance, Recall, and Precision) reveal the advantage of the multi-task
training that forces the network to pay attention to ambiguous boundaries of
organs. A maximum relative improvement of 3.5% and 3.6% is observed in Mean
Dice Score for Pancreas-CT and BTCV datasets, respectively. All source codes
are publically available on https://github.com/samra-irshad/3d-boundary-
constrained-networks.

###### keywords:

Abdominal multi-organ segmentation , Fully convolutional neural networks ,
Boundary-constrained segmentation , Multi-task learning

plaintop table supplementarymaterial

##  1 Introduction

Multi-organ segmentation on abdominal Computed Tomography (CT) scans is an
essential prerequisite for computer-assisted surgery and organ transplantation
[1], [2]. Particularly, quantitative assessment of abdominal regions enables
accurate organ dose calculation, required in numerous radiotherapy treatment
options. Erroneous delineation of abdominal organs prevents harnessing the
benefits of radiotherapeutic advancements. In clinical practice, physicians
delineate abdominal organs using manual segmentation tools, which are time-
consuming, observer-dependent, and error-prone. With the increased use of
imaging facilities and production of a large number of abdominal CT scans, the
utilization of automated, robust, and efficient organ-delineation tools has
become compulsory [2], [3], [4]. Automatic segmentation tools delineate the
abdominal structures much faster and overcome the issues like variability in
human expertise and inherent subjectivity.

Abdominal CT scans often present weak inter-organ boundaries characterized by
regions of similar voxel intensities, which in turn results in low-contrast
representations. Such appearances are usually caused by the representation of
abdominal soft tissues in a narrow band of Hounsfield (HU) values. Another
factor that enhances the already complex representation of abdominal organs is
the existence of artifacts occurring due to blood flow, respiratory, and
cardiac motion. Accurate delineation of abdominal organs with unclear
boundaries and complex geometrical shapes is one of the ongoing challenges
that hurdles the abdominal-related clinical diagnosis.

(a)

(b)

(c)

Figure 1: Exemplary 2D abdominal CT image showing the visual characteristics
of organs. (a) 2D abdominal image, (b) Abdominal organs annotated on CT image:
pancreas (  ), spleen (  ), liver (  ), stomach (  ), gallbladder (  ), (c) 3D
multi-organ voxel map.

Earlier methods proposed for the abdominal multi-organ segmentation mainly
were based on multi-atlas [5], [6] or statistical models [7], [8]. Some
methods also made use of handcrafted or learned features to segment abdominal
organs [9], [10]. However, the recent Fully Convolutional Network (FCN) based
approaches have presented better results due to the improved organ
representation learning [2] [11]. Being able to preserve the image structure
and provision of efficient learning as well as inference, FCN-based methods
are currently considered state-of-the-art for abdominal multi-organ
segmentation [2], [12], [13], [14]. Specifically, these networks follow the
encoder-decoder architectural design [15]. In such networks, the shallow
layers in the encoder aim to extract low-level features, and the deep layers
encode high-level features. While the mirrored-decoder maps back the learned
features to generate an output of the same size as input with skip connections
assisting in retaining the crucial features extracted in the encoding path
[15].

Existing FCN-based methods for abdominal multi-organ segmentation employ
either 2D or 3D convolutional architectures [13], [12]. 2D methods process the
CT scans in a slice-by-slice fashion and predict the organ labels on
individual slices [13]. Despite being memory- and parameter-efficient, 2D
methods are unable to make full use of 3D contextual information [2]. 3D
methods make use of rich volumetric context by processing the whole CT volume
and generating voxel-maps in a single forward propagation pass, leading to
better abdominal CT segmentation performance than 2D approaches [16], [17].

The existing 3D methods have primarily focused on designing better
architectures for improved abdominal multi-organ representation learning [12],
[2]. However, they treat all the anatomical parts within a single organ
equally since they solely rely on voxel-level information and do not
specifically focus on improving the segmentation of voxels in vulnerable
regions/parts of organs. As an example, we highlight some of the important
characteristics of abdominal organs in Fig. 1. From Figures 1(a) and 1(b), it
can be noticed that the adjacent organs have weak contours which sometimes
touch each other. As an example, observe the low-contrasted and touching
boundaries between stomach (  ) and pancreas (  ). Moreover, 3D multi-organ
visualization in 1(c) shows that the adjacent positioning of organs in the
abdominal cavity aggravates the complex spatial relationship among the organs.
Simultaneously segmenting the abdominal organs with soft contours and complex
spatial relationships is a challenging task.

The boundaries of anatomical regions in medical scans serve as an important
cue for facilitating manual and automated delineation [18]. Numerous existing
deep learning-based studies leveraged learning of features corresponding to
boundary of regions for improved medical image segmentation via multitask
learning paradigm [19], [20], [21], [22], [23]. In recent years, deep
multitask learning paradigm has been widely used due to its potential to solve
multiple tasks in one forward propagation and ability to learn better
representations because of the multiple supervisory signals [24], [25]. In
this paper, we propose to improve the segmentation of abdominal organs on CT
scans by enhancing the segmentation of boundary of organs. Particularly, we
train the 3D deep learning networks to simultaneously predict the boundary and
the entire region of organs. The inclusion of boundary information is
motivated by the fact that the voxels on the boundary of organs are more
vulnerable to misprediction because of their ambiguous appearance and complex
relationship with adjacent organs. Specifically, our work makes the following
contributions:

  1. (i)

We develop an end-to-end trainable 3D multi-task learning framework that
simultaneously predicts the voxel-labels of abdominal organs and their
corresponding boundaries. By integrating the boundary features, our proposed
boundary-constrained 3D deep learning framework focuses on the accurate
prediction of the edges of organs in addition to whole organs.

  2. (ii)

Instead of relying on a single network topology, we explore and compare two
network topologies for conducting multi-task learning. In the first topology,
the whole encoder-decoder network is shared with separate task-specific
prediction layers at the end for predicting boundaries and entire organs’
maps. In the second topology, an encoder is shared with separate task-specific
decoders for decoding the features, jointly learned by the shared encoder to
predict the boundary and organ probability maps. With an extensive comparison,
we reveal that integration of boundary features invariably improves the multi-
organ segmentation performance, independent of the multi-task network design.

  3. (iii)

We utilize three state-of-the-art 3D encoder-decoder architectures, i.e., UNet
[26], UNet++ [27], and Attention-UNet [28] as baseline networks for evaluating
the effect of incorporating boundary information. We modifiy each baseline
architecture according to our proposed multi-task topologies. We demonstrate
significant performance improvements with a negligible increase in trainable
parameters.

  4. (iv)

We validate the performance of baseline and counterpart boundary-constrained
models on two publically available datasets (Pancreas-CT [29] and BTCV [30])
using Dice Score, Average Hausdorff Distance, Recall, and Precision.
Furthermore, we conduct additional experiments to evaluate the improvement in
the segmentation of regions around the boundaries. The results show that the
boundary-constrained networks learn feature representations that focus on the
accurate organs segmentation and the challenging parts around the border of
the organs.

The rest of the article is organized as follows. In section 2, we review the
existing methods for abdominal multi-organ segmentation. Section 3 describes
our framework for incorporating the boundary information into the 3D fully
convolutional networks, including the multi-task loss function and the details
of boundary-constrained network topologies. Next, we describe the dataset
specifications and implementation details in section 4. We then present the
experimental results, comparisons with existing single-task approaches, and
in-depth performance analysis of boundary-constrained models in section 5.
Finally, we discuss the important highlights and some directions for future
work in section 6 and present the conclusion in section 7.

(a)

(b)

Figure 2: Multi-task topologies of 3D boundary-constrained network. (a) Multi-
task topology with shared encoder-decoder network and task-specific prediction
layers, and (b) Multi-task topology with shared encoder and task-specific
decoders.

##  2 Related Work

Segmentation of anatomical structures from abdominal scans is a prerequisite
for various high-level CT-based clinical applications. Existing computerized
tools for abdominal image segmentation are either based on deep learning or
non-deep learning methods. In this section, we first briefly discuss the non-
deep learning methods (section 2.1) and then present a review of deep
learning-based methods for abdominal multi-organ segmentation (section 2.2).
We conclude this section with a discussion on multi-task deep neural networks
being employed for complementary boundary learning task to improve medical
image segmentation (section 2.3).

###  2.1 Non-deep learning-based abdominal organs segmentation

Earlier methods proposed for abdominal multi-organ segmentation have primarily
utilized registration-based approaches [7], [8]. Among the registration-based
approaches, the widely used ones include statistical shape models [7], [8] and
multi-atlas label fusion techniques [5], [6]. The development of statistical
models requires registration of training images for estimating the shape or
appearance of anatomical organs followed by fitting constructed models to test
images for generating segmentations [31], [32]. Multi atlas-based methods
utilize an atlas created using multiple labelled images in the training set,
and the test image is segmented by propagating the reference segmentations.
Atlases are constructed by capturing the prior anatomical knowledge relevant
to target organs. However, it is difficult to build an adequate model to
capture the large variability of the deformable organs with limited data [33].
Furthermore, the performance of both these approaches is restricted by image
registration accuracy.

Registration-free approaches train a classifier using either handcrafted or
learned features to segment abdominal images [9]. Extraction of robust and
deformation-invariant features relies on expert knowledge about abdominal
organs [34]. Having the ability to learn the features automatically, FCN-based
methods, have rapidly replaced the traditional solutions that require image
registration or handcrafted features and have shown improved performance for
abdominal CT segmentation [2], [12], [13], [35].

###  2.2 Fully Convolutional Networks for abdominal multi-organ segmentation

In recent years, Fully Convolution Network (FCN) and its variants (e.g., UNet
[15]) have become a common choice for medical image segmentation. This
dominancy can be attributed to their ability to learn effective task
representations and efficient inference. UNet has an encoder-decoder style
architecture and consists of skip connections, joining the encoding and
decoding layers on the same level. Despite being trained from scratch, UNet
demonstrated state-of-the-art performance for various medical image
segmentation tasks [36], [37]. Built on top of UNet, several other modified
architectures were subsequently proposed, e.g., UNet++ [27], Attention-UNet
[28], etc.

Existing deep learning-based studies for abdominal multi-organ segmentation
have utilized 2D or 3D convolutional networks. 2D methods are less parameter-
intensive; however, they cannot exploit the 3D contextual information and
eventually provide sub-accurate organ-delineation performance. 3D
convolutional networks are facilitated with 3D convolutions, 3D pooling, and
3D normalization to exploit the rich volumetric context and generate dense
voxel-wise predictions [26]. Advances in efficient 3D convolutional
implementation and increased GPU memory have enabled the adoption of 3D
convolutional models for abdominal multi-organ segmentation [38], [3].

Roth et al. [16] proposed a cascaded architecture based on two 3D UNets where
the first UNet is trained to separate the abdominal area from the background,
and the latter utilized the output from the first UNet to simultaneously
segment the abdominal organs. Peng et al. [39] delineated abdominal organs
using 3D UNet with residual-learning based units (ResNets) to calculate
patient-specific CT organ dose. In another study [2], abdominal organs are
segmented using a 3D FCN with dilated convolutions based densely connected
units. Heinrich et al. [11] leveraged 3D deformable convolutions to spatially
adapt the receptive field for abdominal multi-organ segmentation. In [40],
abdominal scans were segmented using a 3D deeply supervised patch-based UNet
with grid-based attention gates to encourage the network to focus on useful
salient features propagated through the skip connections. Some existing
methods have employed post-processing steps, including level-sets [3] and
graph-cut [4] to refine initial segmentation maps obtained from 3D deep
convolutional networks.

Through the efforts mentioned above, the existing 3D methods have mostly
emphasized developing better deep learning architectures and did not attempt
to improve the segmentation of challenging parts of abdominal organs, e.g.,
voxels that belong to the contour of organs and regions within the vicinity of
organ-contour. The fuzzy appearance of the boundary of organs and low contrast
between the adjacent abdominal structures makes the voxels belonging to these
regions more susceptible to wrong label prediction.

###  2.3 Boundary-constrained medical image segmentation

Several existing deep learning-based medical image segmentation methods have
utilized the boundary information of regions of interest to overcome the
misprediction of boundary pixels [19], [20], [21], [41]. In these methods, the
networks are trained in a multi-task learning fashion to simultaneously
predict the probability maps of entire organs and their corresponding
boundaries. Most of these methods have resorted to the hard-parameter sharing
technique, where a single network contains shared and task-specific parameters
and is jointly trained to solve multiple tasks.

Chen et al. [19] segmented the glands and their corresponding boundaries via
multi-task training. By training the model to learn the co-representations,
the model achieved better gland segmentation performance than the single-task
models. In [42], a dual-decoder-based network is presented that simultaneously
detects the boundaries and predicts the semantic labels of cells. Features
from the boundary-decoding path were concatenated with those learned in the
entire cell region decoding path via additional skip connections. This led to
the improved histopathological image segmentation performance. In [43],
boundary and distance maps were used for improved polyp and optic disk
segmentation, respectively. Tan et al. [20] proposed a multi-task medical
image segmentation network consisting of a single encoder and separate
dedicated arms for decoding regions and boundaries. The study was evaluated on
numerous applications, including MR femur and CT kidney segmentation. Zhang et
al., [44] presented a edge-based deeply supervised network for predicting the
regions of interest and their corresponding boundaries. The method was
validated for retinal, x-ray, and CT image segmentation. Wang et al. [45]
proposed a two-parallel stream model in which each of the two streams was
trained to segment region and detect boundary followed by fusion of contour
and region prediction maps. Lee et al. [41] proposed a framework that predicts
boundary keypoint maps and makes use of adversarial loss for improved boundary
preserving in medical image segmentation.

Given the challenge presented by voxels on the organs’ boundaries and the
evidence in the literature that focusing on boundaries is beneficial for
performance, we integrate the organs boundary prediction as an auxiliary task
into the training of state-of-the-art 3D medical image segmentation networks.
Since the design choice of network topology impacts the learning process, we
explore two multi-task network designs and analyze their performance. The
boundary co-training resulted in improved performance on abdominal CT
segmentation tasks compared to the several state-of-the-art 3D fully
convolutional baseline architectures.

##  3 Proposed Method

In this section, we first describe the boundary-constrained loss for training
the 3D encoder-decoder network to simultaneously predict the boundaries and
entire abdominal organ regions via multi-task learning (Section 3.1), followed
by an exhibition of our proposed multi-task network topologies (Section 3.2).
After that, we discuss the architecture of the 3D networks that we have as
baselines in our work (Section 3.3). Finally, we present the architectural
design of the counterpart 3D boundary-constrained models (Section 3.4).

###  3.1 Boundary-Constrained Loss

Consider a 3D encoder-decoder network trained to predict the voxel labels of
the abdominal CT scan with $W\times H\times Z$ dimensions, where $W$, $H$, and
$Z$ denote the length, width, and depth of the scan, respectively. Such a
network takes an abdominal multi-organ CT scan as an input and outputs a
labelled voxel map of the same size as the input. To utilize the boundary
information of abdominal organs for improved representation learning, we train
the network to predict the 3D organ-semantic masks and 3D organ-boundaries in
one forward propagation pass. We formulate this problem using a multi-task
learning paradigm where multiple tasks are learned jointly using shared and
task-specific representations. The loss $\mathcal{L}$ for this multi-task
learning problem is a weighted combination of per-task losses, organ
segmentation loss $\mathcal{L}_{RS}$ and organ boundary detection loss
$\mathcal{L}_{BD}$. We use multi-class dice loss [46] for evaluating the
performance of the multi-organ segmentation task, given as

$\displaystyle\mathcal{L}_{RS}$
$\displaystyle=\sum\limits_{c=0}^{C-1}\;\frac{2\;(\hat{y}_{i,c}\times
y_{i,c})}{\hat{y}_{i,c}^{2}\;+\;y_{i,c}^{2}}$ (1)

where, $\hat{y}_{i,c}$ and $y_{i,c}$ denote the 3D multi-organ probability map
and ground-truth mask, respectively, of the $i^{th}$ abdominal CT scan. $C$
denotes the number of organ classes.

$\hat{y}_{i}\;=\;\hat{p}(x_{i};\boldsymbol{\theta_{s}})$ (2)
$\hat{p}(x_{i};\boldsymbol{\theta_{s}})\;=\;\sum\limits_{n=0}^{N-1}\;\hat{p}(x_{i,n};\boldsymbol{\theta_{s}})=\;\sum\limits_{w=0}^{W-1}\sum\limits_{h=0}^{H-1}\sum\limits_{z=0}^{Z-1}\;\hat{p}(x_{i,w,h,z};\boldsymbol{\theta_{s}})$
(3)

where $\hat{p}(x_{i,n})$ represents the label probability of $n^{th}$ voxel in
$i^{th}$ scan and $N$ refers to the total number of voxels in a scan.

To evaluate the model’s performance in predicting the boundaries, we use
binary cross-entropy loss (shown in Eq. Eq. 4). Binary cross-entropy loss for
predicting 3D boundaries is given as

$\displaystyle\mathcal{L}_{BD}$
$\displaystyle=-\sum\limits_{n=0}^{N-1}\;e_{i,n}\log(\hat{e}_{i,n})+(1-e_{i,n})\log(1-\hat{e}_{i,n})$
(4)
$\displaystyle=-\;\sum\limits_{w=0}^{W-1}\sum\limits_{h=0}^{H-1}\sum\limits_{z=0}^{Z-1}\;p(x_{i,w,h,z};\boldsymbol{\theta_{s}})\log(\hat{p}(x_{i,w,h,z};\boldsymbol{\theta_{s}}))$
$\displaystyle+(1-p(x_{i,w,h,z};\boldsymbol{\theta_{s}}))\log(1-\hat{p}(x_{i,w,h,z};\boldsymbol{\theta_{s}}))$

$\hat{e}_{i}$ and $e_{i}$ represent the edge probability map and the
corresponding ground-truth. $\hat{p}(x_{i,w,h,z}$) represents the edge
probability of the $n^{th}$ voxel in $i^{th}$ scan. $\boldsymbol{\theta_{s}}$
represents the weights of the entire deep multi-task encoder-decoder network.

The combined total loss $\mathcal{L}$ is minimized with respect to the
parameters $\boldsymbol{\theta_{s}}$, as shown in Eq. 5. Thus our goal is to
evaluate if a network can learn more robust features and subsequently produce
improved organ segmentations by being trained to explicitly recognize the
boundaries.

$\displaystyle\mathcal{L}(\boldsymbol{\theta_{s}})\;=\;\sum\limits_{i=1}^{M}\mathcal{L}_{RS}\;+\;\lambda\;\sum\limits_{i=1}^{M}\mathcal{L}_{BD}$
(5)

$M$ and $\lambda$ represents the total number of CT scans in the training set
and the weight assigned to the edge detection loss in Eq. 5, respectively.

We hypothesize that the additional boundary loss ($\mathcal{L}_{BD}$) would
impose a larger penalty on erroneous contour voxels, and it subsequently
pushes the optimization of the segmentation network towards the solutions with
more accurate boundaries. Thus, one would potentialize the ability of a
boundary-constrained network to extract features that account for the semantic
abdominal organ regions and boundaries.

###  3.2 Boundary-Constrained Network Topologies

Multi-task learning is generally formulated via hard-parameter sharing and
soft-parameter sharing. In the hard-parameter sharing paradigm, multiple tasks
share a subset of jointly optimized parameters, whereas task-specific
parameters are optimized separately. In soft-parameter sharing, each task is
parameterized using its own set of parameters which are jointly regularized
using constraints [47]. In practice, hard-parameter sharing approaches incur
much less parameter and computational cost. In our work, we formulate the
multi-task learning problem via hard-parameter sharing to train the encoder-
decoder network to do multiple tasks, i.e., organ segmentation and boundary
detection. For deep neural networks, the hard-parameter sharing approach is
realized by sharing some network layers between the tasks while keeping some
layers task-specific.

We explore two different network topologies to conduct multi-task training, as
shown in Figures 2(a) and 2(b). The motivation to explore multiple topologies
is to investigate the impact of sharing the larger and smaller number of
parameters in the network between the two tasks. We explain these multi-task
topologies below.

####  3.2.1 Task-Specific Output Layers (TSOL)

The first multi-task topology that we explore is formulated by appending two
separate prediction layers for predicting the boundaries and semantic organ
masks. This topology employs an encoder-decoder network whose weights are
shared between the tasks, except for the last output layers, as shown in 2(a).
Technically, it encourages the use of compact and tightly shared feature
representations. As evident, this configuration has negligibly fewer more
parameters than the single-task network. We denote this configuration as TSOL.

####  3.2.2 Task-Specific Decoders (TSD)

In second mutli-task topology, we modify the 3D encoder-decoder model to have
a single shared encoder but two separate decoding arms for predicting the
semantic regions and boundaries. The sibling-decoding arms upsample the region
and boundary maps separately. This type of formulation ensures sparse
representation sharing amongst the two tasks since decoders have been
parameterized separately, as shown in 2(b). The presence of two synthesis
paths results in having significantly more parameters than its counterpart
single-task network. We refer to this configuration as TSD.

Figure 3: 3D UNet-MTL-TSOL: Multi-task learning based 3D UNet with task-
specific output layers for simultaneously predicting the organs and their
boundaries. Figure 4: 3D UNet-MTL-TSD: Multi-task learning based 3D UNet with
task-specific decoders for simultaneously predicting the organs and their
boundaries. Figure 5: 3D UNet++-MTL-TSOL: Multi-task learning based 3D UNet++
with task-specific layers for simultaneously predicting the organs and their
boundaries. Figure 6: 3D UNet++-MTL-TSD: Multi-task learning based 3D UNet++
with task-specific decoders for simultaneously predicting the organs and their
boundaries.

###  3.3 3D Baseline models

We use UNet [26], UNet++ [27] and Attention-UNet (Att-UNet) [28] as our
baseline models. We illustrate these models in Supplementary material (see
Figures S1 - S3). These architectures are based on encoder-decoder design and
extended to segment 3D volumes by replacing the 2D convolutions, pooling, and
upsampling with 3D counterparts. Each baseline model processes a 3D input scan
with dimensions $W\times H\times Z$ and outputs a predicted organ-label map of
the same size as input. The encoder of the model contains five convolutional
blocks with pooling layers, and the decoder comprises four upsampling layers.
Each convolutional block in the encoder consists of two convolutional layers
with $3\times 3\times 3$ filters, followed by batch normalization and
Exponential Linear Unit (ELU) activation [48]. We use padded convolutions to
keep the output dimensions of convolutional layers the same as the input
dimensions. A $2\times 2\times 2$ max pooling layer with a stride of two in
each dimension is sandwiched between every two convolutional blocks for
feature maps’ downsampling. The bilinear interpolation layers are used in the
decoder to upsample the extracted feature maps in each dimension. The feature
maps in the decoder are concatenated with the equal-sized representations
learned in the encoder via skip connections. The concatenated feature maps are
then transformed using convolutional blocks, similar to those used in the
encoder. The last $1\times 1\times 1$ convolutional layer maps the feature
channels to the class labels, followed by a softmax activation.

The resolution of the smallest feature map is $9\times 9\times 9$ and the
minimum and maximum feature count at the first and last encoding stage is 16
and 256, respectively. Note that the original UNet++ model is trained with
deep supervision driven by output layers of UNet with varying depths; however,
we train UNet++ without deep supervision to constrain the computational
expense.

Figure 7: 3D Att-UNet-MTL-TSOL: Multi-task learning based 3D Att-UNet with
task-specific prediction layers for simultaneously predicting the organs and
their boundaries. Figure 8: 3D Att-UNet-MTL-TSD: Multi-task learning based 3D
Att-UNet with task-specific decoders for simultaneously predicting the organs
and their boundaries.

###  3.4 3D Boundary-contrained models

To utilize the boundary information of the organs, we train the baseline
models (given in Section 3.3) to predict the organ boundaries along with
organs. We propose two multi-task network topologies (shown in Fig. 2) for
integrating the boundary information and for that, we modify each baseline
model, i.e., 3D UNet, 3D UNet++, and 3D Att-UNet, according to two multi-task
learning-based topologies.

To modify the baseline models according to the first multi-task topology
(TSOL) (shown in 2(a)), we append a separate head at the end to predict
boundaries along with the organs. We refer to these models as UNet-MTL-TSOL,
UNet++-MTL-TSOL, and Att-UNet-MTL-TSOL, as shown in Figures 3, 5 and 7,
respectively. To modify the baseline models according to the second boundary-
constrained multi-task topology (TSD) (shown in 2(b)), we modify the baseline
models (3D UNet, 3D UNet++, and 3D Att-UNet) to have separate decoding paths
followed by prediction layers for predicting boundaries and organs. We refer
to the models modified according to this topology as UNet-MTL-TSD, UNet++-MTL-
TSD, and Att-UNet-MTL-TSD and show them in Figures 4, 6 and 8, respectively.
Note from Fig. 6, we only use the skip connections between the encoder with
the greatest depth and boundary-decoder instead of utilizing feature maps
extracted by nested-UNets of all depths. This design choice is made to
constrain the number of parameters in UNet++-MTL-TSD. Furthermore, observe
from Fig. 8, we do not employ the attention mechanism while decoding the
boundary-features in Att-UNet-MTL-TSD.

##  4 Experimental details

This section first describes the datasets used to validate our study and the
pre-processing we perform on the datasets (Section 4.1), followed by
implementation details (Section 4.2). Finally, we conclude this section by
discussing the metrics used to evaluate baseline and boundary-constrained
models (Section 4.3).

###  4.1 Description of datasets and data preprocessing

We utilize two publically available abdominal CT datasets (Pancreas-CT and
BTCV) to evaluate baseline and boundary-constrained models. Abdominal scans in
Pancreas-CT were acquired at the National Institutes of Health Clinical Center
from pre-nephrectomy healthy kidney donors and subjects with neither major
abdominal pathologies nor pancreatic cancer lesions [29]. The BTCV dataset
consists of abdominal scans acquired at the Vanderbilt University Medical
Center from metastatic liver cancer patients or post-operative ventral hernia
patients [33].

####  4.1.1 Pancreas-CT Dataset (TCIA-43)

The pancreas-CT dataset [29], [49] comprised 82 abdominal contrast-enhanced 3D
CT scans and was initially provided with manually drawn contours of the
pancreas [50], [49]. Recently, 43 scans from this dataset have been re-
annotated to include the segmentation of the liver, duodenum, stomach,
esophagus, spleen, gallbladder, and left kidney [51]. Therefore, we use only
43 scans that have been re-annotated to incorporate labels for multiple
organs.

We first crop a region-of-interest from the CT scans and the corresponding
ground-truth labels using the bounding box coordinates provided with the
dataset [51]. The cropping step ensures the models are only fed with the
foreground inputs without the redundant background region. The cropped region-
of-interest from the CT scans and ground-truth labels are then resampled to a
common dimension of $144\times 144\times 144$ voxels. We randomly divide the
available 43 studies into 28, 5, and 10 for training, validation, and test,
respectively.

####  4.1.2 BTCV Dataset

BTCV was released [30], [33] as a part of a challenge held in conjunction with
MICCAI 2015. The challenge compared the abdominal organs’ segmentation
algorithms on 3D CT scans. Our work focuses on the segmentation of eight
organs from the BTCV dataset, i.e., liver, duodenum, stomach, esophagus,
spleen, gallbladder, left kidney, and pancreas.

For the BTCV dataset, we utilize the bounding box coordinates given with the
dataset for cropping the region-of-interest for both the CT scans and ground-
truth labels [51]. Like the Pancreas-CT dataset, the cropped region-of-
interest is then resampled to a common dimension of $144\times 144\times 144$
voxels. Finally, we randomly divide the available 47 studies into 32, 5, and
10 for training, validation, and test, respectively.

For both Pancreas-CT and BTCV dataset, we applied affine random
transformations to augment the data but did not observe a significant
difference in segmentation performance on the validation set. Therefore, we
did not use any data augmentation. We used the same dataset splits for all the
experiments. To analyze the occurrence of each organ in the dataset, we
present the organs’ occupancy ratio in Figures 9 and 10.

Figure 9: Organ occupancy ratio for Pancreas-ct dataset. Figure 10: Organ
occupancy ratio for BTCV dataset.

###  4.2 Implementation details

All experiments are conducted using Pytorch [52] on two Nvidia Tesla P100,
accessed through the HPC platform111https://supercomputing.swin.edu.au/. We
train all the baseline networks with a mini-batch of size 4, except for 3D
UNet++, which is trained with one batch size. These choices have been made
according to available GPU memory. All baseline (single-task) networks are
trained using multi-class dice loss as expressed in Equations 1, 2 and 3. For
acquiring multi-class dice loss, the 3D predicted organ-segmentation maps are
compared with 3D organ ground-truth maps. All the networks (baseline and
boundary-constrained) are optimized via Adam optimizer [53]. The learning rate
is initially set to 0.001, decaying by a factor of 0.9 after every epoch. To
assess the effect of changing values of training hyperparameters on validation
segmentation performance, we conduct experiments to guide us in selecting the
optimal settings for baseline models. The results for these experiments are
given in Tables S1 and S2 in Supplementary section. We monitor the mean dice
score on the validation set during training and utilize the model for testing
that results in the highest dice coefficient on the validation set.

We use a combination of multi-class dice loss and binary cross-entropy loss,
as illustrated in Eq. 5 for training boundary-constrained models. 3D organ-
boundary predictions are compared with the 3D boundary ground-truths to obtain
the binary cross-entropy loss. Since the datasets do not contain the boundary
annotations of organs, we acquire the ground-truth boundaries by first eroding
the multi-organ ground-truth labels and then taking the difference from the
original ground-truth map. This process gives us boundary annotations of
organs. For UNet-MTL-TSOL, UNet-MTL-TSD, Att-UNet-MTL-TSOL and Att-UNet-MTL-
TSD, we use a batch of size two, whereas, for UNet++-MTL-TSOL and UNet++-MTL-
TSD, we use a single batch size. These choices are made according to the
available GPU memory. Note that the boundary predictions are only used in the
training stage. During the validation stage, we only consider the organ
predictions.

We conduct a grid search (on the range from 0 to 2 with a step of 0.5) to find
the optimal value of $\lambda$ (responsible for balancing the boundary
detection loss). The exact value of $\lambda$ selected to balance the boundary
loss for each boundary-constrained model are given in Tables S3 and S4 in
Supplementary material. These tables also present the standard deviation
between validation dice scores when different values of $\lambda$ are used.

###  4.3 Evaluation metrics

We compare the predicted segmentation masks with the ground-truths to evaluate
the segmentation performance of baseline and boundary-constrained models on
test set. To do that, we utilize Dice Score, Recall, Precision, and Average
Hausdorff Distance as metrics for assessing the quality of predicted
segmentation masks. These metrics are calculated for each organ individually
and then an average is taken across all subjects. All metrics are calculated
by taking a mean of the 5 runs.

| Pancreas-CT dataset | BTCV dataset
---|---|---
Method |  Dice score |  Avg. Hd |  Recall |  Precision |  Dice score |  Avg. Hd |  Recall |  Precision
3D UNet |  0.799± 0.013 |  0.795± 0.088 |  0.786± 0.049 |  0.850± 0.082 |  0.753± 0.008 |  1.085± 0.098 |  0.760± 0.085 |  0.769± 0.116
3D UNet-MTL-TSOL |  0.813± 0.009 |  0.748± 0.130 |  0.807± 0.055 |  0.849± 0.079 |  0.775± 0.004 |  0.989± 0.095 |  0.770± 0.086 |  0.799± 0.106
3D UNet-MTL-TSD |  0.814± 0.005 |  0.703± 0.059 |  0.798± 0.053 |  0.859± 0.079 |  0.776± 0.002 |  0.918± 0.047 |  0.764± 0.081 |  0.802± 0.107
3D UNet++ |  0.772± 0.009 |  1.369± 0.114 |  0.760± 0.089 |  0.825± 0.106 |  0.715± 0.011 |  1.419± 0.088 |  0.707± 0.078 |  0.781± 0.136
3D UNet++-MTL-TSOL |  0.772± 0.014 |  1.170± 0.161 |  0.754± 0.073 |  0.829± 0.110 |  0.741± 0.011 |  1.189± 0.099 |  0.733± 0.080 |  0.789± 0.121
3D UNet++-MTL-TSD |  0.771± 0.008 |  1.308± 0.269 |  0.751± 0.077 |  0.832± 0.111 |  0.723± 0.008 |  1.349± 0.027 |  0.708± 0.084 |  0.783± 0.127
3D Att-UNet |  0.792± 0.015 |  0.825± 0.082 |  0.788± 0.054 |  0.831± 0.093 |  0.752± 0.008 |  1.313± 0.238 |  0.754± 0.091 |  0.778± 0.112
3D Att-UNet-MTL-TSOL |  0.820± 0.004 |  0.673± 0.035 |  0.822± 0.050 |  0.839± 0.076 |  0.769± 0.008 |  1.065± 0.175 |  0.769± 0.086 |  0.790± 0.095
3D Att-UNet-MTL-TSD |  0.807± 0.005 |  0.732± 0.049 |  0.797± 0.050 |  0.847± 0.080 |  0.778± 0.004 |  0.919± 0.052 |  0.759± 0.077 |  0.810± 0.104

Table 1: Quantitative comparison between baseline and boundary-contrained
models for abdominal multi-organ segmentation: Mean values ± Std. Dev. of Dice
Score, Avg. HD, Recall, and Precision are shown. Results in BOLD indicate the
best results corresponding to each baseline.

Method |  No. of parameters |  Inference time (ms)
---|---|---
3D UNet |  5.89M (-) |  44
3D UNet-MTL-TSOL |  5.89M (17) |  53
3D UNet-MTL-TSD |  8.24M (2.35M) |  68
3D UNet++ |  6.87M |  63
3D UNet++ \- MTL-TSOL |  6.87M (17) |  79
3D UNet++ \- MTL-TSD |  9.22M (2.35M) |  81
3D Att-UNet |  6.47M |  77
3D Att-UNet-MTL-TSOL |  6.47M (17) |  78
3D Att-MTL-TSD |  8.82M (2.35M) |  89

Table 2: Comparison of parameter-cost and computational time.

Pancreas-CT dataset

(a)

(b)

(c)

BTCV dataset

(d)

(e)

(f)

Figure 11: Examining the mean dice score computed by comparing predicted
multi-organ segmentation masks with the ground-truth on validation set: First
two rows (a-c) and last two rows (d-f) correspond to the dice score curves for
Pancreas-ct dataset and BTVC dataset, respectively. Each figure compares the
mean DC of the baseline (Blue) and counterpart boundary-constrained model
(Red) on the validation set, computed after each epoch. The baseline models
are trained to predict the masks of the abdominal regions, whereas the
boundary-constrained models are trained to predict the labels of organs and
boundaries.

##  5 Results and Analysis

This section demonstrates the experimental results obtained from boundary-
constrained abdominal segmentation models and compares them against the
performance given by baseline models. For the sake of brevity, we denote the
baseline and boundary-constrained models with the abbreviations below (shown
in bold).

  1. (a)

3D UNet: 3D UNet shown in Fig. S1 in Supplementary material.

  2. (b)

3D UNet-MTL-TSOL: Boundary constrained 3D UNet with task specific output
layers shown in Fig. 3.

  3. (c)

3D UNet-MTL-TSD: Boundary constrained 3D UNet with task specific decoders
shown in Fig. 4.

  4. (d)

3D UNet++: 3D UNet++ shown in Fig. S2 in Supplementary material.

  5. (e)

3D UNet++-MTL-TSOL: Boundary constrained 3D UNet++ with task specific output
layers shown in Fig. 5.

  6. (f)

3D UNet++-MTL-TSD: Boundary constrained 3D UNet++ with task specific decoders
shown in Fig. 6.

  7. (g)

3D Att-UNet: 3D Att-UNet shown in Fig. S3 in Supplementary material.

  8. (h)

3D Att-UNet-MTL-TSOL: Boundary constrained 3D Att-UNet with task specific
output layers shown in Fig. 7.

  9. (i)

3D Att-UNet-MTL-TSD: Boundary constrained 3D Att-UNet with task specific
decoders shown in Fig. 8.

###  5.1 Quantitative Results

Table 1 summarizes the segmentation results for the Pancreas-CT and BTCV
datasets. We report the mean Dice Score, mean Average Hausdorff Distance (Avg.
HD), mean Recall, and mean Precision computed by comparing the predicted
segmentation against the ground-truth. These measures are calculated on test
sets of each dataset. Table 1 shows that the boundary-constrained models
achieve improved multi-organ segmentation on the abdominal CT scans. Firstly,
the mean Dice scores are improved by 1.8% (UNet vs. UNet-MTL-TSOL) and 3.5%
(Att-UNet vs. Att-UNet-MTL-TSOL), for Pancreas-CT dataset. The corresponding
values of mean dice scores for BTCV dataset are improved by 3.1% (UNet vs.
UNet-MTL-TSD), 3.6% (UNet++ vs. UNet++-MTL-TSOL), and 3.5% (Att-UNet vs. Att-
UNet-MTL-TSD). The improved overlap between predicted segmentations and
manually annotated masks can be attributed to the enhanced semantic
representations learned by boundary-constrained models.

Secondly, we observe that boundary-constrained models achieve a lower Avg. HDs
for all the datasets than those obtained from baseline models as shown in
Table 1. For example, after adding boundary information, the Avg. HD values of
UNet, UNet++, and Att-UNet are decreased by 11.5%, 14.5%, and 18.4%,
respectively, for the Pancreas-CT dataset. Likewise, a decrease of 15.4%,
16.2%, and 30%, respectively, is seen for the BTCV dataset. Furthermore, we
notice that the Avg. HD is still lower for the cases where boundary-
constrained models obtained lower or equivalent mean Dice score, e.g.,
UNet++-MTL-TSOL vs. UNet++ and UNet++-MTL-TSD vs. UNet++, for Pancreas-CT
results. This indicates that even for the equivalent dice overlap, the
performance of the boundary-constrained models in accurately predicting the
boundaries is improved.

Thirdly, our boundary-constrained models achieve higher values of mean Recall
and mean Precision for all the models and datasets except for UNet++
corresponding to the Pancreas-CT dataset, as shown in Table 1. The utilization
of boundary information has caused a decrease in false-negative rates and
false-positive rates. Specifically, the mean recall is increased by 1.3% (UNet
vs. UNet-MTL-TSOL) and 4.3% (Att-UNet vs. Att-UNet-MTL-TSOL) for Pancreas-CT
dataset. For BTCV dataset, an increase of 1.3% (UNet vs. UNet-MTL-TSOL), 3.7%
(UNet++-MTL-TSOL vs. UNet++), and 1.9% (Att-UNet vs. Att-UNet-MTL-TSOL) is
observed in Mean Recalls. Finally, we see a maximum improvement of 1.9% and
4.3% in values of mean Precision for Pancreas-CT and BTCV datasets,
respectively. The improvment in mean Recalls and mean Precisions show the
capability of boundary-constrained models in addressing the issues of under-
segmentation and over-segmentation.

Method |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
| Pancreas-CT dataset
3D UNet |  0.935± 0.008 |  0.700± 0.035 |  0.927± 0.014 |  0.793± 0.030 |  0.714± 0.024 |  0.934± 0.009 |  0.829± 0.024 |  0.566± 0.031
3D UNet-MTL-TSD |  0.931± 0.008 |  0.723± 0.028 |  0.937± 0.004 |  0.820± 0.028 |  0.718± 0.014 |  0.947± 0.003 |  0.854± 0.015 |  0.583± 0.021
3D UNet++ |  0.917± 0.006 |  0.613± 0.053 |  0.897± 0.019 |  0.802± 0.026 |  0.712± 0.013 |  0.899± 0.012 |  0.818± 0.014 |  0.516± 0.042
3D UNet++ \- MTL-TSOL |  0.915± 0.005 |  0.621± 0.042 |  0.915± 0.013 |  0.774± 0.051 |  0.697± 0.013 |  0.913± 0.005 |  0.818± 0.032 |  0.519± 0.035
3D Att-UNet |  0.927± 0.007 |  0.688± 0.021 |  0.927± 0.013 |  0.772± 0.037 |  0.700± 0.021 |  0.929± 0.016 |  0.827± 0.029 |  0.565± 0.036
3D Att-UNet-MTL-TSOL |  0.937± 0.008 |  0.724± 0.010 |  0.927± 0.009 |  0.858± 0.017 |  0.734± 0.005 |  0.949± 0.004 |  0.847± 0.018 |  0.589± 0.023
| BTCV dataset
3D UNet |  0.878± 0.016 |  0.659± 0.018 |  0.897± 0.014 |  0.567± 0.020 |  0.698± 0.009 |  0.932± 0.008 |  0.813± 0.012 |  0.576± 0.023
3D UNet-MTL-TSD |  0.897± 0.011 |  0.682± 0.009 |  0.912± 0.004 |  0.608± 0.019 |  0.732± 0.016 |  0.937± 0.007 |  0.837± 0.008 |  0.599± 0.011
3D UNet++ |  0.846± 0.016 |  0.559± 0.032 |  0.851± 0.030 |  0.552± 0.023 |  0.691± 0.014 |  0.882± 0.008 |  0.799± 0.015 |  0.538± 0.028
3D UNet++ \- MTL-TSOL |  0.853± 0.023 |  0.625± 0.018 |  0.891± 0.019 |  0.575± 0.019 |  0.694± 0.014 |  0.918± 0.004 |  0.829± 0.018 |  0.549± 0.032
3D Att-UNet |  0.881± 0.020 |  0.668± 0.016 |  0.882± 0.036 |  0.561± 0.009 |  0.704± 0.023 |  0.939± 0.006 |  0.805± 0.007 |  0.575± 0.007
3D Att-UNet-MTL-TSD |  0.913± 0.009 |  0.674± 0.006 |  0.920± 0.015 |  0.603± 0.034 |  0.720± 0.009 |  0.947± 0.007 |  0.832± 0.022 |  0.608± 0.025

Table 3: Organ-wise Mean Dice scores ± Std. Dev. obtained from baseline and
best performing boundary-constrained models: Bold shows the highest value
corresponding to each baseline.

Fig. 11 shows the performance curves based on the mean Dice scores, computed
by comparing the predicted multi-organ segmentation with the ground-truth
masks on the validation set. Figures 11(a), 11(b) and 11(c) correspond to
performance curves for the Pancreas-CT and Figures 11(d), 11(e) and 11(f) for
the BTCV dataset. It can be seen that the incorporation of boundary
information has improved the mean Dice score as the training progresses.

Method |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
| Pancreas-CT dataset
3D UNet |  0.261± 0.085 |  0.936± 0.279 |  0.184± 0.143 |  0.803± 0.356 |  0.788± 0.275 |  0.234± 0.063 |  0.834± 0.234 |  2.321± 0.316
3D UNet-MTL-TSD |  0.317± 0.083 |  0.911± 0.244 |  0.150± 0.036 |  0.389± 0.074 |  0.683± 0.072 |  0.259± 0.122 |  0.879± 0.304 |  2.038± 0.204
3D UNet++ |  0.419± 0.069 |  1.852± 0.554 |  0.265± 0.153 |  1.569± 0.649 |  1.445± 0.494 |  0.711± 0.218 |  1.371± 0.457 |  3.315± 0.637
3D UNet++ \- MTL-TSOL |  0.389± 0.055 |  1.718± 0.638 |  0.167± 0.048 |  0.873± 0.229 |  1.187± 0.328 |  0.675± 0.104 |  1.296± 0.483 |  3.058± 0.948
3D Att-UNet |  0.363± 0.063 |  0.928± 0.085 |  0.174± 0.106 |  0.527± 0.099 |  0.809± 0.277 |  0.442± 0.175 |  0.969± 0.336 |  2.384± 0.466
3D Att-UNet-MTL-TSOL |  0.255± 0.041 |  0.993± 0.315 |  0.192± 0.033 |  0.305± 0.075 |  0.559± 0.022 |  0.154± 0.054 |  0.766± 0.274 |  2.162± 0.434
| BTCV dataset
3D UNet |  0.474± 0.161 |  1.741± 0.179 |  0.337± 0.077 |  1.750± 0.334 |  0.770± 0.140 |  0.636± 0.206 |  0.941± 0.155 |  2.032± 0.149
3D UNet-MTL-TSD |  0.329± 0.105 |  1.787± 0.326 |  0.253± 0.059 |  1.076± 0.136 |  0.648± 0.168 |  0.619± 0.226 |  0.778± 0.123 |  1.849± 0.199
3D UNet++ |  0.780± 0.220 |  2.423± 0.510 |  0.574± 0.282 |  2.103± 0.284 |  1.034± 0.187 |  0.984± 0.136 |  1.153± 0.148 |  2.306± 0.206
3D UNet++ \- MTL-TSOL |  0.794± 0.089 |  1.842± 0.518 |  0.328± 0.087 |  1.630± 0.479 |  0.908± 0.221 |  1.080± 0.063 |  0.860± 0.248 |  2.073± 0.265
3D Att-UNet |  0.577± 0.245 |  2.017± 0.418 |  0.404± 0.154 |  2.889± 1.387 |  0.981± 0.659 |  0.435± 0.162 |  1.011± 0.133 |  2.193± 0.249
3D Att-UNet-MTL-TSD |  0.273± 0.055 |  1.377± 0.123 |  0.191± 0.062 |  1.631± 0.456 |  0.887± 0.248 |  0.318± 0.202 |  0.789± 0.149 |  1.888± 0.318

Table 4: Organ-wise Mean Avg. HDs ± Std. Dev. obtained from baseline and best
performing boundary-constrained models: Bold shows the highest value
corresponding to each baseline.

###  5.2 Computational Complexity and Architectural Analysis

Table 2 reports parameter count (in million M) and each method’s time to
segment a single CT abdominal volume in the test phase. We also highlight the
increase in parameter count (given in brackets) compared with the baseline
model. Among the single-task baseline models, UNet++ is most parameter-
extensive with 6.87 $\times 10^{6}$ parameters. The parameter count of
boundary-constrained models with TSOL topology have only 17 parameters more
than the baseline models; however, these models showed significantly
considerable improvements in the segmentation of abdominal organs. The shared
encoder and decoder in the TSOL design enable the small parameter count while
capacitating the segmentation algorithm to learn the masks and boundaries
using separate task-specific output layers. Observing the second multi-task
topology TSD, our model requires approximately 2.4 M more parameters than the
baseline.

Furthermore, we can see (from Table 1) that the boundary-constrained models
take more time to segment a single volume at the inference stage. This
behavior can be associated with the extended parameter size of boundary-
constrained models since they are trained to predict a 3D boundary of organs
in addition to region masks.

Analyzing the relationship between the multi-task network design and the
segmentation performance from Table 1, we note there is not a single/fixed
topology that led to the maximum improvement. For example, the TSD showed
maximum improvement in mean DC and Avg HD corresponding to the Pancreas-CT
dataset. Hd over the baseline UNet. However, when the performance of Att-UNet
is compared with boundary-constrained models, we notice that the TSOL
demonstrated the best results. This reveals that the multi-task network
configuration that offers the best results varies depending on each baseline
architecture. All in all, we found that integration of boundary information
improved the multi-organ segmentation, independent of the network topology.

###  5.3 Assesment of organ-wise segmentation performance

To assess which specific organs benefitted greatly from incorporating boundary
information for the segmentation task, we examine the mean Dice scores and
mean Avg. HDs achieved by baseline, and best performing boundary constrained
models (from Table 1) for each abdominal organ. We compute the Dice scores and
Average Hausdorff distances for each organ individually and then average
across all subjects. From Table 3, we can see both baseline and boundary-
constrained models have yielded the highest mean Dice scores for liver (  ),
spleen (  ), and kidney (  ) and lowest for duodenum (  ). However, boundary-
based models have led to the maximum relative improvement for the gallbladder
(  ), pancreas (  ), and duodenum (  ). From Table 4, we observe that the
boundary-constrained models have significantly improved the mean Avg. HD
distances for the spleen (  ), kidney (  ), and gallbladder (  ). Finally,
relating the increase in dice overlap to the organ occurrence (shown in
Figures 9 and 10), we observe that the most significant improvement has
occurred for the underweighted classes. In contrast, the boundary distances
are maximally decreased for both small (e.g., gallbladder (  ) and large
structures like spleen (  ). Furthermore, the boundary-constrained models led
to lower standard deviations of the mean Dice scores and Avg. HDs across
different subjects which show the robustness of proposed models.

(a)

(b)

(c)

(d)

Figure 12: Trimap regions along boundary of organs shown on 2D abdominal
images. (a) Original abdominal image, (b) Organ label-maps corresponding to 2D
image, (c) Trimap region of 5 voxel width around boundaries of organs shown in
gray color, and (d) Trimap region of 7 voxel width shown in gray color.

(a)

(b)

(c)

Figure 13: Evaluation of boundary segmentation via trimaps. In each subfigure,
we have plotted the dice scores (y-axis) obtained by comparing the predicted
and groundtruth segmentations in trimap regions with varying number of voxels
(x-axis). The dice scores for both the baseline (green) and boundary-
constrained counterparts (purple) are plotted. Top row of (a-c) corresponds to
trimap segmentation comparison for Pancreas-CT dataset and bottom row
corresponds to BTCV dataset.

###  5.4 Segmentation performance along boundary of organs

Unclear boundaries of organs and low inter-organ contrast prevent accurate
segmentation of challenging regions around the organ boundaries on abdominal
CT scans. To assess if the incorporation of boundary information has improved
the segmentation of voxels within the close vicinity of organ boundaries, we
evaluate the quality of predicted voxel-labels within these regions and
compare them against the ones acquired via baseline methods. To do this, we
generate trimaps [54] [55] with different voxel-bands surrounding the
boundaries of organs. Trimap is a narrow region along the boundary of an
object which is utilized to evaluate the quality of segmentation within a
specific distance from the object’s contour. First, we generate trimap regions
around the boundary of organs for predicted and ground-truth segmentations,
and then, we compare the 3D trimaps by computing mean Dice scores between
them. We show the exemplary trimap regions on 2D abdominal axial slices in
Fig. 12. For the sake of compendious presentation, we have computed the trimap
Dice scores only for TSOL network topology. Fig. 13 shows the mean Dice score
plotted against the number of voxels the trimap contains. The top row shows
trimap plots for the Pancreas-CT dataset, whereas the bottom row shows the
trimap plots for the BTCV dataset. Our proposed boundary-constrained models
consistently perform better than the baseline models in predicting the
semantic labels within the vicinity of organ-boundaries, except for one case,
i.e., (3D UNet++ vs 3D UNet++-MTL-TSOL).

###  5.5 Qualitative Results

Fig. 14 shows semantic segmentation predictions on a single 2D axial slice of
the 3D scans. The first and second row correspond to segmentation results from
the Pancreas-CT dataset whereas, the third and fourth row corresponds to
results from the BTCV dataset. Each column (from left to right) illustrates
the original abdominal 2D images (Fig. 14), ground-truth masks (Fig. 14),
baseline model (UNet and Att-UNet) segmentations (Fig. 14), and segmentations
acquired from the boundary-constrained counterparts (UNet-MTL-TSD and Att-
UNet-MTL-TSOL) in (Fig. 14). We observe that the baseline models led to under-
segmentations and over-segmentations, indicated in white boxes in Fig. 14.
Furthermore, the segmentations generated by single-task baseline models show
isolated and biologically implausible organs’ parts. Moreover, comparing with
the corresponding boundary-constrained segmentations (Fig. 14), the
incorporation of boundary information has prevented the issue of
mispredictions near boundary of organs and led to generation of biologically
plausible segmentations. Fig. 15 presents the 3D segmentations generated by
baseline (UNet) and boundary-constrained model (UNet-MTL-TSD) along with the
ground-truths from the Pancreas-CT (first row) and BTCV dataset (second row).
Notice that the boundary-constrained segmentations (third column) are more
similar to the ground-truth masks (first column) as compared to the baseline
segmentations (second column). These qualitative results show the improvements
induced by the use of organs borders in training the 3D fully convolutional
models for abdominal organs segmentation.

Figure 14: Qualitative results for multi-organ segmentation between baseline
and boundary-constrained models are shown. Rows 1-2 correspond to results for
the Pancreas-CT dataset, whereas Rows 3-4 show results for the BTCV dataset.
Columns 1-2 show the original image and the corresponding ground-truth mask
overlayed on the 2D image, i.e., spleen (  ), left kidney (  ), gallbladder (
), liver (  ), stomach (  ), and duodenum (  ). Columns 3-4 illustrate the
segmentation results related to baseline UNet and UNet-MTL-TSOL counterparts.
White boxes indicate the segmented regions improved by incorporation of
boundary information.

Figure 15: Visualization of 3D segmentation predictions using Pancreas-CT and
BTCV dataset. First column corresponds to abdominal groundtruth annotations,
i.e., spleen (  ), pancreas (  ), left kidney (  ), gallbladder (  ),
esophagus (  ), liver (  ), stomach (  ), and duodenum (  ), whereas second
and third column corresponds to volume labels predicted by UNet and UNet-MTL-
TSD.

##  6 Discussion

Accurate segmentation of abdominal organs from CT scans is required for
numerous advanced clinical procedures such as computer-assisted surgery and
organ transplantation. The low-contrasted appearance and weak edges of
abdominal organs in CT scans adversely affect the accurate segmentation. In
this paper, we propose to leverage boundary information of organs as an
additional cue for improved 3D abdominal multi-organ segmentation. The
boundary-constrained encoder-decoder network simultaneously learns to
delineate the semantic abdominal regions and detect the boundaries of organs.
This multi-task learning model exploits the statistics from more than one
ground-truth source and subsequently retains features shared between the
tasks. The boundary annotations of abdominal organs can be easily generated
from the ground-truth masks and provide cost-free additional knowledge about
the organs.

As reported by quantitative results in Table 1, the proposed boundary-
constrained 3D encoder-decoder models achieve improved multi-organ
segmentation across the majority of the baselines (3D UNet, 3D UNet++, and 3D
Att-UNet) and datasets (Pancreas-CT and BTCV). We have shown that the
significant improvement in segmentation performance evaluated via Dice Score,
Avg. HD, Recall, and Precision is caused by the improved segmentation of
organs’ boundaries and regions surrounding boundaries. Furthermore,
significant improvements with a negligible increase in parameter count
(0.0002% -TSOL topology) reveal the benefit of regularizing the existing
encoder-decoder segmentation models using boundary information.

The reduction in Avg. HD for both datasets across all baselines depicts the
advantage of informing the model about the vulnerable regions of the organ.
The dramatic decrease in Avg. HD, ranging from 18% to 30%, shows that the
model learned feature combinations that were expressive about the entire
appearance of organs. We believe that training the models with auxiliary
knowledge encourages learning more generalizable and discriminative features.
Notably, the additional experiments that we have conducted to precisely assess
the improvements in segmentation of regions in the vicinity of organ-
boundaries further verify the superiority of training the segmentation model
with complementary boundary information (shown in Fig. 15).

Since there may exist several different configurations through which a fully
convolutional architecture can be designed under a multi-task learning
paradigm, we explore two network topologies based on the extent of parameter-
sharing between the tasks. Through our extensive comparison, we notice that an
overly-shared multi-task network (TSOL) performs on par with the network
designed to have an increased number of task-specific layers (TSD) (Table 1).
This indicates that models with many parameters do not necessarily correspond
to higher performance. Most importantly, we also found that incorporation of
boundary information improved the multi-organ segmentation performance,
regardless of the network topology. One of the critical challenges in
designing 3D multi-task deep learning models is determining which layers
should be shared while keeping the computational expense reasonable. In the
future, we aim to investigate other network topologies for training the
encoder-decoder model in a multi-task learning fashion. Another valuable
extension of our work is to develop a mechanism/policy that can automatically
dictate the sharing pattern of network layers between the two tasks.

As reported in Tables 3 and 4, the improvement in the segmentation performance
of organs rarely occurring in the dataset reveals the efficacy of boundary-
regularized models in compensating for their rare presence. The qualitative
results shown in Figures 14 and 15 verify the positive impact of making the
model aware of organ-boundaries during training. The ability of our model to
simultaneously learn the improved representations of multiple organs is
indicated by the qualitative examples in Figures 14 and 15, where the
boundary-constrain has reduced the occurrence of over- and under-segmented
organs.

##  7 Conclusion

In this paper, we leverage organ boundary information for an improved 3D
abdominal multi-organ segmentation by addressing the challenge of unclear
boundaries in low-contrasted CT scans. We demonstrate that boundary
information can be seamlessly introduced in the training of 3D encoder-decoder
models through different multi-task configurations. The experimental results
show the boundary-constrained multi-organ segmentation outperforms the ones
obtained from several FCN-based baseline models, including 3D UNet, 3D UNet++,
3D Attention-UNet. Furthermore, we found that the multi-task topology that
shows maximum improvement is not fixed and varies depending on the baseline
architecture. This insight shows that the optimal utilization of auxiliary
information cannot always be harvested through a single deep multi-task design
but instead requires the exploration of different multi-task topologies. Our
findings also reveal that leveraging organs boundary features improves the
segmentation of underweighted organs like the gallbladder, pancreas, and
duodenum with a negligible parameter-overhead. Additionally, the experimental
results disclose that the boundary-constrained models improve the labelling of
weak sub-parts of organs in the vicinity of boundaries. We believe the
proposed 3D boundary-constrained models would be valuable for enhancing
abdominal organ segmentation and utilizing those segmentations in relevant
clinical applications.

Conflict of interest statement: On behalf of all authors, the corresponding
author states that there is no conflict of interest.

## References

  * [1] T. Okada, M. G. Linguraru, M. Hori, Y. Suzuki, R. M. Summers, N. Tomiyama, Y. Sato, Multi-organ segmentation in abdominal ct images, in: 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, IEEE, 2012, pp. 3986–3989.
  * [2] E. Gibson, F. Giganti, Y. Hu, E. Bonmati, S. Bandula, K. Gurusamy, B. Davidson, S. P. Pereira, M. J. Clarkson, D. C. Barratt, Automatic multi-organ segmentation on abdominal ct with dense v-networks, IEEE transactions on medical imaging 37 (8) (2018) 1822–1834.
  * [3] P. Hu, F. Wu, J. Peng, Y. Bao, F. Chen, D. Kong, Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets, International journal of computer assisted radiology and surgery 12 (3) (2017) 399–411.
  * [4] H. Kim, J. Jung, J. Kim, B. Cho, J. Jang, S.-w. Lee, J.-G. Lee, S. Yoon, Abdominal multi-organ auto-segmentation using 3d-patch-based deep convolutional neural network, Scientific Reports 10 (2020) 6204.  doi:10.1038/s41598-020-63285-0.
  * [5] Z. Xu, R. P. Burke, C. P. Lee, R. B. Baucom, B. K. Poulose, R. G. Abramson, B. A. Landman, Efficient multi-atlas abdominal segmentation on clinically acquired ct with simple context learning, Medical image analysis 24 (1) (2015) 18–27.
  * [6] M. Suzuki, M. G. Linguraru, K. Okada, Multi-organ segmentation with missing organs in abdominal ct images, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2012, pp. 418–425.
  * [7] J. J. Cerrolaza, M. Reyes, R. M. Summers, M. Á. González-Ballester, M. G. Linguraru, Automatic multi-resolution shape modeling of multi-organ structures, Medical image analysis 25 (1) (2015) 11–21.
  * [8] T. Okada, M. G. Linguraru, M. Hori, R. M. Summers, N. Tomiyama, Y. Sato, Abdominal multi-organ segmentation from ct images using conditional shape–location and unsupervised intensity priors, Medical image analysis 26 (1) (2015) 1–18.
  * [9] P. Campadelli, E. Casiraghi, S. Pratissoli, G. Lombardi, Automatic abdominal organ segmentation from ct images, ELCVIA: electronic letters on computer vision and image analysis 8 (1) (2009) 1–14.
  * [10] M. A. Selver, Segmentation of abdominal organs from ct using a multi-level, hierarchical neural network strategy, Computer methods and programs in biomedicine 113 3 (2014) 830–52.
  * [11] M. P. Heinrich, O. Oktay, N. Bouteldja, Obelisk-net: Fewer layers to solve 3d multi-organ segmentation with sparse deformable convolutions, Medical image analysis 54 (2019) 1–9.
  * [12] M. F. Bobo, S. Bao, Y. Huo, Y. Yao, J. Virostko, A. J. Plassard, I. Lyu, A. Assad, R. G. Abramson, M. A. Hilmes, et al., Fully convolutional neural networks improve abdominal organ segmentation, in: Medical Imaging 2018: Image Processing, Vol. 10574, International Society for Optics and Photonics, 2018, p. 105742V.
  * [13] A. Sinha, J. Dolz, Multi-scale self-guided attention for medical image segmentation, IEEE Journal of Biomedical and Health Informatics (2020).
  * [14] H. Lu, S. Tian, L. Yu, L. Liu, J. Cheng, W. Wu, X. Kang, D. Zhang, DCACNet: Dual context aggregation and attention-guided cross deconvolution network for medical image segmentation, Computer Methods and Programs in Biomedicine 214 (2022) 106566.  doi:10.1016/j.cmpb.2021.106566.
  * [15] O. Ronneberger, P. Fischer, T. Brox, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention, Springer, 2015, pp. 234–241.
  * [16] H. R. Roth, H. Oda, X. Zhou, N. Shimizu, Y. Yang, Y. Hayashi, M. Oda, M. Fujiwara, K. Misawa, K. Mori, An application of cascaded 3d fully convolutional networks for medical image segmentation, Computerized medical imaging and graphics : the official journal of the Computerized Medical Imaging Society 66 (2018) 90–99.
  * [17] X. Zhou, K. Yamada, T. Kojima, R. Takayama, S. Wang, X. Zhou, T. Hara, H. Fujita, Performance evaluation of 2d and 3d deep learning approaches for automatic segmentation of multiple organs on ct images, in: Medical Imaging, 2018\.
  * [18] W. Guo, G. Zhang, Z. Gong, Q. Li, Effective integration of object boundaries and regions for improving the performance of medical image segmentation by using two cascaded networks, Computer Methods and Programs in Biomedicine 212 (2021) 106423.  doi:10.1016/j.cmpb.2021.106423.
  * [19] H. Chen, X. Qi, L. Yu, P.-A. Heng, Dcan: deep contour-aware networks for accurate gland segmentation, in: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2016, pp. 2487–2496.
  * [20] C. Tan, L. Zhao, Z. Yan, K. Li, D. Metaxas, Y. Zhan, Deep multi-task and task-specific feature learning network for robust shape preserved organ segmentation, in: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), 2018, pp. 1221–1224.  doi:10.1109/ISBI.2018.8363791.
  * [21] R. Zhao, W. Chen, G. Cao, Edge-boosted u-net for 2d medical image segmentation, IEEE Access 7 (2019) 171214–171222.  doi:10.1109/ACCESS.2019.2953727.
  * [22] B. Shi, H. Zhang, R. Yan, W. Jing, J. Zang, F. Zhang, Lr-net: A multi-task model using relationship-based contour information to enhance the semantic segmentation of cancer regions, in: 2020 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), 2020, pp. 2371–2378.  doi:10.1109/BIBM49941.2020.9313420.
  * [23] L. Mei, X. Guo, X. Huang, Y. Weng, S. Liu, C. Lei, Dense contour-imbalance aware framework for colon gland instance segmentation, Biomedical Signal Processing and Control 60 (2020) 101988.  doi:https://doi.org/10.1016/j.bspc.2020.101988.
URL https://www.sciencedirect.com/science/article/pii/S1746809420301440

  * [24] A. Amyar, R. Modzelewski, H. Li, S. Ruan, Multi-task deep learning based ct imaging analysis for covid-19 pneumonia: Classification and segmentation, Computers in Biology and Medicine 126 (2020) 104037 – 104037.
  * [25] K. Du, X. Lin, Y. Sun, X. Ma, Crossinfonet: Multi-task information sharing based hand pose estimation, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019) 9888–9897.
  * [26] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, O. Ronneberger, 3d u-net: learning dense volumetric segmentation from sparse annotation, in: International conference on medical image computing and computer-assisted intervention, Springer, 2016, pp. 424–432.
  * [27] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, J. Liang, Unet++: Redesigning skip connections to exploit multiscale features in image segmentation, IEEE transactions on medical imaging 39 (6) (2019) 1856–1867.
  * [28] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, D. Rueckert, Attention gated networks: Learning to leverage salient regions in medical images, Medical image analysis 53 (2019) 197–207.
  * [29] H. Roth, A. Farag, E. B. Turkbey, L. Lu, J. Liu, R. M. Summers, Data from pancreas-ct (2016).  doi:10.7937/K9/TCIA.2016.TNB1KQBU.
  * [30] B. Landman, Z. Xu, J. E. Igelsias, M. Styner, T. R. Langerak, A. Klein, Miccai multi-atlas labeling beyond the cranial vault - workshop and challenge. 2015, accessed August 2020.
  * [31] T. Heimann, H.-P. Meinzer, Statistical shape models for 3d medical image segmentation: a review, Medical image analysis 13 (4) (2009) 543–563.
  * [32] T. F. Cootes, G. J. Edwards, C. J. Taylor, Active appearance models, IEEE Transactions on pattern analysis and machine intelligence 23 (6) (2001) 681–685.
  * [33] Z. Xu, C. P. Lee, M. P. Heinrich, M. Modat, D. Rueckert, S. Ourselin, R. G. Abramson, B. A. Landman, Evaluation of six registration methods for the human abdomen on clinically acquired ct, IEEE Transactions on Biomedical Engineering 63 (8) (2016) 1563–1572.
  * [34] H. Lombaert, D. Zikic, A. Criminisi, N. Ayache, Laplacian forests: semantic image segmentation by guided bagging, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2014, pp. 496–504.
  * [35] Z. Peng, X. Fang, P. Yan, H. Shan, T. Liu, X. Pei, G. Wang, B. Liu, M. Kalra, X. G. Xu, A method of rapid quantification of patient-specific organ dose for ct using coupled deep multi-organ segmentation algorithms and gpu-accelerated monte carlo dose computing code, arXiv preprint arXiv:1908.00360 (2019).
  * [36] Z. Chen, Z. Chen, Rbnet: A deep neural network for unified road and road boundary detection, in: International Conference on Neural Information Processing, Springer, 2017, pp. 677–687.
  * [37] H. Dong, G. Yang, F. Liu, Y. Mo, Y. Guo, Automatic brain tumor detection and segmentation using u-net based fully convolutional networks, ArXiv abs/1705.03820 (2017).
  * [38] L. Zhang, J. Zhang, P. Shen, G. Zhu, P. Li, X. Lu, H. Zhang, S. A. A. Shah, M. Bennamoun, Block level skip connections across cascaded v-net for multi-organ segmentation, IEEE Transactions on Medical Imaging 39 (2020) 2782–2793.
  * [39] Z. Peng, X. Fang, P. Yan, H. Shan, T. Liu, X. Pei, G. Wang, B. Liu, M. K. Kalra, X. G. Xu, A method of rapid quantification of patient-specific organ doses for ct using deep-learning based multi-organ segmentation and gpu-accelerated monte carlo dose computing., Medical physics (2020).
  * [40] Y. Liu, Y. Lei, Y. Fu, T. Wang, X. Tang, X. Jiang, W. J. Curran, T. Liu, P. Patel, X. Yang, Ct-based multi-organ segmentation using a 3d self-attention u-net network for pancreatic radiotherapy, Medical physics 47 (9) (2020) 4316–4324.
  * [41] H. J. Lee, J. U. Kim, S. Lee, H. G. Kim, Y. M. Ro, Structure boundary preserving segmentation for medical image with ambiguous boundary, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [42] H. Oda, H. Roth, K. Chiba, J. Sokolic, T. Kitasaka, M. Oda, A. Hinoki, H. Uchida, J. A. Schnabel, K. Mori, Besnet: Boundary-enhanced segmentation of cells in histopathological images, in: MICCAI, 2018.
  * [43] B. Murugesan, K. Sarveswaran, S. M. Shankaranarayana, K. Ram, M. Sivaprakasam, Psi-net: Shape and boundary aware joint multi-task deep network for medical image segmentation, 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2019) 7223–7226.
  * [44] Z. Zhang, H. Fu, H. Dai, J. Shen, Y. Pang, L. Shao, Et-net: A generic edge-attention guidance network for medical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2019, pp. 442–450.
  * [45] L. Wang, H. Zhen, X. Fang, S. Wan, W. Ding, Y. Guo, A unified two-parallel-branch deep neural network for joint gland contour and segmentation learning, Future Gener. Comput. Syst. 100 (2019) 316–324.
  * [46] F. Milletari, N. Navab, S.-A. Ahmadi, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in: 2016 fourth international conference on 3D vision (3DV), IEEE, 2016, pp. 565–571.
  * [47] S. Ruder, An overview of multi-task learning in deep neural networks, ArXiv abs/1706.05098 (2017).
  * [48] D. Clevert, T. Unterthiner, S. Hochreiter, Fast and accurate deep network learning by exponential linear units (elus), in: Y. Bengio, Y. LeCun (Eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [49] H. R. Roth, L. Lu, A. Farag, H. Shin, J. Liu, E. B. Turkbey, R. M. Summers, Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation, in: Medical Image Computing and Computer-Assisted Intervention \- MICCAI 2015 - 18th International Conference Munich, Germany, October 5-9, 2015, Proceedings, Part I, Vol. 9349 of Lecture Notes in Computer Science, Springer, 2015, pp. 556–564.
  * [50] K. Clark, B. Vendt, K. Smith, J. Freymann, J. Kirby, P. Koppel, S. Moore, S. Phillips, D. Maffitt, M. Pringle, L. Tarbox, F. Prior, The cancer imaging archive (TCIA): Maintaining and operating a public information repository, Journal of Digital Imaging 26 (6) (2013) 1045–1057.  doi:10.1007/s10278-013-9622-7.
  * [51] E. Gibson, F. Giganti, Y. Hu, E. Bonmati, S. Bandula, K. Gurusamy, B. Davidson, S. P. Pereira, M. J. Clarkson, D. C. Barratt, Multi-organ abdominal ct reference standard segmentations (2018).  doi:10.5281/ZENODO.1169361.
  * [52] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch, in: NIPS-W, 2017.
  * [53] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [54] P. Kohli, L. Ladicky, P. H. S. Torr, Robust higher order potentials for enforcing label consistency, in: 2008 IEEE Conference on Computer Vision and Pattern Recognition, 2008, pp. 1–8.  doi:10.1109/CVPR.2008.4587417.
  * [55] B. Cheng, R. B. Girshick, P. Doll’ar, A. C. Berg, A. Kirillov, Boundary iou: Improving object-centric image segmentation evaluation, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2021) 15329–15337.