# HMRNet: High and Multi-Resolution Network with Bidirectional Feature
Calibration for Brain Structure Segmentation in Radiotherapy

Hao Fu  Guotai Wang  Wenhui Lei  Wei Xu  Qianfei Zhao  Shichuan Zhang  Kang Li
and Shaoting Zhang This work was supported by the National Natural Science
Foundation of China under Grant 81771921 and Grant 61901084. (Corresponding
authors: Guotai Wang; Shaoting Zhang.) Hao Fu, Guotai Wang, Wenhui Lei, Wei Xu
and Qianfei Zhao are with the School of Mechanical and Electrical Engineering,
University of Electronic Science and Technology of China, Chengdu, 611731,
China. Guotai Wang is also with Shanghai AI laboratory, Shanghai, 200030,
China. (e-mail: guotai.wang@uestc.edu.cn)Shichuan Zhang is with Department of
Radiation Oncology, Sichuan Cancer Hospital and Institute, University of
Electronic Science and Technology of China, Chengdu, 610042, China. Kang Li is
with West China Hospital-SenseTime Joint Lab, West China Biomedical Big Data
Center, Sichuan University West China Hospital, Chengdu, 610041, China.
Shaoting Zhang is with the School of Mechanical and Electrical Engineering,
University of Electronic Science and Technology of China, Chengdu, 611731,
China, and also with SenseTime Research, Shanghai, 200233, China. (e-mail:
zhangshaoting@uestc.edu.cn).

###### Abstract

Accurate segmentation of Anatomical brain Barriers to Cancer spread (ABCs)
plays an important role for automatic delineation of Clinical Target Volume
(CTV) of brain tumors in radiotherapy. Despite that variants of U-Net are
state-of-the-art segmentation models, they have limited performance when
dealing with ABCs structures with various shapes and sizes, especially thin
structures (e.g., the falx cerebri) that span only few slices. To deal with
this problem, we propose a High and Multi-Resolution Network (HMRNet) that
consists of a multi-scale feature learning branch and a high-resolution
branch, which can maintain the high-resolution contextual information and
extract more robust representations of anatomical structures with various
scales. We further design a Bidirectional Feature Calibration (BFC) block to
enable the two branches to generate spatial attention maps for mutual feature
calibration. Considering the different sizes and positions of ABCs structures,
our network was applied after a rough localization of each structure to obtain
fine segmentation results. Experiments on the MICCAI 2020 ABCs challenge
dataset showed that: 1) Our proposed two-stage segmentation strategy largely
outperformed methods segmenting all the structures in just one stage; 2) The
proposed HMRNet with two branches can maintain high-resolution representations
and is effective to improve the performance on thin structures; 3) The
proposed BFC block outperformed existing attention methods using
monodirectional feature calibration. Our method won the second place of ABCs
2020 challenge and has a potential for more accurate and reasonable
delineation of CTV of brain tumors.

Brain tumor, anatomical brain barriers, segmentation, convolutional neural
networks, attention

##  1 Introduction

Brain cancer has became one of the most common cancers around the world [1].
Radiotherapy plays an important role in clinical treatment of brain cancer,
and is usually used before or after surgery [2, 3, 4]. During radiotherapy
treatment planning, accurate delineation of the Clinical Target Volume (CTV)
[5] is very important for ensuring that the cancer cells receive sufficient
radiation dose and that for healthy structures lie in a safe range. However,
the CTV is not radiographically distinguishable from healthy tissues, and in
practice it is delineated by estimating the extent of microscopic disease
spread based on accumulated knowledge of prior treatment outcomes and
histological evidence on the extent of cancer cell spread [6]. While the CTV
itself is ambiguous, it is much easier to identify structures that do not
belong to the CTV and serve as Anatomical Brain Barriers to Cancer spread
(ABCs) [6]. For gliomas, the most common type of brain tumors, the CTV
boundary is constrained by barrier structures like the falx cerebri, tentorium
cerebelli, brain sinuses, cerebellum and ventricles. Therefore, accurate
segmentation of these barrier structures has a potential to assist more
accurate and reasonable delineation of CTV that can improve the radiotherapy
outcome of brain cancer patients.

Meanwhile, delineation of anatomical structures relies on multi-modal medical
images, such as Computed Tomography (CT) and T1- and T2-weighted Magnetic
Resonance Imaging (MRI). CT is required by radiotherapy dose computation as it
can measure the density of different structures. It can show dense structures
like bones and implants with less distortion, but has a low contrast for soft
tissues [7]. T1- and T2-weighted MRI can better highlight different soft
tissues than CT [7]. Therefore, employing different imaging modalities can
provide complementary information for accurate structure segmentation required
by radiotherapy treatment planning. At present, the delineation of multiple
brain structures is implemented by experienced radiation oncologists through
slice-by-slice manual annotation, which is labor intensive, time consuming,
and subject to the operator’s experience. Thus, accurate automatic
segmentation of the anatomical structures like ABCs is desirable for
reproducibility and reducing the workload of annotators and the waiting time
for patients [8]. Considering the state-of-the-art performance of
Convolutional Nerual Networks (CNNs) in medical image analysis [9, 10] due to
their powerful feature representations capability, it is promising to use CNNs
to automate this task.

To this end, this work aims to obtain automatic segmentation of ABCs of from
medical images, which is important for the accuracy and reasonability of CTV
delineation of the brain cancer. However, this task is challenging due to
several reasons. First, the segmentation involves a set of structures with
various shapes and sizes, as shown in Fig. 1. While some of the barrier
structures are plump in 3D, the shapes of some structures are thin (i.e.,
slice-like) in 3D space, which makes it hard to use a single 2D or 3D CNNs to
deal with all of them well. For example, plump structures like cerebellum and
ventricles are more suitable for using 3D CNNs that have an isotropic
receptive field in 3D, but typical 3D CNNs are suboptimal for dealing with
thin structures like the falx cerebri and brain sinuses that spans only few
slices in the sagittal view. This is due to that the commonly used down- and
up-sampling in 3D CNNs would blur the inter-slice information for thin
structures. Second, there exists a large imbalance between foreground and
background voxels, especially for thin structures. For example, the ratio
between tentorium cerebelli and the background is around 1:500. Therefore,
taking the entire brain volume as the input of a CNN to segment all the
structures simultaneously is unfriendly for such thin structures. Thirdly, the
contrast between these structures and surrounding tissues is still low, which
can easily lead to under- and over-segmentations. Despite that using multi-
modal images can alleviate this problem, it remains challenging to obtain
accurate segmentation results of these structures whose shapes and positions
can be affected by the presence of tumors.

Figure 1: Illustration of Anatomical Brain Barriers to Cancer spread (ABCs).
The bars at the bottom show the proportion of the background and different
structures. Note that the brain sinuses has a sagittal component and a
transverse component.

Recently, various CNNs have been proposed for automatic medical image
segmentation [11, 12, 13, 14, 15, 16]. One of the most successful CNNs in
recent years is the nnU-Net [17] that is an extension of 3D U-Net [18] with an
encoder-decoder architecture. It has been shown good performance in various
segmentation tasks including brain tumor segmentation [17]. Shusharina _et
al._ [6] recently applied 3D U-Net [18] to automatic segmentation of ABCs to
assist CTV delineation of brain tumors. However, they used a single CNN to
segment all the structures simultaneously without considering the large
variance of shapes and positions of structures in this specific task, which
limited the segmentation accuracy.

To address the above challenges, we propose a novel framework to segment five
ABCs structures shown in Fig. 1 from multi-modal images. Considering that
widely used 3D networks (e.g., 3D U-Net [18]) can hardly obtain accurate
results of multiple structures in one step, we employ such a network to
localize each structure roughly, and then use our proposed High and Multi-
Resolution network (HMRNet) with Bidirectional Feature Calibration (BFC)
blocks to get a fine segmentation of each structure, respectively.

The contribution of this work can be summarized as follows: First, we propose
a novel framework for automatic segmentation of anatomical brain barriers to
cancer spread from multi-modality medical images, which has been rarely
studied in the literature so far. Second, we propose a novel network structure
named as HMRNet that combines high-resolution and multi-resolution branches in
parallel to better deal with anatomic structures with various shapes and
scales. We also introduce two variants of HMRNet (i.e., 3D and 2.5D versions,
respectively) to deal with plump and slice-like structures in a unified
framework. Thirdly, we introduce a novel attention block that uses
bidirectional feature calibration to allow features from different branches to
calibrate each other for better feature representation ability. In addition,
we propose a novel loss function that can increase the recall and reduce false
negatives for segmenting objects with a large imbalance between the foreground
and the background.

Our method was originally designed for the task 1 of MICCAI 2020 ABCs
challenge111https://abcs.mgh.harvard.edu/. On the public validation dataset,
our method won the second place among all the
participants222https://abcs.mgh.harvard.edu/index.php/leader-board. This work
has not been published elsewhere, and in this paper, we provide detailed
description of our method and implementation, and validate it with extensive
comparison with existing methods and ablation studies.

##  2 Related Works

###  2.1 CNNs for Medical Image Segmentation

Recent years have seen extensive works on CNNs for image segmentation [9, 10].
Long _et al._ [19] firstly proposed a Fully Convolutional Network (FCN) that
can obtain a full-resolution segmentation of an image in a single forward
pass. Ronneberger _et al._ [11] proposed U-Net that uses symmetric squeeze and
expanding paths to extract features and then recover the spatial resolution of
the final prediction map, where skip connections are introduced to fuse
shallow and deep features. As an extension of U-Net, U-Net++ [13] connects the
encoder and decoder sub-networks through a series of nested dense skip
pathways. V-Net [12] shares a similar structure with U-Net [11] and it uses a
residual connection in each convolutional block. In [20], cascaded FCNs [21]
were used for liver and lesion segmentation. The nnU-Net [17] has achieved
state-of-the-art performance in several 3D medical image segmentation tasks.
Another family of segmentation networks are based on Deeplab [22, 23]. These
networks used dilated convolutions [24] with multiple dilation rates in the
same layer to fuse multi-scale information, which will significantly increase
the GPU memory consumption, limiting their application to 3D medical image
tasks. Recently, Wang _et al._ [25] proposed HRNet that connects high-to-low
resolution convolution streams in parallel and repeatedly exchanges the
information across resolutions. However, it is also memory consuming and can
hardly be used for 3D images.

Besides, attention modules have been widely used to strengthen the
representation power of CNNs. Hu _et al._ [26] proposed the Squeeze-and-
Excitation (SE) block to calibrate different channels of a feature map.
Rickmann _et al._ [27] proposed a Project & Excite (PE) module that squeezes
feature maps along different views of a tensor separately to retain more
spatial information. To highlight useful regions, attention U-Net [28] and
spatial and channel SE (scSE) networks [29] were proposed for segmentation
tasks. The CA-Net [30] further combined spatial, channel and scale attentions
for segmentation of 2D images. However, these methods like the SE [26] often
obtain the calibration coefficients from the input feature map itself, which
may be biased and lead to over-fitting. The attention gate proposed in [28]
avoids this problem by using one high-level feature map to provide the
attention signal to calibrate another low-level feature map. However, this is
a monodirectional calibration, and we propose a bidirectional attention in
this work that enables feature maps from two different branches to calibrate
each other.

Figure 2: Overall framework of our proposed method for segmentation of ABCs.
We first use a coarse localization step to obtain a bounding box of each
structure, and then use our proposed HMRNet to obtain a fine segmentation of
each structure according to the localization result, respectively. Note that
sagittal and transverse brain sinuses are considered respectively, as shown in
Fig. 3. Two variants of HMRNet, i.e., HMRNet-p based on 3D convolutions and
HMRNet-s based on 2.5D convolutions, are used to deal with plump and thin
structures, respectively.

###  2.2 Multi-Organ Segmentation for Radiotherapy

In early studies, automatic multi-organ segmentation are mainly based on
atlases [31]. For example, VaN _et al._ [32] segmented hippocampus and
cerebellum from MRI based on atlas registration and appearance models.
However, these methods require time-consuming image alignment, and the
performance is limited when the similarity between the atlas and target images
is low. Glaister _et al._ [33] proposed a multi-atlas approach to segment the
falx from T1-weighted MRI and susceptibility-weighted MRI.

Recetnly, deep CNNs have been applied to multi-organ segmentation for
radiotherapy planning, such as for Head and Neck (HAN) cancers [34] and lung
cancer [35]. AnatomyNet [34] combined 3D U-Net [18] and SE block to segment
Organs-at-Risks (OARs) for HAN cancers. Considering the imbalance between
large and small OARs, FocusNet [14] and FocusNet-V2 [15] combined ROI-pooling
with small organ localization and segmentation sub-networks while maintaining
accuracy for large organs. Lei _et al._ [16] proposed a 3D SepNet to better
deal with thin structures in images with large inter-slice spacing, and it was
trained with a hardness-weighted loss for better segmentation. Zou _et al._
[36] applied 3D U-Net [18] with a label merging strategy for symmetrical
organs to segment OARs including the brain stem, chiasm, eyes and optic
nerves, etc.

Despite the abundant works on OAR segmentation and multi-class segmentation of
brain tissues [37], methods designed for segmentation of barrier structure for
brain tumors are rare in the literature. To the best of our knowledge, the
only two previous works on this task were reported in [6] and [38], where the
authors simply employed 3D U-Net [18] to segment the barrier structures from
multi-modal images, with limited performance due to the complex shapes and
sizes of different imbalanced structures including the falx cerebri, tentorium
cerebelli, brain sinuses, cerebellum and ventricles.

Figure 3: Illustration of the brain sinuses with a “T” shape. Note that the
sagittal (b) and transverse (c) components span few slices in the sagittal
view and the axial view respectively. Thus, we use two HMRNet-s to deal with
them, respectively. Figure 4: Overview of our proposed HMRNet (a) and BFC (b).
The input image is fed into a high-resolution branch to obtain detailed
features, and a multi-resolution branch to obtain more abstract features from
a larger receptive field. Outputs of the two branches are concatenated to
obtain final segmentation prediction. At different depths of the network,
features from the two branches are calibrated by each other through the BFC
block. The last orange BFC block does not have up- and down-sampling
operations. Note that HMRNet-p and HMRNet-s are two variants of HMRNet, and
they use 3D convolutions and 2.5D convolutions, respectively, where the later
is a combination of intra-slice and inter-slice convolutions.

##  3 Methods

Our proposed framework for ABCs segmentation is illustrated in Fig. 2. As
different structures with various shapes and sizes make it hard to segment
them simultaneously with a single model, we first use a localization model
based on 3D nnU-Net [17] to obtain a rough localization of different
structures, and then propose a High and Multi-Resolution Network (HMRNet) to
segment each structure around its local region respectively, where a
Bidirectional Feature Calibration (BFC) block is introduced for better
interaction between features in the two branches. We introduce two variants of
HMRNet, i.e., HMRNet-p based on 3D convolution and HMRNet-s based on 2.5D
convolution, to deal with plump and thin structures, respectively. A new
weighted loss function is also proposed to increase the recall and reduce
false negatives for tiny structures. In this section, we introduce our HMRNet
with BFC and our proposed loss function, respectively.

###  3.1 High and Multi-Resolution Network (HMRNet)

The widely used 3D U-Net [18] for segmenting volumetric medical images employs
several down-sampling operations to increase the receptive field and obtain
multi-scale features in the encoder, which is helpful for segmentation of
relative large structures, but the down-sampling operations lead to poor
performance for tiny or thin structures. Keeping a high resolution of feature
maps is important for obtaining more detailed results of small structures, but
simply removing the down-sampling operations makes the network computationally
expensive and memory consuming.

To better deal with the brain barrier structures with both plump and thin
structures, we combine the advantages of encoder-decoder-based multi-scale
feature learning and high-resolution representation, therefore propose a High
and Multi-Resolution Network (HMRNet) that consists of a high-resolution
branch and a multi-resolution branch, as shown in Fig. 4(a). The high-
resolution branch does not use any down- and up-sampling operation, and can
keep detailed features for each voxel. The multi-resolution branch is based on
a U-shape backbone and able to learn multi-scale features with more abstract
representations and contextual information. As the two branches are
complementary to each other, we introduce a bidirectional feature calibration
block to enable the interaction between the two branches at different depths
of the network.

Due to the different shapes of ABCs structures, we design two variants of
HMRNet to deal with plump structures (i.e, the cerebellum, the tentorium
cerebelli and the ventricles) and slice-like thin structures (i.e., the falx
cerebri and the brain sinuses), and the corresponding networks are named as
HMRNet-p and HMRNet-s, respectively. Note that these two variants share the
same overall structure, but are implemented with 3D and 2.5D convolutions,
respectively.

HMRNet-p. HMRNet-p is designed for plump anatomical structures. As shown in
Fig. 4(a), the multi-resolution branch of HMRNet-p is a U-shape subnetwork
with seven convolutional blocks, which obtains information of different scales
through down- and up-sampling. The high-resolution branch also contains seven
convolutional blocks while the feature maps’ resolution keeps fixed during the
whole process. Each convolutional block in both branches contains two $3\times
3\times 3$ convolutional layers, each followed by an Instance Normalization
(IN) layer and a Rectified Linear Units (ReLU) layer. In order to reduce the
memory consumption and time cost, we set the number of channels to eight for
all blocks in the high-resolution branch. Meanwhile, in the encoder of the
multi-resolution branch, the output channel number in the first block is 16
and doubled after each max pooling. The channel number is halved after each
block in the decoder.

At each depth of the network, the corresponding features from the two branches
are calibrated by each other through a BFC block, which will be detailed in
Section 3.2. Because of the large similarity between feature maps in the first
convolution blocks of the two branches, we do not apply BFC in the first
convolutional block, as shown in Fig. 4(a). The feature maps at the end of the
two branches have the same spatial size, and they are concatenated and fed
into a $1\times 1\times 1$ convolutional layer followed by softmax to generate
the segmentation probabilities.

HMRNet-s. HMRNet-s is designed for slice-like anatomical structures. Due to
the special shape of some structures such as the falx cerebri, sagittal and
transverse brain sinuses with the slice-like shape that only span few slices,
standard 3D convolutions would blur their boundaries along the through-plane
direction and are harmful to the segmentation accuracy. Therefore, we replace
a standard 3D convolution layer with a spatially separable convolution that
contains two intra-slice convolutions ($3\times 3\times 1$ kernels) and one
inter-slice convolution ($1\times 1\times 3$ kernel). The spatially separable
convolutions are also followed by IN and ReLU, as shown in the legend of Fig.
4(a).

Note that the brain sinuses has a spetial “T”-like shape that consists of a
thin sagittal component and a thin transverse component, as shown in Fig. 3.
To better deal with this problem, we use two HMRNet-s to segment the sagittal
and transverse brain sinuses, respectively, as shown in Fig. 2.

###  3.2 Bidirectional Feature Calibration (BFC)

Due to the complementary feature maps from the high-resolution and multi-
resolution branches, we propose the BFC block to fuse the corresponding
feature maps at the same depth of the two branches and enable a bidirectional
calibration between them. In contrast to self-calibration using SE [26] that
has a risk of over-fitting and attention gate [39, 30] that uses one feature
map to monodirectionally calibrate another feature map, our bidirectional
calibration makes the two branches aware of each other, which has a potential
for more robust learning. Let $F_{h}\in\mathbb{R}^{C_{h}\times D\times H\times
W}$ represent a feature map from the high-resolution branch at a certain depth
of the network, where $C_{h}$ is the channel number and $D\times H\times W$ is
the 3D spatial size. We denote the feature map from the multi-resolution
branch at the same depth as $F_{m}\in\mathbb{R}^{C_{m}\times d\times h\times
w}$ with $C_{m}$ channels and the spatial size of $d\times h\times w$. Note
that $F_{m}$ has a lower resolution than $F_{h}$.

As shown in Fig. 4(b), the BFC block contains two parts: High-Resolution
Branch (HRB) calibration and Multi-Resolution Branch (MRB) calibration. For
HRB calibration, we first upsample $F_{m}$ to the same spatial resolution as
$F_{h}$, and the output is denoted as $\hat{F}_{m}$ and concatenated with
$F_{h}$:

$F^{\prime}_{h}=F_{h}\oplus\hat{F}_{m}$ (1)

where $\oplus$ denotes the concatenation operation, and the output is
$F^{\prime}_{h}\in\mathbb{R}^{(C_{h}+C_{m})\times D\times H\times W}$. Then,
we squeeze $\hat{F}_{m}$ along the channel dimension and excite it in the
spatial dimension to generate a spatial attention map to calibrate
$F^{\prime}_{h}$. Specifically, a linear projection matrix
$W_{m}\in\mathbb{R}^{C_{m}\times 1\times 1\times 1}$ squeezes $\hat{F}_{m}$
into one channel, and then a sigmoid activation $\sigma(\cdot)$ is used to
obtain the spatial attention map for HRB, where $W_{m}$ is implemented by a
pointwise convolution.

$\alpha_{h}=\sigma(W_{m}\cdot\hat{F}_{m})$ (2)

where $\alpha_{h}\in{[0,1]}^{1\times H\times W\times D}$ is the spatial
attention map for HRB. Each value of $\alpha_{h}$ indicates the importance of
spatial information at the corresponding voxel in $F^{\prime}_{h}$. The
calibrated feature in HRB is formulated as:

$F^{c}_{h}=\alpha_{h}\cdot F^{\prime}_{h}$ (3)

Similarly, we use an MRB calibration block to calibrate feature from the
multi-resolution branch. We first downsample $F_{h}$ to the same spatial
resolution as $F_{m}$, and the result is concatenated with $F_{m}$:

$F^{\prime}_{m}=F_{m}\oplus\hat{F}_{h}$ (4)

where $\hat{F}_{h}$ is the downsampled version of $F_{h}$, and
$F^{\prime}_{m}$ is the concatenation output. The attention map is obtained
by:

$\alpha_{m}=\sigma(W_{h}\cdot\hat{F}_{h})$ (5)

where $\alpha_{m}\in{[0,1]}^{1\times h\times w\times d}$ is the spatial
attention map for MRB, and $W_{h}\in\mathbb{R}^{C_{h}\times 1\times 1\times
1}$ is a linear projection matrix implemented by pointwise convolution. The
calibrated feature in MRB is formulated as:

$F^{c}_{m}=\alpha_{m}\cdot F^{\prime}_{m}$ (6)

After using HRB and MRB calibration blocks, the outputs $F^{c}_{h}$ and
$F^{c}_{m}$ in BFC are sent back to the high-resolution branch and multi-
resolution branch respectively, which serve as inputs for the corresponding
next covolutional blocks in HMRNet, as shown in Fig. 4(a). Note that we use
BFC at different depths of the HMRNet, leading to six BFC blocks in total.

###  3.3 Loss Function

The large imbalance between the foreground and background voxels for
small/thin organs tends to increase false negative predictions. To solve this
problem, we design a new loss encouraging a high recall for the fine
segmentation. Inspired by the Tversky loss [40], we add a hyper-parameter
$\theta$ into the Dice loss to control the trade-off between false positives
and false negatives. The standard Dice loss and our proposed false negative
weighted Dice loss are defined as:

$L_{Dice}=1-\frac{1}{\left|K\right|}\sum_{k=1}^{K}\frac{\sum_{i}^{N}2p_{i}^{k}g_{i}^{k}+\epsilon}{\sum_{i}^{N}p_{i}^{k}\;+\;\sum_{i}^{N}g_{i}^{k}+\epsilon\;}$
(7)

$\displaystyle L_{fn-
Dice}=1-\frac{\sum_{i}^{N}2p_{i}g_{i}+\epsilon}{\sum_{i}^{N}p_{i}g_{i}+\theta\cdot\sum_{i}^{N}p_{i}\overline{g}_{i}+\sum_{i}^{N}\overline{p}_{i}g_{i}+\epsilon}$
(8)

where $N$ and $K$ are the numbers of voxels and classes in the image,
respectively. $\epsilon$ is a small number for numerical stability.
$p_{i}^{k}$ and $g_{i}^{k}$ is the predicted and ground truth probability of
pixel $i$ belonging to class $k$. Note that in $L_{fn-Dice}$, we only consider
a binary segmentation formulation. $\overline{g}_{i}=1-g_{i}$ and
$\overline{p}_{i}=1-p_{i}$ denote the background probability in the prediction
and ground truth, respectively. $p_{i}\overline{g}_{i}$ and
$\overline{p}_{i}g_{i}$ denote the False Negative (FN) and the False Positive
(FP) predictions, respectively. $\theta\geq 1$ is a hyper-parameter to
penalize false negatives in the prediction. In our work, due to the large
variance between different structures, an optimal $\theta$ value is selected
for each substructure, which will be described in the experiment. The total
loss $L_{f-seg}$ for training our HMRNet in the fine segmentation stage is a
combination of $L_{fn-Dice}$ and cross entropy loss $L_{CE}$:

$\displaystyle{L}_{f-seg}={L}_{fn-Dice}+L_{CE}$ (9)

where the cross entropy loss is:

${L}_{CE}=-\frac{1}{N}{\sum_{i=1}^{N}}\sum_{k=1}^{K}g_{i}^{k}\log(p_{i}^{k})$
(10)

##  4 Experiments and results

###  4.1 Experimental Setting

####  4.1.1 Dataset and Evaluation Metrics

We used the dataset of MICCAI 2020 ABCs challenge (task 1) for experiments,
which contained images from 45 patients for training, 15 patients for online
validation and 15 patients for final online testing. Because the official
online testing set was not publicly available, we randomly split the official
training data into 36 and 9 for training and testing respectively. The ground
truth for the training data was manually annotated by experts. The data of
each patient consisted of a CT scan, a contrast-enhanced T1-weighted MRI and a
T2-weighted MRI. Images of each patient were registered, and re-sampled to the
size of $164\times 194\times 142$ with the spacing of $1.2~{}mm\times
1.2~{}mm\times 1.2~{}mm$ by the organizers. We used z-score [17] to normalize
the image intensity. The three modalities for each patient were concatenated
into a three-channel volume as the input of CNNs. For quantitative evaluation,
we measured the Dice similarity and Average Symmetric Surface Distance (ASSD)
between segmentation results and the ground truth in 3D space.

Table 1: Quantitative evaluations of different modalities. The best result is shown in bold font. Cer: cerebellum, Fal: falx cerebri, Sinu: sagittal & transverse brain sinuses, Ten: tentorium cerebelli, and Ven: ventricles. Method | Dice (%) |  |  ASSD (mm)
---|---|---|---
Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
T1 | 90.5±2.0 | 75.3±7.9 | 79.8±3.7 | 74.6±5.8 | 84.1±5.9 | 80.9±4.8 |  | .802±.165 | .561±.154 | .711±.182 | .502±.094 | .725±.130 | .661±.178
T1+T2 | 93.3±1.3 | 77.5±6.3 | 81.2±2.6 | 77.3±4.9 | 85.3±4.2 | 82.9±3.0 |  | .799±.151 | .541±.142 | .693±.170 | .499±.089 | .708±.113 | .648±.165
T1+T2+CT | 94.9±1.0 | 78.6±5.1 | 82.1±2.4 | 78.0±4.7 | 87.1±3.6 | 84.1±2.5 |  | .787±.146 | .524±.134 | .679±.161 | .497±.081 | .697±.102 | .637±.151
Table 2: Quantitative evaluations of different methods for ABCs segmentation. The best result is shown in bold font. Cer: cerebellum, Fal: falx cerebri, Sinu: sagittal & transverse brain sinuses, Ten: tentorium cerebelli, and Ven: ventricles. Coarse | Refine | Dice (%) |  |  ASSD (mm)
---|---|---|---|---
Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
| nnU-Net
---
[17]
× | 94.9±1.0 | 78.6±5.1 | 82.1±2.4 | 78.0±4.7 | 87.1±3.6 | 84.1±2.5 |  | .787±.146 | .524±.134 | .679±.161 | .497±.081 | .697±.102 | .637±.151
| nnU-Net
---
[17](sub)
× | 95.0±1.1 | 78.8±4.9 | 82.1±2.4 | 78.5±4.3 | 88.6±3.6 | 85.1±2.4 |  | .643±.137 | .486±.245 | .672±.153 | .456±.079 | .646±.110 | .602±.103
| nnU-Net
---
[17]
| nnU-Net
---
[17]
95.0±0.8 | 79.1±4.8 | 82.4±2.2 | 79.5±3.7 | 88.4±3.3 | 84.8±1.9 |  | .723±.107 | .434±.340 | .655±.135 | .433±.085 | .638±.113 | .576±.132
| nnU-Net
---
[17]
| nnU-Net
---
[17](sub)
95.3±0.8 | 79.3±4.8 | 82.6±2.2 | 79.9±3.7 | 88.6±3.3 | 85.1±1.9 |  | .668±.117 | .444±.345 | .652±.130 | .385±.077 | .614±.101 | .573±.127
| nnU-Net
---
[17]
HMRNet | 95.8±0.7 | 80.1±4.7 | 83.5±2.4 | 82.0±3.4 | 89.5±2.8 | 86.2±2.0 |  | .595±.089 | .371±.101 | .601±.155 | .353±.070 | .576±.052 | .499±.081

####  4.1.2 Implementation Details

The proposed framework was implemented in PyMIC333https://github.com/HiLab-
git/PyMIC [41] using PyTorch on one NVIDIA GTX 2080 Ti GPU with 12 GB memory.
We used the SGD optimizer with a weight decay $1\times 10^{-5}$ for training.
The initial learning rate was $\eta_{0}=0.01$, and was decayed based on
$\eta=\eta_{0}\cdot\left(1-\frac{e-1}{N_{e}}\right)^{0.9}$, where $e$ is the
current epoch number, and $N_{e}=300$ is the total number of epochs. We
employed $L_{Dice}+L_{CE}$ for the coarse location task. The code of our
method is available online444https://github.com/HiLab-git/ABCschallenge2020.

To train the fine segmentation models, we cropped the original input volumes
for each structure based on the bounding box inferred from the ground truth
and expanded by 10 pixels. Random rotation, elastic deformation, gamma
correction and random flipping were applied for data augmentation. Deep
supervision was used for better performance. Specifically, we use a pointwise
convolution layer at the end of each level of the decoding blocks, and
compared it with $G_{s}$ that is a down-sampled version of the ground truth so
that they have the same resolution level. The deep supervision loss is defined
as:

$\displaystyle L_{deep}=\sum_{s=1}^{S}\lambda_{s}L_{f-seg}(P_{s},G_{s})$ (11)

where $S=3$ is the entire number of resolution levels. $\lambda_{s}$ is the
weight of resolution level $s$, and set as 0.3, 0.3 and 0.4 in the experiment,
respectively. During inference, we used the coarse location result to get the
bounding box for each structure and also expanded it for cropping, which
obtains a subvolume as the input of HMRNet. The optimal values of $\theta$ in
$L_{fn-Dice}$ for different structures were determined through five-fold cross
validation, and they were: 2.0 for the cerebellum and the transverse brain
sinuses, 3.0 for the falx cerebri and the sagittal brain sinuses, 3.5 for the
tentorium cerebelli and 2.5 for the ventricles.

In each iteration for training the second-stage model, we randomly selected
the checkpoint of the first-stage model at epoch $e$ ($e$ = 100, 200, 300) to
obtain the initial segmentation results for training data, which serves as an
augmentation strategy, and the training data for the second-stage model
contains both high-quality and low-quality initial segmentation results, which
can improve the second-stage model’s tolerance on relatively low-quality
initial segmentation.

Figure 5: Visual comparison of coarse segmentation and refinement based on nnU-net (sub) and HMRNet, respectively. Red arrows denote the under- and over-segmentation compared with the ground truth.  Table 3: Ablation study of the overall structure of HMRNet when attention mechanisms are not used. HR_branch and MR_branch mean only using the HR branch and only using the MR branch, respectively. HMRNet-p and HMRNet-s represent using HMRNet-p and HMRNet-s for all the structures, respectively. HMRNet denotes using our HMRNet-p for plump structures and HMRNet-s for slice-like structures, respectively. Network | Dice (%) |  |  ASSD (mm)
---|---|---|---
Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
Coarse | 94.9±1.0 | 78.6±5.1 | 82.1±2.4 | 78.0±4.7 | 87.1±3.6 | 84.1±2.5 |  | .787±.146 | .524±.134 | .679±.161 | .497±.081 | .697±.102 | .637±.151
HR_branch | 94.9±0.8 | 79.0±4.7 | 82.0±2.5 | 79.7±3.3 | 87.9±2.2 | 84.7±2.1 |  | .790±.121 | .421±.250 | .714±.213 | .394±.116 | .725±.159 | .609±.123
MR_branch | 95.5±0.8 | 78.9±5.0 | 82.6±2.8 | 79.5±3.5 | 89.1±3.5 | 85.1±1.9 |  | .656±.103 | .436±.338 | .661±.166 | .431±.077 | .617±.157 | .560±.126
HMRNet-p | 95.6±0.6 | 79.5±4.7 | 82.4±2.3 | 81.4±2.9 | 89.4±3.0 | 85.7±1.8 |  | .685±.091 | .360±13.7 | .677±.141 | .364±.066 | .578±.061 | .533±.081
HMRNet-s | 95.2±0.7 | 79.8±4.9 | 82.7±2.4 | 81.0±3.4 | 89.1±3.3 | 85.6±2.0 |  | .737±.103 | .354±.137 | .654±.156 | .389±.141 | .609±.063 | .549±.095
HMRNet | 95.6±0.7 | 79.8±4.7 | 82.7±2.4 | 81.4±2.8 | 89.4±3.1 | 85.8±1.9 |  | .686±.094 | .354±.137 | .649±.154 | .366±.065 | .581±.061 | .527±.086

###  4.2 The Role of Different Modalities

We first investigated the role of different modalities in the segmentation
task, and compared using T1, T1 + T2 and T1 + T2 + CT for the segmentation
task based on nnU-Net[17], respectively. The results are listed in Table 1. It
can be observed that using T1 only obtained an average Dice of 80.9$\%$ for
the five organs, and using the three modalities obtained an average Dice of
84.1$\%$, which outperformed the other two variants. Therefore, it is
necessary to combine the thee available modalities for the segmentation task.

###  4.3 Effectiveness of Our Framework

We first compared two variants of nnU-Net for segmentation: the original nnU-
Net [17] that segments all the organs simultaneously and nnU-Net (sub) that
means segmenting each organ respectively from the entire image using five
models of nnU-Net, each was a binary segmentation model. Then using the
results of nnU-Net [17] as coarse segmentation, we compared three different
ways for fine segmentation: 1) using nnU-Net [17] to refine all the organs
simultaneously, 2) using nnU-Net [17] (sub) that means binary nnU-Net [17]
models are used to refine each organ respectively, and 3) using our HMRNet to
refine each organ respectively. Note that in the fine segmentation stage, both
the 3D nnU-Net [17] and HMRNet used the same cropping strategy based on the
coarse segmentation, as described in Section 4.1.2. All these methods were
trained with $L_{Dice}+L_{CE}$.

Table 4: Ablation study of BFC used in HMRNet. $\times$ : without any attention mechanisms, HRB and MRB denote monondirectional feature calibration methods, where HRB uses multi-resolution branch features to calibrate high-ressolution branch features, and vice versa for MRB. Enco and Deco represent using our BFC block only in the encoder and the decoder, respectively. Bottom, Middle and Top represent using BFC block only for the lowest, middle and highest resolution level of the network, respectively. BFC is our proposed bidirectional feature calibration used at all the resolution levels.  Attention | Dice (%) |  | ASSD (mm)
---|---|---|---
Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
$\times$ | 95.6±0.7 | 79.8±4.7 | 82.7±2.4 | 81.4±2.8 | 89.4±3.1 | 85.8±1.9 |  | .686±.094 | .354±.137 | .649±.154 | .366±.065 | .581±.061 | .527±.086
HRB | 95.7±0.7 | 80.0±4.9 | 82.5±2.4 | 81.4±3.1 | 89.4±3.0 | 85.8±1.9 |  | .643±.087 | .318±.079 | .669±.164 | .382±.079 | .617±.136 | .526±.067
MRB | 95.4±0.8 | 80.0±4.4 | 83.0±2.6 | 81.0±3.6 | 88.9±3.3 | 85.7±2.0 |  | .712±.086 | .421±.306 | .648±.146 | .366±.076 | .603±.080 | .550±.109
Enco | 95.6±0.7 | 79.9±4.8 | 82.7±2.4 | 81.4±2.4 | 89.0±2.9 | 85.7±1.9 |  | .685±.091 | .348±.087 | .648±.154 | .355±.058 | .611±.123 | .529±.071
Deco | 95.6±0.6 | 80.2±4.9 | 82.8±2.4 | 81.8±3.5 | 89.3±3.1 | 85.9±2.1 |  | .682± .094 | .406±.305 | .649±.155 | .355±.081 | .550±.077 | .528±.124
Bottom | 95.7±0.9 | 80.1±4.6 | 83.1±2.2 | 81.1±3.3 | 89.4±2.8 | 85.7±1.9 |  | .601±.117 | .430±.339 | .617±.149 | .363±.066 | .570±.139 | .515±.125
Middle | 95.8±1.0 | 80.0±4.7 | 83.1±2.7 | 80.6±3.7 | 89.0±3.2 | 85.7±2.3 |  | .621±.130 | .431±.342 | .66.0±.200 | .437±.249 | .588±.115 | .549±.184
Top | 95.7±0.9 | 80.1±4.6 | 83.4±2.3 | 81.6±3.0 | 89.4±2.8 | 86.0±1.9 |  | .601±.105 | .421±.346 | .621±.139 | .373±.064 | .595±.060 | .514±.143
BFC | 95.8±0.7 | 80.1±4.4 | 83.5±2.4 | 82.0±3.4 | 89.5±2.8 | 86.2±2.0 |  | .595±.089 | .371±.101 | .601±.155 | .353±.070 | .576±.052 | .499±.081

Quantitative evaluation results in Table 2 show that using another nnU-Net for
refinement can slightly improve the average Dice from 84.1% to 84.8%, but
using nnU-Net (sub) and HMRNet for refinement can improve the avaerge Dice to
85.1% and 86.2%, respectively. Using HRMNet to refine the coarse segmentation
of nnU-Net outperformed the other variants. Compared with the coarse
segmentation of nnU-Net, our proposed HMRNet reduced the average ASSD from
0.637 mm to 0.499 mm, which is a 22% improvement. A visual comparison of
coarse segmentation, nnU-net [17](sub) refine and HMRNet refine is shown in
Fig. 5. From the sagittal view, it could be observed that the prediction of
HMRNet has better connectivity than the other two methods in thin structures
like sagittal & transverse brain sinuses and higher completeness in
ventricles, which is highlighted by red arrows.

###  4.4 Ablation Study of HMRNet

First, to validate the overall structure of HMRNet, we compared it with four
variants: 1) “HR_branch” that means only using the high-resolution branch in
our HMRNet, 2) “MR_branch” that means only applying the multi-resolution
branch in HMRNet, 3) “HMRNet-p” that means using HMRNet-p for all the
structures, and 4) “HMRNet-s” that means using HMRNet-s for all the
structures. Note that our “HMRNet” method uses HMRNet-p for plump structures
(cerebellum, tentorium cerebelli and ventricles) and HMRNet-s for slice-like
structures (falx cerebri, sagittal and transverse brain sinuses),
respectively. All these methods were trained with $L_{Dice}+L_{CE}$ loss for
each structure, and we disabled BFC that will be analyzed in the following. As
shown in Table 3, “HR_branch” and “MR_branch” achieved lower performance than
the other variants. “HMRNet-p” outperformed “HMRNet-s” for the cerebellum,
tentorium cerebelli and venricles that are plump structures, and performed
worse than “HMRNet-s” for the falx cerebri and the brain sinuses that are
slice-like structures. “HMRNet” outperformed all the compared variants, which
demonstrates the effectiveness of combining the two branches and using
different types of convolutoins for the plump and slice-like structures.

Table 5: Effect of $\theta$ in our false negative-weighted loss function. Note that when $\theta=1.0$, our $L_{fn-Dice}$ is equal to $L_{Dice}$.  $\theta$ |  | Dice (%) |  | ASSD (mm)
---|---|---|---|---
Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
1.0 |  | 95.8±0.7 | 80.1±4.7 | 83.5±2.4 | 82.0±3.4 | 89.5±2.8 | 86.2±2.0 |  | .595±.089 | .371±.101 | .601±.155 | .353±.070 | .576±.052 | .499±.081
1.5 |  | 95.9±0.8 | 80.0±4.8 | 84.0±2.3 | 82.1±3.1 | 89.4±2.9 | 86.3±1.9 |  | .618±.106 | .372±.161 | .588±.108 | .355±.057 | .575±.072 | .502±.081
2.0 |  | 96.1±1.0 | 80.1±4.8 | 84.1±2.5 | 82.5±3.2 | 89.6±3.1 | 86.5±1.9 |  | .590±.134 | .341±.098 | .584±.115 | .352±.070 | .565±.061 | .486±.075
Adaptive |  | 96.1±1.0 | 80.4±4.4 | 84.3±2.6 | 82.8±2.5 | 89.9±2.7 | 86.7±1.7 |  | .589±.135 | .311±.064 | .577±.120 | .347±.065 | .557±.057 | .476±.074
Table 6: Quantitative evaluation of different methods for segmentation of ABCs. ∗ denotes significant improvement from the existing methods ($p$-value $<$ 0.05).  Method |  | Dice (%) |  | ASSD (mm)
---|---|---|---|---
| Cer | Fal | Sinu | Ten | Ven | Average |  | Cer | Fal | Sinu | Ten | Ven | Average
3D U-Net [18] |  | 88.3±4.2 | 72.3±6.5 | 76.0±3.8 | 70.1±7.8 | 85.6±4.3 | 78.5±3.8 |  | 1.634±.473 | .516±.252 | .878±.217 | .611±.219 | 2.026±1.075 | 1.133±.191
3D Res U-Net [42] |  | 94.3±0.9 | 75.6±4.9 | 80.0±3.1 | 75.0±6.3 | 84.7±3.4 | 81.9±2.6 |  | .877±.135 | .538±.321 | .763±.192 | .509±.155 | .815±.129 | .700±.133
U-Net cSE [26] |  | 93.3±3.7 | 79.0±5.2 | 81.7±3.1 | 78.0±6.1 | 81.6±6.2 | 82.7±3.9 |  | 1.055±.555 | .467±.171 | .690±.204 | .437±.172 | 1.810±.384 | .892±.239
U-Net sSE [43] |  | 94.6±1.6 | 77.4±6.0 | 80.4±4.0 | 76.7±9.1 | 76.8±17.2 | 81.2±6.8 |  | .875±.301 | .438±.225 | .698±.186 | .476±.238 | 1.286±.389 | .755±.268
U-Net scSE [29] |  | 94.1±1.9 | 76.6±8.5 | 81.4±3.0 | 76.7±8.1 | 85.4±3.3 | 82.8±4.1 |  | .919±.310 | .419±.191 | .733±.198 | .476±.223 | 1.458±.989 | .801±.318
nnU-Net [17] |  | 94.9±0.6 | 78.6±4.9 | 82.1±2.4 | 78.0±5.1 | 87.1±2.9 | 84.1±1.8 |  | .787±.112 | .524±.079 | .679±.136 | .497±.106 | .697±.095 | .637±.074
FocusNet [14] |  | 94.6±0.9 | 78.5±6.6 | 81.2±2.6 | 76.5±5.4 | 87.5±3.4 | 83.7±2.4 |  | .838±.124 | .354±.137 | .695±.138 | .457±.117 | .770±.245 | .623±.108
3D SepNet [16] |  | 95.3±0.6 | 79.5±4.5 | 81.8±2.9 | 78.5±5.3 | 87.5±3.3 | 84.5±2.3 |  | .710±.098 | .316±.079 | .708±.200 | .419±.117 | .693±.110 | .569±.101
Ours ($\theta=1.0$) |  | 95.8±0.7 | 80.1±4.7 | 83.5±2.4 | 82.0±3.4 | 89.5±2.8 | 86.2±2.0 |  | .595±.089 | .371±.101 | .601±.155 | .353±.070 | .576±.052 | .499±.081
Ours |  | 96.1$\textbf{\textpm 0.7}^{*}$ | 80.4$\textbf{\textpm}\textbf{4.4}^{*}$ | 84.3$\textbf{\textpm 2.6}^{*}$ | 82.8$\textbf{\textpm}\textbf{2.5}^{*}$ | 89.9$\textbf{\textpm}\textbf{2.7}^{*}$ | 86.7$\textbf{\textpm}\textbf{1.7}^{*}$ |  | .589$\textbf{\textpm.135}^{*}$ | .311±.064 | .577$\textbf{\textpm}\textbf{.120}^{*}$ | .347±.065 | .557$\textbf{\textpm.057}^{*}$ | .476$\textbf{\textpm}\textbf{.074}^{*}$
Figure 6: Visual comparison of segmentation results obtained by different
methods, which contains three cases with the 2D visualizations in odd columns
and 3D visualizations in even columns. Yellow and red arrows highlight the
difference of local segmentation results.

Then, to validate the effectiveness of our attention mechanism using BFC, we
compared it with eight variants: 1) “$\times$” that means not using any
attention mechanism, 2) “HRB” that means only using the HRB calibration, 3)
“MRB” that means only using the MRB calibration, 4) “Enco” that represents
using our BFC only in the encoder blocks, 5) “Deco” that represents using our
BFC only in the decoder blocks, 6) “Bottom” that means using the BFC in the
smallest resolution level of the multi-resolution branch (channel number 128),
7) “Middle” that represents using BFC only in the middle resolution level of
the multi-resolution branch (channel number 64), and 8) “Top” that means that
using BFC only in the highest resolution level of the multi-resolution branch
(channel number 16). Note that “HRB” and “MRB” are monodirectional feature
calibration, and our proposed BFC is bidirectional feature calibration and
used at all the convolutoinal blocks except the first block. All these methods
were trained with the normal $L_{Dice}+L_{CE}$ loss for each substructure.
Quantitative evaluation results of these methods are listed in Table 4, which
shows that compared with not using attention, our BFC improved the average
Dice score from 85.8$\%$ to 86.2$\%$, and reduced average ASSD from 0.527 mm
to 0.499 mm, respectively. The proposed BFC also outperformed the other
counterparts, which demonstrates the superiority of our bidirectional feature
calibration compared with monodirectional calibration methods.

###  4.5 Effectiveness of Our Weighted Dice Loss

To investigate the performance of our false negative-weighted Dice loss
$L_{fn-Dice}$ combined with $L_{CE}$, we compared different strategies of
setting the value of $\theta$: 1) $\theta=1.0$ that is equal to using typical
$L_{Dice}+L_{CE}$, 2) $\theta=1.5$ for all structures, 3) $\theta=2.0$ for all
structures, and 4) “adaptive” that refers to using adaptive $\theta$ values
for different structures as described in Section 4.1.2. Results in Table 5
show that $\theta=1.5$ and $\theta=2.0$ outperformed $\theta=1.0$, which
demonstrates that assigning higher weights to false negatives could improve
the performance. Table 5 also shows that making $\theta$ adaptive to different
structures performed better than using the same $\theta$ value for all the
structures.

###  4.6 Comparison with State-of-the-art Methods

We compared our proposed framework with two recent methods designed for
multiple OAR segmentation from head and neck images: FocusNet [14] that is an
end-to-end two-stage CNN adopting a segmentation-by-detection strategy, and 3D
SepNet [16] that applies anisotropic convolutions to segment multiple organs
in images with large inter-slice spacing. They were also compared with six
state-of-the-art networks, i.e., 3D U-Net [18], nnU-Net [17], 3D Res U-Net
[42], U-Net cSE [26] that combines 3D U-Net with spatial squeeze and channel
excitation block, U-Net sSE [43] that combines 3D U-Net with channel squeeze
and spatial excitation block, and U-Net scSE [29] that combines 3D U-Net with
concurrent spatial and channel squeeze and excitation block. All the
alternative methods were trained with $L_{Dice}+L_{CE}$. Our method using
$L_{Dice}+L_{CE}$ in the first stage and $L_{f-seg}$ in the second stage was
also compared with using $L_{Dice}+L_{CE}$ for both two stages, which is
referred to as “Ours ($\theta=1.0$)”.

Table 6 shows quantitative comparison of these methods on the same testing
set. It can be observed that 3D U-Net [18] got the worst performance in
average, and the results of U-Net cSE, U-Net sSE and U-Net scSE demonstrate
that using attention networks helped to improve the segmentation accuracy.
FocusNet [14] and 3D SepNet [16] performed better than the other existing
methods, but they are inferior to our method. It should be noted that “Ours
($\theta=1.0$)” already outperformed existing methods when trained with the
same loss function, and using our $L_{fn-Dice}$ further improved the
performance, which achieved an average Dice and ASSD of 86.7$\%$ and 0.476 mm,
respectively. We also did a t-test between our proposed method and the state-
of-the-art methods. For the average Dice, the corresponding p-values were
0.0007(3D U-Net [18]), 0.0003(3D Res U-Net [42]) , 0.0106 ( U-Net cSE [26]),
0.0285 (U-Net sSE [43]), 0.0144 (U-Net scSE [29]), 0.0092 (nnU-Net [17]),
0.0003 (FocusNet [14]) and 0.0021 (3D SepNet [16]), respectively. All the
p-values were less than 0.05, indicating that our proposed method has a
significant improvement, as shown in Table 6. Fig. 6 shows a visual comparison
of these methods. It can be observed that our method can better deal with thin
structures like the falx cerebri and the brain sinuses than the others, as
highlighted by yellow and red arrows.

##  5 Discussion and Conclusion

In this paper, we focus on segmentation of five brain organs for radiotherapy
planning. The method was specifically designed for these organs due to their
special shapes. It would be of interest to investigate the performance of our
method when applied to other segmentation tasks and new data in the future. In
addition, a general problem for multi-modal segmentation is the requirement of
a registration between the different modalities. In this work, the three
modalities had been registered by the ABCs challenge organizers, so we did not
have an extra registration step for preprocessing. When our method is applied
to a new segmentation task, a spatial alignment for data preprocessing is
still necessary, the quality of which may have an impact on the segmentation
results. However, for brain structures, rigid registration is usually enough
to achieve good results for different modalities of the same patient [44].

There are still some limitations of this work. First, our high performance is
obtained at the cost of inference speed. Our two-stage method is less
efficient than one-stage segmentation methods. However, in the refine stage,
each refine model only deals with a subregion of the image, which is fast.
Currently, our refine step runs for each organ sequentially, and the entire
inference time is 285s, compared with 152s for nn-UNet [17] and 106s for 3D
U-Net [18]. If the refinement step for each organ runs in parallel, the
inference time can be 201s, which is only 48s slower than nnU-Net[17]. The
number of parameters of HMRNet is 3.4 million, compared with 2.7 million of
U-Net, and the Multiply–Accumulate Operations (MACs) of HMRNet and U-Net are
234.15 GMac and 116.45 GMac respectively. However, due to the large variance
of shapes and sizes of different organs, the two-stage method is more suitable
for small organs. Second, the network structure is designed manually which
maybe sub-optimal, it is of interest to use Network Architecture Search (NAS)
[45], [46] to find better networks for the segmentation task in the future.

In summary, we propose a novel framework for the segmentation of anatomical
brain barriers to cancer spread from multi-modal brain images. To deal with
multiple structures with different sizes and shapes, we propose HMRNet that
combines a high-resolution branch and a multi-resolution branch to leverage
different contextual information. Two variants of HMRNet, HMRNet-p and
HMRNet-s, are proposed for dealing with plump-like and slice-like structures,
respectively. We also introduce bidirectional feature calibration in HMRNet to
enable interaction and mutual calibration between features from the two
branches. Our framework is built on the top of the state-of-the-art nnU-Net
[17] to obtain a coarse localization result, and further uses our HMRNet
trained with a false negative weighted loss to achieve leading performance on
the MICCAI 2020 ABCs challenge dataset. In the future, it is of interest to
apply our HMRNet to other medical image segmentation tasks.

## References

  * [1] F. Bray, J. Ferlay, I. Soerjomataram, R. L. Siegel, L. A. Torre, and A. Jemal, “Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: A Cancer Journal for Clinicians_ , vol. 68, no. 6, pp. 394–424, 2018.
  * [2] J. Bernier, C. Domenge, M. Ozsahin, K. Matuszewska, J.-L. Lefèbvre, R. H. Greiner, J. Giralt, P. Maingon, F. Rolland, M. Bolla _et al._ , “Postoperative irradiation with or without concomitant chemotherapy for locally advanced head and neck cancer,” _New England Journal of Medicine_ , vol. 350, no. 19, pp. 1945–1952, 2004.
  * [3] J. A. Bonner, P. M. Harari, J. Giralt, R. B. Cohen, C. U. Jones, R. K. Sur, D. Raben, J. Baselga, S. A. Spencer, J. Zhu _et al._ , “Radiotherapy plus cetuximab for locoregionally advanced head and neck cancer: 5-year survival data from a phase 3 randomised trial, and relation between cetuximab-induced rash and survival,” _The lancet oncology_ , vol. 11, no. 1, pp. 21–28, 2010.
  * [4] A. S. Jones, B. Fish, J. E. Fenton, and D. J. Husband, “The treatment of early laryngeal cancers (T1–T2 N0): surgery or irradiation?” _Head & Neck: Journal for the Sciences and Specialties of the Head and Neck_, vol. 26, no. 2, pp. 127–135, 2004.
  * [5] S.-Z. Lai, W.-F. Li, L. Chen, W. Luo, Y.-Y. Chen, L.-Z. Liu, Y. Sun, A.-H. Lin, M.-Z. Liu, and J. Ma, “How does intensity-modulated radiotherapy versus conventional two-dimensional radiotherapy influence the treatment results in nasopharyngeal carcinoma patients?” _International Journal of Radiation Oncology* Biology* Physics_ , vol. 80, no. 3, pp. 661–668, 2011.
  * [6] N. Shusharina, J. Söderberg, D. Edmunds, F. Löfman, H. Shih, and T. Bortfeld, “Automated delineation of the clinical target volume using anatomically constrained 3D expansion of the gross tumor volume,” _Radiotherapy and Oncology_ , vol. 146, pp. 37–43, 2020.
  * [7] G. Bhatnagar, Q. J. Wu, and Z. Liu, “A new contrast based multimodal medical image fusion framework,” _Neurocomputing_ , vol. 157, pp. 143–152, 2015\.
  * [8] G. Sharp, K. D. Fritscher, V. Pekar, M. Peroni, N. Shusharina, H. Veeraraghavan, and J. Yang, “Vision 20/20: perspectives on automated image segmentation for radiotherapy,” _Medical Physics_ , vol. 41, no. 5, 2014.
  * [9] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. Van Der Laak, B. Van Ginneken, and C. I. Sánchez, “A survey on deep learning in medical image analysis,” _Medical Image Analysis_ , vol. 42, pp. 60–88, 2017.
  * [10] D. Shen, G. Wu, and H.-I. Suk, “Deep learning in medical image analysis,” _Annual Review of Biomedical Engineering_ , vol. 19, pp. 221–248, 2017.
  * [11] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in _Proc. MICCAI_. Springer, 2015, pp. 234–241.
  * [12] F. Milletari, N. Navab, and S.-A. Ahmadi, “V-net: Fully convolutional neural networks for volumetric medical image segmentation,” in _Proc. 3DV_. IEEE, 2016, pp. 565–571.
  * [13] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep learning in medical image analysis and multimodal learning for clinical decision support_. Springer, 2018, pp. 3–11.
  * [14] Y. Gao, R. Huang, M. Chen, Z. Wang, J. Deng, Y. Chen, Y. Yang, J. Zhang, C. Tao, and H. Li, “Focusnet: Imbalanced large and small organ segmentation with an end-to-end deep neural network for head and neck CT images,” in _Proc. MICCAI_. Springer, 2019, pp. 829–838.
  * [15] Y. Gao, R. Huang, Y. Yang, J. Zhang, K. Shao, C. Tao, Y. Chen, D. N. Metaxas, H. Li, and M. Chen, “Focusnet-v2: Imbalanced large and small organ segmentation with adversarial shape constraint for head and neck CT images,” _Medical Image Analysis_ , vol. 67, p. 101831, 2020.
  * [16] W. Lei, H. Mei, Z. Sun, S. Ye, R. Gu, H. Wang, R. Huang, S. Zhang, S. Zhang, and G. Wang, “Automatic segmentation of organs-at-risk from head-and-neck CT using separable convolutional neural network with hard-region-weighted loss,” _Neurocomputing_ , vol. 442, pp. 184–199, 2021.
  * [17] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” _Nature Methods_ , vol. 18, p. 203–211, 2021.
  * [18] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger, “3D U-Net: learning dense volumetric segmentation from sparse annotation,” in _Proc. MICCAI_. Springer, 2016, pp. 424–432.
  * [19] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proc. CVPR_ , 2015, pp. 3431–3440.
  * [20] P. F. Christ, M. E. A. Elshaer, F. Ettlinger, S. Tatavarty, M. Bickel, P. Bilic, M. Rempfler, M. Armbruster, F. Hofmann, M. D’Anastasi _et al._ , “Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields,” in _Proc. MICCAI_. Springer, 2016, pp. 415–423.
  * [21] T. Brosch, L. Y. Tang, Y. Yoo, D. K. Li, A. Traboulsee, and R. Tam, “Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 35, no. 5, pp. 1229–1239, 2016.
  * [22] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [23] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proc. ECCV_ , 2018, pp. 801–818.
  * [24] F. Yu and V. Koltun, “Multi-scale context aggregation by dilated convolutions,” in _Proc. ICLR_ , 2015, pp. 1–13.
  * [25] J. Wang, K. Sun, T. Cheng, B. Jiang, C. Deng, Y. Zhao, D. Liu, Y. Mu, M. Tan, X. Wang _et al._ , “Deep high-resolution representation learning for visual recognition,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 43, no. 10, pp. 3349 – 3364, 2021.
  * [26] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proc. CVPR_. IEEE, 2018, pp. 7132–7141.
  * [27] A.-M. Rickmann, A. G. Roy, I. Sarasua, N. Navab, and C. Wachinger, “‘project & excite’modules for segmentation of volumetric medical scans,” in _Proc. MICCAI_. Springer, 2019, pp. 39–47.
  * [28] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz _et al._ , “Attention u-net: Learning where to look for the pancreas,” in _Proc. MIDL_ , 2018, pp. 1–10.
  * [29] A. G. Roy, N. Navab, and C. Wachinger, “Recalibrating fully convolutional networks with spatial and channel “squeeze and excitation” blocks,” _IEEE Transactions on Medical Imaging_ , vol. 38, no. 2, pp. 540–549, 2018\.
  * [30] R. Gu, G. Wang, T. Song, R. Huang, M. Aertsen, J. Deprest, S. Ourselin, T. Vercauteren, and S. Zhang, “CA-Net: Comprehensive attention convolutional neural networks for explainable medical image segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 2, pp. 699–711, 2021\.
  * [31] J. Glaister, A. Carass, D. L. Pham, J. A. Butman, and J. L. Prince, “Automatic falx cerebri and tentorium cerebelli segmentation from magnetic resonance images,” in _Proc SPIE Int Soc Opt Eng._ , vol. 10137, 2017, p. 101371D.
  * [32] F. Van der Lijn, M. De Bruijne, S. Klein, T. Den Heijer, Y. Y. Hoogendam, A. Van der Lugt, M. M. Breteler, and W. J. Niessen, “Automated brain structure segmentation based on atlas registration and appearance models,” _IEEE Transactions on Medical Imaging_ , vol. 31, no. 2, pp. 276–286, 2011\.
  * [33] J. Glaister, A. Carass, D. L. Pham, J. A. Butman, and J. L. Prince, “Falx cerebri segmentation via multi-atlas boundary fusion,” in _Proc. MICCAI_. Springer, 2017, pp. 92–99.
  * [34] W. Zhu, Y. Huang, L. Zeng, X. Chen, Y. Liu, Z. Qian, N. Du, W. Fan, and X. Xie, “Anatomynet: Deep learning for fast and fully automated whole-volume segmentation of head and neck anatomy,” _Medical Physics_ , vol. 46, no. 2, pp. 576–589, 2019.
  * [35] Z. Cao, B. Yu, B. Lei, H. Ying, X. Zhang, D. Z. Chen, and J. Wu, “Cascaded SE-ResUnet for segmentation of thoracic organs at risk,” _Neurocomputing_ , vol. 453, pp. 357–368, 2021.
  * [36] X. Zou and Q. Dou, “Domain knowledge driven multi-modal segmentation of anatomical brain barriers to cancer spread,” _Segmentation, Classification, and Registration of Multi-modality Medical Imaging Data_ , vol. 12587, p. 16, 2021.
  * [37] L. Chen, P. Bentley, K. Mori, K. Misawa, M. Fujiwara, and D. Rueckert, “DRINet for medical image segmentation,” _IEEE Transactions on Medical Imaging_ , vol. 37, no. 11, pp. 2453–2462, 2018.
  * [38] M. Langhans, T. Fechter, D. Baltas, H. Binder, and T. Bortfeld, “Automatic segmentation of brain structures for treatment planning optimization and target volume definition,” _Segmentation, Classification, and Registration of Multi-modality Medical Imaging Data_ , vol. 12587, p. 40, 2021\.
  * [39] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, “Attention gated networks: Learning to leverage salient regions in medical images,” _Medical Image Analysis_ , vol. 53, pp. 197–207, 2019\.
  * [40] S. S. M. Salehi, D. Erdogmus, and A. Gholipour, “Tversky loss function for image segmentation using 3D fully convolutional deep networks,” in _International workshop on machine learning in medical imaging_. Springer, 2017, pp. 379–387.
  * [41] G. Wang, X. Liu, C. Li, Z. Xu, J. Ruan, H. Zhu, T. Meng, K. Li, N. Huang, and S. Zhang, “A noise-robust framework for automatic segmentation of covid-19 pneumonia lesions from CT images,” _IEEE Transactions on Medical Imaging_ , vol. 39, no. 8, pp. 2653–2663, 2020.
  * [42] F. I. Diakogiannis, F. Waldner, P. Caccetta, and C. Wu, “Resunet-a: a deep learning framework for semantic segmentation of remotely sensed data,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 162, pp. 94–114, 2020.
  * [43] A. G. Roy, N. Navab, and C. Wachinger, “Concurrent spatial and channel ‘squeeze & excitation’ in fully convolutional networks,” in _Proc. MICCAI_. Springer, 2018, pp. 421–429.
  * [44] D. De Nigris, D. L. Collins, and T. Arbel, “Multi-modal image registration based on gradient orientations of minimal uncertainty,” _IEEE Transactions on Medical Imaging_ , vol. 31, no. 12, pp. 2343–2354, 2012.
  * [45] D. Guo, D. Jin, Z. Zhu, T.-Y. Ho, A. P. Harrison, C.-H. Chao, J. Xiao, and L. Lu, “Organ at risk segmentation for head and neck cancer using stratified learning and neural architecture search,” in _Proc. CVPR_. IEEE, 2020, pp. 4223–4232.
  * [46] Z. Liu, H. Wang, S. Zhang, G. Wang, and J. Qi, “Nas-scam: Neural architecture search-based spatial and channel joint attention module for nuclei semantic segmentation and classification,” in _Proc. MICCAI_. Springer, 2020, pp. 263–272.