# Achieving Real-Time Object Detection on Mobile Devices with Neural Pruning
Search

1Pu Zhao, 2Wei Niu, 1Geng Yuan, 1Yuxuan Cai, 2Bin Ren, 1Yanzhi Wang, 1Xue Lin
1Northeastern University, Boston, MA
2William & Mary, Williamsburg, VA

###### Abstract

Object detection plays an important role in self-driving cars for security
development. However, mobile systems on self-driving cars with limited
computation resources lead to difficulties for object detection. To facilitate
this, we propose a compiler-aware neural pruning search framework to achieve
high-speed inference on autonomous vehicles for 2D and 3D object detection.
The framework automatically searches the pruning scheme and rate for each
layer to find a best-suited pruning for optimizing detection accuracy and
speed performance under compiler optimization. Our experiments demonstrate
that for the first time, the proposed method achieves (close-to) real-time,
55ms and 99ms inference times for YOLOv4 based 2D object detection and
PointPillars based 3D detection, respectively, on an off-the-shelf mobile
phone with minor (or no) accuracy loss.

##  I Introduction

As the rapid development of the autonomous vehicles, object detection
including 2D and 3D detection is one of the most important prerequisites to
autonomous navigation. It is essential to implement real-time object detection
on autonomous vehicles due to security considerations. However, as 2D and 3D
detection are implemented with deep neural networks (DNNs) such as YOLO [2]
and PointPillars [9], respectively, with tremendous memory and computation
requirements, it is challenging to achieve real-time on autonomous vehicles
with limited memory and computation resources.

To achieve real-time object detection on edge devices with limited resources,
we propose neural pruning search with compiler optimization to implement real-
time 2D detection with YOLO [2] and 3D object detection with PointPillars [9]
on mobile devices. We summarize our contribution as follows,

  * •

We propose to perform a novel _compiler-aware neural pruning search_ with
Bayesian optimization (BO), automatically determining the pruning scheme and
rate (including bypass) for each individual layer. The _objective_ is to
maximize accuracy satisfying an inference latency constraint on the target
mobile device.

  * •

We can achieve (close-to) real-time, 55ms and 99ms inference times for YOLOv4
based 2D detection and PointPillars based 3D detection, respectively, on an
off-the-shelf mobile phone with minor (or no) accuracy loss. Our method on 2D
detection notably outperforms other acceleration frameworks such as TVM [3]
and MNN [1], while we are the first to support 3D detection on mobile.

##  II Automatic Neural Pruning Search

The framework consists of two basic components: a _controller_ and an
_evaluator_. The controller first generates various _pruning proposals_ from
the search space. Then the evaluator evaluates their detection accuracy and
speed performance. Based on the performance, the evaluator provides guidance
for controller about what a satisfying pruning proposal looks like. Next the
controller generates new pruning proposals with the guidance. After
iterations, the controller outputs the best pruning proposal with desirable
detection performance while satisfying the real-time requirement.

###  II-A Controller

The controller generates _pruning proposals_ from the search space. Each
pruning proposal consists of the pruning scheme and rate for each layer of the
model, as shown in Tab. I.

TABLE I: Search space for each DNN layer

Pruning scheme |  {Filter [14], Pattern-based [12], Block-based [5]}
---|---
Pruning rate |  { 1$\times$, 2$\times$, 2.5$\times$, 3$\times$, 5$\times$, 7$\times$, 10$\times$, skip }

Per-layer pruning schemes: The controller can choose from filter (channel)
pruning [14], pattern-based pruning [12] and block-based pruning [5] for each
layer.

Per-layer pruning rate: We can choose from the list
$\\{1\times,2\times,2.5\times,3\times,5\times,7\times,10\times,\text{skip}\\}$,
where $1\times$ means the layer is not pruned, and “skip” means bypassing this
layer.

####  II-A1 Pruning Proposal Updating

The controller generates new proposals following the replacement probability
from the evaluator. It determines whether to replace each node in the
currently best proposal according to the replacement probability. Next if
replaced, the controller chooses randomly from two nodes with the lowest
probabilities as its replacement.

###  II-B Evaluator

The evaluator needs to evaluate pruning proposal performance. We define the
performance measurement (reward) as:

$r=V-\alpha\cdot\mathrm{max}(0,t-T),$ (1)

where $V$ is the validation mean average precision (mAP) of the model, $t$ is
the model inference speed or latency, which is actually measured on a mobile
device with compiler optimizations. $T$ is the threshold for the latency
requirement. Generally, $r$ is high when the model satisfies real-time
requirement ($t<T$) with high mAP. Otherwise $r$ is small, especially when the
real-time latency requirement is violated.

####  II-B1 Evaluation with BO

As evaluating each proposal needs to prune and retrain the model, incurring
large time cost, we use BO [4] to accelerate evaluation. As shown in Algorithm
1, given a proposal pool from the controller, we first adopt BO to select a
part of proposals with potentially better performance and evaluate their
accurate detection and speed performance, while the rest potentially weak
proposals are not evaluated. Thus, we reduce the number of actual evaluated
proposals.

To deal with the non-continuous and graph-like pruning proposals, we build a
Gaussian process (GP) for BO with a Weisfeiler-Lehman (WL) graph kernel [13].
We select the proposals according to their _Expected Improvement_ values.

After selecting $B$ pruning proposals from the pool, we evaluate their
performance using magnitude based framework [7] following their pruning
proposals for each layer.

####  II-B2 Gradients Guidance

To guide the proposal updating, we employ the derivatives of the GP predictive
mean with reference to the number of nodes in the graph. Basically, positive
gradients show that the node is beneficial to improve the reward, while
negative gradients mean that the node decreases the performance and it should
be replaced. To make the gradients more illustrative, we transform the
gradients into a probability distribution (replacement probability) using a
sigmoid transformation on the negative of the gradients and then normalize
them. Thus, negative gradients lead to high replacement probabilities. To
summarize, the evaluator provides the gradient guidance including the best
evaluated pruning proposal and its corresponding replacement probability
obtained from its gradients.

Input: Observation data $\mathcal{D}$, BO batch size $B$, BO acquisition
function $\alpha(\cdot)$ Output: The best pruning proposal $g$ for steps do
Generate a pool of candidate pruning proposals $\mathcal{G}_{c}$;  Select
$\\{\hat{g}^{i}\\}_{i=1}^{B}=\arg\max_{g\in\mathcal{G}_{c}}\alpha(g|\mathcal{D})$;
Evaluate the proposal and obtain reward $\\{r^{i}\\}_{i=1}^{B}$ of
$\\{\hat{g}^{i}\\}_{i=1}^{B}$;  Obtain the gradients guidance information;
$\mathcal{D}\leftarrow\mathcal{D}\cup(\\{\hat{g}^{i}\\}_{i=1}^{B},\\{r^{i}\\}_{i=1}^{B})$;
Update GP of BO with $\mathcal{D}$;  end for

Algorithm 1 Evaluation with BO

##  III Experimental Results

Figure 1: mAP vs. latency for various object detection approaches.

For 2D object detection, we use a YOLOv4 [2] model as starting point and test
on COCO dataset [10]. For 3D detection, we employ the PointPillars as starting
point [9] and test on KITTI dataset [6]. All the acceleration results are
tested on the mobile GPU of a Samsung Galaxy S20 smartphone.

For 2D detection, as shown in Fig. 1, on mobile GPU, our method achieves
5.18$\times$ inference acceleration (285.7ms vs. 55.2ms) compared with the
original model. Compared with other pruning schemes, under the same pruning
rate, our method is a bit slower than filter pruning on mobile GPU but
achieves much higher accuracy (49.3 vs. 25.2 in mAP). With slightly lower
accuracy, our method is 1.79$\times$ faster than unstructured pruning. We
achieve faster speed compared with other acceleration frameworks such as MNN
[1].

Figure 2: mAP vs. latency for various object detection approaches.

For 3D detection with point clouds, we start from PointPillars and test with
different different grid sizes (0.16m and 0.24m). The real-time requirements
are set to 200ms for 0.16m gird size and 100ms for 0.24m. As shown in Tab. II
and Fig. 2, we can observe that, for the same grid size, our method can
significantly reduce the parameter count and computation, thus satisfying the
real-time requirement, while achieving state-of-the-art detection performance.
For a grid size of 0.24m, under the same overall pruning ratio (86%), the
proposed method can achieve the best detection performance compared with other
methods with the same pruning scheme for each layer, demonstrating the
advantages of using flexible pruning scheme for each layer. Besides, with
compiler optimization, filter pruning is the fastest but suffers from obvious
detection performance degradation. The proposed method can process one LiDAR
image within 99ms with the highest precision, achieving (close-to) real-time
inference on mobile.

TABLE II: Comparison of various pruning methods for PointPillars

Methods (grid size) |  Para. # |  Comp. # (MACs) |  Speed (ms) | Car 3D detection
---|---|---|---|---
Easy | Moderate | Hard
PointPillars (0.16) | 5.8M | 60G | 542 | 84.99 | 74.11 | 69.53
Ours (0.16) | 1.1M | 10.7G | 189 | 85.50 | 76.58 | 70.23
PointPillars (0.24) | 5.8M | 28G | 257 | 84.05 | 74.99 | 68.30
Filter [8] (0.24) | 0.8M | 4.0G | 81 | 81.54 | 68.10 | 65.90
Pattern [11] (0.24) | 0.8M | 3.9G | 111 | 80.97 | 73.77 | 68.05
Ours (0.24) | 0.8M | 3.9G | 99 | 85.08 | 75.19 | 68.10

##  IV Acknowledgements

This project is partly supported by National Science Foundation (NSF) under
grants CNS-1932351, CNS-1909172, and CMMI-2013067, Army Research Office (ARO)
Young Investigator Program 76598CSYIP, a grant from Semiconductor Research
Corporation (SRC), and Jeffress Trust Awards in Interdisciplinary Research.
Any opinions, findings, and conclusions or recommendations in this material
are those of the authors and do not necessarily reflect the views of NSF, ARO,
SRC, or Thomas F. and Kate Miller Jeffress Memorial Trust.

## References

  * [1] https://github.com/alibaba/MNN.
  * [2] A. Bochkovskiy, C.-Y. Wang, and H.-Y. M. Liao, “Yolov4: Optimal speed and accuracy of object detection,” _arXiv:2004.10934_ , 2020.
  * [3] T. Chen, T. Moreau _et al._ , “Tvm: An automated end-to-end optimizing compiler for deep learning,” in _USENIX_ , 2018, pp. 578–594.
  * [4] Y. Chen, A. Huang _et al._ , “Bayesian optimization in alphago,” _arXiv:1812.06855_ , 2018.
  * [5] P. Dong, S. Wang _et al._ , “Rtmobile: Beyond real-time mobile acceleration of rnns for speech recognition,” _arXiv:2002.11474_ , 2020.
  * [6] A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in _CVPR_ , 2012.
  * [7] S. Han, J. Pool _et al._ , “Learning both weights and connections for efficient neural network,” in _NeurIPS_ , 2015, pp. 1135–1143.
  * [8] Y. He, P. Liu _et al._ , “Filter pruning via geometric median for deep convolutional neural networks acceleration,” in _CVPR_ , 2019.
  * [9] A. H. Lang, S. Vora _et al._ , “Pointpillars: Fast encoders for object detection from point clouds,” in _CVPR_ , 2019, pp. 12 697–12 705.
  * [10] T.-Y. Lin _et al._ , “Microsoft coco: Common objects in context,” in _ECCV_.
  * [11] X. Ma _et al._ , “Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices,” in _AAAI_ , 2020.
  * [12] W. Niu _et al._ , “Patdnn: Achieving real-time dnn execution on mobile devices with pattern-based weight pruning,” _arXiv:2001.00138_ , 2020.
  * [13] N. Shervashidze, P. Schweitzer _et al._ , “Weisfeiler-lehman graph kernels,” _Journal of Machine Learning Research_ , vol. 12, no. 77, 2011\.
  * [14] Z. Zhuang, M. Tan _et al._ , “Discrimination-aware channel pruning for deep neural networks,” in _NeurIPS_ , 2018, pp. 875–886.


# Iron and gold thin films: first-principles study

Justyna Rychły-Gruszecka, Hubert Głowiński, Justyn Snarski-Adamski, Piotr
Kuświk, Mirosław Werwiński  Institute of Molecular Physics, Polish Academy of
Sciences, Smoluchowskiego 17, 60-179 Poznań, Poland

###### Abstract

Using density functional theory, we carried out systematic calculations for a
series of ultrathin iron layers with thicknesses ranging from one atomic
monolayer to eleven monolayers (up to about 1.5 nm). We considered three
cases: (1) iron layers both on a gold substrate and coated with gold, (2) iron
layers on a gold substrate but without coverage and (3) free-standing iron
layers adjacent to a vacuum. For our models we chose initial bcc Fe(001)
surfaces and fcc Au(001) substrates. Based on the calculations, we determined
the details of the geometry and magnetic properties of the systems. We
calculate lattice parameters, magnetic moments, Curie temperatures and
magnetocrystalline anisotropy energies. From the thickness dependence we
determined the volume and surface contributions to the magnetic anisotropy
constant. The further analysis allowed us to determine the thickness ranges of
the occurrence of perpendicular magnetic anisotropy, as well as the effect of
thickness and the presence of a substrate and cap layer on the direction of
the magnetization easy axis.

###### Index Terms:

iron, gold, thin films, magnetic anisotropy, magnetocrystalline anisotropy,
density functional theory

One of the most important properties of a magnetic thin film is its
magnetocrystalline anisotropy energy (MAE) since it is one of the key factors
determining the orientation of magnetization at remanence. Sufficiently strong
anisotropy with the axis of easy magnetization perpendicular to the surface of
the layers allows perpendicular magnetization of the sample in remanence,
which, for example, is required in magnetic tunnel junctions to maximize
resistance output. Thin film systems with perpendicular anisotropy, due to the
possibility of obtaining magnetic textures with perpendicularly magnetized
very narrow domains, are also important for data storage applications. In this
context, layered systems are particularly interesting because of their ability
to tune effective material parameters, such as magnetic anisotropy. Among such
systems, iron-based layered heterostructures are of significant interest.

First-principles calculations of iron layers on gold have been going on since
the 1980s [1]. It was predicted, for example, that in Fe/Au multilayers above
the thickness of three atomic monolayers of iron, there is a change in the
sign of the MAE interpreted as a change in the direction of magnetization of
the layer from perpendicular (perpendicular magnetic anisotropy - PMA) to
lying in the plane of the layer [2]. The calculations showed a significant
increase in the magnetic moment values of several atomic layers at both the Fe
surface and Fe/Au interfaces. Earlier calculations, however sophisticated,
usually failed to reproduce accurate magnetic results due to the potential
shape approximations used, as well as the most commonly assumed bulk values of
lattice parameters and the lack of optimization of geometries of layers and
heterostructures.

In this work we performed first-principles calculations for different
thicknesses of iron layers deposited on fcc-gold substrates. We have
considered a range of iron layers from single atomic monolayer up to eleven
monolayers. In all the heterostructures considered, the thickness of the gold
layers equals eleven atomic monolayers. We consider three types of structures.
The first type is sandwich-type structures – for which layers of iron and gold
are alternated with each other, see Fig. 1.

Figure 1:  Selected structural models of Au11Fen multilayers. In each case,
several unit cells are shown in the layer plain. In contrast, only one unit
cell is shown in the direction perpendicular to the layer plain.

The second type is iron layers with two different surfaces - surrounded by a
gold substrate on one side and a vacuum on the other. The third type is free-
standing iron layers adjacent to a vacuum.

We performed density functional theory (DFT) calculations using the full-
potential local orbital (FPLO) code [3] in the FPLO18.00-52 version. Using
Hellmann–Feynman forces, we optimized the lattice parameter $a$ and the
Wyckoff positions $z$ which allowed us to determine the lattice parameter $c$.

Figure 2:  Lattice parameters calculated for Fe layers in Au11Fen multilayers
and for Fe layers in Au11Fen heterostructure with vacuum (uncovered Fe
surface). Calculations were performed with the FPLO18 code using the PBE
functional.  Figure 3:  Magnetocrystalline anisotropy energy (MAE) as a
function of Fe layer thickness (expressed as the number of atomic monolayers
n) for Fe layers in Au11Fen multilayers and for Fe layers in Au11Fen
heterostructure with vacuum (uncovered Fe surface). Results for free-standing
Fe layers are included as a reference. Calculations were performed with the
FPLO18 code using the PBE functional.

Figure 2 shows the dependence of lattice parameters on Fe layer thickness.
Assuming perfect epitaxial growth of Fe layers on Au substrate in the model,
the heterostructure model is described by a single unit cell with a lattice
parameter $a$ common to Fe and Au layers. In the case of gold layers, the
presented lattice parameter $a$ defines an fcc Au unit cell in a body centered
tetragonal (bct) representation. Due to the influence of the Au substrate, the
obtained $a$ ($c$) values are higher (lower) than the values for the bulk Fe
structure. The highest (lowest) values of obtained $a$ ($c$) occur for the
thinnest one-atom-thick bcc-Fe layer. (In addition, the figure shows the
values of lattice parameters $a$ for bulk fcc Au and bcc Fe.) The thinner the
Fe layer, the more its structure deviates from cubic toward a flattened
tetragonal cell with a lattice parameter ratio $c/a$ of about 0.9. As is known
from previous studies, such a relatively strong deformation due to the
influence of the substrate and overlay layer can have a large impact on the
value and direction of magnetic anisotropy.

Figure 3 shows the dependence of magnetocrystalline anisotropy energy (MAE) on
the thickness of the Fe layer for both Au/Fe multilayers, as well as for Fe
layers on Au substrate without a cap Au layer. Multilayers tend to magnetize
out-of-plane for thin Fe layers. However, for thicker Fe layers they magnetize
in-plane. This is similar behavior to that found for free-standing Fe layers.
However, this relationship is reversed for Fe layers on Au substrate
surrounded by vacuum. The above result points to the presence of a cap layer
as another important factor affecting the direction and value of magnetic
anisotropy.

Figure 4:  Effective uniaxial anisotropy ($K_{u}$) as a function of inverse
thickness (1/$t$) for free-standing Fe layers with thickness of two to eight
monolayers (bct phase). Fit allows determination of surface magnetic
anisotropy ($K_{s}$).

Figure 4 shows the effective uniaxial anisotropy ($K_{u}$) of thin Fe films as
a function of the inverse of thickness (1/$t$). Fitting with the equation
$K_{u}=K_{v}+2K_{s}/t$ [4] allows us to determine the surface magnetic
anisotropy ($K_{s}$), which is equal to 0.86 mJ m-2 and is in good agreement
with experimental results. It is known that the value of $K_{s}$ strongly
depends on the type of interface. Heinrich et al. [5] observed the strongest
surface anisotropy $K_{s}$ for the Fe(001)/vacuum interface (0.96 mJ m-2 (0.96
ergs/cm2)), followed by the Fe/Ag (0.81 mJ m-2) and the Fe/Au (0.47 mJ m-2)
interfaces, all at room temperature. At lower temperatures (77 K), the latter
two results are slightly higher (no low $T$ data for the Fe/vacuum interface).

## Acknowledgement

We acknowledge the financial support of the National Science Center Poland
under the decision DEC-2018/30/E/ST3/00267 (SONATA-BIS 8). Part of the
computations were performed on resources provided by the Poznan Supercomputing
and Networking Center (PSNC).

## References

  * [1] C. Li, A. J. Freeman, and C. L. Fu, “Monolayer magnetism: Electronic and magnetic properties of Fe/Au(001),” _J. Magn. Magn. Mater._ , vol. 75, no. 3, pp. 201–208, Dec. 1988.
  * [2] L. Szunyogh, B. Újfalussy, and P. Weinberger, “Magnetic anisotropy of iron multilayers on Au(001): First-principles calculations in terms of the fully relativistic spin-polarized screened KKR method,” _Phys. Rev. B_ , vol. 51, no. 15, pp. 9552–9559, Apr. 1995.
  * [3] K. Koepernik and H. Eschrig, “Full-potential nonorthogonal local-orbital minimum-basis band-structure scheme,” _Phys. Rev. B_ , vol. 59, no. 3, pp. 1743–1757, 1999.
  * [4] K. Kyuno, J.-G. Ha, R. Yamamoto, and S. Asano, “First-Principles Calculation of the Magnetic Anisotropy Energies of Ag/Fe(001) and Au/Fe(001) Multilayers,” _J. Phys. Soc. Jpn._ , vol. 65, no. 5, pp. 1334–1339, May 1996.
  * [5] B. Heinrich, Z. Celinski, J. F. Cochran, A. S. Arrott, and K. Myrtle, “Magnetic anisotropies in single and multilayered structures (invited),” _J. Appl. Phys._ , vol. 70, no. 10, pp. 5769–5774, Nov. 1991.


TuNet: End-to-end Hierarchical Brain Tumor Segmentation using Cascaded Networks

Minh H. Vu1 Tufve Nyholm1 Tommy Löfstedt1

Department of Radiation Sciences, Umeå University, Umeå, Sweden

Glioma is one of the most common types of brain tumors; it arises in the glial cells in the human brain and in the spinal cord. In addition to having a high mortality rate, glioma treatment is also very expensive. Hence, automatic and accurate segmentation and measurement from the early stages are critical in order to prolong the survival rates of the patients and to reduce the costs of the treatment. In the present work, we propose a novel end-to-end cascaded network for semantic segmentation that utilizes the hierarchical structure of the tumor sub-regions with ResNet-like blocks and Squeeze-and-Excitation modules after each convolution and concatenation block. By utilizing cross-validation, an average ensemble technique, and a simple post-processing technique, we obtained dice scores of 88.06, 80.84, and 80.29, and Hausdorff Distances (95th percentile) of 6.10, 5.17, and 2.21 for the whole tumor, tumor core, and enhancing tumor, respectively, on the online test set.

§ INTRODUCTION

Glioma is among the most aggressive and dangerous types of cancer [11], leading, for instance, to around 80 % and 75 % of all malignant brain tumors diagnosed in the United States [8] and Sweden [1], respectively. Gliomas with different prognosis and numerous heterogeneous histological sub-regions, such as edema, necrotic core, and enhancing and non-enhancing tumor core, are classified into four world health organisation (WHO) grades according to their aggressiveness: lgg (class I and II, considered as slow-growing), and hgg (class III and IV, considered as fast-growing).

The aim of the brats [16, 4, 5, 3, 2] is in evaluating and finding new state-of-the-art methods for tumor segmentation and to set a gold standard for the segmentation of intrinsically heterogeneous brain tumors. brats provided a large data set comprising multi-institutional pre-operative mri scans with four sequences: t1c, t2, t1, and flair. Masks were annotated manually by one to four raters followed by improvements by expert raters. The segmentation performance of the participants was measured using dsc, sensitivity, specificity, and hd95.

[]Schematic visualization of TuNet. The input is the mri volumes as four different channels, while the output is the predicted masks for the three labels: the necrotic and non-enhancing tumor core (NCR/NET—label 1, green), the peritumoral edema (ED—label 2, yellow) and the GD-enhancing tumor (ET—label 3, red). First, the multi-modal input is fed into the W-Net to generate a probability map of the whole tumor region (orange) including labels 1, 2 and 3. Second, the concatenation of the whole tumor probability map and the input is passed through the C-Net to produce the tumor core probability map (blue) including labels 1 and 3.
Third, the two obtained maps from W-Net and C-Net are concatenated with the multi-modal input and then fed into the E-Net to generate an enhancing tumor probability map including label 3. Last, the outputs of W-Net, C-Net, and E-Net are merged to produce the final brain tumor mask.

Traditional discriminative approaches, such as

Support-vector Machines [6], have been widely used in medical image segmentation. In recent years, cnn have achieved state-of-the-art performance in numerous computer vision tasks.
In the field of medical image segmentation, a fully convolutional encoder-decoder neural network named U-Net, introduced by Ronneberger et al. [18], has received a lot of attention in recent years. With its success, it is unsurprising that the U-Net has motivated many top-ranking teams in segmentation competitions in previous years [13, 7, 17, 19].

[]Our C-Net and E-Net with a U-Net backbone. The W-Net does not include the probability map produced by a previous sub-network. To enrich the feature map at each level in the encoder part, we concatenate the down-sampled original input (light green) with the down-sampled output (grey) of the convolution block (yellow). Each convolution block comprises a convolution operation, an instance normalization followed by a $\relu$ activation function.

The W-Net constrains the C-Net, while the C-Net constrains the E-Net.

The E-Net functions as a regularizer for the C-net, while the C-Net plays the same role for the W-Net.

Kitrungrotsakul et al. [15] presented CasDetNet-CLSTM to detect mitotic events in 4D microscopic images by introducing a connection between a region-based convolutional neural network and convolutional long short-term memory. In the BraTS18, instead of proposing a new network, Isensee et al. [13] focused on the training process, and made only minor modifications to the U-Net, used additional training data, and applied a simple post-processing technique. Myronenko [17] was the winner of BraTS18 by introducing a U-Net-like network with an asymmetrically large encoder and a vae branch to reconstruct the input image to add guidance and regularization to the encoder part of the network.

In another work, Wang et al. [19] proposed an encoder-decoder cascaded anisotropic cnn, that won the second place in the BraTS17, that hierarchically segments the whole tumor, tumor core, and enhancing tumor core sequentially by separating the complex brain tumor segmentation problem into three binary segmentation problems. We argue that: (1) the training process employed in [19] might be time-consuming as there are three separate binary segmentation problems, and (2) the lack of regularization could lead to overfitting.

Motivated by the successes of the cascaded anisotropic network, introduced in [19], and the vae branch, presented in [17], we propose a novel architecture, denoted tunet. tunet exploits separating a complex problem into less challenging sub-problems, and also attempts to avoid overfitting by using heavy regularization through multi-task learning. In the present work, we constructed three U-Net-like networks, each was used to segment a specific tumor region, i.e. whole tumor, core tumor, and enhancing tumor. We connected the three sub-networks to form an end-to-end cascaded network in the hope that three sub-networks could mutually regularize each other to prevent overfitting and reduce the training time.

§ METHODS

Motivated by the drawbacks in [19] and the vae branch in [17], we thus propose a framework that we expect not only utilizes region-based optimization, but we also hypothesise that it will prevent or reduce overfitting.

§.§ Cascaded Architecture

fig:method illustrates the proposed end-to-end cascaded architecture. As in [19], we employed three encoder-decoder networks to cope with the three aforementioned tumor regions. We denote them W-Net (whole tumor network), C-Net (core tumor network), and E-Net (enhancing tumor network). In our proposed approach, instead of using three separate networks, we joined them together to form an end-to-end cascaded network.

§.§ Segmentation Network

fig:unet shows the proposed symmetric encoder-decoder C-Net and E-Net. The patch size was set to $80 \times 96 \times 64$. Note that the probability map (pink) was concatenated with the input image as a first step for the C-Net and the E-Net, but not for the W-Net. We also concatenated the max-pooled feature maps with the down-sampled input (light green) in order to enhance the feature maps at the beginning of each level in the encoder part. The base number of filters was set to 16 and the number of filters were doubled at each level. Skip-connections were used like in the U-Net. The probability maps produced at the end of the decoder part of each sub-network had the same spatial size as the original image and were activated by a logistic sigmoid function.

§.§ Multi-view Fusion

For the 2D version, we trained our proposed network in three individual planes (axial, sagittal and coronal) at the slice-level to take advantage of multi-view contextual information. The trained networks were then used to generate probability maps that were later merged

by a consensus fusion strategy to produce the final brain tumor mask.

It can be seen that the final output probability map is superior, in terms of qualitative and quantitative evaluation, with respect to the maps produced by a single-view model (see Results and Discussion).

§.§ ResNet Block

He et al. [9] proposed the ResNet as a way to make training deep networks easier, and won the 1$^\text{st}$ place in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2015 classification task. It has been shown that ResNet-like networks are not only easier to optimize but also boost accuracy from considerably deeper networks [9, 17]. Inspired by the winner of BraTS18 [17], we replaced the convolution blocks used in the traditional U-Net by res in two of our models (see fig:unet and tab:cv). An illustration of the ResNet-like block can be seen in fig:compare.

§.§ Squeeze-and-Excitation Block

We added a seb as developed by Hu et al. [12]
after each convolution and concatenation block. The seb is a computationally efficient means to incorporate channel-wise inter-dependencies, and has been widely used to improve network performances by significant margins [12]. seb is also illustrated in fig:compare.

[]The convolution blocks used in the different experiments (see tab:cv). seb, res$^1$ and res$^2$ denote seb and two variations of res, respectively. Here, IN and FC stand for instance normalization and fully-connected layer, respectively; while rectified linear unit (ReLU) and Sigmoid are activation functions.

§.§ Preprocessing and Augmentation

We normalized all input images to have mean zero and unit variance. To increase the data set size, we employed simple on-the-fly data augmentation by randomly rotating the images within a range of $-1$ to $1$ degrees and random mirror flips (on the $x$-axis) with a probability of 0.5. We also experimented with median denoising, but it did not demonstrate any additional improvements.

§.§ Post-processing

One of the most difficult tasks of brats is to detect small vessels in the tumor core and to label them as edema or as necrosis. To cope with the fact that lgg patients may have no enhancing tumor region, Isensee et al. [13] proposed to replace all enhancing tumor regions with less than 500 voxels by necrosis. We employed that approach

in the present work as well.
Additionally, we kept decreasing the threshold segmentation from 0.5 to 0.3, 0.1, 0.05, and 0.01, respectively, if no core tumor region was found. This technique resolved several cases where the networks failed to distinguish between core and whole tumor.

§.§ Ensemble of Multiple Models

We employed five-fold cross-validation when training on the 335 cases (259 hgg + 76 lgg) of brats, and did not use any additional in-house data.

We trained 30 models in total and used 15 of them, that correspond to tunet + seb, tunet + res$^1$ + seb and tunet + res$^2$ + seb, to cast votes when predicting the final label maps. We used an average ensemble approach, such that

\begin{equation} \label{eqn:ensemble_avg}
p_{c} = \frac{1}{M} \sum_{m=1}^{M} f_{mc},
\end{equation}

where $p_c \in \mathbb{R}^{|C|}$ and $f_{mc} \in \mathbb{R}^{|C|}$

denote the final probability of label $c$ and the probability of label $c$ generated by model $m=1,\ldots,M$ at an arbitrary voxel, respectively. Here, $C=\{1, 2, 3\}$ is the set of tumor labels, and thus $c \in C$.

The ensemble that we used was computed as the average of the prediction outputs of tunet + seb, tunet + res$^1$ + seb and tunet + res$^2$ + seb (see tab:cv and tab:valid).

§.§ Task 3: Quantification of Uncertainty in Segmentation

In brats, the organizers decided to include a new task that focuses on exploring uncertainty measures in the context of tumor segmentation on three glioma regions: whole, core, and enhancing. This task, called “Quantification of Uncertainty in Segmentation”, aimed at rewarding participating methods with resulting predictions that are: (a) confident when correct and (b) uncertain when incorrect. Participants were called on to generate uncertainty maps associated with the resulting labels at every voxel with values in $[0, 100]$, where 0 represents the most certain prediction and 100 represents the most uncertain.

Our proposed approach, tunet, benefits from this task since, fundamentally, it segments the brain tumor mri images into three aforementioned tumor regions instead of partitioning into three labels. We define an uncertainty score, $u^r_{i,j,k}$, at voxel $(i,j,k)$ as:

\begin{equation}
u^r_{i,j,k} =
\begin{cases}
200 (1-p^{r}_{i,j,k}) & \text{if } p^{r}_{i,j,k} \geq 0.5 \\
200 p^{r}_{i,j,k}     & \text{if } p^{r}_{i,j,k} < 0.5
\end{cases}
\end{equation}

where thus $u^r_{i,j,k} \in [0, 100]^{|\mathcal{R}|}$ and $p^{r}_{i,j,k} \in [0, 1]^{|\mathcal{R}|}$

denote the uncertainty score map and probability map (the network's likelihood outputs) corresponding to tumor region, $r \in \mathcal{R}$, as produced by the TuNet (see fig:method), where $\mathcal{R}$ is the set of tumor regions, i.e. whole, core, and enhancing region.

§ EXPERIMENTS

We implemented our network in  Keras 2.2.4[<https://keras.io>] using TensorFlow 1.12.0[<https://tensorflow.org>] as the backend. This research was conducted using the resources of the High Performance Computing Center North (HPC2N)[<https://www.hpc2n.umu.se/>] at Umeå University, Umeå, Sweden, and the experiments were run on NVIDIA Tesla V100 16 GB GPUs.

We report the results on the brats training set (335 cases) using cross-validation and on the validation set (125 cases) by uploading our predicted masks and corresponding uncertainty maps to the evaluation server. The evaluation metrics of Task 1—Segmentation included: dsc, sensitivity, specificity, and hd95; while the evaluation metrics of Task 3—Quantification of Uncertainty in Segmentation were dauc and rftp.

§.§ Implementation Details and Training

For evaluation of the segmentation performance, we employed the dsc, defined as

\begin{equation}
D(X,Y)=\frac{2 |X \cap Y|}{|X| + |Y|},
\end{equation}

where $X$  and $Y$ denote the output segmentation and its corresponding ground truth, respectively.

The hd95 is defined as the largest value in the set of closest distances between two structures, or

\begin{equation}
H(X,Y)=\max\big\{ \max_{x\in X} \min_{y \in Y} d(x,y), \max_{y\in Y} \min_{x \in X} d(y,x) \big\},
\end{equation}

where $d(x,y)$ denotes the Euclidian distance between two points $x \in X$ and $y \in Y$. It is common practice to report the $95^{th}$ percentile instead of the maximum to compensate for outliers.

The loss function used for training contained three terms,

\begin{equation}
\mathcal{L}(u, v) = \mathcal{L}_{\textit{whole}}(u, v) + \mathcal{L}_{\textit{core}}(u, v) + \mathcal{L}_{\textit{enh}}(u, v),
\end{equation}

where $\mathcal{L}_{\textit{whole}}$,  $\mathcal{L}_{\textit{core}}$, and $\mathcal{L}_{\textit{enh}}$ where the soft dice loss of whole, core, and enhancing tumor regions, respectively, and where the soft dice loss is defined as

\begin{equation}
\mathcal{L}_{dice}(u, v) = \frac{-2 \sum_i u_i v_i}{\sum_i u_i + \sum_i v_i + \epsilon},
\end{equation}

in which $u$ is the softmax output of the network, $v$ is a one-hot encoding
of the ground truth segmentation map, and $\epsilon$ is a small constant added to avoid division by zero.

We used the Adam optimizer [14] with a learning rate of $1 \cdot 10^{-4}$ and momentum parameters of $\beta_1=0.9$ and $\beta_2=0.999$. We also used $L_2$ regularization with a penalty parameter of $1 \cdot 10^{-5}$, that was applied to the kernel weight matrices, for all convolutional layers to cope with overfitting. The activation function of the final layer was the logistic sigmoid function.

All models were trained for 200 epochs, with a mini-batch size of four. To prevent over-fitting, we selected a patience period and dropped the learning rate by a factor of $0.2$ if the validation loss did not improve over six epochs. Further, the training process was stopped if the validation loss did not improve after $15$ epochs. The training time for a single model was around 35 hours on an NVIDIA Tesla V100 GPU.

1 cm
3.7 cm
Mean dsc (higher is better) and hd95 (lower is better) and their sd (in parentheses) computed from the five-folds of cross-validation on the training set (335 cases) for the different models. Details of the variations of the convolution blocks used for each model is shown in fig:compare.

3cdsc        3chd95

Model                                              whole      core       enh.          whole      core       enh.

tunet                                        89.89 (2.07)       84.08 (4.02)         74.92 (7.24)            6.22 (2.66)      6.82 (2.93)      4.72 (2.21)

tunet + seb                            91.38 (2.02)       85.93 (3.77)         78.11 (7.29)            4.50 (2.34)      6.70 (2.72)      4.24 (2.07)

tunet + res$^1$                        90.68 (1.98)       86.07 (3.84)         75.88 (7.36)            5.14 (2.07)      5.16 (1.99)      4.20 (1.99)

tunet + res$^1$ + seb            90.86 (2.12)       86.30 (3.72)         76.20 (7.32)            5.10 (2.29)      5.72 (2.62)      3.89 (1.61)

tunet + res$^2$                        90.77 (2.18)       85.84 (3.96)         75.22 (7.30)            5.42 (2.50)      6.00 (2.38)      4.56 (2.09)

tunet + res$^2$ + seb            91.90 (2.00)       86.09 (3.81)         77.43 (7.28)            5.07 (2.38)      6.20 (2.45)      3.98 (1.87)

Ensemble                                           91.92 (2.18)       86.35 (3.81)         78.01 (7.30)            5.30 (2.19)      5.80 (2.69)      3.45 (1.93)

Ensemble + post-process                            91.92 (2.18)       86.45 (3.79)         78.72 (7.10)            5.30 (2.19)      5.75 (2.65)      3.08 (1.90)

§ RESULTS AND DISCUSSION

The key strengths of the proposed method are: (1) it takes advantage of the hierarchical structure of the tumor sub-regions, since the whole tumor region must contain the core tumor, and the core tumor region must contain the enhancing tumor region, and (2) it consists of three connected sub-networks that mutually regularize each other, i.e. the E-Net functions as a regularizer for the C-net, C-Net plays that role for the W-Net, and the W-Net and the E-Net constrain each other through the C-Net.

1. cm
3.7 cm
Results of Segmentation Task on brats validation data (125 cases). The results were obtained by computing the mean of predictions of five models trained over the folds. “UmU” denotes the name of our team and the ensemble of tunet + seb, tunet + res$^1$ + seb and tunet + res$^2$ + seb.  The metrics were computed by the online evaluation platform. All the predictions were post-processed before submitting to the server. Bottom rows correspond to the top-ranking teams from the online system.

3cdsc        3chd95

Model                                              whole        core       enh.            whole      core       enh.

tunet + seb                              90.41      81.67      78.97           4.35       6.12       3.35

tunet + res$^1$ + seb              90.06      79.43      78.12           4.66       7.98       3.34

tunet + res$^2$ + seb              90.34      79.14      77.38           4.29       8.80       3.57

UmU                                                  90.34     81.12     78.42             4.32          6.28      3.70

ANSIR                                                90.09     84.38     80.06             6.94          6.00      4.52

lfn_                                                90.91     85.48     80.24             4.35          5.32      3.88

SVIG1                                                91.16     85.79     81.33             4.10          5.92      4.21

NVDLMED                                              91.01     86.22     82.28             4.42          5.46      3.61

1. cm
3.7 cm
Results of Segmentation Task on brats test data (166 cases).

3cdsc        3chd95

Model                                              whole        core       enh.            whole      core       enh.

UmU                                                  88.06     80.84     80.29             6.10       5.17       2.20

1. cm
3.7 cm
Results of Quantification of Uncertainty Task on brats validation data (125 cases) including mean dauc (higher is better) and rftp (lower is better). The results were obtained by computing the mean of predictions of five models trained over the folds. “UmU” denotes the name of our team and the ensemble of tunet + seb, tunet + res$^1$ + seb and tunet + res$^2$ + seb.  The metrics were computed by the online evaluation platform. The bottom rows correspond to the top-ranking teams from the online system. The proposed method was ranked among the top that was evaluated on the online test set.

3cDAUC        3cRFTPs

Model                                              whole        core       enh.          whole      core       enh.

tunet + seb                              87.52      79.90      75.97         4.50       12.97      6.59

tunet + res$^1$ + seb              86.13      80.01      75.13         5.80       16.50      8.02

tunet + res$^2$ + seb              87.40      79.32      75.90         5.50       15.30      8.53

UmU                                                  87.47      79.86      75.89         5.83       13.72      8.46

ANSIR                                                89.42      88.72      87.67         1.23       6.16       4.63

NVDLMED                                              89.95      88.03      84.67         1.68       2.43       2.86

TEAM_ALPACA                                         90.93      88.08      87.90         1.54       3.82       2.59

tab:cv shows the mean dsc and hd95 scores and sd computed from the five-folds of cross-validation on the training set. As can be seen in tab:cv, with dsc of $89.89/84.08/74.92$ (whole/core/enh.) using cross-validation on the training set and $89.70/76.96/77.90$ (whole/core/enh.) on the validation set, our baseline (tunet) produced acceptable results. Adding seb to  tunet (tunet + seb) improved the dsc on all tumor regions; however, only core and core+enhancing dsc scores were boosted when adding seb to two variations, that used ResNet-like blocks, i.e. tunet + res$^1$ + seb and tunet + res$^2$ + seb.

We gained a few dsc points when the average ensemble technique (Ensemble), i.e. dsc reached $91.92/86.45/78.72$ (whole/core/enh). The post-processing step only improved slightly the Ensemble model on the core region, but boosted the dsc of the enhancing region by a large margin, from 78.01 to 78.72, making it (Ensemble + post-processing) the best-performing model of all proposed models on the training set.

[rotate=90, text width=0.2]   Success;

[rotate=90, text width=0.2]   Failure;

[rotate=90, text width=0.0]  ;

[subfigure]labelformat=empty, justification=centering

Ground truth

[]A comparison of the ground truth masks and the results. The two examples show the input t1c (left), the ground truth masks (middle) and the results of the proposed method (right).

[]The uncertainty score maps generated from probability maps corresponding to whole, core, and enhancing tumor regions. Top row (from left to right): input, probability map of the whole, core and enhancing tumor regions, respectively. Bottom row (from left to right): multi-class label map (output), uncertainty map of the whole, core and enhancing tumor regions, respectively.

tab:valid shows the mean dsc and hd95 scores on the validation set by uploading our predicted masks to the evaluation server[<https://www.cbica.upenn.edu/BraTS19/lboardValidation.html>] (team name UmU). What is interesting in this table is: (1) dsc of core region on the validation set was lower compared to the cross-validation score on the training set, and (2) tunet + seb, perhaps surprisingly, performed slightly better than the ensemble model of tunet + seb, tunet + res$^1$ + seb and tunet + res$^2$ + seb. tab:test shows that our BraTS19 testing dataset results are 88.06, 80.84 and 80.29 average dice for whole tumor, tumor core and enhanced tumor core, respectively.

tab:valid_unc provides the mean dauc and rftp scores on the validation set obtained after uploading our predicted masks and corresponding uncertainty maps to the evaluation server[<https://www.cbica.upenn.edu/BraTS19/lboardValidationUncertainty.html>]. Similar to tab:valid, it can be seen in tab:valid_unc that tunet + seb is the best-performing model of our models. Though our best-performing model performs slightly worse than the top-ranking teams on the validation set, it was one of the top-performing models on the test set

This indicates that the proposed ensemble model might generalize the problem well.

fig:qualitative illustrates four qualitative examples generated from the Ensemble + post-process model on the training set. As can be seen in the first and second rows, our model detected all three regions well. However, it struggled to correctly differentiate between the necrotic and non-enhancing tumor core and the enhancing tumor (third and last rows). This is most likely due to difficulties in labeling the homogeneous areas.

fig:unc illustrates the uncertainty score maps generated from the corresponding output probability maps, that are produced by the tunet, for three tumor regions (whole, core, and enhancing). As can be seen in fig:unc, the uncertainty scores tends to be: (i) more obvious at the borderline or overlapping areas between tumor regions, and (ii) less apparent on the background or non-tumor regions.

An adapted version of the tunet was also used by the authors in the kits [10].

§ CONCLUSION

In conclusion, we developed a cascaded architecture by connecting three U-Net-like networks to segment glioma sub-regions from multimodal brain mri images. We separated the complex brain tumor segmentation into three simpler binary tasks to segment the whole tumor, tumor core, and enhancing tumor core, respectively. Our network used an encoder-decoder structure with ResNet-like blocks and Squeeze-and-Excitation blocks after each convolution and concatenation block. Dice scores on the training set and validation set were 91.92/86.45/78.72 and 90.34/81.12/78.42 for the whole tumor, tumor core, and enhancing tumor core, respectively.

[1]
Asklund, T., Björ, O., Malmström, A., Blomquist, E., Henriksson, R.:
Överlevnanden vid maligna gliom har ökat senaste tio åren. analys
av kvalitetsregisterdata. Läkartidningen  109(17-18),  875–878

[2]
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.,
Freymann, J., Farahani, K., Davatzikos, C.: Segmentation labels and radiomic
features for the pre-operative scans of the tcga-gbm collection. the cancer
imaging archive (2017) (2017)

[3]
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.,
Freymann, J., Farahani, K., Davatzikos, C.: Segmentation labels and radiomic
features for the pre-operative scans of the tcga-lgg collection. The Cancer
Imaging Archive  286 (2017)

[4]
Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J.S.,
Freymann, J.B., Farahani, K., Davatzikos, C.: Advancing the cancer genome
atlas glioma MRI collections with expert segmentation labels and radiomic
features. Scientific data  4,  170117 (2017)

[5]
Bakas, S., Reyes, M., Jakab, A., Bauer, S., Rempfler, M., Crimi, A., Shinohara,
R.T., Berger, C., Ha, S.M., Rozycki, M., et al.: Identifying the best machine
learning algorithms for brain tumor segmentation, progression assessment, and
overall survival prediction in the BRATS challenge. arXiv preprint
arXiv:1811.02629  (2018)

[6]
Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin
classifiers. In: Proceedings of the fifth annual workshop on Computational
learning theory. pp. 144–152. ACM (1992)

[7]
Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder
with atrous separable convolution for semantic image segmentation. In:
Proceedings of the European conference on computer vision (ECCV). pp.
801–818 (2018)

[8]
Gholami, A., Mang, A., Biros, G.: An inverse problem formulation for parameter
estimation of a reaction–diffusion model of low grade gliomas. Journal of
mathematical biology  72(1-2),  409–433 (2016)

[9]
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image
recognition. In: Proceedings of the IEEE conference on computer vision and
pattern recognition. pp. 770–778 (2016)

[10]
Heller, N., Sathianathen, N., Kalapara, A., Walczak, E., Moore, K., Kaluzniak,
H., Rosenberg, J., Blake, P., Rengel, Z., Oestreich, M., Dean, J., Tradewell,
M., Shah, A., Tejpaul, R., Edgerton, Z., Peterson, M., Raza, S., Regmi, S.,
Papanikolopoulos, N., Weight, C.: The KiTS19 challenge data: 300 kidney
tumor cases with clinical context, CT semantic segmentations, and surgical
outcomes (2019)

[11]
Holland, E.C.: Progenitor cells and glioma formation. Current opinion in
neurology  14(6),  683–688 (2001)

[12]
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of
the IEEE conference on computer vision and pattern recognition. pp.
7132–7141 (2018)

[13]
Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H.: No
new-net. In: International MICCAI Brainlesion Workshop. pp. 234–244.
Springer (2018)

[14]
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv
preprint arXiv:1412.6980  (2014)

[15]
Kitrungrotsakul, T., Iwamoto, Y., Han, X.H., Takemoto, S., Yokota, H.,
Ipponjima, S., Nemoto, T., Wei, X., Chen, Y.W.: A cascade of CNN and LSTM
network with 3D anchors for mitotic cell detection in 4D microscopic
image. In: ICASSP 2019-2019 IEEE International Conference on Acoustics,
Speech and Signal Processing (ICASSP). pp. 1239–1243. IEEE (2019)

[16]
Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby,
J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., et al.: The multimodal
brain tumor image segmentation benchmark (BRATS). IEEE transactions on
medical imaging  34(10),  1993–2024 (2014)

[17]
Myronenko, A.: 3D MRI brain tumor segmentation using autoencoder
regularization  (2018), <http://arxiv.org/abs/1810.11654>

[18]
Ronneberger, O., Fischer, P., Brox, T.: U-Net: Convolutional networks for
biomedical image segmentation. In: International Conference on Medical image
computing and computer-assisted intervention. pp. 234–241. Springer (2015)

[19]
Wang, G., Li, W., Ourselin, S., Vercauteren, T.: Automatic brain tumor
segmentation using cascaded anisotropic convolutional neural networks.
Lecture Notes in Computer Science p. 178–190 (2018)