# Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for
Efficient Training

Anup Sarma Sonali Singh Huaipan Jiang
Rui Zhang Mahmut T. Kandemir Chita R. Das
Pennsylvania State University
{avs6194, sms821, hzj5142, rmz5227, mtk2<EMAIL_ADDRESS>

###### Abstract

Recurrent Neural Networks (RNNs), more specifically their Long Short-Term
Memory (LSTM) variants, have been widely used as a deep learning tool for
tackling sequence-based learning tasks in text and speech. Training of such
LSTM applications is computationally intensive due to the recurrent nature of
hidden state computation that repeats for each time step. While sparsity in
Deep Neural Nets has been widely seen as an opportunity for reducing
computation time in both training and inference phases, the usage of non-ReLU
activation in LSTM RNNs renders the opportunities for such dynamic sparsity
associated with neuron activation and gradient values to be limited or non-
existent. In this work, we identify dropout induced sparsity for LSTMs as a
suitable mode of computation reduction. Dropout is a widely used
regularization mechanism, which randomly drops computed neuron values during
each iteration of training. We propose to structure dropout patterns, by
dropping out the same set of physical neurons within a batch, resulting in
column (row) level hidden state sparsity, which are well amenable to
computation reduction at run-time in general-purpose SIMD hardware as well as
systolic arrays. We provide a detailed analysis of how the dropout-induced
sparsity propagates through the different stages of network training and how
it can be leveraged in each stage. More importantly, our proposed approach
works as a direct replacement for existing dropout-based application settings.
We conduct our experiments for three representative NLP tasks: language
modelling on the PTB dataset, OpenNMT based machine translation using the
IWSLT De-En and En-Vi datasets, and named entity recognition sequence
labelling using the CoNLL-2003 shared task. We demonstrate that our proposed
approach can be used to translate dropout-based computation reduction into
reduced training time, with improvement ranging from 1.23$\times$ to
1.64$\times$, without sacrificing the target metric.

##  1 Introduction

Recurrent Neural Networks (RNNs) are an important class of machine learning
approaches for analyzing sequential data including language, speech, time-
series data, etc. Given an input training sequence, RNNs cells pass
information from the previous time-step to the current time-step, in the form
of a hidden-state, which helps the network to learn inherent temporal
constructs. To further enhance the ability to learn from past information,
Long Short-Term Memory(LSTM) (Hochreiter and Schmidhuber, 1997) cells have
been proposed, which consist of multiple internal gates and a cell-state in
addition to the hidden-state. Such LSTM cell-based RNNs have been successfully
deployed in a variety of application domains, e.g., language modelling
(Mikolov and Zweig, 2012), automatic speech recognition (Hannun et al., 2014;
Amodei et al., 2016; Graves et al., 2013), sentiment analysis (Ma et al.,
2018), and machine translation (Luong et al., 2015). However, training such
RNNs is a time- and compute-intensive task. This is because, for each input
sequence, a given RNN is logically unrolled into a sequence equal to the
sequence length, and computation is performed following the dependencies. Once
the training loss is calculated, it is back-propagated over the sequence in
the reverse direction to compute the weight gradients, which is known as back-
propagation through time (BPTT) (Werbos, 1990).

One of the promising ways of reducing the effective amount of computation is
to exploit sparsity present in data structures. Sparsity refers to the
fraction of zero-valued operands to the total size of the operands involved in
a computation step. Thus, efficiently skipping zero-valued operands from
computation leads to faster training time and energy efficiency. Convolutional
Neural Networks (CNNs), owing to their prominent use of ReLU (Nair and Hinton,
2010) as the activation function, generate significant dynamic sparsity
associated with the activation and neuron gradient values, which can be
leveraged during training. However, due to the non-ReLU activation functions,
e.g., tanh and sigmoid, dynamic sparsity in LSTM based networks is not
inherent in nature. Such activation functions do not directly induce zero
values, and hence the opportunity of exploiting sparsity is only limited to
thresholding and approximation-based approaches (Gupta et al., 2019). Although
such techniques have been shown to work relatively well during the inference
phase (after the network is fully trained), training still relies on high
precision arithmetic operations. Also, the derivatives of the tanh and sigmoid
functions do not lend themselves to sparsity as in ReLU. For the same reasons,
the back-propagated gradient values are not sparse either, thus ruling out the
opportunity for sparsity exploitation during the backward pass.

Motivated by these observations, in this work, we explore the opportunity of
using the dropout-induced zero values as a source of sparsity in LSTMs.
Dropout, a widely used regularizing technique, works by randomly dropping
activation values in the forward pass. Since the dropped activation map
becomes sparse in nature, we can take advantage of this sparsity towards
achieving computation speed-up. However, such dropout masks are sampled
randomly, which creates bottlenecks in accelerating the resultant sparse
computation in a general-purpose SIMD hardware owing to the storage and memory
access overheads. Therefore, we propose to structure such dropout patterns
which makes it amenable for hardware acceleration. We distinguish between
sparsity types, namely, input and output sparsity, depending on whether the
input operands are zero-valued or an output operand is going to be zero. Since
the dropout mask can be sampled ahead of time, exploiting output sparsity may
seem straightforward. However, applying it naively to the hidden state can
result in unintended consequences. This is because the sparsity also
propagates to the cell states due to the inherent computational dependencies,
resulting in possible loss of functional performance(e.g. accuracy). Further,
we carefully analyze the computational flow in the backward pass of LSTM to
identify sparse operands. We also extend the sparsity technique to SIMD GPU-
based matrix-multiplication using shared memory, and show that that our
proposed approach is exploitable in existing software based techniques. Our
proposed sparsity pattern is also well-suited to be leveraged in systolic
array-based computations, resulting in energy-efficiency benefits of dense
matrix operations combined with high-performance sparse operations. In
summary, this paper makes the following key contributions:

  * •

We outline a generic framework for applying dropout in the context of LSTMs,
considering uniformity of dropout patterns within a batch and across time
steps. In addition to commonly used non-recurrent(NR) direction, we also
extend dropout structural pattern to recurrent (NR+RH) directions to maximize
the scope for potential training speedup.

  * •

We provide detailed opportunities for exploiting the uniform sparsity in
forward as well as backward passes of LSTMs based on sparsity propagation
during training. We also identify the sparsity types based on input/output
sparsity, and leverage it towards faster computation operation.

  * •

We conduct extensive experiments with three representative applications
including Language Modelling, Machine Translation, and Named Entity
Recognition to evaluate the general applicability and performance benefits of
our proposed approach. Our results show that uniform dropout pattern within a
batch and random in time steps achieves significant training speedup ranging
from 1.23x-1.64x, while not compromising on the target accuracy metric.

##  2 Related Work

#### Dropout and DropConnect.

Several prior works study the scope of Dropout towards reducing model over-
fitting and achieve regularization. Hinton et al. (2012) first proposed the
idea of dropout, which randomly removes the neurons in a given layer of a
feed-forward neural network for a given probability $p$. Wan et al. (2013) on
the other hand proposed DropConnect, which, instead of neurons, randomly drops
the weight connections between two layers. Pham et al. (2014) and Zaremba et
al. (2014) applied Dropout towards LSTM RNN regularization, where the dropout
operator is specifically applied on the non-recurrent direction in LSTM.
Kingma et al. (2015) and Gal and Ghahramani (2016) explored dropout strategies
based on Bayesian Approximation. Gal and Ghahramani (2016) in particular also
extended Dropout to the recurrent direction of LSTMs. Moon et al. (2015) have
proposed using dropout on the LSTM cell-state instead of the hidden-states;
however, their approach is application-specific towards automatic speech
recognition (ASR) tasks and doesn’t discuss generality. Semeniuta et al.
(2016) investigated a variant of recurrent dropout that does not cause a loss
of long term memory by letting the dropout mask operate directly on one of the
internal state vectors. Krueger et al. (2016) proposed to regularize RNNs by
randomly preserving hidden units from the previous time-step, instead of
dropping them out. Merity et al. (2017) used DropConnect along hidden-to-
hidden weight matrices as a form of recurrent regularization for LSTMs, while
using Dropout on the non-recurrent dimension. However, all of these prior
works focus on achieving better regularization to avoid over-fitting while our
approach seeks to (a) act as a simple, plug-in replacement in existing LSTM
applications without requiring to alter other hyper-parameter settings, and
(b) reduce the overall training time by structuring the dropout pattern
itself, while achieving the same degree of regularization effect.

#### Weight Sparsity in Deep Neural Networks.

Sparsity in Deep Neural Nets has also been extensively explored, and the work
in this area can be categorized as unstructured or structured types. In
particular, unstructured pruning approaches (Han et al., 2015a, b; Dai et al.,
2019; Mao et al., 2017; Narang et al., 2017a) result in random sparsity in
weight matrices, which is difficult to accelerate on general-purpose hardware
due to storage and memory access overheads. This has motivated structured
sparsity-based approaches for both CNNs (Yu et al., 2017; Wen et al., 2016;
Narang et al., 2017a) and RNNs (Lu et al., 2016; Wen et al., 2017; Narang et
al., 2017b). However, these works mainly aim to improve model size and speedup
during model inference; in contrast, our work is aimed at achieving speedup
during the training phase. We also believe our approach can be adapted in
structured pruning approaches as well, towards faster training on general-
purpose hardware.

##  3 A General Framework of Using Dropout for Efficient LSTM Training

In this section, we introduce a generic dropout framework to distinguish
different possibilities of dropout application followed by the scope of the
dropout-induced sparsity throughout different phases of training. We first
highlight the computational dynamics of an LSTM cell (Hochreiter and
Schmidhuber, 1997), which is a variant of RNN, consisting of four different
types of gates:

$\displaystyle\text{input gate}:\bm{i}_{t}$
$\displaystyle=\operatorname{sigmoid}(\bm{i}^{*}_{t}),$
$\displaystyle\bm{i}^{*}_{t}=\bm{h}^{l-1}_{t}\bm{W}_{i}+\bm{h}^{l}_{t-1}\bm{U}_{i}+\bm{b}_{i}$
(1) $\displaystyle\text{forget gate}:\bm{f}_{t}$
$\displaystyle=\operatorname{sigmoid}(\bm{f}^{*}_{t}),$
$\displaystyle\bm{f}^{*}_{t}=\bm{h}^{l-1}_{t}\bm{W}_{f}+\bm{h}^{l}_{t-1}\bm{U}_{f}+\bm{b}_{f}$
(2) $\displaystyle\text{output gate}:\bm{o}_{t}$
$\displaystyle=\operatorname{sigmoid}(\bm{o}^{*}_{t}),$
$\displaystyle\bm{o}^{*}_{t}=\bm{h}^{l-1}_{t}\bm{W}_{o}+\bm{h}^{l}_{t-1}\bm{U}_{o}+\bm{b}_{o}$
(3) $\displaystyle\text{input modulation gate}:\bm{g}_{t}$
$\displaystyle=\tanh(\bm{g}^{*}_{t}),$
$\displaystyle\bm{g}^{*}_{t}=\bm{h}^{l-1}_{t}\bm{W}_{c}+\bm{h}^{l}_{t-1}\bm{U}_{c}+\bm{b}_{c}$
(4)

where $\bm{h}_{t}^{l}$ notation refers to hidden-state from layer $l$ and time
step $t$. Based on the above four internal gate states, the following two
output states are computed:

$\displaystyle\text{cell state}:\bm{c}^{l}_{t}$
$\displaystyle=\bm{f}_{t}\odot\bm{c}_{t-1}+\bm{i}_{t}\odot\bm{g}_{t}$ (5)
$\displaystyle\text{hidden state}:\bm{h}^{l}_{t}$
$\displaystyle=\bm{o}_{t}\odot\tanh(\bm{c}^{l}_{t})$ (6)

Figure 1: (a) A unifying sparsity framework based on dropout characteristics
within a batch and across time steps. Based on two possible choices in each
dimension, there are four possible cases. The prior works fall under case-I
and case-II, while our proposed structured dropout mechanism belongs to Case-
III. It allows taking advantage of structural sparsity due to dropout towards
computation speed-up, while difference across time-steps leads to the
necessary variation needed for achieving regularization objectives. (b)
Graphical illustration of the four cases with dropout pattern applied to the
two-dimensional hidden state matrix (dark squares indicate dropped out
values).

Equations (1) through (4) form the core of an LSTM cell and are also the
computationally intensive part of the operation, during both the forward and
backward passes of training. Parameters
$\\{\bm{W}_{*},\bm{U}_{*},\bm{b}_{*}\\}$ constitute the weight of the network,
where $\bm{W}_{*}$ and $\bm{U}_{*}$ refer to the input-to-hidden and hidden-
to-hidden weight projection matrices, respectively, and $\bm{b}_{*}$ is the
bias. Each weight matrix is given by size $H\times H$, where $H$ is the LSTM
hidden size.

###  3.1 Reframing the Scope of Dropout Structuring in LSTMs

Considering an LSTM cell with hidden state dimension $H$ and an input batch
size of $B$, at each time step $t$, the cell generates a hidden-state matrix
($\bm{h}_{t}$) and cell-state matrix ($\bm{c}_{t}$), each of size $B\times H$.
Each row of this matrix corresponds to the feature vector of the corresponding
input. The hidden state ($\bm{h}_{t}$) is used as recurrent input to the next
time-step $t+1$, and also as input to the subsequent LSTM cell/layer as a non-
recurrent input.

The idea of regularizing recurrent neural networks by using dropouts has
initially been introduced by Zaremba et al. (2014). The two key aspects of the
said work were – (a) dropouts were only used along the non-recurrent
direction, and (b) at different time steps, it was sampled using different
dropout masks making it time-variant. Subsequent work from Gal and Ghahramani
(2016) has shown that dropout can be extended to the recurrent direction as
well, thereby increasing network regularization scope. Similar to the work of
Zaremba et al. (2014), the dropout mask itself was random. However, unlike
Zaremba et al. (2014), the sampled dropout masks remained the same across time
steps.

Given this possible choice of application of dropout, we propose a unifying
framework considering the two dimensions of dropout pattern – dropout within a
batch and dropout across time steps. Each dimension in turn offers two
possible choices, resulting in four possible combinations, as summarized in
Fig. 1(a) and illustrated in Fig. 1(b). Case-I refers to the scenario where
dropout is random within a batch and also different across time-steps. This
refers to the most common application of dropout including Zaremba et al.
(2014)’s method. Case-II on the other hand refers to the scenario in which
dropout pattern is random within a batch, but the same pattern is repeated
across time steps. This approach is similar to Gal and Ghahramani (2016)’s
work and has also been adopted later by Merity et al. (2017) in AWD-LSTM.
Case-III refers to a dropout pattern that is structured within a batch, but
varies across time steps. Finally, Case-IV is the most restricted form of
dropout, where the pattern is not only structured within a batch, but also
remains the same across time-steps.

These different dropout patterns exhibit different trade-offs. For example,
while a mask, which is structured in a batch (B) and same across time-steps
(T), is most amenable for hardware acceleration due to least metadata
overhead, it also offers the least variability to the training model
underneath. In this work, we stick to Case-III, where a structure in dropout
mask within batch allows us to take advantage of computation speed-up, while
variation across time-steps injects sufficient randomness into the training
process, helping to retain the functional accuracy. Also, at a given time
step, it is possible to view dropout being applied to only non-recurrent
dimension (NR) or both non-recurrent as well as recurrent-hidden (NR+RH)
dimensions. Therefore, we can further distinguish those cases as NR+ST or
NR+RH+ST, and so on. In this study, we explore both NR+ST and NR+RH+ST
configurations under Case-III towards maximizing potential training speedup,
while retaining functional accuracy.

###  3.2 Dropout Propagation during Network Training

Next, we analyze the opportunity for dropout-based computation acceleration
across the three key phases of network training: forward pass, backward pass,
and weight gradient computation.

#### Scope of Dropout Sparsity in Forward Pass (FP).

During Forward Pass (FP), the most compute-intensive parts of LSTM cell are
captured by Equations (1) through (4), in which inputs $\bm{h}_{t}^{l-1}$ and
$\bm{h}_{t-1}^{l}$ are transformed by matrices $\bm{W}$ and $\bm{U}$,
respectively. Since both recurrent and non-recurrent inputs to the cell would
be structured-drop, this will result in the first input operand of the matrix
multiplication being column sparse. Consequently, we can perform a sparsity-
aware matrix-multiplication. This is shown in Fig 2(a). The resulting output
matrix is a dense type on which subsequent reduction operations corresponding
to Equations (5) and (6) could be performed.

Since dropout masks can be sampled ahead of time, ideally, we should also be
able to apply the notion of output sparsity to prevent computation from taking
place in the first place. Note however that, when output sparsity is applied
to the hidden state, it is also inherently gets applied to the cell-state.
This is because both the hidden state $h_{t}$ and cell state $\bm{c}_{t}$ is a
combination of the same internal gate states. Therefore, any computation
saving intended in the form of output sparsity for $\bm{h}_{t}$, also gets
applied to the cell state $\bm{c}_{t}$. However, simply applying dropout to
the LSTM cell state can be detrimental to the LSTM’s ability to learn( Note
that Moon et al. (2015) work to drop $\bm{c}_{t}$ doesn’t warrant a generic
deployment). Therefore, the scope of dropout sparsity exploitation in the
forward-pass is only limited to input type in our studies, where the first
input operand is column-sparse.

#### Scope of Dropout Sparsity in Backward Pass (BP).

To understand the feasibility of types of sparsity exploitation during the BP,
we need to carefully take into account the effect of the back-propagation
operator. In the case of FP, while input sparsity exploitation is
straightforward, output sparsity is contingent upon applying dropout to the
cell state as well. In the case of BP, the effect is closely related to the
non-linearity used in Equations (1) through (4), which is
$\operatorname{sigmoid}$ and $\tanh$ type. For both these non-linearity types,
the derivatives are non-zero except at far ends from the origin, where
derivatives tend to be zero asymptotically and are likely to result in random
patterns of sparsity, if any. Therefore, we will look at the opportunity for
sparsity exploitation due to the dropout operator from the FP.

We start by applying the back-propagation operator on the set of equations
starting from Equation (6):

$\delta\bm{o}_{t}=\delta\bm{h}_{t}\odot\tanh(\bm{c}_{t}),\quad\quad\delta\bm{c}_{t}=\delta\bm{h}_{t}\odot\bm{o}_{t}\odot{(1-\tanh^{2}(\bm{c}_{t}))}+\delta\bm{c}_{t+1}$
(7)

Here notation $\delta\bm{x}$ refer to gradient of error with respect to
variable $x$. Note that, in the above sets of equations, $\delta\bm{h}_{t}$ is
structurally sparse, due to the application of the same dropout mask as in the
forward pass on the set of computed gradient values. Thus, $\delta\bm{o}_{t}$
is sparse due to the only element-wise operator involving $\delta\bm{h}_{t}$.
However, $\delta\bm{c}_{t}$ is not sparse in the same manner, in particular
due to the recursive relationship involved.

The back-propagation operator for Equation (5):

$\displaystyle\delta\bm{f}_{t}$
$\displaystyle=\delta\bm{c}_{t}\odot\bm{c}_{t-1},$
$\displaystyle\delta\bm{c}_{t-1}=\delta\bm{c}_{t}\odot\bm{f}_{t}$ (8)
$\displaystyle\delta\bm{i}_{t}$
$\displaystyle=\delta\bm{c}_{t}\odot\bm{g}_{t},$
$\displaystyle\delta\bm{g}_{t}=\delta\bm{c}_{t}\odot\bm{i}_{t}$ (9)

Since $\delta\bm{c}_{t}$ is not sparse (in general), from the above
expression, gradients arriving at different gate positions, i.e.,
$\delta\bm{f}_{t}$, $\delta\bm{i}_{t}$, and $\delta\bm{g}_{t}$ are also not
sparse at this stage, except $\delta\bm{o}_{t}$.

Now, referring back to Equations (1) through (4), these gradients must back-
propagate through the corresponding non-linearities resulting in
$\delta\bm{i}_{t}^{*}$, $\delta\bm{f}_{t}^{*}$, $\delta\bm{o}_{t}^{*}$ and
$\delta\bm{g}_{t}^{*}$. Therefore, the only sparse gradient operand
$\delta\bm{o}_{t}$ also loses sparsity after back-propagating through the
$\operatorname{sigmoid}$ activation. These non-sparse gradient values are used
to compute the output gradient for $\delta\bm{h}^{l}_{t-1}$ and
$\delta\bm{h}^{l-1}_{t}$ using the following methodology:

$\delta\bm{h}^{l}_{t-1,i}=\bm{U}^{\intercal}_{i}\times\delta\bm{i}_{t}^{*},\quad\quad\delta\bm{h}^{l-1}_{t,i}=\bm{W}^{\intercal}_{i}\times\delta\bm{i}_{t}^{*}$
(10)

The above calculation step is also repeated with respect to the other three
gates (Equations (2), (3), and (4)), and the corresponding hidden state
gradients are added up in an element-wise fashion, before being passed through
the same dropout mask used during the forward pass. Since the hidden state
values eventually get dropped out before being passed to the previous state,
we can take advantage of output sparsity here according to the dropout mask
being applied. As shown in Figure 2(b), the output operator goes to zero
according to the specific dropout pattern applied during the FP. Therefore, we
can safely skip computing those locations by treating the second input operand
column-sparse.

Figure 2: Resulting sparsity types due to dropout application during different
stages of training.

#### Scope of Dropout Sparsity in Weight Gradient (WG) Computation.

After computing the neuron gradients, it is now feasible to compute the weight
gradients which are given by the inner product of the transposed input
activation’s map during the FP and input gradients during the BP. Thus, the
weight gradient computation stage is applicable to the expressions involving
$\bm{W}$ and $\bm{U}$ parameters from Equations (1), (2), (3), and (4).
Corresponding to Equation (1), the weight gradient step can be written as
follows:

$\delta\bm{W}_{i}=(\bm{h}^{l-1}_{t})^{\intercal}\times\delta\bm{i}_{t}^{*},\quad\quad\indent\delta\bm{U}_{i}=(\bm{h}^{l}_{t-1})^{\intercal}\times\delta\bm{i}_{t}^{*}$
(11)

In both the expressions above, we observe that first input operands on the
right hand side, i.e., $\bm{h}^{l-1}_{t}$ and $\bm{h}^{l}_{t-1}$, are column-
sparse (from the FP application of structured dropout) operands. Therefore, we
can leverage the notion of input sparsity with respect to these operands to
accelerate computation in the WG computation phase. However, due to the
application of the transposition operator, the first input operand for the
matrix-multiplication becomes row-sparse as depicted in Figure 2(c). As a
result, the generated output map also becomes row-sparse, correlating to the
fact when a neuron is dropped during the FP, and it does not contribute to the
WG computation in the BP.

In summary, the application of dropout creates opportunities for computation
skipping throughout the LSTM training. Following the structured application of
dropout pattern according to Case-III, the FP stage can support column-sparse
input sparsity, the BP supports column-sparse output sparsity, and row-sparse
input sparsity again during the WG computation stage. We next experimentally
demonstrate the effectiveness of our proposed approach from the perspectives
of both its regularization ability and training speed-up.

##  4 Experimental Results

Our proposed method can serve as a plug-in optimization in various
applications using the LSTM-based models with dropout. We discuss the
effectiveness and generality of our structured dropout approach by considering
three representative application domains – Language Modelling (Section 4.1),
Machine Translation (Section 4.2), and Sequence Labelling tasks (Section 4.3).
Our training is conducted using an NVIDIA TITAN V GPU card hosted on a 12-core
Intel Xeon Bronze 3104 CPU platform with Pytorch v1.8 as the training
framework and CUDA version 11.2. The reported speedup measurements are based
on using matrix-matrix multiplication time of the LSTM and FC layers using the
NVIDIA CuBLAS library using the same GPU platform after performing matrix
compaction. In general, we refer to cases when only random dropout is applied
in the non-recurrent direction as NR+Random, and the dropout pattern which is
structured along non-recurrent direction as NR+ST (no recurrent dropout
applied). Finally, NR+RH+ST refers to dropout which is applied to both the
non-recurrent and recurrent directions and is also structured.

###  4.1 Language Modelling

Table 1: Results of Language Modelling on Penn Treebank (PTB) showing
perplexity on the validation and test sets, and the speedup for different
training phases (FP, BP, and WG).

Model | Perplexity | Speedup
---|---|---
| Validation | Test | FP | BP | WG | Overall
Zaremba et al. (2014) \- Medium (Baseline) | 86.20 | 82.70 | 1 | 1 | 1 | 1
This work(NR+ST) | 86.60 | 83.01 | 1.29 | 1.01 | 1.42 | 1.17
This work(NR+RH+ST) | 83.64 | 80.84 | 1.66 | 1.10 | 1.57 | 1.45
Zaremba et al. (2014) \- Large (Baseline) | 82.20 | 78.40 | 1 | 1 | 1 | 1
Krueger et al. (2016)-Large | 81.40 | 77.40 | 1 | 1 | 1 | 1
This work (NR+ST) | 85.42 | 82.39 | 1.47 | 1.09 | 1.31 | 1.27
This work (NR+RH+ST) | 79.38 | 76.05 | 2.45 | 1.28 | 1.41 | 1.64
Merity et al. (2017) AWD-LSTM (Baseline) | 63.31 | 61.21 | 1 | 1 | 1 | 1
This work(NR+RH+ST) | 63.95 | 61.77 | 1.63 | 1.04 | 1.53 | 1.38

#### Model and Datasets.

We conduct language modelling experiments using the Penn Treebank (PTB)
dataset (Marcus et al., 1993). It consists of 929k training words, 73k
validation words, and 82k test words, with a vocabulary size of 10k. We
discuss the single model perplexity results corresponding from Zaremba et al.
(2014) and AWD LSTM by Merity et al. (2017). In each case, we keep the
original training framework consistent, and the only modification we make is
to change the pattern of the dropout application as discussed previously.

#### Implementation Details.

Zaremba et al. (2014) experiment with two configurations: medium and large.
Each configuration uses two layers of LSTM cells and a minibatch size of 20,
and is unrolled for 35 time steps. For medium, each LSTM layer consists of 650
hidden units with parameters uniformly initialized in [-0.05, 0.05]. A 50%
dropout is applied only on the non-recurrent dimension of LSTM cells, which is
essentially random in nature and also varies across time-steps. After 39
epochs of training, this baseline network achieves validation and test
perplexity results of 86.2 and 82.7, respectively. For large, each LSTM layer
has 1500 hidden units uniformly initialized in [-0.04,0.04]. The large LSTM
variant also employs a larger dropout pattern of 0.65 (random) along the non-
recurrent direction. The large regularized LSTM achieves validation and test
perplexity values of 82.2 and 78.4, respectively, after 55 epochs of training.
Our dropout modifications to these networks involve (a) structuring the
existing dropout levels (50%/65%) on the non-recurrent direction, and (b)
introducing an additional (50%/65%) structured dropout along the recurrent
direction. For the large model, we also report the corresponding results from
Krueger et al. (2016) which improves Zaremba et al. (2014) test perplexity
from 78.4 to 77.4.

We also experiment with Merity et al. (2017)’s ASGD Weight Dropped LSTM(AWD-
LSTM), which consists of a three-layer LSTM with an embedding size of 400, and
each LSTM hidden layer has a size of 1150. The LSTM weight parameters are
initialized uniformly between $[-\frac{1}{\sqrt{H}},\frac{1}{\sqrt{H}}]$,
where $H$ is the LSTM cell size, while all embedding weights are initialized
in [-0.1, 0.1]. For AWD-LSTM, the dropout rate applied at different levels of
networks varies, which can be expressed by dropout vector of [0.4, 0.1, 0.25,
0.4] corresponding to input dropout, embedding layer dropout, non-recurrent
LSTM dropout, and output dropout(pre-FC layer). In addition, it applies a
dropout value of 0.5 on the recurrent weight matrices. Note that when we
structure the recurrent weight matrix dropout, it essentially becomes
structured dropout applied on the recurrent feature maps due to the
computational symmetry of matrix multiplication.

We also analyze Gal and Ghahramani (2016)’s proposed approach in regularizing
recurrent direction on top of Zaremba et al. (2014)’s prior work. However, due
to the lack of a consistent framework to replicate these results (original Lua
implementation), we avoid further quantitative comparison. Also, Moon et al.
(2015) PTB language modelling network using 256 hidden units per layer is not
compatible with either Zaremba-medium or large configurations, and also has
higher perplexity numbers as compared to Zaremba-medium baseline. So, we omit
these comparisons as well.

#### Results and Discussion.

Table 1 shows that our NR+RH+ST modifications improve the validation and test
perplexity levels to 83.64 and 80.84, respectively, for the medium
configuration. Similarly, we observe an improvement in the case of the large
LSTM configuration as well, with validation and test perplexity scores
reaching 79.38 and 76.05, respectively, which also represents an improvement
over Krueger et al. (2016). For AWD-LSTM, our proposed optimization yields
comparable validation and test perplexity with respect to the baseline after
100 epochs of training.

Table 1 also shows the speedups achieved on these particular set of
benchmarks, with a breakdown of speedup during the FP, BP and WG stages of
training. Our proposed structured dropout mechanism in the recurrent direction
results in an overall speedup of 1.45$\times$ for the Zaremba-medium
configuration, and 1.64$\times$ for the Zaremba-large configuration, while
providing improved/competitive perplexity levels with respect to the state-of-
the-art baseline. For the AWD-LSTM workload, the observed speedup is
1.38$\times$ over the baseline. Intuitively, this implies that we can achieve
significant speedups (in the range of 45% to 64%) in the training of these
workloads, while maintaining almost similar accuracy. For the Zaremba-medium
and large cases, we also report the perplexity and speedup metrics obtained by
running NR+ST configuration. As shown in the table, for both the
configurations, the test and validation perplexity values are marginally worse
than their baseline, while speedup-improvements – 1.17x for medium and 1.27x
for large – are also lower than NR+RH+ST case, which is as expected.

Figure 3: Perplexity during training.

Another important point that can be observed from this table is that each of
the phases results in a different amount of speedup, although the total
computation size and sparsity fraction are the same. More specifically, we
note that the FP and WG phases have higher speedups compared to the BP phase.
This essentially stems from the fact that sparsity is applicable for different
matrix multiplication operands (refer to Fig. 2) in different phases,
resulting in differently-shaped MM operations, which in turn leads to the
difference in speedup.

Fig. 3 shows an interesting trend with respect to the validation perplexity
during the course of training, corresponding to the Zaremba-Medium
configuration. The plot shows that our proposed NR+RH+ST scheme starts with a
higher level of perplexity as compared to Baseline(NR+Random) and NR+ST
configurations. However, as the training proceeds, the perplexity continues to
get lower for the NR+RH+ST case, while the Baseline and NR+ST cases start to
get flatten. This further indicates the effectiveness of regularization
achieved in our final approach.

###  4.2 Machine Translation

Table 2: Results of Machine Translation on IWSLT’14 De-En and IWSLT’15 En-Vi.

Model | BLEU | Speedup(De-En/En-Vi)
---|---|---
| De-En | En-Vi | FP | BP | WG | Overall
Luong et al. (2015) | 28.13 | 25.26 | 1 | 1 | 1 | 1
This work (NR + ST) | 27.54 | 26.08 | 1.17/1.16 | 1.13/1.01 | 1.22/1.14 | 1.17/1.09
This work (NR + RH + ST) | 28.46 | 26.20 | 1.35/1.33 | 1.17/1.07 | 1.45/1.37 | 1.31/1.23

#### Model and Datasets.

We evaluate our framework on two machine translation datasets, IWSLT’14
German-English (De-En) and IWSLT’15 English-Vietnamese (En-Vi) parallel
corpus. After pre-processing and data-cleanup, IWSLT’14 De-En results in 153k
sentence pairs, while IWSLT’15 En-Vi results in 133k training sentence pairs.
For development and testing, we utilize the 2012 and 2013 test sets for the
language pairs, respectively. We experiment with the NMT model proposed by
Luong et al. (2015) that consists of a 2-layer unidirectional LSTM encoder-
decoder with attention.

#### Implementation Details.

Each LSTM layer consists of 512 hidden units and a maximum vocabulary size of
50,000 for both the source and target languages. It uses a batch size of 64,
and a dropout probability of 0.3 (only in the non-recurrent direction) for
both the encoder and decoder. The evaluation metric is bi-lingual evaluation
understudy (BLEU), which is evaluated on the development set at each 5K
training steps. Our implementation is based on the popular OpenNMTPy (Klein et
al., 2017), a Pytorch-based framework for training NMT models. In this set of
experiments, we run each dataset for a total of 65k steps under the baseline
configurations. Without affecting any other hyper-parameter setting, we first
structure the existing dropout pattern along the non-recurrent direction.
Then, we insert an additional structured dropout of 0.3 along the recurrent
direction as a part of enhancing the computation reduction opportunities. To
further enhance it, we added dropout layers (0.3) to the final output of the
encoder and decoder layers.

#### Results and Discussion.

Table 2 shows the results corresponding to the baseline and our proposed
approach. The baseline model achieves a BLEU score of 28.13 and 25.26 for the
De-En and En-Vi datasets, respectively. Our NR+ST extension achieves BLEU
scores of 27.54 and 26.08, while achieving speedups of 1.17x and 1.09x, for
the two datasets. Our further modifications involving NR+RH+ST yield BLEU
scores of 28.46 and 26.20 for the same two datasets, indicating that our
proposed dropout extension is not only able to preserve the original learning
dynamics, but also marginally improve upon it. Table 2 also shows the
corresponding speedups achieved for the two datasets, which are 1.31x for De-
En and 1.23x for En-Vi. Although the model and dropout configurations are the
same for both the datasets, the difference arises in the decoder’s final
projection FC layer configuration, which is unique for each language depending
on its vocabulary size.

###  4.3 Sequence Labelling for Named Entity Recognition

#### Model and Datasets.

In this part, we experiment with a sequence labelling task for Named Entity
Recognition (NER) to locate and classify named entities in an unstructured
text. We refer to the work from Ma and Hovy (2016), which uses Bidirectional
LSTM-CNNs-CRF towards achieving state-of-the-art results in sequence
labelling. The proposed model architecture consists of a convolutional neural
network (CNN) layer to encode the character-level information of a word into
its character-level representation. Next, the character and word level
representations are combined and fed as an input into the bi-directional LSTM
layer. Finally, a sequential CRF is used to jointly decode the labels for the
entire sentence. We perform our experiments using the CoNNL 2003 shared task
dataset, which consists of four different types of named entities. The total
number of word tokens for training, development and test in this dataset are
204k, 51k and 46k, respectively.

Table 3: Results for Named Entity Recognition on the CoNLL-2003 shared task.

Model | NER Metrics | Speedup
---|---|---
| Acc. | Prec. | Recall | F1-score | FP | BP | WG | Overall
Ma and Hovy (2016) | 97.57 | 89.29 | 89.23 | 89.26 | 1 | 1 | 1 | 1
This work (NR + ST) | 97.67 | 89.86 | 89.46 | 89.32 | 1.43 | 1.06 | 1.18 | 1.21
This work (NR + RH + ST) | 97.64 | 89.34 | 89.64 | 89.49 | 1.70 | 1.20 | 1.32 | 1.39

#### Implementation Details.

We use the publicly available framework (Chernodub et al., 2019) to refer to
the baseline and implement our modifications on top of it. This is a single-
layer bidirectional LSTM network, and dropout was only applied at the input
level. More specifically, input to the CNN layer is applied with a dropout of
50% (random) and its output is concatenated with embedding layer output, which
is dropped at 50%. Since the CNN output contributes more towards the overall
input feature dimension, the input to the BiLSTM layer is only $\sim$12%
sparse. In our setting, we move the dropout layer from the input of the CNN
layer to the concatenated output, thereby increasing the input sparsity level
to 50%. In addition, we apply 50% structured dropout to the recurrent
dimension of both the forward and reverse directions of BiLSTM, which is
originally not present. Training results are shown for a maximum of 60 epochs.

#### Results and Discussion.

Table 3 shows the values of the different evaluation metrics associated with
this application for the baseline and the modified scenario. It can be
observed from these results that, our proposed modifications with both NR+ST
and NR+RH+ST fare equal or marginally better than the baseline configuration
across the accuracy, precision, recall, and F1-score metrics. The proposed
scheme also leads to an overall speed-up of 1.21x and 1.39x over the baseline
corresponding to NR+ST and NR+RH+ST schemes, which further highlights the
efficacy of our approach.

##  5 Conclusion

In this work, we holistically review and analyze the scope of leveraging
dropouts induced sparsity for computation reduction of LSTM-RNN based neural
network training. We explore the potential of structuring both the non-
recurrent and recurrent dimensions associated with neural networks, which can
bring potential speed-up during all phases of network training involving
forward, backward, and weight-gradient computation stages. We also
systematically outline the scope and nature of sparsity patterns in each
phase. We validate the efficacy of our proposed approach by applying it to
three popular and broad classes of NLP tasks targeting state-of-the-art GPU
stack, and show that it achieves nearly identical(or even better in cases)
accuracy compared to the baseline scenario, with significant execution
speedup, ranging from 1.23x to 1.64x, in all three NLP tasks.

## References

  * Amodei et al. (2016) Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al.  Deep speech 2: End-to-end speech recognition in english and mandarin.  In _International conference on machine learning_ , pages 173–182, 2016.
  * Chernodub et al. (2019) Artem Chernodub, Oleksiy Oliynyk, Philipp Heidenreich, Alexander Bondarenko, Matthias Hagen, Chris Biemann, and Alexander Panchenko.  Targer: Neural argument mining at your fingertips.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 195–200, 2019.
  * Dai et al. (2019) Xiaoliang Dai, Hongxu Yin, and Niraj K Jha.  Grow and prune compact, fast, and accurate lstms.  _IEEE Transactions on Computers_ , 69(3):441–452, 2019.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , pages 1050–1059. PMLR, 2016.
  * Graves et al. (2013) Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton.  Speech recognition with deep recurrent neural networks.  In _2013 IEEE international conference on acoustics, speech and signal processing_ , pages 6645–6649. IEEE, 2013.
  * Gupta et al. (2019) Udit Gupta, Brandon Reagen, Lillian Pentecost, Marco Donato, Thierry Tambe, Alexander M Rush, Gu-Yeon Wei, and David Brooks.  Masr: A modular accelerator for sparse rnns.  In _2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT)_ , pages 1–14. IEEE, 2019.
  * Han et al. (2015a) Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  _arXiv preprint arXiv:1510.00149_ , 2015a.
  * Han et al. (2015b) Song Han, Jeff Pool, John Tran, and William J Dally.  Learning both weights and connections for efficient neural networks.  _arXiv preprint arXiv:1506.02626_ , 2015b.
  * Hannun et al. (2014) Awni Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, et al.  Deep speech: Scaling up end-to-end speech recognition.  _arXiv preprint arXiv:1412.5567_ , 2014.
  * Hinton et al. (2012) Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  Improving neural networks by preventing co-adaptation of feature detectors.  _arXiv preprint arXiv:1207.0580_ , 2012.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Kingma et al. (2015) Diederik P Kingma, Tim Salimans, and Max Welling.  Variational dropout and the local reparameterization trick.  _arXiv preprint arXiv:1506.02557_ , 2015.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada, July 2017. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/P17-4012.
  * Krueger et al. (2016) David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Chris Pal.  Zoneout: Regularizing rnns by randomly preserving hidden activations.  _arXiv preprint arXiv:1606.01305_ , 2016.
  * Lu et al. (2016) Zhiyun Lu, Vikas Sindhwani, and Tara N Sainath.  Learning compact recurrent neural networks.  In _2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 5960–5964. IEEE, 2016.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning.  Effective approaches to attention-based neural machine translation.  _arXiv preprint arXiv:1508.04025_ , 2015.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy.  End-to-end sequence labeling via bi-directional lstm-cnns-crf.  _arXiv preprint arXiv:1603.01354_ , 2016.
  * Ma et al. (2018) Yukun Ma, Haiyun Peng, and Erik Cambria.  Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm.  In _Aaai_ , pages 5876–5883, 2018.
  * Mao et al. (2017) Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally.  Exploring the regularity of sparse structure in convolutional neural networks.  _arXiv preprint arXiv:1705.08922_ , 2017.
  * Marcus et al. (1993) Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz.  Building a large annotated corpus of English: The Penn Treebank.  _Computational Linguistics_ , 19(2):313–330, 1993\.  URL https://www.aclweb.org/anthology/J93-2004.
  * Merity et al. (2017) Stephen Merity, Nitish Shirish Keskar, and Richard Socher.  Regularizing and optimizing lstm language models.  _arXiv preprint arXiv:1708.02182_ , 2017.
  * Mikolov and Zweig (2012) T. Mikolov and G. Zweig.  Context dependent recurrent neural network language model.  In _2012 IEEE Spoken Language Technology Workshop (SLT)_ , pages 234–239, 2012.  doi: 10.1109/SLT.2012.6424228.
  * Moon et al. (2015) Taesup Moon, Heeyoul Choi, Hoshik Lee, and Inchul Song.  Rnndrop: A novel dropout for rnns in asr.  In _2015 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU)_ , pages 65–70. IEEE, 2015.
  * Nair and Hinton (2010) Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In _Icml_ , 2010.
  * Narang et al. (2017a) Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta.  Exploring sparsity in recurrent neural networks.  _arXiv preprint arXiv:1704.05119_ , 2017a.
  * Narang et al. (2017b) Sharan Narang, Eric Undersander, and Gregory Diamos.  Block-sparse recurrent neural networks.  _arXiv preprint arXiv:1711.02782_ , 2017b.
  * Pham et al. (2014) Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour.  Dropout improves recurrent neural networks for handwriting recognition.  In _2014 14th international conference on frontiers in handwriting recognition_ , pages 285–290. IEEE, 2014.
  * Semeniuta et al. (2016) Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth.  Recurrent dropout without memory loss.  _arXiv preprint arXiv:1603.05118_ , 2016.
  * Wan et al. (2013) Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.  Regularization of neural networks using dropconnect.  In _International conference on machine learning_ , pages 1058–1066. PMLR, 2013.
  * Wen et al. (2016) Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li.  Learning structured sparsity in deep neural networks.  _arXiv preprint arXiv:1608.03665_ , 2016.
  * Wen et al. (2017) Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, and Hai Li.  Learning intrinsic sparse structures within long short-term memory.  _arXiv preprint arXiv:1709.05027_ , 2017.
  * Werbos (1990) Paul J Werbos.  Backpropagation through time: what it does and how to do it.  _Proceedings of the IEEE_ , 78(10):1550–1560, 1990.
  * Yu et al. (2017) Jiecao Yu, Andrew Lukefahr, David Palframan, Ganesh Dasika, Reetuparna Das, and Scott Mahlke.  Scalpel: Customizing dnn pruning to the underlying hardware parallelism.  _ACM SIGARCH Computer Architecture News_ , 45(2):548–560, 2017.
  * Zaremba et al. (2014) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals.  Recurrent neural network regularization.  _arXiv preprint arXiv:1409.2329_ , 2014.


11institutetext: DAMO Academy, Alibaba Group 22institutetext: Hupan Lab,
310023, Hangzhou, China 33institutetext: Department of Radiology, Shengjing
Hospital of China Medical University, Shenyang 110004, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# Liver Tumor Screening and Diagnosis in CT with Pixel-Lesion-Patient
Network††thanks: Partially supported by the National Natural Science
Foundation of China (grant 82071885), Basic Research Projects of Liaoning
Provincial Department of Education (LJKMZ20221160), the National Youth Talent
Support Program of China, and Science and Technology Innovation Talent Project
in Shenyang (RC210265).

Ke Yan ✉ 1122 Xiaoli Yin 33 Yingda Xia 11 Fakai Wang 11 Shu Wang 33
Yuan Gao 1122 Jiawen Yao 1122 Chunli Li 112233 Xiaoyu Bai 1122
Jingren Zhou 1122 Ling Zhang 11 Le Lu 11 Yu Shi 33

###### Abstract

Liver tumor segmentation and classification are important tasks in computer
aided diagnosis. We aim to address three problems: liver tumor screening and
preliminary diagnosis in non-contrast computed tomography (CT), and
differential diagnosis in dynamic contrast-enhanced CT. A novel framework
named Pixel-Lesion-pAtient Network (PLAN) is proposed. It uses a mask
transformer to jointly segment and classify each lesion with improved anchor
queries and a foreground-enhanced sampling loss. It also has an image-wise
classifier to effectively aggregate global information and predict patient-
level diagnosis. A large-scale multi-phase dataset is collected containing 939
tumor patients and 810 normal subjects. 4010 tumor instances of eight types
are extensively annotated. On the non-contrast tumor screening task, PLAN
achieves 95% and 96% in patient-level sensitivity and specificity. On
contrast-enhanced CT, our lesion-level detection precision, recall, and
classification accuracy are 92%, 89%, and 86%, outperforming widely used CNN
and transformers for lesion segmentation. We also conduct a reader study on a
holdout set of 250 cases. PLAN is on par with a senior human radiologist,
showing the clinical significance of our results.

###### Keywords:

Liver tumor Lesion segmentation and classification CT.

##  1 Introduction

Liver cancer is the third leading cause of cancer death world-wide in 2020
[14]. Early detection and accurate diagnosis of liver tumors may improve
overall patient outcomes, in which imaging plays a key role [11]. Computed
tomography (CT) is one of the most important imaging modalities for liver
tumors. Dynamic contrast-enhanced (DCE) CT is widely used for diagnostics, but
it requires iodine contrast injection which can cause reaction and potential
risks in patients. Recently, non-contrast (NC) CT scans are gaining attention
as they are cheaper and safer to acquire, thus can be potential tools for
opportunistic tumor screening [18, 20]. Meanwhile, finding and diagnosing
tumors in NC CTs is also extremely challenging because of the poor contrast
between tumors and normal tissues compared to those in DCE CTs. Prior works on
pancreas [18] and esophagus [20] have shown that latest deep learning
techniques can detect subtle texture and shape changes in NC CT that even
human eyes may miss. Thus, we aim to investigate the performance of liver
tumor segmentation and classification in NC CTs. Such an approach will be
helpful to discover asymptomatic incidental tumors [12] from routine NC CT
scans indicated for general diagnostic purposes at no additional cost and
radiation exposure. After an incidental tumor is found, the patient may
undergo further imaging examination such as a multi-phase DCE CT for
differential diagnosis [11], which can provide useful discriminative
information such as the vascularity of lesions and the pattern of contrast
agent enhancement [19]. Liver is largest solid organ in body and is the site
of many tumor types [11]. Therefore, accurate tumor type classification is
important for the decision of treatment plans and prognosis.

Many researchers have developed algorithms to automatically segment [1, 9, 13,
15, 23] or classify [25, 21, 19] liver tumors in CT to help radiologists
improve their accuracy and efficiency. For example, public datasets such as
the Liver Tumor Segmentation Benchmark (LiTS) [1] fostered a series of works
aiming to segment liver tumors with improved convolutional neural network
(CNN) backbones [9, 13] and lesion edge information [15]. LiTS only has
single-phase CTs (venous phase). Several studies investigated methods to
exploit multi-phase CT by methods such as hetero-phase fusion [5] and
modality-aware mutual learning [23]. There are few work discussing liver tumor
analysis in NC CT [5]. Besides lesion segmentation, CNN-based lesion
classification algorithms have been studied to distinguish common lesion types
[19, 25, 21].

In this paper, we build a comprehensive framework to address both tumor
screening and diagnosis. (1) Tumor screening involves finding tumor patients
in a large pool of healthy subjects and patients. Most existing works in tumor
segmentation and detection did not explicitly consider it since their training
and testing images are all tumor patients. Such models may generate false
positives in real-world screening scenario when facing diverse tumor-free
images. We collect a large-scale dataset with both tumor and non-tumor
subjects, where the non-tumor subjects includes not only healthy ones, but
also patients with various diffuse liver diseases such as steatosis and
hepatitis to improve the robustness of the algorithm. (2) Most works studied
liver tumor segmentation alone without differentiating tumor types, while a
few works classify liver tumors on cropped tumor patches [19, 25, 21].
Meanwhile, we learn tumor segmentation and classification with one network
using an instance segmentation framework [3]. We train two networks for NC and
multi-phase DCE CTs, respectively. (3) For evaluation, previous segmentation
works typically use _pixel-level_ metrics such as Dice coefficient. Such
metrics cannot reflect the _lesion-level_ accuracy (how many lesion instances
are correctly detected and classified) and may bias to large lesions when a
patient has multiple tumors. _Patient-level_ metrics (e.g. classifying whether
a subject has malignant tumors) are also useful for treatment recommendation
in clinical practice [18, 20]. Therefore, we assess our algorithm thoroughly
with pixel, lesion, and patient-level metrics.

Algorithms for liver tumor segmentation have focused on improving the feature
extraction backbone of a fully-convolutional CNN [9, 13, 15, 23]. The pixel-
wise segmentation architectures may not be optimal for lesion and patient-
level evaluation metrics since they cannot consider a lesion or an image
holistically. Recently, a series of mask transformer algorithms [17, 4, 3]
have emerged in the computer vision community and achieved the state-of-the-
art performance in instance segmentation tasks. In brief, they use object
queries to interact with image feature maps and with each other to produce
mask and class predictions for each instance. Inspired by them, we propose a
novel end-to-end framework named Pixel-Lesion-pAtient Network (PLAN) for
lesion segmentation and classification, as well as patient classification. It
contains three branches with bottom-up cooperation: The segmentation map from
the _pixel branch_ helps to initialize the _lesion branch_ , which is an
improved mask transformer aiming to segment and classify each lesion; The
_patient branch_ aggregates information from the whole image and predicts
image-level labels of each lesion type, with regularization terms to encourage
consistency with the lesion branch.

We collected a large-scale multi-phase dataset containing 810 non-tumor
subjects and 939 tumor patients. 4010 tumor instances of eight types are
extensively annotated based on pathological reports. On the non-contrast tumor
screening and diagnosis task, PLAN achieves 95.0%, 96.4%, and 0.965 in
patient-level sensitivity, specificity, and average AUC for malignant and
benign patients, in contrast to 94.4%, 93.7%, and 0.889 for the widely-used
nnU-Net [8]. On multi-phase DCE CT, our lesion-level detection precision,
recall, and classification accuracy are 92.2%, 89.0%, 85.9%, outperforming
nnU-Net [8] and Mask2Former [3]. We further conduct a reader study on a
holdout set of 250 cases. Our algorithm is on par with a senior radiologist
(16 yrs experience), showing the clinical significance of our results. Our
codes will be made public upon institutional approval.

##  2 Method

###  2.1 Preliminary on Mask Transformer

Figure 1: Framework of the Pixel-Lesion-pAtient Network (PLAN).

Mask transformers are a series of latest works achieving superior accuracy on
various segmentation tasks [17, 4, 3, 22]. Different from traditional fully-
convolutional segmentators [8] that predict a class label for each pixel, mask
transformers predict a class label and a binary mask for each object. Take
Mask2Former [3] as an example. It includes a pixel encoder and a pixel decoder
that extract a high-resolution pixel embedding tensor
$\mathbf{P}\in\mathbb{R}^{M\times D\times H\times W}$ from the image, where
$M$ is the embedding dimension, $D\times H\times W$ is the shape of the 3D
image. A group of $Q$ learnable feature vectors
$\\{\mathbf{q}_{i}\in\mathbb{R}^{M}\\}_{i=1}^{Q}$ are randomly initialized as
object queries. They are processed by a transformer decoder to interact with
multi-scale image features and each other using cross and self-attention
operations. After processing, each query is supposed to contain information of
one object, which can be used to predict the class probability
$\mathbf{c}\in\mathbb{R}^{C+1}$ of the object. Here $C$ is the number of
object classes, and we add 1 to indicate an additional “no-object” class if
the query does not match with any object. In training, Mask2Former uses
bipartite matching [2] to assign each query to a ground-truth object (or “no-
object”). Multiplying $\mathbf{q}_{i}$ with $\mathbf{P}$ gives the binary mask
$\mathbf{m}_{i}\in\mathbb{R}^{D\times H\times W}$ of object $i$. During
inference, the class and mask predictions of all queries can be merged by
matrix multiplication to obtain the final semantic segmentation result
$\hat{\mathbf{Y}}\in\mathbb{R}^{C\times D\times H\times W}$. We refer readers
to [3] for more details.

Mask transformers have various advantages when applied to our task. They can
classify a lesion as a whole instead of classifying each pixel, thus can view
each lesion holistically. Cross-attention is used to aggregate global features
for each lesion. Inter-lesion relation can also be exploited by self-attention
operations. In liver CT, inter-lesion relation is diagnostically useful, e.g.,
metastases and cysts are often multiple. Therefore, We pioneer mask
transformers’ adaptation for lesion segmentation and classification in 3D
medical images. Given a ground-truth or a predicted lesion mask image, we
perform connected component (CC) analysis and treat each CC as a lesion
instance for training and evaluation.

###  2.2 Pixel-Lesion-Patient Network (PLAN)

Our goal is to segment the mask and classify the type of each tumor in a liver
CT. We also hope to make patient-level diagnoses for each CT scan. PLAN is
inspired by Mask2Former [3] with three key improvements: (1) A pixel branch is
added to provide anchor queries to the lesion branch. (2) The lesion branch is
composed of the transformer decoder in Mask2Former, and we improve its
segmentation loss to enhance recall of small lesions. (3) A patient branch is
attached to make dedicated image-level predictions with a proposed lesion-
patient consistency loss. Our framework is shown in Fig. 1.

Pixel branch and anchor queries. The pixel branch is a convolutional layer
after the pixel decoder and learns to predict pixel-wise segmentation maps
similar to traditional segmentators. We do CC analysis to the predicted mask
to extract lesion instances, and then average the pixel embeddings inside each
predicted lesion to obtain a feature vector. The feature vectors are regarded
as anchor queries and work the same way as the randomly initialized queries in
the lesion branch. Compared to the random queries in the original Mask2Former,
the anchor queries contain prior information of the lesions to be segmented,
helping the lesion branch to match with the lesion targets more easily [10].

Lesion branch and foreground-enhanced sampling loss. Similar to Mask2Former,
the lesion branch predicts a binary mask and a class label for each query, see
Fig. 1. Mask2Former calculates its segmentation loss on $K$ sampled pixels
instead of on the whole image, which is shown to both improve accuracy and
reduce GPU memory usage [3]. However, in lesion segmentation, some tumors are
very small compared to the whole 3D image. The importance sampling strategy
[3] can hardly select any foreground pixels in such cases, so the loss only
contains background pixels, degrading the segmentation recall of small
lesions. We propose a simple approach to remedy this issue by sampling an
extra $n$ foreground pixels for each lesion.

Patient branch. A patient-level diagnosis is useful for triage. For example,
diagnosing the subject as normal, benign, or malignant will result in
completely different treatments [24]. Intuitively, we can also infer patient-
level labels from segmentation results by checking if there is any lesion in
the predicted mask. However, certain tumors are often related to signs outside
the tumor, e.g. hepatocellular carcinoma and cirrhosis, cholangiocarcinoma and
bile duct dilatation, etc. We equip PLAN with a dedicated patient branch to
aggregate such global information to make better patient-level prediction.
Since one patient can have multiple liver tumors of different types, in our
problem, we give each image several hierarchical binary labels. The first
label classifies normal and tumor subjects (whether the image contains any
tumor); The second and third labels indicate the existence of respectively
benign and malignant tumors; The rest $C$ labels suggest the existence of $C$
fine-grained types of tumors. We employ the dual-path transformer block [17]
to fuse multi-scale features from the pixel encoder and decoder to generate a
feature map, followed by global average pooling and a linear classification
layer to predict the $C+3$ labels.

A lesion-patient consistency loss is further proposed to encourage coherence
of the lesion and patient-level predictions. Inspired by multi-instance
learning [6], we compute a pseudo patient-level prediction
$\tilde{\mathbf{c}}\in\mathbb{R}^{C}$ from the lesion-level predictions by
max-pooling the class probability of each class across all lesion queries
(discarding the no-object class). We also have the probability vector from the
patient branch $\tilde{\mathbf{p}}\in\mathbb{R}^{C}$ corresponding to the $C$
fine-grained classes. Then, we compute the L2 loss between them:
$\mathcal{L}_{\mathrm{consist}}=\|\tilde{\mathbf{p}}-\tilde{\mathbf{c}}\|^{2}$.

The overall loss of PLAN is listed in Eq. 1, where
$\mathcal{L}_{\mathrm{pixel}}$ is the combined cross-entropy (CE) and Dice
loss for the pixel branch as in nnU-Net [8];
$\mathcal{L}_{\mathrm{lesion\mathchar 45\relax class}}$ is the CE loss [3] for
lesion classification in the lesion branch;
$\mathcal{L}_{\mathrm{lesion\mathchar 45\relax mask}}$ is the combined CE and
Dice loss [3] for binary lesion segmentation in the lesion branch with the
foreground-enhanced sampling strategy; $\mathcal{L}_{\mathrm{patient}}$ is the
binary CE loss for the multi-label classification task in the patient branch.

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\mathrm{pixel}}+\lambda_{2\mathrm{c}}\mathcal{L}_{\mathrm{lesion\mathchar
45\relax class}}+\lambda_{2\mathrm{m}}\mathcal{L}_{\mathrm{lesion\mathchar
45\relax
mask}}+\lambda_{3}\mathcal{L}_{\mathrm{patient}}+\lambda_{4}\mathcal{L}_{\mathrm{consist}}.$
(1)

##  3 Experiments

Data. Our dataset contains 810 normal subjects and 939 patients with liver
tumors. Each normal subject has a non-contrast (NC) CT, while each patient has
a dynamic contrast-enhanced (DCE) CT scan with NC, arterial, and venous
phases. We use DEEDS [7] to register NC and arterial phases to the venous
phase, and then invite a senior radiologist with 10 years of experience to
annotate on the multi-phase CTs using CT Labeler [16]. The 3D mask and the
type of all liver tumors are annotated based on pathological reports and
magnetic resonance scans if necessary. Eight tumor types are considered in our
study: hepatocellular carcinoma (HCC), intrahepatic cholangiocarcinoma (ICC),
metastasis (meta), hepatoblastoma (hepato), hemangioma (heman), focal nodular
hyperplasia (FNH), cyst, and others (all other tumor types). If a lesion’s
type cannot be determined according to image signs [11] and pathology, it will
be marked as “unknown” and ignored in training and evaluation. In total, 4010
tumor instances are annotated, whose volumes range from 11 to $3.7\times
10^{6}$ mm3. Detailed statistics and examples of the lesions are shown in the
Appendix. We train two separate networks for NC and DCE CTs. In the former
setting, both normal and patient data are used and randomly split into 1149
training, 100 validation, and 500 testing. In the latter one, only patient
data are used with 641 training, 100 validation, and 200 testing. Another
hold-out set of 150 patients and 100 normal CTs are used for reader study to
compare our accuracy with two radiologists.

Implementation Details. Each CT is resampled to $0.7\times 0.7\times 5$mm in
spacing. We first train an nnU-Net on public datasets to segment liver and
surrounding organs (gallbladder, hepatic vein, spleen, stomach, and pancreas),
and then crop the liver region to train PLAN. To help PLAN differentiate liver
tumors and other organs, we train the network to segment both tumors and
organs using the predicted organ labels. PLAN is built on top of the nnU-Net
framework [8]. Its pixel encoder is a U-Net encoder, whereas its pixel decoder
is a light-weight feature pyramid network [3]. The lesion branch incorporates
three transformer decoder blocks with masked attention [3] which use feature
maps of strides 16, 8, 4 from the pixel decoder. The number of random queries
is $Q=20$; the embedding dimension is $M=64$; the number of sampled pixels is
$K=12544$ [3], foreground pixels $n=3$; the loss weight is 0.1 for the no-
object class while 1 for other classes in the lesion branch [3]. The weights
in Eq. 1 are
$\lambda_{1}=\lambda_{2\mathrm{c}}=2,\lambda_{2\mathrm{m}}=5,\lambda_{3}=1,\lambda_{4}=0.1$.
We use the RAdam optimizer with an initial learning rate of 0.0001. Each
training batch contains two patches of size $256\times 256\times 24$. For DCE
CT, the three phases form a 3-channel image as the network input. Extensive
data augmentation is applied including random cropping, scaling, flipping,
elastic deformation, and brightness adjustment [8]. During training, we first
pretrain the backbone and the pixel branch for 500 epochs, and then train the
whole network for another 500 epochs.

Table 1: Patient-level performance on the test set of 500 cases. Spec. 1: specificity on the 202 completely normal cases; Spec. 2: specificity on the 100 hard non-tumor cases. | NC tumor screening (%) | NC diagnosis AUC | DCE diagnosis AUC
---|---|---|---
|  Sens. |  Spec. 1 |  Spec. 2 |  Malignant |  Benign | 8-class Average
nnU-Net [8] |  94.4 |  95.1 |  91.0 |  0.948 |  0.829 | 0.863
Mask2Former [3] |  93.9 |  97.0 |  94.0 |  0.924 |  0.828 | 0.873
PLAN (ours) |  95.0 |  97.5 |  94.0 |  0.961 |  0.968 | 0.898

Patient-level results. This paper has three major goals: tumor screening in NC
CT (classifying a subject as normal or tumor), preliminary diagnosis in NC CT
(predicting the existence of malignant and benign tumors), and fine-grained
diagnosis in DCE CT (predicting the existence of 8 tumor types). Among the 8
tumor types, HCC, ICC, meta, and hepato are malignant; heman, FNH, and cyst
are benign. “Others” can be either malignant or benign, thus are excluded in
the preliminary diagnosis task. The NC test set contains 198 tumor cases, 202
completely normal cases, and 100 “hard” non-tumor cases which may have larger
image noise, artifact, ascites, diffuse liver diseases such as hepatitis and
steatosis. These cases are used to test the robustness of the model in real-
world screening scenario with diverse tumor-free images. We compare PLAN with
a widely-used strong baseline, nnU-Net [8]. The recent mask transformer,
Mask2Former [3], is also adapted to 3D for comparison. For the baselines,
patient-level labels are inferred from their predicted masks by counting
lesion pixels. As displayed in Table 1, PLAN achieves the best accuracy on all
tasks, especially in NC preliminary diagnosis tasks, which demonstrates the
effectiveness of its dedicated patient branch that can explicitly aggregate
features from the whole image.

Table 2: Lesion-level performance (precision, recall, recall of lesions with different radius, classification accuracy of 8 tumor types), and pixel-level performance (Dice per case). Precision, recall, and Dice are computed without considering the tumor types. |  |  Prec. |  Recall |  R<5mm |  5$\sim$10 |  10$\sim$20 |  >20mm |  Acc. |  Dice
---|---|---|---|---|---|---|---|---|---
NC | nnU-Net |  78.8 |  77.3 |  19.7 |  63.6 |  90.1 |  96.5 |  75.7 |  78.3
Mask2Former |  85.7 |  74.0 |  10.0 |  60.5 |  91.9 |  97.4 |  77.9 |  76.4
PLAN |  80.1 |  81.9 |  21.9 |  64.6 |  90.1 |  98.3 |  78.5 |  77.2
DCE | nnU-Net |  88.1 |  88.3 |  22.5 |  76.4 |  93.7 |  98.3 |  83.1 |  84.2
Mask2Former |  90.3 |  83.5 |  11.7 |  74.4 |  94.6 |  97.4 |  84.8 |  82.9
PLAN |  92.2 |  89.0 |  25.6 |  74.9 |  94.6 |  98.3 |  85.9 |  84.2

Lesion and pixel-level results. In lesion-level evaluation, we treat a
prediction as a true positive if its overlap with a ground-truth lesion is
>0.2 in Dice. Lesions smaller than 3mm in radius are ignored. As shown in
Table 2, the pixel-level accuracy of nnU-Net and PLAN are comparable, but
PLAN’s lesion-level accuracy is consistently higher than nnU-Net. In this
work, we focus more on patient and lesion-level metrics. Although NC images
have low contrast, they can still be used to segment and classify lesions with
$\sim$ 80% precision, recall, and classification accuracy. It implies the
potential of NC CT, which has been understudied in previous works. Mask2Former
has higher precision but lower recall in NC CT, especially for small lesions,
while PLAN achieves the best recall using the foreground-enhanced sampling
loss. Both PLAN and Mask2Former achieve better classification accuracy, which
illustrates the mask transformer architecture is good at lesion-level
classification.

| NC |  DCE
---|---|---
| Sens. | Spec. |  3-class Acc. |  8-class Acc.
Radiologist 1 | 94.1 | 99.0 |  90.8 |  75.6
Radiologist 2 | 85.5 | 99.0 |  72.0 |  40.5
PLAN | 96.7 | 98.0 |  91.3 |  75.6
Table 3: Reader study results on 150 tumor cases and 100 normal cases. 3-class
acc. means classification accuracy of normal vs. benign vs. malignant.

Figure 2: ROC curve of our method versus 2 radiologists’ performance.

Comparison with radiologists. In the reader study, we invited a senior
radiologist with 16 years of experience in liver imaging, and a junior
radiologist with 2 years of experience. They first read the NC CT of all
subjects and provided a diagnosis of normal, benign, or malignant. Then, they
read the DCE scans and provided a diagnosis of the 8 tumor types. We consider
patients with only one tumor type in this study. Their reading process is
without time constraint. In Table 2 and Fig. 2, all methods get good
specificity probably because the normal subjects are completely healthy. Our
model achieves comparable accuracy with the senior radiologist but outperforms
the junior one by a large margin in sensitivity and classification accuracy.

Table 4: Ablation study on NC data. FES loss: foreground enhanced sampling loss. | Tumor screening (%) | Prelim. diagnosis AUC | Lesion and pixel-level (%)
---|---|---|---
|  Sens. |  Spec. |  Malignant |  Benign |  Precision |  Recall |  Acc. |  Dice
PLAN (proposed) |  95.0 |  96.4 |  96.1 |  96.8 |  80.1 |  81.9 |  78.5 |  77.2
w/o anchor queries |  94.4 |  95.4 |  94.9 |  93.5 |  78.9 |  78.1 |  77.1 |  75.0
w/o FES loss |  93.4 |  96.0 |  94.0 |  96.4 |  86.6 |  75.1 |  77.7 |  77.2
w/o consistency loss |  93.9 |  96.7 |  95.4 |  96.3 |  79.1 |  80.7 |  78.2 |  76.6

An ablation study for our method is shown in Table 4. It can be seen that our
proposed anchor queries produced by the pixel branch, FES loss, and lesion-
patient consistency loss are useful for the final performance. The efficacy of
the lesion and patient branches has been analyzed above based on the lesion
and patient-level results. We will show the accuracy for each tumor type and
more qualitative examples in the Appendix.

Comparison with literature. In the pixel level, we obtain Dice scores of 77.2%
and 84.2% using NC and DCE CTs, respectively. The current state of the art
(SOTA) of LiTS [1] achieved 82.2% in Dice using CTs in venous phase; [23]
achieved 81.3% in Dice using DCE CT of two phases. In the lesion level, our
precision and recall are 80.1% and 81.9% for NC CT, 92.2% and 89.0% for DCE
CT, at 20% overlap. [25] achieved 83% and 93% for DCE CT. SOTA of LiTS
achieved 49.7% and 46.3% at 50% overlap. [21] classified lesions into 5
classes, achieving 84% accuracy for DCE and 49% for NC CT. We classify lesions
into 8 classes with 85.9% accuracy for DCE and 78.5% for NC CT. In the patient
level, [5] achieved AUC=0.75 in NC CT tumor screening, while our AUC is 0.985.
In summary, our results are superior or comparable to existing works.

##  4 Conclusion

Three tasks are investigated in this paper: liver tumor screening and
preliminary diagnosis in NC CT, and the diagnosis of 8 tumor types in DCE CT.
The pixel-lesion-patient network is proposed that can accomplish lesion-level
segmentation and classification, and patient-level classification.
Comprehensive evaluation on a large-scale dataset confirms the effectiveness
and clinical significance of our method. It can serve as a powerful tool for
automated screening and diagnosis of various liver tumors. Our future work
includes further improving the specificity of hard non-tumor cases and
sensitivity of small lesions.

##  5 Appendix

Table 5: Statistics of lesion types and sizes in our dataset of 939 patients and 4010 annotated lesions. The last row shows the number of patients with the tumor type. Note that one patient may have tumors of multiple types. | HCC | ICC | Meta | Hepato | Heman | FNH | Cyst | Others | Total
---|---|---|---|---|---|---|---|---|---
R<5mm | 19 | 6 | 516 | 3 | 45 | 0 | 1081 | 127 | 1797
5$\sim$10 | 101 | 4 | 504 | 3 | 118 | 13 | 224 | 45 | 1012
10$\sim$20 | 186 | 21 | 241 | 1 | 82 | 14 | 43 | 33 | 621
>20mm | 191 | 59 | 90 | 20 | 138 | 28 | 15 | 39 | 580
Total # lesions | 497 | 90 | 1351 | 27 | 383 | 55 | 1363 | 244 | 4010
# Patients | 414 | 79 | 218 | 20 | 171 | 43 | 376 | 174 | 939
Table 6: Lesion-level performance (type-agnostic precision, type-agnostic recall, and recall of 8 tumor types). We find recall is negatively correlated with average size of each tumor type. NC: non-contrast; DCE: dynamic contrast-enhanced (NC+arterial+venous). |  |  Prec. |  Recall |  HCC |  ICC |  Meta |  Hepato |  Heman |  FNH |  Cyst |  Others
---|---|---|---|---|---|---|---|---|---|---|---
NC | nnUNet |  78.8 |  77.3 |  85.7 |  100.0 |  60.5 |  66.7 |  84.0 |  80.0 |  76.2 |  65.7
Mask2Former |  85.7 |  74.0 |  83.8 |  100.0 |  59.0 |  66.7 |  81.3 |  90.0 |  48.6 |  62.9
PLAN |  80.1 |  81.9 |  82.9 |  100.0 |  63.6 |  66.7 |  85.3 |  100.0 |  77.1 |  80.0
DCE | nnUNet |  88.1 |  88.3 |  91.4 |  100.0 |  70.8 |  100.0 |  94.7 |  100.0 |  78.0 |  71.4
Mask2Former |  90.3 |  83.5 |  91.4 |  100.0 |  65.1 |  100.0 |  86.7 |  100.0 |  56.0 |  68.6
PLAN |  92.2 |  89.0 |  89.5 |  100.0 |  72.8 |  100.0 |  92.0 |  100.0 |  86.2 |  71.4
Figure 3: Confusion matrix of lesion-level tumor classification in DCE CT.
Rows and columns indicate true and predicted tumor types, respectively. Only
the correctly detected lesions are considered here for evaluation. Cyst,
heman, HCC, meta, and FNH can be better classified. ICC and others may be
confused with HCC and meta. Figure 4: Qualitative examples of lesion
segmentation and classification in NC CT using different methods. Ground-truth
and predicted tumor types are shown in each result. Ground-truth masks were
annotated on registered venous phase. If a tumor is not predicted, it will be
marked as “missed”. Red texts indicate wrong predictions. Some challenging
cases can be detected by PLAN, such as the subtle FNH (e), small meta (c) and
cyst (f), and the heman which resembles adjacent vessels (d). We also show
failure cases in (g) where a very subtle HCC is missed, and in (c) where a
tiny meta (among three metas) is missed.

## References

  * [1] The Liver Tumor Segmentation Benchmark (LiTS). Medical Image Analysis 84 (2023)
  * [2] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-End Object Detection with Transformers. In: ECCV (2020)
  * [3] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention Mask Transformer for Universal Image Segmentation. In: CVPR. pp. 1280–1289 (2022)
  * [4] Cheng, B., Schwing, A.G., Kirillov, A.: Per-Pixel Classification is Not All You Need for Semantic Segmentation. In: NeurIPS. vol. 22, pp. 17864–17875 (2021)
  * [5] Cheng, C.T., Cai, J., Teng, W., Zheng, Y., Huang, Y.T.: A Flexible Three-Dimensional Hetero-phase Computed Tomography Hepatocellular Carcinoma ( HCC ) Detection Algorithm for Generalizable and Practical HCC Screening. Hepatology Communications (2022)
  * [6] Cheplygina, V., de Bruijne, M., Pluim, J.P.: Not-So-Supervised: A Survey of Semi-Supervised, Multi-Instance, And Transfer Learning in Medical Image Analysis. Medical Image Analysis 54, 280–296 (may 2019)
  * [7] Heinrich, M.P., Jenkinson, M., Brady, M., Schnabel, J.A.: MRF-Based Deformable Registration and Ventilation Estimation of Lung CT. IEEE Transactions on Medical Imaging 32(7), 1239–1248 (2013)
  * [8] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnU-Net: A Self-Configuring Method for Deep Learning-Based Biomedical Image Segmentation. Nature Methods 18(2), 203–211 (2021)
  * [9] Li, X., Chen, H., Qi, X., Dou, Q., Fu, C.W., Heng, P.A.: H-DenseUNet: Hybrid Densely Connected UNet for Liver and Tumor Segmentation from CT Volumes. IEEE Transactions on Medical Imaging 37(12), 2663–2674 (2018)
  * [10] Liu, S.: DAB-DETR : Dynamic Anchor Boxes Are Better Queries for DETR. In: ICLR. pp. 1–19 (2022)
  * [11] Marrero, J.A., Ahn, J., Rajender Reddy, K., Americal College of Gastroenterology: ACG Clinical Guideline: The Diagnosis and Management of Focal Liver Lesions. The American journal of gastroenterology 109(9), 1328–1347 (2014)
  * [12] Semaan, A., Branchi, V., Marowsky, A.L., Websky, M.V., Kupczyk, P., Enkirch, S.J., Kukuk, G., Bölke, E., Stoffels, B., Kalff, J.C., Schäfer, N., Lingohr, P., Matthaei, H.: Incidentally Detected Focal Liver Lesions-A Common Clinical Management Dilemma Revisited. Anticancer Research 36(6), 2923–2932 (2016)
  * [13] Seo, H., Huang, C., Bassenne, M., Xiao, R., Xing, L.: Modified U-Net (mU-Net) with Incorporation of Object-Dependent High Level Features for Improved Liver and Liver-Tumor Segmentation in CT Images. IEEE Transactions on Medical Imaging 39(5), 1316–1325 (2020)
  * [14] Sung, H., Ferlay, J., Siegel, R.L., Laversanne, M., Soerjomataram, I., Jemal, A., Bray, F.: Global Cancer Statistics 2020 : GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA: A Cancer Journal for Clinicians 71(3), 209–249 (2021)
  * [15] Tang, Y., Tang, Y., Zhu, Y., Xiao, J., Summers, R.M.: E2Net: An Edge Enhanced Network for Accurate Liver and Tumor Segmentation on CT Scans. In: MICCAI (2020)
  * [16] Wang, F., Cheng, C.t., Peng, C.w., Yan, K., Wu, M., Lu, L., Liao, C.h., Zhang, L.: A Cascaded Approach for Ultraly High Performance Lesion Detection and False Positive Removal in Liver CT Scans (2023), http://arxiv.org/abs/2306.16036
  * [17] Wang, H., Adam, H., Yuille, A., Chen, L.c.: MaX-DeepLab : End-to-End Panoptic Segmentation with Mask Transformers. In: CVPR. pp. 5463–5474 (2021)
  * [18] Xia, Y., Yao, J., Lu, L., Huang, L., Xie, G., Xiao, J., Yuille, A., Cao, K., Zhang, L.: Effective Pancreatic Cancer Screening on Non-contrast CT Scans via Anatomy-Aware Transformers. In: MICCAI. vol. 12905 LNCS, pp. 259–269 (2021)
  * [19] Xu, X., Zhu, Q., Ying, H., Li, J., Cai, X., Li, S.: A Knowledge-Guided Framework for Fine-Grained Classification of Liver Lesions Based on Multi-Phase CT Images. IEEE Journal of Biomedical and Health Informatics 27(1), 386–396 (2023)
  * [20] Yao, J., Ye, X., Xia, Y., Zhou, J., Shi, Y., Yan, K., Wang, F., Lin, L., Yu, H., Hua, X.S., Lu, L., Jin, D., Zhang, L.: Effective Opportunistic Esophageal Cancer Screening Using Noncontrast CT Imaging. In: MICCAI. vol. 13433 LNCS, pp. 344–354 (2022)
  * [21] Yasaka, K., Akai, H., Abe, O., Kiryu, S.: Deep Learning with Convolutional Neural Network for Differentiation of Liver Masses at Dynamic Contrast-Enhanced CT: A Preliminary Study. Radiology 286(3), 887–896 (2018)
  * [22] Yu, Q., Wang, H., Qiao, S., Collins, M., Zhu, Y., Adam, H., Yuille, A., Chen, L.C.: K-means Mask Transformer. In: ECCV. pp. 288–307 (2022)
  * [23] Zhang, Y., Yang, J., Tian, J., Shi, Z., Zhong, C., Zhang, Y., He, Z.: Modality-Aware Mutual Learning for Multi-modal Medical Image Segmentation. In: MICCAI. vol. 12901 LNCS, pp. 589–599 (2021)
  * [24] Zhao, T., Cao, K., Yao, J., Nogues, I., Lu, L., Huang, L., Xiao, J., Yin, Z., Zhang, L.: 3D Graph Anatomy Geometry-Integrated Network for Pancreatic Mass Segmentation, Diagnosis, And Quantitative Patient Management. In: CVPR. pp. 13738–13747 (2021)
  * [25] Zhou, J., Wang, W., Lei, B., Ge, W., Huang, Y., Zhang, L., Yan, Y., Zhou, D., Ding, Y., Wu, J., Wang, W.: Automatic Detection and Classification of Focal Liver Lesions Based on Deep Convolutional Neural Networks: A Preliminary Study. Frontiers in Oncology 10, 1 (2021)