# Neural network relief: a pruning algorithm based on neural activity

Aleksandr Dekhovich
Department of Materials Science and Engineering
Delft University of Technology
&David M.J. Tax
Pattern Recognition and Bioinformatics Laboratory
Delft University of Technology
&Marcel H.F. Sluiter
Department of Materials Science and Engineering
Delft University of Technology
&Miguel A. Bessa
Department of Materials Science and Engineering
Delft University of Technology
Corresponding author, e-mail<EMAIL_ADDRESS>
###### Abstract

Current deep neural networks (DNNs) are overparameterized and use most of
their neuronal connections during inference for each task. The human brain,
however, developed specialized regions for different tasks and performs
inference with a small fraction of its neuronal connections. We propose an
iterative pruning strategy introducing a simple importance-score metric that
deactivates unimportant connections, tackling overparameterization in DNNs and
modulating the firing patterns. The aim is to find the smallest number of
connections that is still capable of solving a given task with comparable
accuracy, i.e. a simpler subnetwork. We achieve comparable performance for
LeNet architectures on MNIST, and significantly higher parameter compression
than state-of-the-art algorithms for VGG and ResNet architectures on
CIFAR-10/100 and Tiny-ImageNet 111Code is available at:
https://github.com/adekhovich/NNrelief. Our approach also performs well for
the two different optimizers considered – Adam and SGD. The algorithm is not
designed to minimize FLOPs when considering current hardware and software
implementations, although it performs reasonably when compared to the state of
the art.

††Author contributions: A.D. and M.A.B. designed research; A.D. performed
research; A.D., D.M.J.T., M.H.F.S. and M.A.B. analyzed data; A.D. and M.A.B.
wrote the paper with the contribution of D.M.J.T. and M.H.F.S.

##  1 Introduction

Pruning is a common technique for neural network compression [1], where the
main goal is to reduce memory and computational costs of inference. Pruning
assumes particular relevance for deep neural networks because modern
architectures involve several millions of parameters. Existing pruning methods
are based on different strategies, e.g Hessian analysis [2, 3], magnitudes of
weights [4], data-driven approaches [5, 6], among others [7, 8]. Pruning can
be done in one shot [9] or in an iterative way [10], and it is possible to
prune connections [2, 3, 11, 12], neurons [13, 14, 6] or filters for
convolutional layers [15, 16]. The typical pruning pipeline includes three
stages: training the original network, pruning parameters and fine-tuning.
Recently, interesting solutions for the third stage have been suggested that
involve weight rewinding [10] and learning rate rewinding [17].

We have developed an algorithm that aims to prune a network without a
significant decrease in accuracy after every iteration by keeping the signal
in the network at some predefined level close to the original one. This
contrasts with strategies that use a pre-defined ratio of parameters to prune
[5, 6, 10] which may lead to a drastic drop in accuracy for relatively high
ratios. We look at the local behaviour of a particular connection and its
contribution to the neuron, but not at the output or the loss function. Our
aim is to deactivate unimportant connections for a given problem in order to
free them for other tasks. Therefore, we want to reduce the number of
parameters, although we expect a reduction of the computational complexity
(FLOPs) as well.

Our iterative pruning algorithm is based on an importance score metric
proposed herein that quantifies the relevance of each connection to the local
neuron behaviour. We show compressions of more than 50 times for VGG [18]
architectures on CIFAR-10 and Tiny-ImageNet datasets with a marginal drop of
accuracy. We also apply our method to ResNets [19], achieving better parameter
compression than state-of-the-art algorithms with a comparable decrease of
accuracy on CIFAR-10 dataset. In addition, we visualise the effects of our
pruning strategy in the information propagation through the network, and we
observe a significant homogenization of the importance of the pruned neurons.
We associate this homogenization to the notion of neural network relief: using
fewer neuronal connections and distributing importance among them.

##  2 Related work

One of the first works eliminating unimportant connections in relatively small
networks proposed analyzing the Hessian of the loss function [2, 3] without
network retraining. This idea was further developed for convolutional networks
[11, 20]. However, computing second-order derivatives by calculating the
Hessian is costly.

The magnitude-based approach [4] is simple and fast because it only involves
pruning the weights with smallest magnitude. This assumes that parameters with
small magnitudes do not contribute significantly to the resulting performance.
For convolutional layers, the sum of kernels’ elements in the filters is
considered and filters with the smallest sum are pruned [21]. Iterative
magnitude pruning is applied for finding winning tickets [10] – the minimal
subnetwork that can be trained at least as well as the original one with the
same hyperparameters.

Other algorithms use input data to reduce the number of parameters. In [5]
pruning depends on the ratio of zero activations using ReLU function. So, the
neurons that do not fire frequently enough are eliminated. A greedy approach
is used for ThiNet [22], where channels that do not affect the resulting sum
in convolutional layers are removed by solving an optimization problem. A
similar algorithm is used in [15], but the optimization problem is solved with
LASSO regression for determining ineffective channels. The property of
convexity and sparsity that ReLU produces provides analytical boundaries for
Net-trim [23] by solving a convex optimization problem. The game-theoretic
approach with Shapley values [6] demonstrates good performance in a low-data
regime, i.e. one-shot pruning without retraining, where neurons within one
layer are considered as players in a cooperative game.

NISP [24] finds the contribution of neurons to the last layer before
classification, while iSparse framework [25] trains sparse networks
eliminating the connections that do not contribute to the output. A pruning
strategy where the connections’ contribution are based on computing the loss
function derivatives was also suggested in [26].

Additional methods of interest include Bayesian weight pruning [8] and
Bayesian compression [14] to prune neurons and aiming at computational
efficiency. SSS [27] introduces a scaling factor and sparsity constraints on
this factor to scale neurons’ or blocks’ outputs. Similarly, SNLI [28] uses
ISTA[29] to update the scaling parameter in Batch Normalization of
convolutional layers to obtain a sparse representation. HRank [30] calculates
the average rank of feature maps and prunes filters with the lowest ones.

##  3 Our pruning approach

Our goal is to eliminate connections in a layer that, on average, provide a
weak contribution to the next layer. We believe that a low magnitude of a
weight does not mean that information passing through the connection has to
have a negligible contribution. The converse is also assumed to be true: if a
high weight magnitude is multiplied by a weak signal from the neuron (or even
zero) then the contribution of that product is relatively insignificant in
comparison with other input signals in the neuron, as long as this holds for
most data. Therefore, our strategy contrasts significantly with magnitude-
based pruning [4].

###  3.1 Fully connected layers

Figure 1: Neural network layers $l-1$ and $l$ with $m_{l-1}=4$ and $m_{l}=2$
neurons, respectively. The weights associated to a connection between neurons
$i$ and $j$ in layer $l-1$ to $l$ are $w_{ij}^{(l)}$.

Assume that we have a pruning set
$\mathbf{X}^{(l-1)}=\\{\mathbf{x}^{(l-1)}_{1},\ldots,\mathbf{x}^{(l-1)}_{N}\\}$
with $N$ samples, where each datapoint
$\mathbf{x}^{(l-1)}_{n}=(x_{n1}^{(l-1)},\ldots,x_{nm_{l-1}}^{(l-1)})\in\mathbb{R}^{m_{l-1}}$
is the input for layer $l-1$ with dimension $m_{l-1}$ and where $\ 1\leq l\leq
L$ . We define the importance of the connection between neuron $i$ of layer
$l-1$ and neuron $j$ of layer $l$ as:

$s_{ij}^{(l)}=\frac{\overline{|w_{ij}^{(l)}x_{i}^{(l-1)}|}}{\sum_{k=1}^{m_{l-1}}\overline{|w^{(l)}_{kj}x_{k}^{(l-1)}|}+|b_{j}^{(l)}|},$
(1)

where
$\overline{|w_{ij}^{(l)}x_{i}^{(l-1)}|}=\frac{1}{N}\sum_{n=1}^{N}|w_{ij}^{(l)}x_{ni}^{(l-1)}|$,
and $w_{ij}^{(l)}$ is the corresponding weight between neurons $i$ and $j$
(see Figure 1) and $b_{j}^{(l)}$ is the bias associated to neuron $j$. The
importance score for the bias of neuron $j$ is
$s^{(l)}_{m_{l-1}+1,j}=\frac{|b^{(l)}_{j}|}{\sum_{k=1}^{m_{l-1}}\overline{|w^{(l)}_{kj}x_{k}^{(l-1)}|}+|b_{j}^{(l)}|}$.
The denominator corresponds to the total importance in the neuron $j$ of layer
$l$ that we denote as
$S^{(l)}_{j}=\sum_{k=1}^{m_{l-1}}\overline{|w^{(l)}_{kj}x_{k}^{(l-1)}|}+|b_{j}^{(l)}|,\
1\leq j\leq m_{l}$.

1:function FC pruning(network, X, $\alpha$)

2: $\mathbf{X}^{(0)}\leftarrow\mathbf{X}$

3: for every fc_layer $l$ in FC_Layers do

4: $\mathbf{X}^{(l)}$ $\leftarrow$ fc_layer$\big{(}\mathbf{X}^{(l-1)}\big{)}$

5: for every neuron $j$ in fc_layer $l$ do

6: compute importance scores $s^{(l)}_{ij}$ for every incoming connection $i$
using (1).

7: $\hat{s}^{(l)}_{ij}=Sort(s^{(l)}_{ij},order=descending)$

8: $p_{0}=\min\\{p:\sum_{i=1}^{p}\hat{s}^{(l)}_{ij}\geq\alpha\\}$

9: prune connections with importance score
$s^{(l)}_{ij}<\hat{s}^{(l)}_{p_{0}j}$

10: end for

11: end for

12: return pruned network

13:end function

Algorithm 1 Fully connected layers pruning

The idea behind our approach is to deactivate the connections
$i=1\ldots,m_{l-1}$ that on average do not carry important information to the
neuron $j$ of layer $l$ in comparison with other connections. As a result of
Algorithm 1, we simplify each neuron by using a smaller number of input
connections, while causing minimal effective changes to the input signal of
that neuron and subsequent minimal impact on the network.

(a) LeNet-300-100 pretrained on MNIST

(b) LeNet-300-100 pruned on MNIST

Figure 2: LeNet-300-100 architecture on MNIST before and after pruning, where
connections are coloured with respect to importance score: blue (least
important) $\to$ red (most important)

Figure 2 shows the importance score of every connection in the LeNet-300-100
[31] architecture before and after pruning when applied to the MNIST dataset.
Connections are represented by thin blue lines when they have the lowest
importance score $s_{ij}^{(l)}$, and go up to red thick lines when they are
the most important. The figure demonstrates an interesting phenomena besides
the fact that there are much fewer connections after applying our pruning
strategy: the importance scores of the connections become much closer to each
other and there are no longer highly important and highly unimportant
connections. We eliminate connections (contributors to the neuron’s signal)
that on average do not contribute in terms of the strength of the signal that
they bring to the neuron for a given dataset.

###  3.2 Convolutional layers

Our pruning approach for convolutional layers is similar to the one conducted
on fully connected layers. We consider kernels and a bias in a particular
filter as contributors to the signal produced by this filter.

function CONV pruning(network, X, $\alpha$)

2: $\mathbf{X}^{(0)}\leftarrow\mathbf{X}$

for conv layer $l$ in CONV_Layers do

4: $\mathbf{X}^{(l)}\leftarrow$ conv_layer$\big{(}\mathbf{X}^{(l-1)}\big{)}$

for every filter $\mathbf{F}_{j}$ in conv_layer $l$ do

6: compute importance scores $s_{ij}$ $\forall$ kernel $\mathbf{K}^{(l)}_{ij}$
and bias $b^{(l)}_{j}$ in filter $\mathbf{F}^{(l)}_{j}$ using (2), (3).

$\hat{s}^{(l)}_{ij}=Sort(s^{(l)}_{ij},order=descending)$

8: $p_{0}=\min\\{p:\sum_{i=1}^{p}\hat{s}^{(l)}_{ij}\geq\alpha\\}$

prune kernel $\mathbf{K}^{(l)}_{ij}$ with importance score
$s^{(l)}_{ij}<\hat{s}^{(l)}_{p_{0}j}$

10: end for

end for

12: return pruned network

end function

Algorithm 2 Convolutional layers pruning

Assume we have $m_{l-1}$-channeled input samples
$\mathbf{X}^{(l-1)}=\\{\mathbf{x}^{(l-1)}_{1},\ldots,\mathbf{x}^{(l-1)}_{N}\\}$,
where
$\mathbf{x}^{(l-1)}_{k}=(x^{(l-1)}_{k1},\ldots,x^{(l-1)}_{km_{l-1}})\in\mathbb{R}^{m_{l-1}\times
h_{l-1}^{1}\times h^{2}_{l-1}}$, where $h^{1}_{l-1}$ and $h^{2}_{l-1}$ are the
height and width of input images (or feature maps) for convolutional layer
$l$. For every kernel
$\mathbf{K}^{(l)}_{1j},\mathbf{K}^{(l)}_{2j},\ldots,\mathbf{K}^{(l)}_{m_{l}j},\
\mathbf{K}^{(l)}_{ij}=(k^{(l)}_{ijqt})\in\mathbb{R}^{r_{l}\times r_{l}}$,
$1\leq q,t\leq r_{l}$, $r_{l}$ is a kernel size, and a bias $b^{(l)}_{j}$ in
filter $\mathbf{F}^{(l)}_{j}$, we define
$\mathbf{\hat{K}}^{(l)}_{ij}=\Big{(}\absolutevalue{k^{(l)}_{ijqt}}\Big{)}$ as
a matrix consisting of the absolute values of the matrix
$\mathbf{K}^{(l)}_{ij}$.

Then we compute importance scores $s^{(l)}_{ij},i\in\\{1,2,\ldots,m_{l}\\}$ of
kernels $\mathbf{K}^{(l)}_{ij}$ as follows:

$\displaystyle s^{(l)}_{ij}$
$\displaystyle=\frac{\frac{1}{N}\sum_{n=1}^{N}\norm{\mathbf{\hat{K}}^{(l)}_{ij}*\absolutevalue{x^{(l-1)}_{ni}}}_{F}}{S^{(l)}_{j}}.$
(2) $\displaystyle s^{(l)}_{m_{l}+1,j}$
$\displaystyle=\frac{\absolutevalue{b^{(l)}_{j}}\sqrt{h^{1}_{l}h^{2}_{l}}}{S^{(l)}_{j}},$
(3)

where
$S^{(l)}_{j}=\sum_{i=1}^{m_{l-1}}\Big{(}\frac{1}{N}\sum_{n=1}^{N}\norm{\mathbf{\hat{K}}^{(l)}_{ij}*\absolutevalue{x^{(l-1)}_{ni}}}_{F}\Big{)}+\absolutevalue{b^{(l)}_{j}}\sqrt{h^{1}_{l}h^{2}_{l}}$
is the total importance score in filter $\mathbf{F}^{(l)}_{j}$ of layer $l$,
and where $*$ indicates a convolution operation, and $\norm{\cdot}_{F}$ the
Frobenius norm.

In (2) we compute the amount of information that every kernel produces on
average, analogously to what we do in fully connected layers.

##  4 Experiments

We test our pruning method for LeNet-300-100 and LeNet-5 [31] on MNIST, VGG-19
[18] and VGG-like [32] on CIFAR-10/100 [33] and Tiny-ImageNet datasets. We
evaluate our method on classification error, the percentage of pruned
parameters ($\frac{|w=0|}{|w|}(\%)$) or the percentage of remaining parameters
($\frac{|w\neq 0|}{|w|}(\%)$) or compression rate ($\frac{|w|}{|w\neq
0|}(\%)$) and pruned FLOPs (floating-point operations), where $|w\neq 0|$
refers to the number of unpruned connections. For the details about training
and pruning hyperparameters, and FLOPs computation see Section A.1 and Section
A.2. We use the initialization method introduced in [34] to initialize
parameters.

###  4.1 LeNets

We experiment with LeNets on MNIST dataset. LeNet-300-100 is a fully connected
network with 300 neurons and 100 neurons in two hidden layers respectively,
and LeNet-5 Caffe, which is a modified version of [31], has two convolutional
layers followed by one hidden layer and an output layer. We perform iterative
pruning by retraining the network starting from initial random parameters,
instead of fine-tuning. Table 1 shows that our approach is among the best for
both LeNets in terms of pruned parameters.

Network | Method | Error $(\%)$ | $\frac{|w\neq 0|}{|w|}(\%)$
---|---|---|---
LeNet-300-100 | DNS [35] | 1.99 | 1.79
L-OBS [11] | 1.96 | 1.5
SWS [7] | 1.94 | 4.3
Sparse VD [8] | 1.92 | 1.47
| Ours | 1.98 $\pm 0.07$ | 1.51 $\pm$ 0.07
LeNet-5 | DNS [35] | 0.91 | 0.93
L-OBS [11] | 1.66 | 0.9
SWS [7] | 0.97 | 0.5
Sparse VD [8] | 0.75 | 0.36
| Ours | 0.97 $\pm$ 0.05 | 0.65 $\pm$ 0.02
Table 1: Results for LeNet-300-100 and LeNet-5 trained and pruned on MNIST.
For pruning, 1000 random training samples are chosen,
$\alpha_{\text{fc}}=0.95,\ \alpha_{\text{conv}}=0.9$.

###  4.2 VGG

We perform our experiments on VGG-13 and VGG-like (adapted version of VGG-16
for CIFAR-10 dataset that has one fully connected layer less) on CIFAR-10,
CIFAR-100 and Tiny-ImageNet datasets. We also show in this section the effect
of two optimizers – Adam and SGD and two weight decay values on our pruning
technique.

#### CIFAR-10/100.

Table 2 shows that after the reduction of the weights to less than $2\%$ of
the original number, the accuracy only drops by $0.1\%$, and that our approach
outperforms others both in terms of pruned parameters and pruned FLOPs.

Network | Method |  Acc $(\%)$ (baseline) | Acc drop $(\%)$ | $\frac{|w\neq 0|}{|w|}(\%)$ | FLOPs pruned $(\%)$
---|---|---|---|---|---
VGG-like | Pruning [21] | 93.25 | -0.15 | 36 | 34.2
Sparse VD [8] | 92.3 | 0.0 | 2.1 | N/A
BC-GNJ [14] | 91.6 | 0.2 | 6.7 | 55.6
BC-GHS [14] | 91.6 | 0.6 | 5.5 | 61.7
SNIP [26] | 91.7 | -0.3 | 3.0 | N/A
HRank [30] | 93.96 | 1.62 | 17.9 | 65.3
| Ours | 92.5 | 0.1 | 1.92 $\pm$ 0.02 | 75.5 $\pm$ 0.4
Table 2: Results for VGG-like trained on CIFAR-10 with Adam optimizer. During
retraining 60 epochs are used; the learning rate is decreased by 10 every 20
epochs. For Ours $\alpha_{\text{conv}}=\alpha_{\text{fc}}=0.95$ and 1000
samples are chosen for the importance scores computation. Sign ’-’ means that
accuracy has increased after pruning.

We also perform pruning for standard VGG-13 on CIFAR-100 using 5 iterations,
and considered three randomization seeds to provide reasonable statistical
significance. The results from the different optimizers are presented in
Figure 3. The results show that Adam seems to perform the compression more
aggressively, resulting in a higher compression rate, and also a slightly
lower accuracy. The compression of about 14.5$\times$ after the fifth
iteration without loss in accuracy is presented for Adam and weight decay
$5\cdot 10^{-4}$, while SGD allows to train a network with higher accuracy,
but after 5 iterations we obtain smaller compression (about $4.5\times$) and
the decrease of accuracy by $0.5\%$ with the same weight decay. We obtain
lower compression rate for both optimizers when the weight decay is lower
($10^{-4}$). Figure 5 shows the final architecture and sparsity that we
achieve with Adam optimizer at the end of pruning procedure. We observe a high
level of filter sparsity with Adam, as also reported in [36], but good
compression is also obtained for the SGD optimizer.

Figure 3: Results for VGG-13 over three seeds; mean values are used to compute
dots and standard deviation are shown with bars. We compare two optimizers,
SGD and Adam, and two different values of weight decay for evaluation after 5
iterations.

(a) An architecture before and after pruning by seed.

(b) Sparsity by layer after pruning

Figure 4: Architecture structure for VGG-like on CIFAR-10 with Adam optimizer.

(a) SGD, weight decay = $5\cdot 10^{-4}$

(b) SGD, weight decay = $10^{-4}$

(c) Adam, weight decay = $5\cdot 10^{-4}$

(d) Adam, weight decay = $10^{-4}$

Figure 5: VGG-13 architecture on CIFAR-100 trained with SGD (top) and Adam
(bottom), and weight decay equal to $5\cdot 10^{-4}$ (left) and $10^{-4}$
(right) and pruned with 5 iterations. The results for three different seeds
are presented.

#### Tiny-ImageNet.

We also apply our approach to Tiny-ImageNet dataset that is a subset of the
ImageNet dataset with 200 classes and image spatial resolution of
64$\times$64\. Following SNIP strategy, we use strides $[2,2]$ in the first
convolutional layer to reduce the size of images. The results are reported in
Table 3. We observe a compression by more than 40 times almost without any
loss of accuracy ($-0.03\%$) using Adam optimizer.

Network | Method |  Acc $(\%)$ (baseline) | Acc drop $(\%)$ | $\frac{|w\neq 0|}{|w|}(\%)$ | FLOPs pruned $(\%)$
---|---|---|---|---|---
VGG-like | SNIP [26] | 45.14 | 0.87 | 5.0 | N/A
Ours | 45.63 | 0.03 | 2.32 $\pm$ 0.06 | 75 $\pm$ 0.32
Table 3: Results for VGG-like trained on Tiny-ImageNet with Adam optimizer.
During retraining 50 epochs are used; the learning rate is decreased by 10
every 20 epochs. For Ours $\alpha_{\text{conv}}=\alpha_{\text{fc}}=0.95$ and
2000 samples are chosen for the importance scores computation.

###  4.3 ResNets

In addition, we test our approach on ResNet architectures for CIFAR-10/100 and
Tiny-ImageNet datasets. Our main objective is to prune as many parameters as
we can without significant loss of accuracy. For ResNet-20/56 on CIFAR-10 we
perform iterative pruning over 10 iterations, training the model with SGD and
Adam. In Table 4 we compare our results with other approaches, and Figure 6
displays the pruning history. We achieve higher percentage of pruned
parameters for ResNet-20 and ResNet-56, using both optimizers with comparable
final accuracy.

ResNet | Method |  Acc $(\%)$ (baseline) | Acc drop $(\%)$ | $\frac{|w=0|}{|w|}(\%)$ |  FLOPs pruned $(\%)$
---|---|---|---|---|---
20 | SFP [16] | 92.2 | 1.37 | 30 | 42.2
SNLI [28] | 92.0 | 1.1 | 37.2 | N/A
SSS [27] | 92.8 | 2 | 45 | 60
CNN-CFC [37] | 92.2 | 1.07 | 42.75 | 41.6
Ours (SGD) | 92.25 $\pm$ 0.12 | 1.15 $\pm$ 0.13 | 63.68 $\pm$ 1.52 | 25.85 $\pm$ 1.57
Ours (Adam) | 91.83 $\pm$ 0.16 | 0.39 $\pm$ 0.27 | 68.75 $\pm$ 1.26 | 13.81 $\pm$ 2.6
56 | Pruning-B [21] | 93.04 | -0.02 | 13.7 | 27.6
SFP [16] | 93.59 | -0.19 | 30 | 41.1
NISP [24] | N/A | 0.03 | 42.6 | 43.61
CNN-CFC[37] | 93.14 | -0.24 | 43.09 | 42.78
93.14 | 1.22 | 69.74 | 70.9
HRank [30] | 93.36 | 0.09 | 42.4 | 50
93.36 | 2.54 | 68.1 | 74.1
Ours (SGD) | 93.6 $\pm$ 0.08 | 1.14 $\pm$ 0.22 | 76.2 $\pm$ 0.44 | 35.8 $\pm$ 2.69
Ours (Adam) | 92.8 $\pm$ 0.08 | 0.03 $\pm$ 0.23 | 75.95 $\pm$ 0.22 | 32.5 $\pm$ 0.31
Table 4: Results for ResNets trained on CIFAR-10 with SGD and Adam optimizers.
During retraining 60 epochs are used; the learning rate is decreased by 10
every 20 epochs. For Ours $\alpha_{\text{conv}}=0.95,\alpha_{\text{fc}}=0.99$.

A notable observation for some seeds when training with SGD is that separate
convolutional blocks are pruned and only residual blocks remain, i.e. the
signal propagates through skip connections and not through main convolutional
layers since they do not contribute to the sum (Figure 8).

(a) CIFAR-10

(b) CIFAR-100

(c) CIFAR-10

(d) CIFAR-100

Figure 6: ResNet-20 on CIFAR-10 and CIFAR-100 by iteration averaged over three
seeds. The red dot corresponds to the selected best iteration.

(a) CIFAR-10

(b) CIFAR-100

(c) CIFAR-10

(d) CIFAR-100

Figure 7: ResNet-56 on CIFAR-10 and CIFAR-100 by iteration averaged over three
seeds. The red dot corresponds to the selected best iteration. Figure 8:
ResNet-56 architecture on CIFAR-10 by seed with SGD optimizer.

For ResNet-20/56 on CIFAR-100 the results are presented in Table 5 and by
iteration in Figure 7.

Network | Optimizer |  Acc $(\%)$ (baseline) | Acc drop $(\%)$ | $\frac{|w=0|}{|w|}(\%)$ | FLOPs pruned $(\%)$
---|---|---|---|---|---
ResNet-20 | SGD | 68.75 $\pm$ 0.15 | 1.26 $\pm$ 0.33 | 54.54 $\pm$ 0.24 | 12.49 $\pm$ 1.52
Adam | 67.34 $\pm$ 0.2 | 0.41 $\pm$ 0.23 | 51.82 $\pm$ 0.23 | 3.42 $\pm$ 1.21
ResNet-56 | SGD | 71.43 $\pm$ 0.16 | 0.43 $\pm$ 0.26 | 58.62 $\pm$ 0.26 | 31.2 $\pm$ 3.09
Adam | 71.68 $\pm$ 0.09 | 0.88 $\pm$ 0.12 | 67.72 $\pm$ 0.29 | 13.79 $\pm$ 0.94
Table 5: Results for ResNets trained on CIFAR-100 with SGD and Adam
optimizers. During retraining 80 epochs are used; the learning rate is
decreased by 10 every 30 epochs. For Ours
$\alpha_{\text{conv}}=0.95,\alpha_{\text{fc}}=0.99$.

#### Tiny-ImageNet.

Training of a modified ResNet-18 with 16, 32, 64 and 128 output channels
indicates that we can prune more than 50% of the parameters with both
optimizers (see Figure 9). Adam, however, maintains a higher level of accuracy
during retraining than SGD. We use 2000 samples to evaluate importance scores,
and weight decay for (re-)training equal to $5\cdot 10^{-4}$.

#### Sparsity.

In contrast to VGG results, we do not observe significant difference between
optimizers in terms of produced sparsity for CIFAR-10/100 and Tiny-ImageNet.
In the works that explore the question of sparsity [36], ResNet architecture
was not explored. It seems that the skip connections allow the signal to pass
unhindered to the next layers, making it harder for the Adam optimizer to find
which activations are significant, and aggressively prune the less significant
connections.

##  5 Discussion

Figure 9: ResNet-18 on Tiny-ImageNet.

To illustrate the meaning of the importance scores we compare our approach
with the magnitude-based one. Also, we present error bounds for the difference
of a signal in the layer before and after pruning.

###  5.1 Comparison with Magnitude-based approach

In order to show the difference between our approach and the magnitude-based
approach [4] we build a heatmap for importance scores for the connections of
the first fully connected layer, computed both with our rule and magnitude
rule. The importance scores are computed by equation (1).

Figure 10 displays difference between our approach and magnitude-based one
when determining the significance of network connections. According to our
approach, in the original well-trained network there are just a few
connections which contribute much more than others. However, the contrast is
much more significant for the magnitude-based approach.

Figure 10: Importance scores (left) and Magnitudes (right) of weights
visualization for trained Lenet-300-100 on MNIST from the input to the first
hidden layer with heatmaps. The highest value of each of the color bars
corresponds to the maximum value in the corresponding matrix.

###  5.2 Error estimation

To complement the analysis of our approach we derive error bounds by
considering the difference between a signal in the trained neuron and a
reduced one without retraining, before and after the activation function.
Without loss of generality we may assume the first $p$ connections in the
neuron $j$ of layer $l$ are kept and that the bias $b^{(l)}_{j}$ as well as
weights $w^{(l)}_{p+1,j},\ldots,w^{(l)}_{m_{l}j}$ are pruned in a particular
neuron $j$. Then, for any input
$x_{n}^{(l-1)}\in\mathbf{X}^{(l-1)}=\\{x_{1}^{(l-1)},\ldots,x^{(l-1)}_{N}\\}\subset\mathbb{R}^{m_{l-1}}$
we obtain:

$\delta_{p}=\absolutevalue{\sum_{k=1}^{m_{l-1}}w^{(l)}_{kj}x^{(l-1)}_{k}+b^{(l)}_{j}-\sum_{k=1}^{p}w^{(l)}_{kj}x^{(l-1)}_{k}}=\absolutevalue{\sum_{k=p+1}^{m_{l-1}}w^{(l)}_{kj}x^{(l-1)}_{k}+b^{(l)}_{j}}\leq\\\
\absolutevalue{b^{(l)}_{j}}+\sum_{k=p+1}^{m_{l-1}}\absolutevalue{w^{(l-1)}_{kj}x^{(l-1)}_{k}}.$

By averaging over all samples:

$\overline{\delta_{p}}\leq\absolutevalue{b^{(l)}_{j}}+\sum_{k=p+1}^{m_{l-1}}\overline{\absolutevalue{w^{(l)}_{kj}x^{(l-1)}_{k}}}=S^{(l)}_{j}\cdot\Bigg{(}\frac{\absolutevalue{b^{(l)}_{j}}}{S^{(l)}_{j}}+\frac{\sum_{k=p+1}^{m_{l-1}}\overline{\absolutevalue{w^{(l)}_{kj}x^{(l-1)}_{k}}}}{S^{(l)}_{j}}\Bigg{)}=\\\
S^{(l)}_{j}\cdot\sum_{k=p+1}^{m_{l-1}}s^{(l)}_{kj}=S^{(l)}_{j}(1-\alpha).$ (4)

Therefore, from (4) we see that the sequence of residuals
$\\{\overline{\delta_{p}}\\}_{p=1}^{m}$ decreases with respect to $p$ and
increases with $\alpha$. Also, for any Lipschitz continuous activation
function $\varphi\in\text{Lip}(X),\ X\subset\mathbb{R}$ with Lipschitz
constant $C$ we obtain:

$\Delta_{p}=\absolutevalue{\varphi\Big{(}\sum_{k=1}^{m_{l-1}}w^{(l)}_{kj}x^{(l-1)}_{k}+b^{(l)}_{j}\Big{)}-\varphi\Big{(}\sum_{k=1}^{p}w^{(l)}_{kj}x^{(l-1)}_{k}\Big{)}}\leq
C\absolutevalue{\sum_{k=p+1}^{m_{l-1}}w^{(l)}_{kj}x^{(l-1)}_{k}+b^{(l)}_{j}}=C\delta_{p}.$
(5)

As a result, from (4) and (5):

$\overline{\Delta}_{p}\leq CS^{(l)}_{j}(1-\alpha).$ (6)

Since ReLU, ELU, sigmoid and, tanh are Lipschitz continuous functions then (6)
holds for them, meaning that the sequence of residuals
$\\{\overline{\Delta}_{p}\\}_{p=1}^{m_{l}}$ decreases as well with the
increase of $\alpha$. Moreover, in the case of ReLU and ELU, Lipschitz
constant $C=1$. All this means that by increasing the $\alpha$ we indeed
reduce the approximation error of the neuron.

Analogously, for input $m_{l-1}$-channeled pruning set
$\mathbf{X}^{(l-1)}=\\{\mathbf{x}^{(l-1)}_{1},\ldots,\mathbf{x}^{(l-1)}_{N}\\}$,
where
$\mathbf{x}^{(l-1)}_{k}=(x^{(l-1)}_{k1},\ldots,x^{(l-1)}_{km})\in\mathbb{R}^{m_{l-1}\times
h^{1}_{l-1}\times h^{2}_{l-1}}$, where $h^{1}_{l-1}$ and $h^{2}_{l-1}$ are
height and width of input images (or feature maps) for convolutional layer
$l-1$, and $\mathbf{B}^{(l)}_{j}\in\mathbb{R}^{h^{1}_{l}\times h^{2}_{l}}$ is
a matrix that consists of the same bias value $b^{(l)}_{j}$ at every element
we obtain:

$\begin{split}\delta_{p}=\norm{\sum_{i=1}^{m_{l-1}}\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}+\mathbf{B}^{(l)}_{j}-\sum_{i=1}^{p}\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}}_{F}=\norm{\sum_{i=p+1}^{m_{l-1}}\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}+\mathbf{B}^{(l)}_{j}}_{F}\leq\\\
\sum_{i=p+1}^{m_{l-1}}\norm{\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}}_{F}+\norm{\mathbf{B}^{(l)}_{j}}_{F}\leq\sum_{i=p+1}^{m_{l-1}}\norm{\mathbf{K}^{(l)}_{i}*\absolutevalue{x^{(l-1)}_{ki}}}_{F}+\sqrt{h^{1}_{l}h^{2}_{l}}\absolutevalue{b^{(l)}_{j}}\end{split}$
(7)

Then, by averaging over all pruning sample similarly to fully connected case
we obtain:

$\overline{\delta}_{p}\leq\sum_{i=p+1}^{m_{l-1}}\frac{1}{N}\sum_{k=1}^{N}\norm{\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}}_{F}=S^{(l)}_{j}(1-\alpha),$
(8)

from which for any Lipschitz continuous activation functions $\varphi$ with
Lipschitz constant $C$, we obtain:

$\overline{\Delta}_{p}=\frac{1}{N}\sum_{k=1}^{N}\norm{\varphi\Big{(}\sum_{i=1}^{m_{l-1}}\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}+\mathbf{B}^{(l)}_{j}\Big{)}-\varphi\Big{(}\sum_{i=1}^{p}\mathbf{K}^{(l)}_{i}*x^{(l-1)}_{ki}\Big{)}}_{F}\leq
CS^{(l)}_{j}(1-\alpha).$ (9)

Inequalities (6), (9) show how the signal in a neuron or filter, respectively,
changes after the layer is pruned. Let us consider how this change influences
the network output. We define
$N^{(l)}_{i_{l}}\big{(}x;\mathbf{W}^{(1:l)}\big{)}$ as a value in the
$i_{l}^{\text{th}}$ neuron in layer $l$, where
$\mathbf{W}^{(1:l)}=(\mathbf{W}^{(1)},\ldots,\mathbf{W}^{(l)})$ are parameters
in layers $1,\ldots,l$. The matrix that we obtain after pruning layer $l$ we
define as $\mathbf{\hat{W}}^{(l)}$, and $m_{l}$ is the number of neurons in
layer $l,\ 1\leq l\leq L$. Then, for $1\leq l<L$ (for $l=L$ see (4) in Section
5.2):

$\varepsilon=\absolutevalue{N^{(L)}_{i_{L}}\big{(}x;\mathbf{W}^{(1:L)}\big{)}-N^{(L)}_{i_{L}}\big{(}x;\mathbf{W}^{(1:l-1)},\mathbf{\hat{W}}^{(l)},\mathbf{W}^{(l+1:L)}\big{)}}=\\\
\absolutevalue{\sum_{i_{L-1}=1}^{m_{L-1}}w^{(L)}_{i_{L-1}i_{L}}\Big{(}\varphi\big{(}N^{(L-1)}_{i_{L-1}}\big{(}x;\mathbf{W}^{(1:L-1)}\big{)}-\varphi(N^{(L-1)}_{i_{L-1}}\big{(}x;\mathbf{W}^{(1:l-1)},\mathbf{\hat{W}}^{(l)},\mathbf{W}^{(l+1:L-1)}\big{)}\big{)}\Big{)}}\leq\\\
C\sum_{i_{L-1}=1}^{m_{L-1}}\absolutevalue{w^{(L)}_{i_{L-1}i_{L}}}\absolutevalue{N^{(L-1)}_{i_{L-1}}\big{(}x;\mathbf{W}^{(1:L-1)}\big{)}-N^{(L-1)}_{i_{L-1}}\big{(}x;\mathbf{W}^{(1:l-1)},\mathbf{\hat{W}}^{(l)},\mathbf{W}^{(l+1:L-1)}\big{)}}$
(10)

Performing similar transformations as in (10) until layer $l$, we obtain:

$\varepsilon\leq
C^{L-l}\sum_{i_{L-1}=1}^{m_{L-1}}\absolutevalue{w^{(L)}_{i_{L-1}i_{L}}}\sum_{i_{L-2}=1}^{m_{L-2}}\absolutevalue{w^{(L-1)}_{i_{L-2}i_{L-1}}}\ldots\\\
\sum_{i_{l}=1}^{m_{l}}\absolutevalue{w^{(l)}_{i_{l}i_{l+1}}}\absolutevalue{\varphi\big{(}N^{(l)}_{i_{i_{l}}}\big{(}x;\mathbf{W}^{(1:l)}\big{)}\big{)}-\varphi\big{(}N^{(l)}_{i_{l}}\big{(}x;\mathbf{W}^{(1:l-1)},\mathbf{\hat{W}}^{(l)}\big{)}\big{)}}$
(11)

Figure 11: ResNet-20 after pruning its fully connected layer
($\alpha_{\text{fc}}=0.95$)

Then, averaging (11) over all pruning samples from $\mathbf{X}$ and using (6):

$\begin{split}\overline{\varepsilon}\leq
C^{L-l}\sum_{i_{L-1}=1}^{m_{L-1}}\absolutevalue{w^{(L)}_{i_{L-1}i_{L}}}\sum_{i_{L-2}=1}^{m_{L-2}}\absolutevalue{w^{(L-1)}_{i_{L-2}i_{L-1}}}\ldots\sum_{i_{l}=1}^{m_{l}}\absolutevalue{w^{(l)}_{i_{l}i_{l+1}}}\overline{\Delta}^{(l)}_{p_{i_{l}}}\leq\\\
(1-\alpha)C^{L-l}\sum_{i_{L-1}=1}^{m_{L-1}}\absolutevalue{w^{(L)}_{i_{L-1}i_{L}}}\sum_{i_{L-2}=1}^{m_{L-2}}\absolutevalue{w^{(L-1)}_{i_{L-2}i_{L-1}}}\ldots\sum_{i_{l}=1}^{m_{l+1}}\absolutevalue{w^{(l)}_{i_{l}i_{l+1}}}S^{(l)}_{i_{l}}.\end{split}$
(12)

Inequality (12) shows the bounds of the network error change after pruning on
the pruning set $\mathbf{X}$. Figure 11 shows theoretical bounds for output
signal found by (6) and experimentally for ResNet-20, where only the fully
connected layer is pruned on CIFAR-10. We computed it for 10 random data
points out of each of the classes taken from the test set.

##  6 Conclusion

We propose a new pruning strategy and associated error estimation that is
applicable to different neural network architectures. Our algorithm is based
on eliminating layers’ kernels or connections that do not contribute to the
signal in the next layer. In this method, we estimate the average signal
strength coming through each connection via proposed importance scores and try
to keep the signal level in the neurons close to the original level. Thus, the
number of removed parameters in a layer is selected adaptively, rather than
being predetermined. As a result, the algorithm obtains a subnetwork capable
of propagating most of the original signal while using fewer parameters than
other reported strategies. We explore the effect of Adam and SGD optimizers on
our pruning strategy. The results show that for VGG architecture Adam
optimizer gives much higher compression than SGD, however, for ResNet we do
not observe similar behaviour. We observe that the obtained subnetwork tends
to homogenize connection importance, hinting that every remaining connection
is approaching similar importance for the dataset on average.

## References

  * [1] Ehud D Karnin.  A simple procedure for pruning back-propagation trained neural networks.  IEEE transactions on neural networks, 1(2):239–242, 1990.
  * [2] Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In Advances in neural information processing systems, pages 598–605, 1990.
  * [3] Babak Hassibi and David G Stork.  Second order derivatives for network pruning: Optimal brain surgeon.  In Advances in neural information processing systems, pages 164–171, 1993.
  * [4] Song Han, Jeff Pool, John Tran, and William Dally.  Learning both weights and connections for efficient neural network.  In Advances in neural information processing systems, pages 1135–1143, 2015.
  * [5] Hengyuan Hu, Rui Peng, Yu-Wing Tai, and Chi-Keung Tang.  Network trimming: A data-driven neuron pruning approach towards efficient deep architectures.  arXiv preprint arXiv:1607.03250, 2016.
  * [6] Marco Ancona, Cengiz Öztireli, and Markus Gross.  Shapley value as principled metric for structured network pruning.  arXiv preprint arXiv:2006.01795, 2020.
  * [7] Karen Ullrich, Edward Meeds, and Max Welling.  Soft weight-sharing for neural network compression.  arXiv preprint arXiv:1702.04008, 2017.
  * [8] Dmitry Molchanov, Arsenii Ashukha, and Dmitry Vetrov.  Variational dropout sparsifies deep neural networks.  In International Conference on Machine Learning, pages 2498–2507. PMLR, 2017.
  * [9] Chenglong Zhao, Bingbing Ni, Jian Zhang, Qiwei Zhao, Wenjun Zhang, and Qi Tian.  Variational convolutional neural network pruning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2780–2789, 2019.
  * [10] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In International Conference on Learning Representations (ICLR), 2019\.
  * [11] Xin Dong, Shangyu Chen, and Sinno Pan.  Learning to prune deep neural networks via layer-wise optimal brain surgeon.  In Advances in Neural Information Processing Systems, pages 4857–4867, 2017.
  * [12] Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  In International Conference on Learning Representations (ICLR), 2016\.
  * [13] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz.  Importance estimation for neural network pruning.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11264–11272, 2019.
  * [14] Christos Louizos, Karen Ullrich, and Max Welling.  Bayesian compression for deep learning.  In Advances in neural information processing systems, pages 3288–3298, 2017.
  * [15] Yihui He, Xiangyu Zhang, and Jian Sun.  Channel pruning for accelerating very deep neural networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1389–1397, 2017.
  * [16] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  In International Joint Conference on Artificial Intelligence (IJCAI), pages 2234–2240, 2018.
  * [17] Alex Renda, Jonathan Frankle, and Michael Carbin.  Comparing rewinding and fine-tuning in neural network pruning.  arXiv preprint arXiv:2003.02389, 2020.
  * [18] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In International Conference on Learning Representations (ICLR), 2015\.
  * [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [20] Vadim Lebedev and Victor Lempitsky.  Fast convnets using group-wise brain damage.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2554–2564, 2016.
  * [21] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf.  Pruning filters for efficient convnets.  In International Conference of Learning Representation (ICLR), 2017\.
  * [22] Jian-Hao Luo, Jianxin Wu, and Weiyao Lin.  Thinet: A filter level pruning method for deep neural network compression.  In Proceedings of the IEEE international conference on computer vision, pages 5058–5066, 2017.
  * [23] Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg.  Net-trim: Convex pruning of deep neural networks with performance guarantee.  Advances in Neural Information Processing Systems, 30:3177–3186, 2017.
  * [24] Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis.  Nisp: Pruning networks using neuron importance score propagation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9194–9203, 2018.
  * [25] Yash Garg and K Selçuk Candan.  isparse: Output informed sparsification of neural network.  In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 180–188, 2020.
  * [26] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr.  Snip: Single-shot network pruning based on connection sensitivity.  In International Conference on Learning Representations (ICLR), 2019\.
  * [27] Zehao Huang and Naiyan Wang.  Data-driven sparse structure selection for deep neural networks.  In Proceedings of the European conference on computer vision (ECCV), pages 304–320, 2018.
  * [28] Jianbo Ye, Xin Lu, Zhe Lin, and James Z Wang.  Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers.  arXiv preprint arXiv:1802.00124, 2018.
  * [29] Amir Beck and Marc Teboulle.  A fast iterative shrinkage-thresholding algorithm for linear inverse problems.  SIAM journal on imaging sciences, 2(1):183–202, 2009.
  * [30] Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao.  Hrank: Filter pruning using high-rank feature map.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1529–1538, 2020.
  * [31] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324, 1998.
  * [32] Sergey Zagoruyko.  92.45 on cifar-10 in torch, 2015.  URL http://torch. ch/blog/2015/07/30/cifar. html.
  * [33] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [34] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015.
  * [35] Yiwen Guo, Anbang Yao, and Yurong Chen.  Dynamic network surgery for efficient dnns.  In Advances in neural information processing systems, pages 1379–1387, 2016.
  * [36] Dushyant Mehta, Kwang In Kim, and Christian Theobalt.  On implicit filter level sparsity in convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 520–528, 2019.
  * [37] Tuanhui Li, Baoyuan Wu, Yujiu Yang, Yanbo Fan, Yong Zhang, and Wei Liu.  Compressing convolutional neural networks via factorized convolutional filters.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3977–3986, 2019.
  * [38] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  arXiv preprint arXiv:1912.01703, 2019.
  * [39] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz.  Pruning convolutional neural networks for resource efficient inference.  arXiv preprint arXiv:1611.06440, 2016.

##  Appendix A Appendix

###  A.1 Pruning setups

We implement our approach with PyTorch [38]. Table 6 shows the training
parameters for each network.

Network | dataset | optimizer |  learning rate (by epoch) |  weight decay
---|---|---|---|---
LeNet-5/300-100 | MNIST | Adam | $\begin{cases}10^{-3},&1\leq epoch\leq 30\\\ 10^{-4},&31\leq epoch\leq 60\end{cases}$ | $5\cdot 10^{-4}$
VGG-like/13 |  CIFAR-10/100 Tiny-ImageNet | SGD/Adam | $\begin{cases}10^{-1}/10^{-3},&1\leq epoch\leq 80\\\ 10^{-2}/10^{-4},&81\leq epoch\leq 120\\\ 10^{-3}/10^{-5},&121\leq epoch\leq 150\end{cases}$ | $5\cdot 10^{-4}$
ResNet-20/56 | CIFAR-10/100 | SGD | $\begin{cases}0.1,&1\leq epoch\leq 80\\\ 0.01,&81\leq epoch\leq 120\\\ 0.001,&121\leq epoch\leq 150\end{cases}$ | $5\cdot 10^{-4}$
Adam | $\begin{cases}10^{-3},&1\leq epoch\leq 120\\\ 10^{-4},&121\leq epoch\leq 160\\\ 10^{-5},&161\leq epoch\leq 200\end{cases}$ | $5\cdot 10^{-4}$
Table 6: Training setups

Table 7 shows the parameters for retraining on every iteration.

Network | dataset | optimizer |  learning rate (by epoch) | weight decay
---|---|---|---|---
LeNet-5/300-100 | MNIST | Adam | $\begin{cases}10^{-3},&1\leq epoch\leq 30\\\ 10^{-4},&31\leq epoch\leq 60\end{cases}$ | $5\cdot 10^{-4}$
VGG-like/13 |  CIFAR-10/100 Tiny-ImageNet | SGD/Adam | $\begin{cases}10^{-1}/10^{-3},&1\leq epoch\leq 20\\\ 10^{-2}/10^{-4},&21\leq epoch\leq 40\\\ 10^{-3}/10^{-5},&41\leq epoch\leq 60\end{cases}$ | $5\cdot 10^{-4}$
ResNet-20/56 | CIFAR-10 | SGD/Adam | $\begin{cases}10^{-1}/10^{-3},&1\leq epoch\leq 20\\\ 10^{-2}/10^{-4},&21\leq epoch\leq 40\\\ 10^{-3}/10^{-5},&41\leq epoch\leq 60\end{cases}$ | $5\cdot 10^{-4}$
CIFAR-100 | SGD/Adam | $\begin{cases}10^{-1}/10^{-3},&1\leq epoch\leq 30\\\ 10^{-2}/10^{-4},&21\leq epoch\leq 60\\\ 10^{-3}/10^{-5},&41\leq epoch\leq 80\end{cases}$ | $5\cdot 10^{-4}$
Table 7: Retraining parameters

Table 8 shows the pruning parameters and total number of iteration.

Network | dataset |  $(\alpha_{\text{conv}}$, $\alpha_{\text{fc}})$ | number of iterations |  best iteration SGD/Adam
---|---|---|---|---
LeNet-300-100 | MNIST | (-, 0.95) | 15 | 11
LeNet-5 | MNIST | (0.9,0.95) | 20 | 20
VGG-like | CIFAR-10 | (0.95,0.95) | 6 | 6
Tiny-ImageNet | (0.95,0.95) | 5 | 5
ResNet-20 (SGD/Adam) | CIFAR-10 | (0.95,0.99) | 10 | 7/10
CIFAR-100 | (0.95,0.99) | 10 | 7/8
ResNet-56 (SGD/Adam) | CIFAR-10 | (0.95,0.99) | 10 | 6/2
CIFAR-100 | (0.95,0.99) | 10 | 6/9
Table 8: Pruning parameters

###  A.2 FLOPs computation

According to [39], we compute FLOPs as follows:

  * •

for a convolutional layer: FLOPs $=2HW(C_{in}K^{2}+1)C_{out}$ where $H,W$ and
$C_{in}$ are height, width and number of channels of the input feature map,
$K$ is the kernel width (and height due to symmetry), and $C_{out}$ is the
number of output channels.

  * •

for a fully connected layer: FLOPs $=(2I-1)O$, where $I$ is the input
dimensionality and O is the output dimensionality.

###  A.3 Runtime

We perform our computations on a single Quadro P-5000 GPU. In Table 9 we
present approximate time to do a prunining step and retraining step for one
iteration.

Network | dataset | pruning step | retraining step
---|---|---|---
LeNet-5/300-100 | MNIST |  $\sim$3 s |  $\sim$6.5 min
VGG-like | CIFAR-10 |  $\sim$90 s |  $\sim$60 min
ResNet-20 | CIFAR-10 |  $\sim$40 s |  $\sim$40 min
CIFAR-100 |  $\sim$45 s |  $\sim$ 60 min
ResNet-56 | CIFAR-10 |  $\sim$120 s |  $\sim$60 min
CIFAR-100 |  $\sim$120 s |  $\sim$80 min
Table 9: Approximate runtime for one iteration.


# PPGN: PHRASE-GUIDED PROPOSAL GENERATION NETWORK FOR REFERRING EXPRESSION
COMPREHENSION

###### Abstract

Reference expression comprehension (REC) aims to find the location that the
phrase refer to in a given image. Proposal generation and proposal
representation are two effective techniques in many two-stage REC methods.
However, most of the existing works only focus on proposal representation and
neglect the importance of proposal generation. As a result, the low-quality
proposals generated by these methods become the performance bottleneck in REC
tasks. In this paper, we reconsider the problem of proposal generation, and
propose a novel phrase-guided proposal generation network (PPGN). The main
implementation principle of PPGN is refining visual features with text and
generate proposals through regression. Experiments show that our method is
effective and achieve SOTA performance in benchmark datasets.

Index Terms—  Referring expression comprehension, Proposal generation methods,
Multimodel representation learning

##  1 Introduction

Reference expression comprehension (REC) acts as one of core tasks in human-
machine interaction. REC can be typically formulated as locating the entity
involved in an expression (e.g.,  a little girl wearing a pink shirt and
holding a red umbrella) through a bounding box. REC is challenging because it
requires not only to understand the fine-grained semantic information of image
and natural language, but also to align and associate them for locating the
true region.

Fig. 1: The generated region proposals with different proposal generation
methods on COCO. (a) Selective Search (SS); (b) Faster-RCNN pre-trained on
COCO; (c) Our phrase-guided proposal generation network.The red boxes
represent the proposals, and the green boxes represent the ground truth.

In recent years, mainstream REC works can be categorized into two-stage
methods[1, 2, 3] and one-stage methods[4, 5, 6]. Two-stage REC methods were
firstly proposed, which can be formulated as follows: Given an input image, a
proposal generator is adopted to generate a certain number of region proposals
and then the visual features of each proposal are extracted. Simultaneously, a
language model (such as Bert[7]) encodes the corresponding referring phrase
into language features. The visual features and language features then are fed
into the multi-modal fusion module that aims to generate integrated features.
After that, the proposal ranking module is utilized to generate the proposal’s
location with the highest ranking score by using integrated features. Many
effective two-stage REC methods have been proposed in recent years, for
example, MattNet[8] parses the phrase into parts of subject, location, and
relationship, and links each part with the related object regions for matching
score calculation. NMTREE[9] parses the phrase via a dependency tree parser
and links each tree node with a visual region. DGA[10] parses the phrase with
text self-attention and uses dynamic graph attention to link the text with
regions. Nevertheless, recent proposed two-stage REC methods usually focus on
the proposal representation and ranking, especially on how to extract more
robust visual and text features. In contrast, the problem of proposal
generation is rarely explored. Therefore, the resulted proposals are often of
low quality that limits the model performance. Thus, in order to avoid the
proposal generation stage, the one-stage methods[4, 5, 6] have recently been
proposed. They fuse visual-text features at the image level and directly
predict the boundary box to locate the object they refer to, which
significantly improves the model performance and soon becomes prevailing.
However, when facing with some complicated scenarios, such as content-rich
images or complex semantic expressions, the one-stage methods are sometimes
inferior to the two-stage methods[6].

Considering the above problems, it is desirable to pay more attention to
proposal generation in two-stage methods. Most existing proposal generation
methods can be categorized into non-training methods (e.g., Selective
Search[11], Edgebox[12]) and class-aware object detector (e.g., Faster-
RCNN[13] trained on COCO with $80$ classes). Non-training methods often
generate region proposals with some features of the image itself, such as edge
information[12]. As such, proposals generated by these methods have poor
discriminability, hence they are difficult to hit the ground truth in a
limited number (See Fig. 1(a)). Class-aware object detector can generate
proposals discriminatively and accurately only if the referring object
pertaining to preset categories[13]. However, the vocabulary of referring
phrase is usually open, and if the referring object is not in the preset
categories, such as “window” is not in the preset categories of COCO, the
correct region proposal cannot be generated (See Fig. 1(b)). Moreover, both
kinds of methods have a fatal flaw: they generate proposals based only on the
image information without considering the referring phrase, which may lead to
numerous redundant proposals unrelated to the phrase.

To address the above problems, we propose a novel phrase-guided proposal
generation network (PPGN), which is an end-to-end deep regression network.
More specifically, PPGN utilizes visual features refined by text features to
predict the proposals through regression. In order to make the proposed
generation not limited by the preset categories, PPGN is only pretrained on
the REC training set with two loss functions that control anchor box offset
and confidence respectively. Compared to existing proposal generators, PPGN
bears the merits of class-agnostic and high discriminability. Moreover, due to
the consideration of phrase information, the generated proposals will change
accordingly even for the same image as long as the referring phrase is
different (See Fig. 1(c)).

The main contributions of this work are as follows. 1) We propose a novel
proposal generation paradigm of REC task, in which the referring phrase
directly participates in the proposal generation. 2) We propose a novel
phrase-guided proposal generator (PPGN) to generate high-quality proposals. 3)
We evaluate our model on the benchmark datasets, and our experimental results
show that our model achieves significant performance improvements in the test
dataset.

Fig. 2: Overall structure of our model. The part of stage 1 is our phrase-
guided proposal generation network model, and the part of stage 2 is proposal
representation and ranking module. These two parts are combined to make up a
complete REC model.

##  2 METHOD

In this section, we will introduce our phrase-guided proposal generation
network (PPGN), which is an end-to-end deep regression network. Firstly, we
apply feature pyramid network to extract visual feature $V_{k}(k=1,2,3)$ with
three different resolutions of the image, and utilize language model to
extract text feature $Q$ of referring phrase. In order to involve the phrase
information in proposal generation, all the visual features are refined with
text to obtain the text-conditional visual features $V^{t}_{k}$ . Then, the
model is trained with two loss functions that control anchor box offset and
confidence respectively. Finally, the model uses $V^{t}_{k}$ to predict
proposals by regression.

###  2.1 Text-conditional Visual Embedding

In order to obtain the visual features involving phrase information, we adopt
the conditional normalization method in image-level tasks [14] to refine three
different resolution visual features with the text feature. First, text
feature $Q$ is mapped into a scaling vector $p$ and a shifting vector $q$ by
two MLPs:

$p=tan(W_{p}Q+b_{p}),q=tan(W_{q}Q+b_{q}),$ (1)

where $W_{p}$, $b_{p}$, $W_{q}$, $b_{q}$ are learnable parameter. Then, we
obtain the text-conditional visual feature $V^{t}_{k}$ via Equation (1) :

$V_{k}^{t}(i,j)=f_{2}\left\\{ReLU[f_{1}(V_{k}(i,j)\odot
p+q)+V_{k}(i,j)\right\\},$ (2)

where $\odot$ denotes hadamard product, $(i,j)$ is the spatial coordinate of
visual features, $f_{1}$ and $f_{2}$ are learnable mapping layers as shown in
Fig. 2. $f_{1}$ is composed of $1\times 1$ convolution following an instance
normalization layer, $f_{2}$ is composed of a $3\times 3$ convolution
following a batch normalization layer and ReLU activation function. Each
coordinate $(i,j)$ of the visual feature goes through the same operation.

###  2.2 Grounding Module

As shown in the stage 1 of Fig. 2, grounding module takes the text-conditional
visual features $V_{k}(k=1,2,3)$as input and outputs multiple bounding boxes
as the proposals. There are $8\times 8+16\times 16+32\times 32=1344$ different
locations, and each location corresponds to a vector of a 512-dimensional
text-conditional visual feature. Follow YOLOV3[15], in each location, PPGN
will set three anchor boxes, and the size of the anchor boxes is obtained by
K-means clustering based on the width and height of the ground truth in the
training set with (1-IOU) as the distance. Thus, we have a total of $N=3\times
1344=4032$ anchor boxes. For each anchor box, the prediction of PPGN is four
values though regression for shifting the width, height, and center of the
anchor box, together with the fifth value via a sigmoid function about the
confidence of the shifted box.

For the prediction of confidence scores, the problem is how to design the loss
function to make the predicted confidence scores $S$ consistent with the
ground truth confidence scores $S^{*}$. Most existing methods[6, 16] define
the confidence score
$S^{*}=[s_{1}^{*},\ldots,s_{N}^{*}]\in\left\\{0,1\right\\}^{N}$ , which is an
one-hot vector that has one only element is set to 1 when the anchor box has
the largest IOU with the ground truth box and 0 otherwise. Then they apply the
one-hot label to implement the cross entropy loss to train the model.

In this paper, we improve the aforementioned method. Specifically, We set a
threshold $\eta$, calculate the IOU scores between each anchor box and the
ground truth box, if it exceeds the threshold $\eta$, set the IOU score as the
label of confidence score, otherwise 0. By doing this, we get the smooth
confidence label $S^{*}=[s_{1}^{*},\ldots,s_{N}^{*}]\in\mathbb{R}^{N}$, and
then perform a L1 regularization to ensure that $\sum S^{*}=1$. In this way,
we apply the Kullback-Leibler Divergence (KLD) as our loss function to make
the smooth confidence label $S^{*}$ gradually close to the predicted
confidence scores $S$, as shown in Equation(3). Note that $S$ is also
performed L1 regularization to satisfy the same probability distribution.

$L_{conf}=\dfrac{1}{N}\sum_{n=1}^{N}s_{i}^{*}log(\dfrac{s_{i}^{*}}{s_{i}}).$
(3)

The benefits of smooth label are as follows: 1) Some anchors can also provide
useful context information besides the anchor with the largest IOU; 2) The
soft label actually regularizes the model and alleviates overfitting.

For the prediction of the the anchor box offset, We adopt the MSE loss
function as follows:

$\displaystyle L_{coord}=$
$\displaystyle\sum_{n=1}^{N}\mathbbm{1}^{IOU}_{n}\Big{[}(\sigma(t_{x})_{n}-\sigma(\hat{t_{x}})_{n})^{2}$
(4) $\displaystyle+(\sigma(t_{y})_{n}-\sigma(\hat{t_{y}})_{n})^{2}$
$\displaystyle+((t_{w})_{n}-(\hat{t_{w}})_{n})^{2}$
$\displaystyle+((t_{h})_{n}-(\hat{t_{h}})_{n})^{2}\Big{]},$

where $\mathbbm{1}^{IOU}_{n}$denotes if the IOU between $n$-th anchor box and
ground truth exceeds the threshold $\eta$, $\sigma$ denotes sigmoid function.
Note that the loss function only penalizes coordinate error if the anchor box
will be selected for a proposal.

The overall loss of PPGN is defined as:

$L=L_{conf}+\gamma L_{coord},$ (5)

where $\gamma$ is a trade-off parameter.

During testing, we first set $K$ to the number of proposals, and then output
the shifted anchor boxes in turn according to the confidence of anchor from
high to low as proposals. In order to avoid the interference of exception
values, we remove the boxes with extremely small height or width. Noted that
although our method is similar to recently one-stage REC methods[4, 5, 6, 16],
which also adopt deep regression network, we have a different motivation. One-
stage REC methods directly output the only region with the highest confidence
as the final result, while our model output multiple regions as proposals.

Table 1: Performance Comparison of different REC methods on RefCOCO, RefCOCO+
and RefCOCOg (acc@0.5$\%$). Methods marked with (*) are one-stage methods.

Method | Proposal Method | Features | RefCOCO | RefCOCO+ | RefCOCOg
---|---|---|---|---|---
val | testA | testB | val | testA | testB | val | test
CMN[1] | FRCN Detc. | VGG16-COCO | - | 71.03 | 65.77 | - | 54.32 | 47.76 | - | -
ParallelAttn[17] | FRCN Detc. | VGG16-ImageNet | - | 75.31 | 65.52 | - | 61.34 | 50.86 | - | -
VC[18] | SSD Detec. | VGG16-COCO | - | 73.33 | 67.44 | - | 58.40 | 53.18 | - | -
LGRAN[3] | FRCN Detc. | VGG16-ImageNet | - | 76.60 | 66.40 | - | 64.00 | 53.40 | - | -
SLR[19] | SSD Detec. | Res101-COCO | 69.48 | 73.71 | 64.96 | 55.71 | 60.74 | 48.80 | 60.21 | 59.63
MattNet[8] | FRCN Detc. | Res101-COCO | 76.40 | 80.43 | 69.28 | 64.93 | 70.26 | 56.00 | 66.67 | 67.01
DGA[10] | FRCN Detc. | Res101-COCO | - | 78.42 | 65.53 | - | 69.07 | 51.99 | - | 63.28
SSG* [4] | - | Darknet53-COCO | - | 72.51 | 67.50 | - | 62.14 | 49.27 | 58.80 | -
FAOA* [6] | - | Darknet53-COCO | 72.05 | 74.81 | 67.59 | 55.72 | 60.37 | 48.54 | 59.03 | 58.70
Imp-FAOA*[20] | - | Darknet53-COCO | 77.63 | 80.45 | 72.30 | 63.59 | 68.36 | 56.81 | 67.30 | 67.20
Ours-SoftMAX | Phrase-guided | Darknet53-COCO | 77.14 | 80.16 | 72.37 | 64.01 | 69.73 | 57.12 | 66.61 | 66.29
Ours-KLD | Phrase-guided | Darknet53-COCO | 77.98 | 81.35 | 73.02 | 64.80 | 70.42 | 57.76 | 67.14 | 67.31

Table 2: Performance Comparison of different REC methods on Refrit
(acc@0.5$\%$). Methods marked with (*) are one-stage methods.

Method | Proposal Method | Visual Features | Acc
---|---|---|---
VC[18] | SSD Detec. | VGG16-COCO | 31.13
CITE-Resnet[21] | Edgebox N=200 | Res101-COCO | 35.07
Similarity Net[22] | Edgebox N=200 | Res101-COCO | 34.54
MattNet[8] | FRCN Detc. | Res101-COCO | 29.04
SSG* [4] | - | Darknet53-COCO | 54.24
ZSGNet*[5] | - | Res50-FPN | 58.63
FAOA*[6] | - | Darknet53-COCO | 59.30
Imp-FAOA* [20] | - | Darknet53-COCO | 64.60
Ours-SoftMAX | Pharse-guided | Darknet53-COCO | 63.89
Ours-KLD | Pharse-guided | Darknet53-COCO | 65.65

###  2.3 Framework Details

Visual and text feature encoder. PPGN is an end-to-end network, inputting an
image and its corresponding referring phrase, and then outputs a series of
image areas as proposals. For the image, we first resize the original image to
$256\times 256$, and then utilize the Darknet53[15] with feature pyramid
networks, which is pre-trained on the COCO object detection dataset, to
extract visual features. The extracted features have three spatial
resolutions, which are $8\times 8\times D_{1}$, $16\times 16\times D_{2}$, and
$32\times 32\times D_{3}$. $D_{1}=1024$, $D_{2}=512$, $D_{3}=256$ are the
number of feature channels under the corresponding resolution. Finally, we add
a 1×1 convolution layer with batch normalization and RELU to map them to the
uniform dimension $D=512$. For the referring phrase, we embed it to a vector
of 768 dimensions via the uncased version of Bert[7], and then make it through
two full connection layers of 512 neurons to get a $512D$ text feature. On
account of spatial feature encoding will be better achieved in proposal
representation and ranking module, we do not repeat this operation in PPGN.

Proposal representation and ranking module. To fully implement an REC model,
we follow MattNet[8], using a modular network to realize the proposal feature
representation and ranking. As shown in the stage 2 of Fig. 2, MAttNet applys
three modular components related to the appearance, location, and relationship
of an object to other objects. The subject module deals with attributes such
as categories, colors, and so on. The location module deals with absolute and
relative locations, and the relationship module deals with subject-object
relationships. Each module has a different structure, learning parameters in
its own module space without affecting each other. Instead of using an
external language parsers, this module learns to parse phrases automatically
by a soft attention mechanism. The matching scores of the three modules are
calculated to measure the compatibility between the object and the referring
phrase. More details can be found in [8].

##  3 EXPERIMENTS

###  3.1 Implementation details

Training setting. When we resize an input image, we keep the original image
ratio and resize its long edge to $256$. We then pad the mean pixel value of
the image along the short edge. We adopt the RMSProp optimization method to
train the model. We begin with a learning rate of $10^{-4}$ and adopt a
polynomial schedule with a power of 1. Since Darknet is pre-trained, we reduce
the main learning rate of the Darknet portion in the model by 10 times. We set
the IOU threshold $\eta$ as 0.7, trade-off parameter $\gamma$ of loss function
as 1. We choose K = 7 as the default number of generated proposals, and
relevant ablation experimental studies can be seen in the quantitative
results. The batch size is 32 in all of our experiments and we complete our
training on a 1080Ti GPU. The training setting of the proposal representation
and ranking module is the same as MAttNet[8].

Evaluation setting. We fellow the evaluation method in previous studies[8, 23,
4, 24], for a given referring phrase, if the IOU between the predict box and
the ground truth is not less than 0.5, the predict box is considered correct.

###  3.2 Dataset

We use $4$ classic REC datasets: Refrit[24], RefCOCO[25], RefCOCO+[25] and
RefCOCOg[26]. Refrit contains $20,000$ images from the SAIAPR-12[27], and we
apply a cleaned split version[23] with $9,000$, $1,000$, and $10,000$ images
in the train, validation, and test sets, respectively. RefCOCO has $50,000$
target entities collected from $19,994$ images. RefCOCO+ has $49,856$ target
entities collected from 19,992 images. These two datasets are split into four
parts of train, val, testA and testB. RefCOCOg includes $49,822$ target
entities from 25799 images, which are split into three parts of train, val and
test.

Table 3: Comparison of the performance of different proposal generators based
on the same proposal representation and ranking<EMAIL_ADDRESS>
Proposal method | Referit | RefCOCO | RefCOCO+
---|---|---|---
test | val | testA | testB | val | testA | testB
Edgebox[12] | 46.54 | 59.32 | 59.09 | 56.32 | 50.24 | 55.29 | 45.02
Selec. Search[11] | 56.45 | 56.38 | 55.63 | 54.34 | 51.01 | 53.14 | 42.67
FRCN Dectc.[13] | 29.04 | 76.40 | 80.43 | 69.28 | 64.93 | 70.26 | 56.00
Ours | 65.65 | 77.98 | 81.35 | 73.02 | 64.80 | 70.42 | 57.76

Table 4: Ablation studies on proposal number<EMAIL_ADDRESS>
Num(K) | 1 | 4 | 7 | 10 | 13 | 16
---|---|---|---|---|---|---
Acc | 57.25 | 64.57 | 65.65 | 64.78 | 63.09 | 62.22

###  3.3 Quantitative Results

Referring expression comprehension results. We perform a comparison of our
method with other SOTA methods[1, 17, 18, 3, 19, 8, 10, 4, 6, 20, 21, 22, 5].
Table. 1 and Table. 2 report the referring expression comprehension results on
COCO-series datasets and Referit dataset respectively. The results show that
our model outperforms the existing SOTA methods, both two-stage and one-stage.
Especially on Referit datasets, which is not a COCO-series dataset, our method
exceeds the existing two-stage methods by a large margin. Moreover, the tables
also report the performance of our model trained with different losses, which
shows that training with the KLD loss result in a 0.6$\sim$1.8-point
improvement over the models with conventional onehot-label softmax loss.

Proposal generator performance comparison. To further investigate the
performance of PPGN, we utilize the same proposal representation and ranking
method applied in MAttnet[8], changing only the proposal generator for a
comparative experiment. We compare three mainstream proposal generators,
namely Edgebox[12], Select Search[11] and FRCN object detector[13] trained on
COCO, and the final results are shown in Table. 3. We can see that our
proposed method is superior to other methods in all datasets. FRCN trained on
COCO also shows fine performance on the COCO-series datasets (i.e. RefCOCO,
RefCOCO+). For example, FRCN performs as well as PPGN in Val and testA of
RefCOCO+, though our model significantly outperforms FRCN on RefCOCO and testB
of RefCOCO+. However, the performance of FRCN drops dramatically in the
Referit dataset. Since RefCOCO/ RefCOCO+ are subsets of COCO and have shared
images and entities, the COCO-trained detector can generate almost perfect
region proposals on the COCO-series datasets. When we apply it on other
datasets, e.g., Referit datasets, their performance will drop significantly.
Nevertheless, PPGN works commendably on all datasets.

Ablation studies. We conducte ablation studies on different proposal
generation number $K$ on the Referit dataset, and the results are shown in
Table. 4. We observe that increasing the number of generated proposals no
longer resulted in improved accuracy after reaching a certain threshold (for
example, $K\geq 7$ at Referit). Therefore, in our experiment, we choose $K=7$
as the default value. According to our analysis, since our method generates
proposals according to the confidence of anchor from high to low, when $K=7$,
the hit ratio to ground truth is already pretty high. If $K$ increases
further, the hit ratio increase will not be enough to counteract the negative
impact of the proposal redundancy.

###  3.4 Visualization

We visualize the REC results on Referit in Fig. 3. It illustrates that PPGN
generates different proposals for the same picture with different referring
expressions, and the generated proposals have high discriminability and
accuracy. For example, in the first column, under the guidance of the phrase
3nd person black shirt, PPGN generates proposals about all the people in the
image, while under the guidance of the phrase glass being held by man in blue
shirt, the model generates proposals about all the glass-like objects in the
image.

##  4 CONCLUSION

In this paper, we introduce a new phrase-guided REC task proposal generation
paradigm in the first time, aiming to solve the problem of ignoring phrase
information during proposal generation. Then, a phrase-guided proposal
generation network (PPGN) is proposed to produce high-quality proposals. By
utilizing visual features refined by phrase information, PPGN has the
advantages of generating class-agnostic and high-discriminability proposals.
Meanwhile, we design two loss functions to control anchor box offset and
confidence respectively. PPGN alleviates the performance bottleneck caused by
low-quality proposals in the conventional two-stage REC methods. Extensive
experiments on four banchmark datasets show that our model outperforms other
SOTA methods on most evaluation indicators. Based on our work, more efficient
two-stage REC methods can be further explored.

Fig. 3: The example in Referit dataset.The ground-truth(green), the final top-
ranked predicted proposal(red), and the other proposals (blue) are visualized
respectively. For better viewing, we removed some highly overlapping bounding
boxes.We also show 3 examples of incorrect predictions(IoU $\leq$ 0.5).Best
viewed in color.

## References

  * [1] R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko,  “Modeling relationships in referential expressions with compositional modular networks,”  in CVPR, 2017.
  * [2] K. Chen, R. Kovvuri, J. Gao, and R. Nevatia,  “Msrc: Multimodal spatial regression with semantic context for phrase grounding,”  in ICMR, 2017.
  * [3] P. Wang, Q. Wu, J. Cao, C. Shen, L. Gao, and A. Hengel,  “Neighbourhood watch: Referring expression comprehension via language-guided graph attention networks,”  in CVPR, 2019.
  * [4] X. Chen, L. Ma, J. Chen, Z. Jie, W. Liu, and J. Luo,  “Real-time referring expression comprehension by single-stage grounding network,”  arXiv preprint arXiv:1812.03426, 2018.
  * [5] A. Sadhu, K. Chen, and R. Nevatia,  “Zero-shot grounding of objects from natural language queries,”  in ICCV, 2019.
  * [6] Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo,  “A fast and accurate one-stage approach to visual grounding,”  in ICCV, 2019.
  * [7] J. Devlin, M. Chang, K. Lee, and K. Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [8] L. Yu, Z. Lin, X. Shen, J. Yang, . Lu, M. Bansal, and T. Berg,  “Mattnet: Modular attention network for referring expression comprehension,”  in CVPR, 2018.
  * [9] D. Liu, H. Zhang, F. Wu, and Z. Zha,  “Learning to assemble neural module tree networks for visual grounding,”  in ICCV, 2019.
  * [10] S. Yang, G. Li, and Y. Yu,  “Dynamic graph attention for referring expression comprehension,”  in ICCV, 2019.
  * [11] J. Uijlings, K. Van De Sande, T. Gevers, and A. Smeulders,  “Selective search for object recognition,”  IJCV, 2013.
  * [12] C. Zitnick and P. Dollár,  “Edge boxes: Locating object proposals from edges,”  in ECCV, 2014.
  * [13] S. Ren, K. He, R. Girshick, and J. Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  TPAMI, 2016.
  * [14] H. de Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. Courville,  “Modulating early visual processing by language,”  in ANIPS, 2017.
  * [15] J. Redmon and A. Farhadi,  “Yolov3: An incremental improvement,”  arXiv preprint arXiv:1804.02767, 2018.
  * [16] X. Liu, Z. Wang, J. Shao, X. Wang, and H. Li,  “Improving referring expression grounding with cross-modal attention-guided erasing,”  in CVPR, 2019.
  * [17] B. Zhuang, Q. Wu, C. Shen, I. Reid, and A. Van Den Hengel,  “Parallel attention: A unified framework for visual object discovery through dialogs and queries,”  in CVPR, 2018.
  * [18] H. Zhang, Y. Niu, and S. Chang,  “Grounding referring expressions in images by variational context,”  in CVPR, 2018.
  * [19] L. Yu, H. Tan, M. Bansal, and T. Berg,  “A joint speaker-listener-reinforcer model for referring expressions,”  in CVPR, 2017.
  * [20] Z. Yang, T. Chen, L. Wang, and J. Luo,  “Improving one-stage visual grounding by recursive sub-query construction,”  ECCV, 2020.
  * [21] Bryan A Plummer, P. Kordas, M. Hadi Kiapour, S. Zheng, R. Piramuthu, and S. Lazebnik,  “Conditional image-text embedding networks,”  in ECCV, 2018.
  * [22] L. Wang, Y. Li, J. Huang, and S. Lazebnik,  “Learning two-branch neural networks for image-text matching tasks,”  TPAMI.
  * [23] K. Chen, R. Kovvuri, and R. Nevatia,  “Query-guided regression network with context policy for phrase grounding,”  in ICCV, 2017.
  * [24] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg,  “Referitgame: Referring to objects in photographs of natural scenes,”  in EMNLP, 2014.
  * [25] L. Yu, P. Poirson, S. Yang, A. Berg, and T. Berg,  “Modeling context in referring expressions,”  in ECCV.
  * [26] J. Mao, J. Huang, A. Toshev, O. Camburu, A. Yuille, and K. Murphy,  “Generation and comprehension of unambiguous object descriptions,”  in CVPR, 2016.
  * [27] H. Escalante, C. Hernández, J. Gonzalez, A. López-López, M. Montes, E. Morales, L. Sucar, L. Villaseñor, and M. Grubinger,  “The segmented and annotated iapr tc-12 benchmark,”  Computer vision and image understanding, 2010.