#  On The Energy Statistics of Feature Maps in Pruning of Neural Networks with
Skip-Connections††thanks: This work has been supported in part by the Army
Research Office grant No. W911NF-15-1-0479.

Mohammadreza Soltani∗ Suya Wu∗ Yuerong Li∗
Jie Ding† and Vahid Tarokh∗
∗Department of Electrical and Computer Engineering, † School of Statistics
∗Duke University, †University of Minnesota

###### Abstract

We propose a new structured pruning framework for compressing Deep Neural
Networks (DNNs) with skip-connections, based on measuring the statistical
dependency of hidden layers and predicted outputs. The dependence measure
defined by the energy statistics of hidden layers serves as a model-free
measure of information between the feature maps and the output of the network.
The estimated dependence measure is subsequently used to prune a collection of
redundant and uninformative layers. Model-freeness of our measure guarantees
that no parametric assumptions on the feature maps distribution are required,
making it computationally appealing for very high dimensional feature space in
DNNs. Extensive numerical experiments on various architectures show the
efficacy of the proposed pruning approach with competitive performance to
state-of-the-art methods.

##  1 Introduction

Modern deep convolutional neural networks (CNNs) with competitive performance
in image understanding competitions (e.g., ImageNet [1] and COCO [2]) are
over-parameterized with millions of parameters, resulting in a large memory
utilization and computational complexity [3, 4]. As a result, deploying deep
models with high accuracy in devices with limited hardware is a challenge, due
to both memory requirements and a large number of floating point operations
per second (FLOPs). Many empirical observations have also demonstrated that
the test accuracy of DNNs is not affected drastically by distorting their
trained weights and even removing the hidden units [5, 6]. Balancing the
trade-off between the size of a deep model and achieving high performance has
recently fueled research activity in developing various model compression
techniques, and quantifying the notion of redundancy in DNNs.

Among existing algorithms, a higher compression ratio. In general, pruning
techniques are divided into two categories: Structured and Unstructured
methods. Unstructured methods directly remove individual weights in a deep
model, while structured methods apply pruning in the level of filters and
layers by imposing some structure on the topology of weights. Unstructured
methods provide a higher compression ratio and accuracy in terms of memory
requirement, but less gain in the FLOPs number; however, structured methods
are much easier to be deployed in parallel processors, and usually provide a
significant reduction in the FLOPs count. In this work, we study structured
pruning techniques for deep networks with skip-connections e.g., ResNet [7]
and DenseNet [8] by finding a set of informative hidden units. We cast the
pruning problem as a feature selection problem These architectures are very
robust against removing a set of layers referred to here as _skip-units_
(please see Section 3), making them ideal candidates for pruning. In order to
select the most informative feature maps, we need to measure the dependency
between these features and the output of the model. Measuring the dependency
between random variables is an important problem in statistics, information
theory, and machine learning [9], which can be used to determine the
information of one random variable about the other one. The most well-known
dependence measure is the Shannon mutual information. Other measures include
Maximum Mean Discrepancy (MMD) [10], and Energy Dependence (Distance-based
Correlation) [11], to name a few.

In this paper, we use Energy Dependence as a model-free measure between a set
of hidden layers and the output of a DNN with skip-connections as a feature
selection criterion in order to assess the importance and relevance of the
feature maps. Using this information, we propose a structured pruning
algorithm to iteratively remove a set of less important redundant layers from
the network.

##  2 Related Work

Here, we only review the pruning techniques most relevant to our method. Among
existing approaches in network compression, pruning techniques are quite
popular [12, 13, 14, 15, 16, 17, 18]. These methods reduce the network
complexity through weights/kernels removal.

Pruning through the removal of weights is accomplished by determining and
removing less important weights in the final prediction of the model. However,
finding the less important weights is a prohibitively costly problem due to
its combinatorial nature. To get around this issue, various criteria such as
the minimum norm of weights, the activation of the feature maps, information
gain, and etc have been proposed in literate [19]. However, these approaches
often either ignore the dependency between layers or solve a more difficult
optimization objective than the original loss. Evaluating the importance of a
filter just from its weights is not a reliable measure as dependency between
filters is not taken into account. In addition, higher weight values do not
necessarily mean the true importance of a filter since the filter’s
contribution can be compensated elsewhere in the network [18].

The most relevant pruning technique with our approach is the structured method
of dynamic path selection. In this method, a sub-graph or a group of weights
with a specific structure is selected from the original network such that the
chosen graph has a comparable inference (test) performance with the original
model. The common practice is to train an additional module which selects
where to forward between channels, blocks, or other parts of the graph [20,
21, 22, 23, 16, 24]. In particular, approaches proposed by [23] and [22] have
mostly focused on the residual networks, and their goal is to dynamically
select a set of layers using reinforcement learning. however, they are less
effective with deeper neural networks. In addition, the additional module may
increase the complexity and running time of the whole model.

##  3 Background and Proposed Method

###  3.1 Deep Neural Models with Skip-Connection

As mentioned, architectures with skip-connections are more robust to weight
distortion than the feed-forward networks [25]. Here, we take this phenomenon
one step further and develop a structured pruning technique based on removing
a collection of redundant layers instead of individual random layers. First,
we define a _skip-unit_ as a set of layers where its output is a function of
sequential application of operations in the unit including Conv, Pooling,
ReLU, BN, Dropout, etc and feature maps of previous unit(s). Specifically,
consider a DNN with $L$ skip-units. We denote the input of $l^{th}$ unit as
$U_{l-1}$. Let $T_{l}=f_{l}(U_{l-1})$ denote the output of sequential
application of the aforementioned operations in a skip-unit, summarized by
$f_{l}$. We call $T_{l}$ as the feature map of the unit $l$. Hence, each skip-
unit is mathematically expressed by: $U_{l}=\Psi(T_{l},U_{l:l-1},\alpha_{l}),\
\ l=2,3,\ldots,L,$ where $\alpha_{l}\in\\{0,1\\}$ is referred to as _policy_
for unit $l$, and $U_{1:l-1}$ denotes all the skip-units from unit $1$ to the
$(l-1)^{th}$ unit. $\Psi$ also denotes an operation that combines $T_{l}$ and
$U_{1:l-1}$. In ResNet and DenseNet models, $\Psi_{res}$ and $\Psi_{den}$ are
respectively given by

$\displaystyle
U_{l}=\Psi_{res}(T_{l},U_{1:l-1},\alpha_{l})=\alpha_{l}T_{l}+\mathcal{A}_{l-1}U_{l-1},$
(1) $\displaystyle
U_{l}=\Psi_{dense}(T_{l},U_{1:l-1},\alpha_{l})=Con(\alpha_{l}T_{l},\
U_{1:l-1}),$ (2)

where $Con$ is a concatenation operation, and $\mathcal{A}_{l-1}$ is an
identity, down-sampling, or some operation such as convolution operator(s).
Furthermore, $U_{0}$, the input for the first unit is usually given by a
convolution operator (e.g., in ResNet, and DenseNet).

###  3.2 Model-Free Measurement of Dependency

Our main idea for compression of deep models with skip-units is based on
selecting the most informative hidden units by measuring their statistical
dependency to the output of the model and then removing less important units.
A popular information measure is Shannon’s _Mutual Information_ (MI).
Computing the mutual information between the output and the features of a
machine learning model has been a research topic with a long history in
unsupervised feature learning [26]. In the context of deep learning, MI
between the hidden layers and the output of a network has been investigated by
the _Information Bottleneck_ theory [27]. While Shannon MI is an appealing
mathematical measure, its estimation may be a difficult task. The estimation
of MI for high-dimensional data needs a model assumption for the underlying
probability distribution. The selected model may be statistically unrealistic.
Here, we use a model-free measure with similar properties of MI that is not
computationally intense [28]. We introduce a measure that quantifies the
dependency between two random vectors $T$ and $Y$, a feature map and output
labels, respectively. Our proposed measure is based on the Energy Distance
between two random vectors.

######  Definition 1 (Energy Distance (ED) [29])

: Suppose that the characteristic functions and distribution functions of two
random vectors $T_{i}\in\mathbb{R}^{d}$ with finite first moment (i.e.,
$E||T_{i}||<\infty$) are respectively given by $\phi_{i}(\cdot)$ and
$F_{i}(\cdot)$ ($i=1,2$). The ED between $F_{1}$ and $F_{2}$ is given by
$\mathcal{E}(F_{1},F_{2})=\frac{1}{c_{d}}\int_{\mathbb{R}^{d}}\frac{|\phi_{1}(s)-\phi_{2}(s)|^{2}}{\|s\|^{d+1}}ds,$
where $c_{d}=\pi^{(1+d)/2}/\Gamma((1+d)/2)$ only depends on $d$.

###### Theorem 1

The energy distance in (1) can be written as [29]

$\displaystyle\mathcal{E}(F_{1},F_{2})$
$\displaystyle=2E\|T_{1}-T_{2}\|-E\|T_{1}-T_{1}^{\prime}\|-E\|T_{2}-T_{2}^{\prime}\|,$
(3)

where $T_{1}^{\prime}$ and $T_{2}^{\prime}$ are i.i.d copies of $T_{1}$ and
$T_{2}$, respectively, and $\|\cdot\|$ denotes the Euclidean norm. Energy
Distance is non-negative and $\mathcal{E}(F_{1},F_{2})=0$ iff $F_{1}=F_{2}$.

Now suppose that we have $n_{i}$ observations of $T_{i}$, $i=1,2$, where
$t_{i,j}$ denotes the $j$-th observation of $T_{i}$. Then an unbiased
estimator of ED in (3) is given by:

$\displaystyle\hat{\mathcal{E}}(F_{1},F_{2})$
$\displaystyle=\frac{2}{n_{1}n_{2}}\sum_{1\leq j_{1}\leq n_{1},1\leq j_{2}\leq
n_{2}}\|t_{1,j_{1}}-t_{2,j_{2}}\|-\frac{1}{n_{1}^{2}}\sum_{1\leq
j_{1},j_{1}^{\prime}\leq n_{1}}\|t_{1,j_{1}}-t_{1,j_{1}^{\prime}}\|$
$\displaystyle\hskip 113.81102pt-\frac{1}{n_{2}^{2}}\sum_{1\leq
j_{2},j_{2}^{\prime}\leq n_{2}}\|t_{2,j_{2}}-t_{2,j_{2}^{\prime}}\|.$ (4)

Applying the theory of V-statistics [30], it can be proved under mild
assumptions that the above $\hat{\mathcal{E}}(F_{1},F_{2})$ is a consistent
estimator of $\mathcal{E}(F_{1},F_{2})$, that is
$\hat{\mathcal{E}}(F_{1},F_{2})\rightarrow\mathcal{E}(F_{1},F_{2})$ in
probability as $n=\min\\{n_{1},n_{2}\\}\rightarrow\infty$. This implies that
with sufficiently large data size, $\hat{\mathcal{E}}(F_{1},F_{2})$ vanishes
for $T_{1}$ identically distributed as $T_{2}$, and is bounded away from zero
otherwise, without the need of specifying any parametric model for the
underlying probability distribution. We now define the dependency.

######  Definition 2 (Energy Dependence)

Consider random vectors/variables $T\in\mathbb{R}^{d}$ and $Y\in\mathcal{Y}$,
where $\mathcal{Y}$ has $p$ elements, say $\\{1,2,\ldots,p\\}$ for the
notational convenience. The Energy Dependence between $T$ and $Y$ is defined
by $D(T,Y)=\max_{1\leq i,j\leq p}\mathcal{E}(F_{i},F_{j}),$ where $F_{i}$
denotes the distribution of $T$ conditional on $Y=i$.

###### Theorem 2

For random vector/variable $T\in\mathbb{R}^{d}$ and $Y\in\mathcal{Y}$ with a
finite alphabet $\mathcal{Y}$, $D(T,Y)=0$ if $T$ and $Y$ are independent.

###### Proof 1

The independence of $T$ and $Y$ is equivalent to $F_{T\mid Y=j}=F_{T\mid
Y=j^{\prime}}$ for any $1\leq j,j^{\prime}\leq p$. This is further equivalent
to $\mathcal{E}(F_{i},F_{j})=0$, which implies $D(T,Y)=0$, according to
Definition 2 and Theorem 1.

The larger the Energy Distance, the larger is the dissimilarity. As a
consequence, the Energy Dependence in Definition 2 may be interpreted as a
quantification of the dependency between $T$ and $Y$. Based on (4), we define
a consistent estimator of $D(T,Y)$ by $\hat{D}(T,Y)=\max_{1\leq i,j\leq
p}\hat{\mathcal{E}}(F_{i},F_{j}),$ where $\hat{\mathcal{E}}(F_{i},F_{j})$ is
given by (4). We use this to measure the statistical dependency between $T$
and $Y$. The larger its value, the more information $T$ reveals about $Y$.
Similar to the consistency of (4), $\hat{D}(T,Y)$ is a consistent estimator of
$D(T,Y)$ according to the standard theory of V-statistics.

###  3.3 Proposed Algorithm

The general idea is to measure the relevance and importance of different skip-
units to the output of the network, and then select the less important ones.
If we show the Energy Dependence between the unit $T_{l}$ and the output of
the network, $Y$ by $D(T_{l},Y),\ l=1,2,\dots,L$, ideally, we need to to
select a set $S^{*}\subseteq\\{T_{1},T_{2},\dots,T_{L}\\}$ of informative
units, maximizing the average dependencies:
$S^{*}=\underset{S}{\mathrm{argmax}}\frac{1}{|S|}\sum_{T_{l}\in S}D(T_{l},Y),$
where $|S|$ demotes the size of the set $S$. However, the solution of this
optimization problem is a set with highly redundant units which does not take
into account the dependency between the units. In addition, due to the fact
that the existence of the skip-units in ResNet and DenseNet architectures, the
Markov Chain property among the layers is violated. This in turn implies that
the mutual information between the skip-units and the output does not decrease
monotonically according to the information processing inequality. As a result,
selecting units with the largest energy values (information) may remove some
important skip-units. We will show an experiment to verify this argument in
below. Hence, we need to solve alternatively the following optimization
problem to simultaneously maximize the relevance of units and minimizing the
redundancy among them:

$\displaystyle S^{*}=\underset{S}{\mathrm{argmax}}\frac{1}{|S|}\sum_{T_{l}\in
S}D(T_{l},Y)-\frac{1}{|S|^{2}}\sum_{T_{l},T_{l^{\prime}}\in
S}D(T_{l},T_{l^{\prime}}).$

However, solving the above problem is computationally expensive if it is not
possible due to its combinatorial nature; as a result, we propose our
algorithm which we call _Pruning with Energy Dependence_ (PED). The pseudocode
of our algorithm is given in Algorithm 1. The idea of PED can be summarized by
3 steps: (I) Clustering the Energy Dependence of units, (II) Selecting the
most informative unit as a cluster head in each cluster with the largest
Energy Dependence value, and (III) Removing other units from each cluster in
order to prune the redundant units. In Algorithm 1 $|A|$ is the size of a set
$A$ and $S^{t}$ is the index set of active skip-units at stage $t$, and
$\mathbf{DNN}^{t}$ means a trained pruned model with weights initialized from
stage $t-1$.

In each stage $t$, PED determines the units whose feature maps $T_{l}$ are
more informative about the output of the model for the given input by
computing $\hat{D}(T_{l},Y)$, $l=1,2,\ldots,L$, defined in Definition 2. Next,
PED uses $\hat{D}(T_{l},Y)$ to select active skip-units for which
$\alpha_{l}=1$ by clustering the $\hat{D}(T_{l},Y)$ values. For clustering
purpose, we have used an optimal $k$-means algorithm based on dynamic
programming [31] (by invoking $\mathrm{Clust}(K^{t},\mathbf{D}^{t},S^{t})$
sub-routine)111Other clustering algorithms can be used here; however, since
the clustering is applied on 1-D data, using the approach by [31] guarantees
optimality of clustering..

INPUT:

$\mathbf{DNN}^{0}$: Pre-trained Deep Neural Network

$S^{0}$: The index set of skip-units in $\mathbf{DNN}^{0}$

$T_{l}^{0}$: Feature maps at $t=0$, $l=1,2,\dots,|S^{0}|$

$K^{0}$: Number of Clusters at $t=0$

$N$: Number of stages

for $t=0,1,\ldots,N-1$ do

Compute $\hat{D}(T_{l}^{t},Y),\ \ l=1,\dots,|S^{t}|\ $ using
$\mathbf{DNN}^{t}$

Construct
$\mathbf{D}^{t}=[\hat{D}(T_{1}^{t},Y),,\ldots,\hat{D}(T_{|S^{t}|}^{t},Y)]$

$\displaystyle\\{Cluster_{1},\ldots,Cluster_{K^{t}}\\}=\mathrm{Clust}(K^{t},\mathbf{D}^{t},S^{t})$

for $j=1,2,\ldots,K^{t}$ do

$a_{c}=1,\ \ \ c=\mathrm{centroid\ index\ of\ j^{th}\ cluster}$

$a_{u}=0,\ \ \forall u\in Cluster_{j}\setminus c$

end for

Update $S^{t}$ using only $K^{t}$ units and remove the rest

Update $\mathbf{DNN}^{t}$ by re-training the model with $K^{t}$ units

$K^{t}=|S^{t}|-1$

end for

Return pruned model with $|S^{N-1}|$ active skip-units

Algorithm 1 Pruning with Energy Dependence (PED)

Next, PED only keeps $K^{t}$ units with the largest ED values and removes all
other skip-units At the end of stage $t$, PED re-trains the new compressed
model with weights in the active units and initialized with their values from
the previous stage. This process continues until the desired size of the
compressed network is met. For choosing the number of clusters ($K^{t}$) in
each stage, while simply trying all the possible values from $1$ to
$|S^{t}|-1$ is an acceptable approach (In ResNet or DenseNet, $|S^{t}|$ is at
most $60$, and it decreases after each stage), one can start with a coarse
pruning in the early stages and shifts gradually to the finer pruning in the
later stages.

|  |  |
---|---|---|---
(a) |  |  | (b)
Figure 1: Comparison of clustering scheme in PED against pruning with largest
energy values, and random selection. The experiments are based on the ResNet56
and CIFAR-10.

An important question here is that what if we choose the units with the
largest energy values, or selecting them randomly instead of clustering
approach. We first note that the model-free measure of information is a random
variable which depends on the data samples. When these values are close to
each other, a test of hypothesis fails to reject if an information value is
larger than the other. Thus, we cluster the values that cannot be
distinguished from each other statistically. To support the preference of
clustering over two other schemes, we have compared pruning using clustering
with the two above schemes. Plots (a) and (b) of Figure 1 show the comparison
for pruning ResNet56 model in classification of CIFAR-10 data set. To be fair
in comparison with the clustering scheme used in PED, we have used the same
number of units used in the clustering for removing units randomly and with
the smallest energy. As we can see in Figure 1, clustering approach results in
better test accuracy both with respect to the percentage of remaining
parameters, and the percentage of remaining FLOPs compared to the pruning
methods with the largest Energy Dependence and random selection specifically
for the later stages.

Models | Acc. | Par. | Red.(%) | FLOPs | Red.(%)
---|---|---|---|---|---
ResNet32 (SNIP) [32]U | 0.9259 | 0.19 | 90.00 | - | -
ResNet56 (PFEC-B) [33]S | 0.9306 | 0.73 | 13.70 | 90.90 | 27.60
ResNet56 (SFP) [34]S | 0.9359 | - | - | 59.40 | 52.60
ResNet56 (CNN-FCF) [35]S | 0.9338 | - | 43.09 | 72.40 | 42.78
ResNet56 (FPGM-mix 40%) [36]S | 0.9359 | - | - | 59.40 | 52.60
ResNet56 (HRank) [16]S | 0.9317 | 0.49 | 42.40 | 62.72 | 50.00
ResNet56 (ours) | 0.9336 | 0.32 | 62.96 | 45.73 | 63.86
ResNet110 (BlockDrop) [22]S | 0.9360 | - | - | 173.00 | 65.00
ResNet110 (SkipNet) [23]S | 0.9330 | - | - | 126.00 | 50.47
ResNet164-pruned [37]S | 0.9492 | 1.44 | 14.90 | 381.00 | 23.70
ResNet164-a (ours) | 0.9519 | 0.65 | 61.78 | 71.06 | 72.13
ResNet164-b (ours) | 0.9426 | 0.48 | 71.57 | 48.93 | 80.81
DenseNet40-pruned [37]S | 0.9481 | 0.66 | 35.70 | 381.00 | 28.40
IGC-V2*C416 [38]S | 0.9451 | 0.65 | - | - | -
CondenseNet86 [6]S | 0.9496 | 0.52 | - | 65.00 | -
DenseNet100-k12-a (ours) | 0.9432 | 0.29 | 61.98 | 117.73 | 59.89
DenseNet100-k12-b (ours) | 0.9425 | 0.27 | 64.50 | 116.51 | 60.31
Table 1: Classification test accuracy, number of parameters (Par), and FLOPs on CIFAR-10 between PED and those of the state-of-the-art methods. Par and FLOPs are in million. Models | Acc. | Par. | Red.(%) | FLOPs | Red.(%)
---|---|---|---|---|---
ResNet32 (SET) [39]U | 0.6966 | 0.19 | 90.00 | - | -
ResNet32 (GraSP) [40]U | 0.6924 | 0.19 | 90.00 | - | -
Resnet110 (BlockDrop) [22]S | 0.7370 | - | - |  $\sim$284.00 |  $\sim$56.00
ResNet110 (SkipNet) [23]S | 0.7250 | - | - | - | 37.00
ResNet164-pruned [37]S | 0.7713 | 1.46 | 15.50 | 333.00 | 33.30
ResNet164-a (ours) | 0.7499 | 0.58 | 67.70 | 100.09 | 60.74
ResNet164-b (ours) | 0.7402 | 0.47 | 72.80 | 91.01 | 64.30
DenseNet40-pruned [37]S | 0.7472 | 0.66 | 37.50 | 371.00 | 30.30
CondenseNet86 [6]S | 0.7636 | 0.52 | - | 65.00 | -
DenseNet100-k12-a (ours) | 0.7526 | 0.51 | 36.64 | 239.27 | 21.32
DenseNet100-k12-b (ours) | 0.7488 | 0.47 | 40.78 | 221.89 | 27.04
Table 2: The top-1 test accuracy, number of parameters (Par), and FLOPs on CIFAR-100 between PED and those of the state-of-the-art methods. Par and FLOPs are in million. “$\sim$” means approximate value. The accuracy reported in [40] is given by $0.6924\pm 0.24$. Model (ResNet50) | Top1 Acc. | Top5 Acc. | Par. | Red.(%) | FLOPs | Red.(%)
---|---|---|---|---|---|---
Channel Prune [41]S | 0.7230 | 0.9080 | - | - | 5.22 | 28.00
SkipNet [23]S | 0.7200 | - | - | - | - | 12.00
SSS [42]S | 0.7182 | 0.9079 | 15.60 | 38.82 | 2.33 | 43.32
GAL-0.5 [43]S | 0.7180 | 0.9082 | 19.31 | 24.74 | 1.84 | 55.01
HRank [16]S | 0.7198 | 0.9101 | 13.77 | 46.95 | 1.55 | 62.10
PED (ours) | 0.7280 | 0.9094 | 12.40 | 51.49 | 2.03 | 50.53
Table 3: The top-1 and top-5 test accuracy, number of parameters (Par), and
FLOPs on ImageNet data set between PED and those of the state-of-the-art
methods.

##  4 Experimental Results

In this section, we present the performance of PED on the classification task
of CIFAR-10/100 [44], and ImageNet [1] data sets. For the ImageNet data set,
we have used the common data-augmentation scheme at training time [7], and
perform a re-scaling to $256\times 256$ followed by a $224\times 224$ center
crop at test time. We have compared the performance of PED with the state-of-
the-art methods for CIFAR-10, CIFAR-100, and ImageNet datsets respectively in
Table 1, Table 2, and Table 3. The third/fourth and fifth/sixth columns
labeled ’Red’ in all tables represent the percentage of reduction in the
number of parameters and FLOPs, respectively. Also, the numbers given in
columns labeled as ”Par.” (i.e., Parameters) and ”FLOPs” are in million (Only
FLOPs in Table 3 is in Billion) and rounded by two-decimal digits. In all
tables, “–” means no reported value, and the superscript “S” and “U” denote
the structured and unstructured methods, respectively.

The ResNet56 architecture consists of $56$ layers with $27$ residual units and
the total number of $0.85$ (M) training parameters and $126.54$ (M) FLOPs. In
order to run PED on ResNet56, we first train it on CIFAR-10 to achieve
$0.9334$ test accuracy. The results of applying PED on ResNet56 with $N=5$
stages are shown in Table 1. As we can see, without almost any dropping in the
test accuracy, we can reduce the number of trainable parameters and FLOPs by
$62.96\%$ and $63.86\%$, respectively. We next consider the ResNet164 which
consists of $18$ units with total number of $1.70$ (M) trainable parameters
and $254.94$ (M) FLOPs. Similar to the ResNet56, we first train this model on
the CIFAR-10 and achieve $0.9569$ test accuracy. In Table 1, we have listed
two ResNet164 pruned models. The first one, ResNet164-a is a compressed
version of the ResNet164 by running PED for $N=6$ stages, while ResNet164-b
corresponds to $N=9$ stages. As we can see, ResNet164-a achieves $0.9519$
accuracy with $0.65$ (M) parameters and $71.06$ (M) FLOPs. For the CIFAR-100,
a full ResNet164 achieves $77.93\%$ top-1 accuracy. Table 2 also presents
ResNet164-a and ResNet164-b corresponding to $N=9$ and $N=11$ stages,
respectively.

Next we focus on the DenseNet100-k12 with $100$ layers, $k=12$ growth-rate
(i.e., the number of output channels in each unit), and $48$ skip-units. We
apply PED to DenseNet100-k12 by first training DenseNet100-k12 on both the
CIFAR-10 with $0.9531$ test accuracy, $0.77$ (M) number of parameters, and
$293.55$ (M) FLOPs (Tabel 1) and the CIFAR-100 with $0.7793$ as the top-1 test
accuracy, $0.80$ (M) number of parameters, and $304.10$ (M) FLOPs (Table 2).
DenseNet100-k12-a and DenseNet100-k12-b correspond to running PED with $N=10$
and $N=11$ stages, respectively. Also we have two models of DenseNet100-k12-a
and DenseNet100-k12-b in Table 2 corresponding to running PED with $N=5$ and
$N=6$ stages, respectively. These experiments suggest that our proposed
algorithm is competitive to the state-of-the-art methods in one or more
criteria of accuracy/parameters/FLOPs. Finally, Table 3 illustrates
experiments on the ImagaNet data set using the ResNet50. The model consists of
4 blocks with 3, 4, 6, and 3 skip-units in each block. ResNet50 has $25.56$
(M) trainable parameters and $4.11$ (B) FLOPs. Compared to methods of SkipNet,
SSS, GAL-0.5, and HRank, our method has better performance.

##  5 References

## References

  * [1] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2009, pp. 248–255.
  * [2] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. Zitnick,  “Microsoft coco: Common objects in context,”  in European Conf. Comp. vision, 2014, pp. 740–755.
  * [3] A. Krizhevsky, I. Sutskever, and G. Hinton,  “Imagenet classification with deep convolutional neural networks,”  in Adv. Neural Inf. Process. Sys., 2012.
  * [4] K. Simonyan and A. Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [5] Y. Cheng, D. Wang, P. Zhou, and T. Zhang,  “A survey of model compression and acceleration for deep neural networks,”  arXiv preprint arXiv:1710.09282, 2017.
  * [6] G. Huang, S. Liu, L. Van der Maaten, and K. Weinberger,  “Condensenet: An efficient densenet using learned group convolutions,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2018, pp. 2752–2761.
  * [7] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2016, pp. 770–778.
  * [8] G. Huang, Z. Liu, L. Van Der Maaten, and K. Weinberger,  “Densely connected convolutional networks,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2017, pp. 4700–4708.
  * [9] B Poczos, Z Ghahramani, and J Schneider,  “Copula-based kernel dependency measures,”  in Int. Conf. Machine Learning, 2012.
  * [10] K. Borgwardt, A. Gretton, Ma. Rasch, H. Kriegel, B. Schölkopf, and A. Smola,  “Integrating structured biological data by kernel maximum mean discrepancy,”  Bioinformatics, vol. 22, no. 14, pp. e49–e57, 2006.
  * [11] G. Székely, M. Rizzo, and N. Bakirov,  “Measuring and testing dependence by correlation of distances,”  Annals of Statistics, vol. 35, no. 6, pp. 2769–2794, 2007.
  * [12] X. Xiao, Z. Wang, and S. Rajasekaran,  “Autoprune: Automatic network pruning by regularizing auxiliary parameters,”  in Adv. Neural Inf. Process. Sys., 2019.
  * [13] Z. Zhuang, M. Tan, B. Zhuang, J. Liu, Y. Guo, Q. Wu, J. Huang, and J. Zhu,  “Discrimination-aware channel pruning for deep neural networks,”  in Adv. Neural Inf. Process. Sys., 2018, pp. 875–886.
  * [14] S. Lin, R. Ji, Y. Li, Y. Wu, F. Huang, and B. Zhang,  “Accelerating convolutional networks via global & dynamic filter pruning.,”  in IJCAI, 2018, pp. 2425–2432.
  * [15] E. Malach, G. Yehudai, S. Shalev-Shwartz, and O. Shamir,  “Proving the lottery ticket hypothesis: Pruning is all you need,”  arXiv preprint arXiv:2002.00585, 2020.
  * [16] M. Lin, R. Ji, Y. Wang, Y. Zhang, B. Zhang, Y. Tian, and L. Shao,  “Hrank: Filter pruning using high-rank feature map,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2020, pp. 1529–1538.
  * [17] C. Herrmann, Richard S. Bowen, and R. Zabih,  “Channel selection using gumbel softmax,”  in Europ. Conf. Comp. Vision, 2020, pp. 241–257.
  * [18] M. Ganesh, J. Corso, and S. Sekeh,  “Mint: Deep network compression via mutual information-based neuron trimming,”  arXiv preprint arXiv:2003.08472, 2020.
  * [19] P. Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz,  “Pruning convolutional neural networks for resource efficient inference,”  Int. Conf. Learning. Rep.., 2017.
  * [20] L. Liu and J. Deng,  “Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution,”  in PAAAI Conf. Artificial Intelligence, 2018, vol. 32.
  * [21] J. Hu, L. Shen, and G. Sun,  “Squeeze-and-excitation networks,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2018, pp. 7132–7141.
  * [22] Z. Wu, T. Nagarajan, A. Kumar, S. Rennie, L. Davis, K. Grauman, and R. Feris,  “Blockdrop: Dynamic inference paths in residual networks,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2018, pp. 8817–8826.
  * [23] X. Wang, F. Yu, Z. Dou, T. Darrell, and J. Gonzalez,  “Skipnet: Learning dynamic routing in convolutional networks,”  in Europ. Conf. Comp. Vision, 2018, pp. 409–424.
  * [24] S. Lee, S. Chang, and N. Kwak,  “Urnet: User-resizable residual networks with conditional gating module,”  in AAAI Conf. Artificial Intelligence, 2020, vol. 34.
  * [25] A. Veit, M. Wilber, and S. Belongie,  “Residual networks behave like ensembles of relatively shallow networks,”  in Adv. Neural Inf. Process. Sys., 2016, pp. 550–558.
  * [26] R Linsker,  “Self-organization in a perceptual network. computer,”  pp. 105–117, 1988.
  * [27] N. Tishby and N. Zaslavsky,  “Deep learning and the information bottleneck principle,”  in IEEE Inf. Theory. Workshop. (ITW), 2015, pp. 1–5.
  * [28] L. Bottou, M. Arjovsky, D. Lopez-Paz, and M. Oquab,  “Geometrical insights for implicit generative modeling,”  in Braverman Readings in Machine Learning. Key Ideas from Inception to Current State, pp. 229–268. Springer, 2018.
  * [29] G. Székely and M. Rizzo,  “Energy statistics: A class of statistics based on distances,”  Journal of stat. Plan. Inference, vol. 143, no. 8, pp. 1249–1272, 2013.
  * [30] A. Lee,  U-statistics: Theory and Practice,  Routledge, 2019.
  * [31] H. Wang and M. Song,  “Ckmeans. 1d. dp: Optimal k-means clustering in one dimension by dynamic programming,”  2011, vol. 3, p. 29.
  * [32] N. Lee, T. Ajanthan, and P. Torr,  “SNIP: Single shot network pruning based on connection sensitivity,”  in Int. Conf. Learning. Rep., 2019.
  * [33] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf,  “Pruning filters for efficient convnets,”  in Int. Conf. Learning. Rep., 2017.
  * [34] Y. He, G. Kang, X. Dong, Y. Fu, and Y. Yang,  “Soft filter pruning for accelerating deep convolutional neural networks,”  in Int. Joint Conf. Artificial Intelligence, 2018.
  * [35] T. Li, B. Wu, Y. Yang, Y. Fan, Y. Zhang, and W. Liu,  “Compressing convolutional neural networks via factorized convolutional filters,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2019, pp. 3977–3986.
  * [36] Y. He, P. Liu, Z. Wang, Z. Hu, and Y. Yang,  “Filter pruning via geometric median for deep convolutional neural networks acceleration,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2019, pp. 4340–4349.
  * [37] Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang,  “Learning efficient convolutional networks through network slimming,”  in Int. Conf. Comp. Vision, 2017.
  * [38] G. Xie, J. Wang, T. Zhang, J. Lai, R. Hong, and G. Qi,  “Interleaved structured sparse convolutional neural networks,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2018.
  * [39] D. Mocanu, E. Mocanu, P. Stone, P. Nguyen, M. Gibescu, and A. Liotta,  “Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science,”  Nature communications, vol. 9, no. 1, pp. 1–12, 2018.
  * [40] Chaoqi W., Guodong Z., and Roger G.,  “Picking winning tickets before training by preserving gradient flow,”  in Int. Conf. Learning. Rep., 2020.
  * [41] Y. He, X. Zhang, and J. Sun,  “Channel pruning for accelerating very deep neural networks,”  in IEEE Int. Conf. Comp. Vision, 2017, pp. 1389–1397.
  * [42] Z. Huang and N. Wang,  “Data-driven sparse structure selection for deep neural networks,”  in Europ. Conf. Comp. Vision, 2018, pp. 304–320.
  * [43] S. Lin, R. Ji, C. Yan, B. Zhang, L. Cao, Q. Ye, F. Huang, and D. Doermann,  “Towards optimal structured cnn pruning via generative adversarial learning,”  in IEEE Conf. Comp. Vision. Patt. Recog., 2019, pp. 2790–2799.
  * [44] A. Krizhevsky and G. Hinton,  “Convolutional deep belief networks on cifar-10,”  Unpublished manuscript, vol. 40, no. 7, pp. 1–9, 2010.