# Neural Network Pruning Through Constrained Reinforcement Learning

###### Abstract

Network pruning reduces the size of neural networks by removing (pruning)
neurons such that the performance drop is minimal. Traditional pruning
approaches focus on designing metrics to quantify the usefulness of a neuron
which is often quite tedious and sub-optimal. More recent approaches have
instead focused on training auxiliary networks to automatically learn how
useful each neuron is however, they often do not take computational
limitations into account. In this work, we propose a general methodology for
pruning neural networks. Our proposed methodology can prune neural networks to
respect pre-defined computational budgets on arbitrary, possibly non-
differentiable, functions. Furthermore, we only assume the ability to be able
to evaluate these functions for different inputs, and hence they do not need
to be fully specified beforehand. We achieve this by proposing a novel pruning
strategy via constrained reinforcement learning algorithms. We prove the
effectiveness of our approach via comparison with state-of-the-art methods on
standard image classification datasets. Specifically, we reduce $83-92.90\%$
of total parameters on various variants of VGG while achieving comparable or
better performance than that of original networks. We also achieved $75.09\%$
reduction in parameters on ResNet18 without incurring any loss in accuracy.

##  1 Introduction

Deep neural networks typically have large memory and compute requirements,
making it difficult to deploy them on small devices such as mobiles and
tablets [1]. These cumbersome networks are pruned by removing their redundant
weights, layers, filters and blocks [2, 3, 4, 5, 6, 7]. Neural network pruning
strategies can be grouped into three categories namely i) Offline pruning, ii)
Online pruning and iii) Pruning via Reinforcement Learning.

Offline Pruning requires multiple iterations of training, pruning, and fine-
tuning. magnitude-based pruning [8] which works on ‘magnitude equals salience’
principle. [9, 10] uses second derivative of weights to rank connections, More
recent approaches suggest look-ahead pruning strategies [11, 5] where
magnitude of previous connected layers is also considered. Layers are pruned
independently to speed up the pruning process in [12]. Instead of reducing the
original, deep network (called the “teacher”) by pruning connections,
knowledge distillation [13] trains another, smaller network (called the
“student”) to mimic the output of the teacher network. However, it requires
designing an ad hoc student network which is a tedious task.

Online Pruning: A more recent class of techniques poses the problem of pruning
as a learning problem by introducing a mask vector that acts as a gate or an
indicator function to turn on/off a particular component (connection, filter,
layer, block) [14, 15, 16, 7]. The mask vector can be treated as a trainable
parameter and is directly optimized through gradient descent. The mask can
also be obtained via an auxiliary neural network [17, 3]. Pruning via
Surrogate Lagrangian Relaxation (P-SLR) [18] utilizing surrogate gradient
algorithm for Lagrangian Relaxation [19]. [4, 20, 2, 20] uses Budget Aware
Pruning hat tries to reduce the size of a network to respect a specific budget
on computational and space complexitie these budgets introduce an arbitrary
function that is non-differentiable. However, one shortcoming of current
budget-aware techniques is that they require the function on which the budget
is being imposed to be either differentiable or fully specified [4]. This,
however, is not always possible when we, for example, want to impose budgets
on metrics such as inference time. The main question, then, is the following:
can we prune neural networks to respect budgets on arbitrary, possibly non-
differentiable, functions? One way to solve this problem is to leverage some
recent techniques in reinforcement learning (RL) [21, 22].

Pruning via Reinforcement Learning: In reinforcement learning [22] an agent
interacts with the environment around them by taking different actions. Each
action results in the agent receiving a reward depending upon how good or bad
the action is. If the agent is trained to predict sparsity as action, then the
accuracy of the obtained model can be used as a reward, resulting in RL based
neural network pruning [3]. Reinforcement learning (RL) [21, 22] is known to
optimize arbitrary non-differentiable functions while respecting budget on
computational and space complexities. One such example is AutoML for Model
Compression (AMC) [3] in which an agent is trained to predict the sparsity
ratio for each layer. Accuracy is returned as a reward to encourage the agent
to build smaller, faster, and more accurate networks. Similarly, Conditional
Automated Channel Pruning (CACP) [23] is another RL-based method that
simultaneously produces compressed models under different compression rates
through a single layer-by-layer channel pruning process. While these methods
allow to include both sparsity and accuracy, they lack fine-grained control
over these constraints.

Our main contributions: In this work, we formulate the problem of pruning in
the constrained reinforcement learning framework (CRL). We propose a general
methodology for pruning neural networks, which can prune neural networks to
respect specified budgets on arbitrary, possibly non-differentiable,
functions. Furthermore, we only assume the ability to be able to evaluate
these functions for different inputs, and hence they do not need to be fully
specified beforehand. Our proposed CRL framework outperform state-of-the-art
methods in compressing popular models such as VGG [24] and ResNets [25] on
benchmark datasets.

##  2 Pruning via Constrained Reinforcement Learning

Constrained reinforcement learning [26] is an extension of the reinforcement
learning problem in which agents, in addition to the reward, also receive a
cost. The agent’s goal is to maximize its cumulative reward subject to its
cumulative cost being less than some pre-defined threshold. One interesting
property here is that neither the reward nor the cost function needs to be
differentiable or fully specified. The agent simply needs to be fed a scalar
reward and a scalar cost value each time it performs an action.

Let $\boldmath\theta$ denote the parameters of a neural network. Each element
of $\boldmath\theta$ represents the weight of a single connection. Removing a
connection is thus equivalent to multiplying its weight by $0$. Let
$\mathcal{L}$ denote the loss function of the neural network. Pruning, in its
most general form, tries to find a mask $\boldmath{M}\in\\{0,1\\}^{|\theta|}$
that solves the following optimization:

$\begin{split}&\underset{\boldmath\theta,\boldmath{M}\in\\{0,1\\}^{|\theta|}}{\text{minimize}}\;\mathcal{L}(\boldmath\theta\odot\boldmath{M})\\\
&\text{subject to }f(\boldmath{M})\leq\alpha,\end{split}$ (1)

where $\odot$ is the element-wise Hadamard product, $f$ be some arbitrary,
possibly non-differentiable function, and $\alpha$ is a known constant. Here
$f$ represents the computational and space complexity of the network. For
example, if we want to compress the network by at least $50\%$ (in terms of
the space it occupies), we can let $f$ be the $\ell_{1}$-norm and $\alpha$ to
be equal to $0.5|\boldmath\theta|$. Similarly, if we wanted to optimize for
speed, we could set $f$ to be the number of flops the network consumes (or the
time it takes) when it is pruned according to $\boldmath{M}$.

In our proposed approach, the mask $\boldmath{M}$ is modeled via action
performed by an agent who is interacting with the environment. In the
beginning, the environment loads a pre-trained network (e.g., VGG16 [24]). The
agent is then fed the layer representations of the first convolutional layer
in the network (as defined in the previous section). The agent then specifies
an appropriate action. This action is then recorded, and the agent is then fed
the next state. Once the agent has finished predicting its actions for all
convolutional layers, the network is pruned according to the agent’s specified
actions and fine-tuned on the training set (this is the same training set that
was used to pre-train the network). The accuracy of the fine-tuned network is
returned as the agent’s reward along with the cost (the reward and cost at all
other time-steps are $0$). Figure 1 shows an illustration of our pruning
process.

Fig. 1: A graphical visualization of the environment consisting of a pre-
trained network that we are interested in pruning. At time-step $t$, the
policy outputs an action that specifies how the $t^{th}$ convolutional layer
should be pruned. The environment executes this action and returns the next
state (i.e., layer $(t+1)^{th}$ specifications) and a reward and a cost.

###  2.1 Constrained Markov Decision Process

Markov Decision Processes Mathematically, the notion of an environment is
captured through a Markov Decision Process (MDP). A finite-horizon MDP
$\mathcal{M}$ is a tuple $(\mathcal{S},\mathcal{A},p,r,\gamma,T)$, where
$\mathcal{S}\in\mathbb{R}^{|\mathcal{S}|}$ is a set of states,
$\mathcal{A}\in\mathbb{R}^{|\mathcal{A}|}$ is a set of actions,
$p:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$ is the
transition probability function (where $p(s^{\prime}|s,a)$ denotes the
probability of transitioning to state $s^{\prime}$ from state $s$ by taking
action $a$), $r:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward
function, $\gamma$ is the discount factor and $T$ is the time-horizon. A
trajectory $\tau=\\{s_{1},a_{1},\ldots,s_{T},a_{T}\\}$ denotes a sequence of
states-action pairs such that $s_{t+1}\sim p(\cdot|s_{t},a_{t})$.

The goal of an agent is to learn a policy about which action to take in each
state. Formally, a policy $\pi:\mathcal{S}\mapsto\mathcal{P}(\mathcal{A})$ is
a map from states to probability distributions over actions, with $\pi(a|s)$
denoting the probability of taking action $a$ in state $s$.

Let $r(\tau)=\sum_{t=1}^{T}\gamma^{t}r(s_{t},a_{t})$ denote the total
discounted reward of a trajectory. The problem of RL is to find a policy
$\pi^{*}$ that maximizes the expected total discounted reward, i.e.,

$\pi^{*}=\text{arg}\max_{\pi}J(\pi)=\mathbb{E}_{\tau\sim\pi}[r(\tau)].$ (2)

We parameterize $\pi$ as a neural network to deal with large state and action
spaces. One common approach is to model $\pi$ as a Gaussian distribution [27]
$\mathcal{N}(\mu_{\theta}(\boldmath{s}),\boldmath\sigma I)$, where
$\mu_{\theta}:|\mathcal{S}|\mapsto|\mathcal{A}|$ is a neural network.
$\boldmath\sigma\in\mathcal{R}^{|\mathcal{A}|}$ is a trainable vector and $I$
is the identity matrix. We will denote all of the parameters of $\pi$ with
$\boldmath\theta$. The policy gradients algorithm [28] finds $\pi^{*}$ through
stochastic gradient descent. That is, we update $\boldmath\theta$ as

$\boldmath\theta:=\boldmath\theta+\nabla_{\boldmath\theta}J(\pi_{\boldmath\theta}).$
(3)

The gradient of loss $J$ with respect to $\boldmath\theta$ can be shown to be

$\nabla_{\boldmath\theta}J(\boldmath\theta)=\nabla_{\boldmath\theta}\mathbb{E}_{\tau\sim\pi_{\boldmath\theta}}[r(\tau)]=\mathbb{E}_{\tau\sim\pi_{\boldmath\theta}}[\nabla_{\boldmath\theta}\log\pi_{\theta}(\tau)r(\tau)].$
(4)

The proximal policy algorithm (PPO) [29] makes a first-order approximation to
this optimization problem and proposes to update $\pi_{\boldmath\theta}$ for
some known constant $\epsilon$ by solving

$\pi^{*}=\text{arg}\max_{\pi}\mathbb{E}_{\tau\sim\pi_{\bar{\bm{\theta}}}}(x)$
(5)
$x=\left[\min\left(\frac{\pi_{\bm{\theta}}(\tau)}{\pi_{\bar{\bm{\theta}}}(\tau)}r(\tau),\text{clip}\left(\frac{\pi_{\bm{\theta}}(\tau)}{\pi_{\bar{\bm{\theta}}}(\tau)},1-\epsilon,1+\epsilon\right)r(\tau)\right)\right]$
(6)

Here clip bounds its first argument between the other two. The objective
function above essentially removes any incentive to move the ratio between the
two policies outside of the interval $[1-\epsilon,1+\epsilon]$.

Constrained Markov Decision Processes: In order to formulate our problem as a
constrained RL problem, we need to first define our constrained MDP (CMDP).
Let $g=g_{F}\circ g_{T}\circ g_{T-1}\circ\ldots g_{1}$ define a neural network
with $T$ convolutional layers, $g_{1},\ldots,g_{T}$ followed by a few fully
connected layers collectively denoted as $g_{F}$. We wish to prune $g$.
Furthermore, let $\boldmath\theta_{t}$ denote the parameters corresponding to
layer $t$. We define the key components of the CMDP as follows:

  1. 1.

State, $s$: Each convolutional layer corresponds to a single state. Similar to
the scheme in [3] each of these layers is represented by the following tuple:

(layer ($t$), input channels, number. of filters, kernel size, stride,
padding)

where $t$ is the index of that layer and the remaining entries are the
attributes of a convolutional layer.

  2. 2.

Action, $a$: for the action, representations we end up with a mask vector
$M_{t}$ at each layer $g_{t}$. The length of this mask vector will be equal to
the number of filters for $g_{t}$. We use $M$ to collectively denote the mask
vectors for all layers.

  3. 3.

Transition function: The agent always transitions from state $g_{t}$ to state
$g_{t+1}$, and so the transitions are fixed and independent of the agent’s
actions.

  4. 4.

Reward function, $r$: Let $B$ be a batch of input examples (uniformly) sampled
from the training dataset. We define the reward as follows:

$r(s_{t},a_{t})=\begin{cases}-\mathcal{L}(\theta\odot M)&\text{if }t=T\\\
0&\text{otherwise}\end{cases}$ (7)

where $\mathcal{L}$ is the loss function of the network evaluated on the batch
$B$.

  5. 5.

Cost function, $c$: the cost function is defined as

$c(s_{t},a_{t})=\begin{cases}f(M)&\text{if }t=T\\\
0&\text{otherwise}\end{cases}$ (8)

where $f$ is our constraint function evaluated on the batch $B$.

  6. 6.

Budget, $\alpha$: This the budget on $f$ we wish our pruned network to
respect.

The policy predicts a sparsity ratio for each layer. Filters are then pruned
via magnitude-based pruning [8] upto the desired sparsity.

###  2.2 Algorithm

Let $d_{a}$ be the dimension of the action. Also, let $d_{s}$ denote the
dimension of the state vector (recall that each state vector corresponds to a
single layer).

We model our policy as a (diagonal) multivariate Gaussian distribution
$\mathcal{N}(\mu_{\boldmath{\theta}},\boldmath{\sigma}\boldmath{I})$. Here
$\mu_{\boldmath{\theta}}:\mathbb{R}^{d_{s}}\mapsto\mathbb{R}^{d_{a}}$ is a
neural network with parameters $\boldmath{\theta}$,
$\sigma\in\mathbb{R}^{d_{a}}$ is a trainable vector and
$\boldmath{I}\in\mathbb{R}^{d_{a}\times d_{a}}$ is the identity matrix (hence
$\boldmath{\sigma}\boldmath{I}$ is the covariance matrix of the distribution).
The network $\mu_{\boldmath{\theta}}$ takes in as input a state vector $s$
provided by environment and outputs a vector of dimension $d_{a}$. We then
simply sample an action from
$\mathcal{N}(\mu_{\boldmath{\theta}}(\boldmath{s}),\boldmath{\sigma}\boldmath{I})$
and feed it to the environment. In parallel, we also train two other neural
networks, $V_{\boldmath{\phi_{r}}}^{r}:\mathbb{R}^{d_{s}}\mapsto\mathbb{R}$
and $V_{\boldmath{\phi_{c}}}^{c}:\mathbb{R}^{d_{s}}\mapsto\mathbb{R}$ with
parameters $\boldmath{\phi_{r}}$ and $\boldmath{\phi_{c}}$. The cost value
function is also defined similarly for the cost function. Recall that having
these networks helps us reduce variance.

We initialize our policy and value networks randomly and collect data
$\mathcal{D}$ from the environment. Each data point is essentially a tuple
$({s}_{t},{a_{t}},{s}_{t+1},r,c)$ where ${s}_{t}$ and ${s}_{t+1}$ are the
states at time $t$ and $t+1$ respectively, ${a}_{t}$ is the action taken at
time $t$ and $r$ and $c$ are the reward and cost received as a consequence of
taking action ${a}_{t}$. We use this dataset to update our parameters.
Furthermore, we initialize our Lagrange multiplier $\lambda$ with a constant
value and also update it using this dataset. This entire process is repeated
until convergence.

Recall that at each iteration, we are interested in optimizing (here we
decompose the expectation over trajectories into expectation over states and
actions):

$J_{\text{LAG}}^{\text{PPO}}(\lambda,\pi_{\boldmath\theta})=\sum_{t=1}^{T}\sum_{{s}_{t},{a}_{t}\sim\mathcal{D}}\left[\frac{\pi_{\boldmath\theta}({s}_{t},{a}_{t})}{\pi_{\bar{\boldmath\theta}}({s}_{t},{a}_{t})}J^{r}({s}_{t},{a}_{t},{s}_{t+1})-\lambda(\mathbb{E}_{\tau\sim\pi_{\boldmath\theta}}[J^{c}({s}_{t},{a}_{t},{s}_{t+1})]-\alpha)\right],$
(9)

where $\alpha$ is the budget and the losses $J^{(.)}$ are:

$J^{r}({s}_{t},{a}_{t},{s}_{t+1})=\sum_{t^{\prime}=t}^{T}r({s}_{t},{a}_{t})-V_{\boldmath{\phi_{r}}}^{r}({s}_{t+1}),$
(10)
$J^{c}({s}_{t},{a}_{t},{s}_{t+1})=\sum_{t^{\prime}=t}^{T}c({s}_{t},{a}_{t})-V_{\boldmath{\phi_{c}}}^{c}({s}_{t+1}).$
(11)

All parameters are updated via to gradient descent using their respective
learning rates $\eta_{i}$. Specifically, the policy network is updated as:

$\boldmath{\theta}:=\boldmath{\theta}-\eta_{1}\nabla_{\boldmath\theta}J_{\text{LAG}}^{\text{PPO}}(\lambda,\pi_{\boldmath\theta}),$
(12)

and the Lagrange multiplier as:

$\lambda:=\lambda-\eta_{2}\nabla_{\lambda}J_{\text{LAG}}^{\text{PPO}}(\lambda,\pi_{\boldmath\theta}),$
(13)

Furthermore, we define the loss for the reward value function network as:

$\mathcal{L}^{r}=\sum_{t=1}^{T}\sum_{{s}_{t},{a}_{t}\sim\mathcal{D}}||r(s_{t},a_{t})+\gamma_{r}V_{\boldmath{\phi_{r}}}(s_{t+1})-V_{{\boldmath\phi_{r}}}(s_{t})||_{2}^{2},$
(14)

and update its parameters as:

$\boldmath{\phi_{r}}:=\boldmath{\phi_{r}}-\eta_{3}\nabla_{\boldmath{\phi_{r}}}\mathcal{L}^{r}(\boldmath{\phi_{r}}),$
(15)

Similarly, the loss for the cost value function network is defined as:

$\mathcal{L}^{c}=\sum_{t=1}^{T}\sum_{\boldmath{s}_{t},\boldmath{a}_{t}\sim\mathcal{D}}||c(s_{t},a_{t})+\gamma_{c}V_{\boldmath{\phi_{c}}}(s_{t+1})-V_{{\boldmath\phi_{c}}}(s_{t})||_{2}^{2},$
(16)

and its parameters are updated as:

$\boldmath{\phi_{c}}:=\boldmath{\phi_{c}}-\eta_{4}\nabla_{\boldmath{\phi_{c}}}\mathcal{L}^{c}(\boldmath{\phi_{c}}),$
(17)

##  3 Results

###  3.1 Experimental Setup

We evaluated our approach using CIFAR-10 [30] dataset on ResNet18 [25] and
variants of VGG [24] network. The training was performed using Adam optimizer
[31] using learning rates $\eta_{i}=3.0\times 10^{-4}$ and batch size $60$.
Experiments were conducted on VGG11, VGG16, VGG19, and ResNet18. For each
model, we ran the experiments using different budget values. Initially, the
policy network was trained for a fixed number of iterations. The policy
network then pruned the original model by predicting the sparsity ratio for
every convolutional layer using the PPO-Lagrangian algorithm. The Lagrangian
multiplier was initialized with a fixed value as $\lambda=1$, and $\gamma_{r}$
and $\gamma_{c}$ are initialized as $0.99$ and $1.00$ respectively. Once the
network was pruned, it was fine-tuned for a certain number of iterations.
Since fine-tuning is computationally expensive, we adopted a schedule with
hyperparameter values of 0, 32, 128, respectively. It means that we fine-tune
less in the beginning and more towards the end. In practice, we normalize our
rewards and cost values with a running mean and standard deviation which is
continually updated as more data is collected. Furthermore, we also normalize
the state vector in a similar way to improve stability.

###  3.2 Ablative Study

To demonstrate the efficacy of the proposed constrained RL method, we
performed experiments on VGG11, VGG16, and VGG19 [24] and compared the results
with the magnitude-based pruning method. We train each of these networks on
the CIFAR-10 dataset [30]. Pretrained models of VGG were used to train the
policy network for $40$K iterations. During pruning, the models were fine-
tuned by $25$K, $35$K, $40$K iterations, respectively, in a fine-tuning
schedule. We experimented with budget values $\alpha=10$ and $\alpha=20$.
Increasing the $\alpha$ value decreases the sparsity of the pruned network.
The ablative study showed that the proposed constrained RL method is
significantly more optimal than magnitude-based pruning (see Table 1). Our
method achieved higher accuracy than magnitude-based pruning in all
experiments. In fact, it even outperformed the unpruned network in 4 out of 6
cases.

Table 1: Comparison of sparsity and accuracy between unpruned network,
baseline magnitude-based pruning [8] and proposed constrained reinforcement
learning (CRL).

|  | $\alpha=20$ | $\alpha=10$
---|---|---|---
|  | Sparsity ($\%$) | Acc. ($\%$) | Sparsity ($\%$) | Acc. ($\%$)
VGG11 | Unpruned | ”-” | 89.23 | ”-” | 89.23
Magnitude-based Pruning | 80.00 | 85.50 | 90.00 | 83.80
Proposed CRL | 83.48 | 89.11 | 90.75 | 88.09
VGG16 | Unpruned | ”-” | 90.69 | ”-” | 90.69
Magnitude-based Pruning | 80.00 | 88.40 | 90.00 | 87.10
Proposed CRL | 83.81 | 90.96 | 92.90 | 89.89
VGG19 | Unpruned | ”-” | 90.59 | ”-” | 90.59
Magnitude-based Pruning | 80.00 | 88.40 | 90.00 | 86.90
Proposed CRL-Coarse | 83.48 | 91.06 | 92.31 | 91.31

###  3.3 Comparison with State-of-the-art

To prove the effectiveness of the proposed constrained RL approach, we also
conducted experiments with ResNet18, VGG16 and Resnet50 to compared our
results with state-of-the-art methods on the CIFAR-10 dataset.

In the case of VGG16, we compared our method with two state-of-the-art
methods, namely Conditional Automated Channel Pruning (CACP) [23], and
GhostNet [32]. Our pruned VGG16 model performed better than both state-of-the-
art methods. Note that when $\alpha=20$, our pruned model contained 2.38M
parameters, which are fewer than the pruned models of both CACP and GhostNet,
which contained 4.41M and 3.30M parameters. Despite this, our pruned VGG16
still produced an accuracy change of $+0.27$ which is better than both state-
of-the-art methods and our baseline unpruned VGG16 model. Moreover, when
$\alpha=10$, our pruned model contained only 1.05M parameters, but it had an
accuracy change similar to GhostNet, having 3.30M parameters.

Table 2 also shows the comparison of proposed method with four state-of-the-
art methods on ResNet18 architecture and CIFAR-10 dataset. The state-of-the-
art methods include Prune it Yourself (PIY) [5], Conditional Automated Channel
Pruning (CACP) [23], Lagrangian Relaxation (P-SLR) [18], and PCNN [16]. It can
be seen that the proposed method outperformed all the methods in terms of
accuracy and three out of four state-of-the-art methods in terms of
compression. Table 2 also shows the comparison of proposed method with state-
of-the-art AMC [3] method on ResNet50. Our method at $\alpha=55$ outperforms
it.

Table 2: Comparison of proposed Constrained Reinforcement Learning (CRL)
method with state-of-the-art approaches on VGG16 [24], ResNet18 [25] and
ResNet50 [25] architectures using CIFAR-10 dataset.

|  | Unpruned | Pruned
---|---|---|---
|  | Acc. $\%$ | Params. | Acc. $\%$ | Params. | $\Delta$ Acc. $\%$
|  |  | in millions |  | in millions |
VGG16 | CACP [23] | 93.02 | 14.73 | 92.89 | 4.41 | -0.13
Ghost Net [32] | 93.60 | 14.73 | 92.90 | 3.30 | -0.70
CRL-$\alpha=20\%$ | 90.69 | 14.73 | 90.96 | 2.38 | 0.27
CRL-$\alpha=10\%$ | 90.69 | 14.73 | 89.89 | 1.05 | -0.80
ResNet18 | PIY [5] | 91.78 | 11.18 | 91.23 | 6.11 | -0.55
CACP [23] | 93.02 | 11.68 | 92.03 | 3.50 | -0.99
P-SLR [18] | 93.33 | 11.68 | 90.37 | 1.34 | -2.96
PCNN [16] | 96.58 | 11.20 | 96.38 | 3.80 | -0.20
CRL-$\alpha=30\%$ | 91.82 | 11.68 | 92.09 | 2.91 | 0.27
CRL-$\alpha=20\%$ | 91.82 | 11.68 | 90.97 | 2.52 | -0.85
ResNet50 | AMC [3] | 93.53 | 25.56 | 93.55 | 15.34 | 0.02
CRL-$\alpha=55\%$ | 92.97 | 25.56 | 93.60 | 12.34 | 0.63

##  4 Conclusions

We propose a novel framework for neural network pruning via constrained
reinforcement learning that allows respecting budgets on arbitrary, possibly
non-differentiable functions. Ours is a pro-Lagrangian approach that
incorporates budget constraints by constructing a trust region containing all
policies that respect constraints. Our experiments show that the proposed CRL
strategy significantly outperform the state-of-the-art methods in terms of
producing small and compact while maintaining the accuracy of unpruned
baseline architecture. Specifically, our method reduces nearly
$75.08\%-92.9\%$ parameters without incurring any significant loss in
performance.

## References

  * [1] Y.D. Kim amd E. Park, S.J. Yoo, L. Yang T. Choi, and D.J. Shin,  “Compression of deep convolutional neural networks for fast and low power mobile applications,”  in Int. Conf. on Learning Representations, 2016.
  * [2] M. Tan and Q.V. Le,  “EfficientNet: Rethinking model scaling for convolutional neural networks,”  in Int. Conf. on Machine Learning, 2019.
  * [3] Y. He, J. Lin, Z. Liu, H. Wang, L. Li, and S. Han,  “AMC: AutoML for model compression and acceleration on mobile devices,”  in European Conf. on Computer Vision. 2018, Springer International Publishing.
  * [4] C. Lemaire, A. Achkar, and P. Jodoin,  “Structured pruning of neural networks with budget-aware regularization,”  in IEEE/CVF Int. Conf. on Computer Vision and Pattern Recognition, 2019.
  * [5] Z. Yan, P. Xing, Y. Wang, and Y. Tian,  “Prune it yourself: Automated pruning by multiple level sensitivity,”  in IEEE Int. Conf. on Multimedia Information Processing and Retrieval, 2020, pp. 73–78.
  * [6] D. Blalock, G. Ortiz, J. Javier, J. Frankle, and J. Guttag,  “What is the state of neural network pruning?,”  in Proceedings of Machine Learning and Systems, I. Dhillon, D. Papailiopoulos, and V. Sze, Eds., 2020, vol. 2, pp. 129–146.
  * [7] M.U. Haider and M. Taj,  “Comprehensive online network pruning via learnable scaling factors,”  in IEEE Int. Conf. on Image Processing, 2021.
  * [8] S. Han, J. Pool, J. Tran, and W. J. Dally,  “Learning both weights and connections for efficient neural network,”  in Advances in Neural Information Processing Systems, 2015.
  * [9] Y. LeCun, J. Denker, and S. Solla,  “Optimal brain damage,”  in Advances in Neural Information Processing Systems. 1990, Morgan-Kaufmann.
  * [10] V. Lebedev and V. Lempitsky,  “Fast convnets using group-wise brain damage,”  in IEEE Int. Conf. on Computer Vision and Pattern Recognition, 2016, pp. 2554–2564.
  * [11] J. Lee, S. Park, and J. Shin,  “Learning bounds for risk-sensitive learning,”  in Advances in Neural Information Processing Systems. 2020, Curran Associates, Inc.
  * [12] X. Dong, S. Chen, and S.J. Pan,  “Learning to prune deep neural networks via layer-wise optimal brain surgeon,”  in Advances in Neural Information Processing Systems. 2017, Curran Associates, Inc.
  * [13] G. Hinton, O. Vinyals, and J. Dean,  “Distilling the knowledge in a neural network,” 2015,  arXiv:1503.02531.
  * [14] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H.P. Graf,  “Pruning filters for efficient convnets,”  in Int. Conf. on Learning Representation, 2017.
  * [15] Z. Huang and N. Wang,  “Data-driven sparse structure selection for deep neural networks,”  in European Conf. on Computer Vision, 2018.
  * [16] Z. Tan, J. Song, and Others,  “PCNN: Pattern-based fine-grained regular pruning towards optimizing cnn accelerators,”  in ACM/IEEE Design Automation Conference, 2020, pp. 1–6.
  * [17] J. Lin, Y. Rao, J. Lu, and J. Zhou,  “Runtime neural pruning,”  in Advances in Neural Information Processing Systems. 2017, Curran Associates, Inc.
  * [18] D. Gurevin, S. Zhou, L. Pepin, B. Li, M. Bragin, C. Ding, and F. Miao,  “Enabling retrain-free deep neural network pruning using surrogate lagrangian relaxation,” 2021.
  * [19] X. Zhao, P.B. Luh, and J. Wang,  “The surrogate gradient algorithm for lagrangian relaxation method,”  in IEEE Conf. on Decision and Control, 1997, vol. 1, pp. 305–310 vol.1.
  * [20] L. Enderich, F. Timm, and W. Burgard,  “Holistic filter pruning for efficient deep neural networks,”  in IEEE/CVF Winter Conf. on Applications of Computer Vision, January 2021, pp. 2596–2605.
  * [21] R.S. Sutton and A.G. Barto,  Reinforcement Learning: An Introduction,  The MIT Press, second edition, 2018.
  * [22] L. P. Kaelbling, M. L. Littman, and A. W. Moore,  “Reinforcement learning: A survey,”  Journal of Artificial Intelligence Research, vol. 4, pp. 237–285, 1996.
  * [23] Y. Liu, Y. Guo, J. Guo, L. Jiang, and J. Chen,  “Conditional automated channel pruning for deep neural networks,”  IEEE Signal Processing Letters, 2021.
  * [24] K. Simonyan and A. Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  in Int. Conf. on Learning Representations, 2015.
  * [25] K. He, X. Zhang, S. Ren, and J. Sun,  “Deep residual learning for image recognition,”  in IEEE Int. Conf. on Computer Vision and Pattern Recognition, 2016\.
  * [26] E. Altman,  Constrained Markov Decision Processes,  Chapman and Hall, 1999.
  * [27] C. Williams and C. Rasmussen,  “Gaussian processes for regression,”  in Advances in Neural Information Processing Systems, D. Touretzky, M. C. Mozer, and M. Hasselmo, Eds. 1996, vol. 8, MIT Press.
  * [28] R.J. Williams,  “Simple statistical gradient-following algorithms for connectionist reinforcement learning,”  Machine Learning, 1992.
  * [29] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,  “Proximal policy optimization algorithms,” 2017,  arXiv:1707.06347.
  * [30] A. Krizhevsky and G. Hinton,  “Learning multiple layers of features from tiny images,”  Tech. Rep., Univ. of Toronto, 2009.
  * [31] D.P. Kingma and J. Ba,  “Adam: A method for stochastic optimization,”  in Int. Conf. for Learning Representations, 2015.
  * [32] K. Han, Y. Wang, and Others,  “Ghostnet: More features from cheap operations,”  in IEEE/CVF Int. Conf. on Computer Vision and Pattern Recognition, 2020.