11institutetext: Beijing University of Posts and
Telecommunications22institutetext: Beijing Key Laboratory of Network System
and Network Culture, China 33institutetext: China Mobile Research Institute
33email<EMAIL_ADDRESS>
33email<EMAIL_ADDRESS>
# Bag of Tricks for Out-of-Distribution Generalization

Zining Chen 11 Weiqiu Wang 11 Zhicheng Zhao 1122 Aidong Men 11 Hong Chen 33

###### Abstract

Recently, out-of-distribution (OOD) generalization has attracted attention to
the robustness and generalization ability of deep learning based models, and
accordingly, many strategies have been made to address different aspects
related to this issue. However, most existing algorithms for OOD
generalization are complicated and specifically designed for certain dataset.
To alleviate this problem, nicochallenge-2022 provides NICO++, a large-scale
dataset with diverse context information. In this paper, based on systematic
analysis of different schemes on NICO++ dataset, we propose a simple but
effective learning framework via coupling bag of tricks, including multi-
objective framework design, data augmentations, training and inference
strategies. Our algorithm is memory-efficient and easily-equipped, without
complicated modules and does not require for large pre-trained models. It
achieves an excellent performance with Top-1 accuracy of 88.16$\%$ on public
test set and 75.65$\%$ on private test set, and ranks $1^{st}$ in domain
generalization task of nicochallenge-2022.

###### Keywords:

Out-of-Distribution Generalization, Domain Generalization, Image Recognition

##  1 Introduction

Deep learning based methods usually assume that data in training set and test
set are independent and identically distributed (IID). However, in real world
scenario, test data may have large distribution shifts to training data,
leading to significant decrease on model performance. Thus, how to enable
models to tackle data distribution shifts and better recognize out-of-
distribution data is a topic of general interest nowadays. Nicochallenge-2022
is a part of ECCV-2022 which aims at facilitating the out-of-distribution
generalization in visual recognition, searching for methods to increase model
generalization ability, and track 1 mainly focuses on Common Context
Generation (Domain Generalization, DG).

Advancements in domain generalization arise from multiple aspects, such as
feature learning, data processing and learning strategies. However, as
distribution shifts vary between datasets, most of the existing methods have
limitations on generalization ability. Especially NICO++ dataset is a large-
scale dataset containing 60 classes in track 1, with hard samples including
different contexts, multi-object and occlusion problems, etc. Therefore, large
distribution shifts between current domain generalization datasets and NICO++
may worsen the effect of existing algorithms.

In this paper, without designs of complicated modules, we systematically
explore existing methods which improve the robustness and generalization
ability of models. We conduct extensive experiments mainly on four aspects:
multi-objective framework design, data augmentations, training and inference
strategies. Specifically, we first compare different ways to capture coarse-
grained information and adopt coarse-grained semantic labels as one of the
objective in our proposed multi-objective framework. Secondly, we explore
different data augmentations to increase the diversity of data to avoid
overfitting. Then, we design a cyclic multi-scale training strategy, which
introduces more variations into the training process to increase model
generalization ability. And we find that enlarging input size is also helpful.
Moreover, we merge logits of different scales to make multi-scale inference
and design weighted Top-5 voting to ensemble different models. Finally, our
end-to-end framework with bag of simple and effective tricks, as shown in
Figure 2, gives out valuable practical guidelines to improve the robustness
and generalization ability of deep learning models. Our solution achieves
superior performance on both public and private test set of domain
generalization task in nicochallenge-2022, with the result of 88.16$\%$ and
75.65$\%$ respectively, and ranks $1^{st}$ in both phases.

##  2 Related Work

###  2.1 Domain Generalization

Domain Generalization aims to enable models to generalize well on unknown-
distributed target domains by training on source domains. Domain-invariant
feature learning develops rapidly in past few years, IRM [1] concentrates on
learning an optimal classifier to be identical across different domains, CORAL
[17] aims at feature alignment by minimizing second-order statistics of source
and target domain distributions. Data processing methods including data
generation (e.g. Generative Adversarial Networks [10]) and data augmentation
(e.g. Rotation) are simple and useful to increase the diversity of data, which
is essential in domain generalization. Other strategies include Fish [15], a
multi-task learning strategy that consists the direction of descending
gradient between different domains. StableNet [24] aims to extract essential
features from different categories and remove irrelevant features and fake
associations by using Random Fourier Feature. SWAD [4] figures out that flat
minima leads to smaller domain generalization gaps and suffers less from
overfitting. Several self-supervised learning methods [12] [7] [3] are also
proposed these years to effectively learn intrinsic image properties and
extract domain-invariant representations. Although these methods make great
progress on domain generalization, most of them are complicatedly designed and
may only benefit on certain dataset.

###  2.2 Fine-grained Classification

Fine-grained Classification aims to recognize sub-classes under main classes.
Difficulty mainly lies in finer granularity for small inter-class variances,
large intra-class similarity and different image properties (e.g. angle of
view, context and occlusion). Attention mechanisms are mainstream of fine-
grained classification which aim at more discriminative foreground features
and suppress irrelevant background information [8] [21] [11]. Also, network
ensemble methods (e.g. Multiple granularity CNN [20]) including dividing
classes into sub-classes or using multi-branch neural networks are proposed.
Meanwhile, high-order fine-grained feature is another aspect, which Bilinear
CNN [13] uses second-order statistics to fuse context of different channels.
However, as fine-grained based methods may have different effects between
networks and several with high computational complexity, we only adopt light-
weight ECA channel attention mechanism in eca-nfnet-l0 backbone network [2]
and SE channel attention mechanism in efficientnet-b4 backbone network [19].

###  2.3 Generalization Ability

Generalization Ability refers to the adaptability of models on unseen data,
which is usually relevant to model overfitting in deep learning based
approaches. Reduce model complexity can avoid model fitting into a parameter
space only suitable for training set. For example, use models with less
parameters and add regularization terms (e.g. L1 and L2 regularization) to
limit the complexity of models [9]. Diverse data distribution can also
increase generalization ability by using abundant data for pre-training (e.g.
Imagenet [6]), applying data augmentation methods [16], and using re-balancing
strategies to virtually set different-distributed dataset [27].

##  3 Challenge Description

###  3.1 Dataset

The data of domain generalization task in nicochallenge-2022 is from NICO++
dataset [25], a novel domain generalization dataset consisting of 232.4k
images for total 80 categories, including 10 common domains and 10 unique
domains for each category. The data in domain generalization task is
reorganized to 88,866 samples for training, 13,907 for public test and 35,920
for private test with 60 categories. While images from most domains are
collected by searching a combination of a category name and a phrase extended
from the domain name, there exists hard samples with multi-target, large
occlusions and different angle of views, as shown in Figure 1.

(a) Multi-target.

(b) Occlusions.

(c) Angle of views.

Figure 1: Hard samples with difficult image properties, such as multi-target,
occlusions and angle of views, which are easily classified incorrectly for
many models.

###  3.2 Task

Track 1 of nicochallenge-2022 is a common context generation competition on
image recognition which aims at facilitating the OOD generalization in visual
recognition, whose contexts of training and test data for all categories are
aligned and domain labels for training data are available. This task is also
known as domain generalization, to perform better generalization ability on
unknown test data distribution. Specifically, its difficulty mainly lies in no
access to target domains with different distributions during training phase.
Thus, the key for this challenge is to improve the robustness and
generalization ability of models based on images with diverse context and
properties in NICO++ dataset.

##  4 Method

Our proposed end-to-end framework is illustrated in Figure 2. Firstly, we
input multi-scale images based on a cyclic multi-scale training strategy and
apply data augmentations to increase the diversity of training data. Then we
adopt efficient and light-weight networks (e.g. eca-nfnet-l0 [2]) as backbones
to extract features and training with our designed multi-objective head, which
can capture coarse-grained and fine-grained information simultaneously.
Finally, during inference stage, we merge logits of different scales and
design weighted Top-5 voting to ensemble different models.

Figure 2: Overview of model framework: Input cyclic multi-scale images with
data augmentation methods into backbone network to extract features; Extracted
Features are fed into multi-objective head to extract coarse-grained and fine-
grained logits; Logits are ensembled through TTA methods to output the final
result of a single model.

###  4.1 Multi-objective Framework Design

To capture multi-level semantic information in images, we propose a multi-
objective framework. Firstly, domain labels provided by NICO++ dataset
naturally contain coarse-grained information and we have considered using them
as auxiliary targets to train the backbone network. However, it worsens the
performance probably because domain labels focus on the context of images,
which may impair the feature learning of foreground objects. Furthermore, we
analyse many bad cases, examples from which are illustrated in Figure 3. We
find that bicycle is misclassified as horse, wheat is misclassified as monkey
and bicycle is misclassified as gun, respectively, which are far from the
correct answer. Therefore, we aim to introduce coarse-grained information to
assist model training. Specifically, we manually divide 60 categories into 4
coarse categories according to their properties, denoted as plant, animal,
vehicle and object as coarse semantic labels and design a coarse classifier to
enable model to learn coarse-grained features. The output dimension is the
number of coarse categories, 4, while the output dimension of fine-grained
classifier is the number of classes, 60. Under this circumstances, our network
can utilize various information from multi-objective, thus increasing
robustness and generalization ability of backbone network with barely no
computational consumption.

Figure 3: Examples of bad cases which are easily misclassified as ridiculous
results.

Besides, we also have explored self-supervised objective to increase the
generalization ability of models. However, due to large GPU memory consumption
and little improvement on test accuracy, we leave detailed self-supervised
task design as future work in domain generalization.

###  4.2 Data Augmentation

Data Augmentation is one of the most significant series of methods in domain
generalization, for its simplicity but effectiveness on increasing diversity
of data. During training stage, except for common augmentations such as
resized-crop, horizontal-flip and normalization, we perform multiple
combinations of augmentations and find that the combination of Random
Augmentation [5], Cutmix [22] and Mixup [23] and Label Smoothing [18] is the
most effective one.

####  4.2.1 Random Augmentation [5]

Random Augmentation aims to solve the problem of Auto Augmentation for its
high computational cost for the separate search phase on a proxy task, which
may be sub-optimal for divergent model and dataset size. It reduces search
space for data augmentation and propose one with only 2 hyper-parameters. In
this challenge, we set magnitudes, the strength of transformation to 9 and the
mean standard deviation to 0.5.

####  4.2.2 Cutmix [22] and Mixup [23]

Cutmix randomly selects two training images and cuts them into patches with
the scale of $\sqrt{1-\gamma}$ on height and width. Then patches from one
sample are pasted to another while labels are transformed into one-hot format
and mixed proportionally to the area of patches. Mixup also randomly selects
two training images and mix them pixel-wise and label-wise with a random
number $\lambda$. Both $\gamma$ and $\lambda$ are random numbers, calculated
from Beta distribution. In this challenge, we apply these two methods on all
batches, with an alternative probability of 0.5. Also, we set $\gamma$ and
$\lambda$ to 0.4 and 0.4 by empirical practice, where
$\gamma\in\mathbb{[}0,1]$ and $\lambda\in\mathbb{[}0,1]$.

####  4.2.3 Label Smoothing [18]

Hard label is prone to overfitting practically in deep learning based
approaches, and label smoothing was first proposed to change the ground truth
label probability to,

$p_{j}=\left\\{\begin{aligned} &1-\epsilon,&ifj=y_{j},\\\
&\epsilon/(N-1),&otherwise.\\\ \end{aligned}\right.$ (1)

where $\epsilon$ is a constant, $N$ denotes the number of classes, $j$ is the
index of class, $y_{j}$ is the index of ground truth for current image. In
this challenge, we set $\epsilon$ to 0.1, where $\epsilon\in\mathbb{[}0,1)$.

####  4.2.4 Others

Except the above methods, we have also exerted Gaussian Blur, Random Erasing
and Image-cut, but fail to improve on public test set probably because of
conflicts and overlaps between augmentations. For example, Image-cut is a data
extension method to cut original images into five images offline, containing
four corners and a center one, which has similar effects with multi-scale
training and five-crop. Random Erasing may conflict with Cutmix and Mixup for
introducing noise on augmented images and Gaussian Blur may impair the quality
of images especially with small objects.

###  4.3 Training Strategy

Different training strategies may lead to severe fluctuations in deep learning
based models. In this section, we propose innovative and effective training
strategies to enhance the process of model training.

####  4.3.1 Cyclic Multi-scale Training

Due to various scales of objects in NICO++ dataset, we employ Cyclic Multi-
scale Training strategy to increase the robustness and generalization ability
of our model. Different from multi-scale strategy in object detection which
applies multi-scale input in each batch, we propose to change the input size
of data periodically for every 5 epochs to better learn representations of
objects at different scales, which is suitable for models without pre-training
and consume less GPU memory. Also, as we figure out that larger scale is
helpful to improve model performance, we set large multi-scales for light-
weight eca-nfnet-l0, and small multi-scales for the rest of backbone networks.

####  4.3.2 Others

Considering the constraints of GPU memory, we adopt gradient accumulation [14]
to increase batch size, which calculates the gradient of a single batch and
accumulate for several steps before the update of network parameters and zero-
reset of gradient. Besides, we have also verified two-stage training
strategies, which CAM [26] is utilized to extract foreground region during
second-stage to fine-tune the model. However, little improvement on test set
with longer fine-tuning epochs is not worthy.

###  4.4 Inference Strategy

Inference strategies consume little computational resources but may increase
model performance significantly with proper design. In this section, we will
introduce our multi-scale inference strategy and weighted Top-5 voting
ensemble method.

####  4.4.1 Test-Time Augmentation

Test-Time Augmentation (TTA) aims to enhance images in test set with proper
data augmentation methods and enable models to make predictions on different
augmented images to improve model performance. Typical TTA methods including
resize, crop, flip, color jitter are used in this challenge, where we first
use resize with an extension of 64 pixels on input size. Then we apply
different crop strategies, five-crop with an additional extension of 32
pixels, center-crop with an additional extension of 64 pixels. Besides, for
center-crop based TTA we use horizontal flip with a probability of 0.5, color
jitter with a scope of 0.4, and conduct fused TTA methods based on above.
Also, we design multi-scale logits ensemble strategy for multi-scale test.
Specifically, we input three different size corresponding to different
networks, and apply average-weighted (AW) and softmax-weighted (SW), two
different ensemble methods to fuse logits of three scales, as shown in Eq. 2
and Eq. 3, respectively. Finally, we compare different TTA combinations to get
the best strategy and remove TTAs contradicting with previous strategies (e.g.
five-crop and Image-cut)

$L_{AW}=[L_{1},L_{2},L_{3}]*[1/3,1/3,1/3]^{T}$ (2)
$L_{SW}=[L_{1},L_{2},L_{3}]*Softmax(Max(L_{1}),Max(L_{2}),Max(L_{3}))^{T}$ (3)

where $L_{AW}$ denotes the ensemble logits by average-weighted method,
$L_{SW}$ denotes the ensemble logits by softmax-weighted method, $L_{i}$
denotes the logits of $i$-th scale after applying TTA methods.

####  4.4.2 Model Ensemble

As diverse model may capture different semantic information due to its unique
architecture, model ensemble methods are used to better utilize different
context of models to make improvement. Logits ensemble and voting are
mainstream methods for their simplicity and efficiency. In this challenge, we
propose weighted Top-5 voting strategy on diverse models. Specifically, we get
the Top-5 class predictions of each model and then assign voting weights for
each prediction according to its rank. The voting weights for the top-5
predictions of each model can be formulated as Eq. 4,

$W_{Top5}=[1,1/2,1/3,1/4,1/5]$ (4)

where $W_{Top5}$ denotes the voting weights from 1-st to 5-th. While voting,
we sum the voting weights for the same class prediction from different models
and finally take the class prediction with the maximum sum of voting weights
as the final result.

##  5 Experiments

###  5.1 Implementation Details

Models are trained on 8 Nvidia V100 GPUs, using AdamW optimizer with cosine
annealing scheduler. Learning rate is initialized to $1e^{-3}$ for 300 epochs
and weight decay is $1e^{-3}$ for all models. Batch size is 8 and gradient
accumulation is adopted to restrict GPU memory.

###  5.2 Results

Three backbone networks, eca-nfnet-l0, eca-nfnet-l2 and efficientnet-b4 are
trained with cyclic multi-scale training strategy to enrich the diversity of
models for better ensemble results. Data augmentations including Cutmix and
Mixup, Random Augmentation and Label Smoothing are adopted with empirical
hyper-parameters to get diverse training data. With inference strategies of
TTA and model ensemble, including multi-scale logits ensemble, five-crop and
weighted Top-5 voting, we further improve test set performance. During phase 1
on public test set, the evaluation metric is Top-1 accuracy, and finally we
rank $1^{st}$ with a result of 88.16$\%$. The results are shown in Table 1.

Model | Input size | Top-1 Accuracy
---|---|---
eca-nfnet-l0 | Multi-scale(large) | 86.87$\%$
eca-nfnet-l2 | Multi-scale(small) | 86.55$\%$
efficientnet-b4 | Multi-scale(small) | 81.43$\%$
ensemble |  | 88.16$\%$
Table 1: Top-1 accuracy of models on public test set. Multi-scale(small)
indicates input size as (448, 384, 320), Multi-scale(large) indicates input
size as (768, 640, 512).

###  5.3 Ablation Studies

We conduct ablation studies to demonstrate the effectiveness of our methods on
multi-objective framework design, data augmentations, training and inference
strategies, illustrated in Table 2. These methods are added to Baseline step-
by-step and effectively improve performance on public test set with negligible
computational resources. For example, Cutmix and Mixup improves 12.19$\%$,
Multi-scale(small) improves 7.72$\%$ and Multi-scale(large) further improves
2.55$\%$. Except the above methods, other strategies basically improve
performance for around 2$\%$ without any mutual conflicts.

Besides, as mentioned above in Section 4.3, Image-cut has similar effects with
multi-scale training and five-crop. Thus, when applying them together, Image-
cut can only further improve accuracy of 0.1$\%$ with weighted Top-5 voting
strategy. CAM based approach can improve accuracy of 0.4$\%$ but it requires
second-stage training, consuming extra 60 epochs. Therefore, we exclude it
from our framework for simplicity. However, the local feature view of Image-
cut and the object-sensitive features of CAM based methods are still worth to
be explored in future research.

Furthermore, we apply several recent state-of-the-art domain generalization
methods, including CORAL, SWAD and StableNet, but they decrease the
performance by 0.98$\%$, 2.41$\%$, 1.24$\%$ respectively. It further
demonstrates that existing algorithms on domain generalization may only
benefit on certain dataset and perform worse than heuristic data
augmentations.

Methods | Top-1 Accuracy
---|---
Baseline | 56.61$\%$
+Multi-scale(small) | 64.33$\%$
+Cutmix and Mixup | 76.52$\%$
+Random Augmentation | 78.92$\%$
+Test-time Augmentation | 80.53$\%$
+Multi-objective Framework | 81.53$\%$
+Multi-scale(large) - Multi-scale(small) | 84.08$\%$
+Longer Epochs | 86.87$\%$
+Model Ensemble | 88.16$\%$
Table 2: Ablation studies on different strategies. Baseline indicates a
classic eca-nfnet-l0 backbone network. Except for Model Ensemble, the backbone
network of all other strategies is eca-nfnet-l0, and $+$ denotes adding the
method based on the previous experimental settings, while $-$ denotes removing
the method from the previous experimental settings.

##  6 Conclusions

In this paper, we comprehensively analyse bag of tricks to tackle image
recognition on domain generalization. Methods including multi-objective
framework design, data augmentations, training and inference strategies are
shown to be effective with negligible extra computational resources. By
exerting these methods in a proper way to avoid mutual conflicts, our end-to-
end framework consumes low-memory usage, but largely increases robustness and
generalization ability, which achieves a significantly high accuracy of
88.16$\%$ on public test set and 75.65$\%$ on private test set, and ranks
$1^{st}$ in domain generalization task of nicochallenge-2022.

##  7 Acknowledgments

This work is supported by Chinese National Natural Science Foundation
(62076033), and The Key R$\&$D Program of Yunnan Province
(202102AE09001902-2).

## References

  * [1] Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019)
  * [2] Brock, A., De, S., Smith, S.L., Simonyan, K.: High-performance large-scale image recognition without normalization. In: International Conference on Machine Learning. pp. 1059–1071. PMLR (2021)
  * [3] Carlucci, F.M., D’Innocente, A., Bucci, S., Caputo, B., Tommasi, T.: Domain generalization by solving jigsaw puzzles. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2229–2238 (2019)
  * [4] Cha, J., Chun, S., Lee, K., Cho, H.C., Park, S., Lee, Y., Park, S.: Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems 34, 22405–22418 (2021)
  * [5] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 702–703 (2020)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [7] Feng, Z., Xu, C., Tao, D.: Self-supervised representation learning from multi-domain data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3245–3255 (2019)
  * [8] Fu, J., Zheng, H., Mei, T.: Look closer to see better: Recurrent attention convolutional neural network for fine-grained image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4438–4446 (2017)
  * [9] Goodfellow, I., Bengio, Y., Courville, A.: Regularization for deep learning. Deep learning pp. 216–261 (2016)
  * [10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. Advances in neural information processing systems 27 (2014)
  * [11] Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7132–7141 (2018)
  * [12] Kim, D., Yoo, Y., Park, S., Kim, J., Lee, J.: Selfreg: Self-supervised contrastive regularization for domain generalization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9619–9628 (2021)
  * [13] Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear cnn models for fine-grained visual recognition. In: Proceedings of the IEEE international conference on computer vision. pp. 1449–1457 (2015)
  * [14] Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016)
  * [15] Shi, Y., Seely, J., Torr, P.H., Siddharth, N., Hannun, A., Usunier, N., Synnaeve, G.: Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 (2021)
  * [16] Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of big data 6(1), 1–48 (2019)
  * [17] Sun, B., Saenko, K.: Deep coral: Correlation alignment for deep domain adaptation. In: European conference on computer vision. pp. 443–450. Springer (2016)
  * [18] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [19] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [20] Wang, D., Shen, Z., Shao, J., Zhang, W., Xue, X., Zhang, Z.: Multiple granularity descriptors for fine-grained categorization. In: Proceedings of the IEEE international conference on computer vision. pp. 2399–2406 (2015)
  * [21] Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., Tang, X.: Residual attention network for image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3156–3164 (2017)
  * [22] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)
  * [23] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [24] Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y., Shen, Z.: Deep stable learning for out-of-distribution generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5372–5382 (2021)
  * [25] Zhang, X., He, Y., Xu, R., Yu, H., Shen, Z., Cui, P.: Nico++: Towards better benchmarking for domain generalization (2022)
  * [26] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)
  * [27] Zhou, B., Cui, Q., Wei, X.S., Chen, Z.M.: Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9719–9728 (2020)


# DropNet: Reducing Neural Network Complexity via Iterative Pruning

John Tan Chong Min  Mehul Motani

###### Abstract

Modern deep neural networks require a significant amount of computing time and
power to train and deploy, which limits their usage on edge devices. Inspired
by the iterative weight pruning in the Lottery Ticket Hypothesis (Frankle &
Carbin, 2018), we propose DropNet, an iterative pruning method which prunes
nodes/filters to reduce network complexity. DropNet iteratively removes
nodes/filters with the lowest average post-activation value across all
training samples. Empirically, we show that DropNet is robust across diverse
scenarios, including MLPs and CNNs using the MNIST, CIFAR-10 and Tiny ImageNet
datasets. We show that up to 90% of the nodes/filters can be removed without
any significant loss of accuracy. The final pruned network performs well even
with reinitialization of the weights and biases. DropNet also has similar
accuracy to an oracle which greedily removes nodes/filters one at a time to
minimise training loss, highlighting its effectiveness.

Machine Learning, Pruning, Iterative Pruning, ICML

##  1 Introduction

The surprising effectiveness of neural networks in domains such as image
recognition in ImageNet (Russakovsky et al., 2015) has inspired much recent
research. Current state-of-the-art deep models include Transformers (Vaswani
et al., 2017) for language modelling, InceptionNet (Szegedy et al., 2015) for
image modelling, and ResNets (He et al., 2016) which include over 100 layers.
In fact, the number of configurable parameters per model has risen
significantly from hundreds to tens of millions. The increased computational
complexity required for modern neural networks has made deployment in edge
devices challenging.

Previous Work on Complexity Reduction: Current methods of reducing complexity
include quantization to 16-bit (Gupta et al., 2015), group L1 or L2
regularization (Alemu et al., 2019), node pruning (Castellano et al., 1997;
Zhang & Qiao, 2010; Alvarez & Salzmann, 2016; Wen et al., 2016), filter
pruning for CNNs (Li et al., 2016; Wen et al., 2016; Alvarez & Salzmann, 2016;
He et al., 2017; Liu et al., 2017), weight pruning using magnitude-based
methods (Han et al., 2015) or second-order Hessian-based methods such as
Optimal Brain Damage (LeCun et al., 1990) or Optimal Brain Surgeon (Hassibi et
al., 1993).

Previous Work on Node/Filter Pruning: For node pruning, previous work includes
introducing regularization terms based on input weights using group lasso to
the loss function (Alvarez & Salzmann, 2016; Wen et al., 2016) and selecting
the least important node to drop based on mutual information (Zhang & Qiao,
2010). In CNNs, previous work includes layer-wise pruning such as layer-wise
lasso regression on filters (He et al., 2017), and global pruning methods such
as pruning filters with the lowest sum of absolute input weights globally (Li
et al., 2016), using second-order Taylor expansion to prune unimportant
filters (Molchanov et al., 2016; Lin et al., 2018), introducing structured
sparsity using a particle filter approach (Anwar et al., 2017), or to
repeatedly perform an L1 regularization of the batch normalization layer’s
scaling factor in CNNs (Liu et al., 2017).

Previous Work on Iterative Pruning: Most techniques that can perform one-shot
pruning can also be applied recursively using iterative pruning. It has been
shown that iterative pruning achieves better performance than one-shot pruning
(Li et al., 2016). More notably, iterative pruning with reinitialization can
reduce parameter counts by over 90% (Frankle & Carbin, 2018). Such iterative
pruning methods shed new insight into how more effective pruning approaches
might be developed.

Comparison with Similar Work: DropNet iteratively removes nodes/filters with
the lowest average post-activation values across all training samples. Similar
work to ours includes removing nodes with the highest average percentage of
zero activation values across a validation set - Average Percentage of Zeros
(APoZ) (Hu et al., 2016), which measures sparsity of a node’s activations. In
contrast, DropNet (i) utilizes average magnitude of post-activation values,
and (ii) does so over the training set. Another similar metric is to prune
channels to a filter using variance of post-activation values (Polyak & Wolf,
2015). DropNet instead prunes the entire filter using average post-activation
values.

Comparison with Dropout: Dropout (Srivastava et al., 2014) randomly drops a
fraction $p$ of nodes during training, but keeps the entire network intact
during test time. DropNet, however, drops nodes/filters permanently during
training time and test time.

Our Contributions:

  * •

We propose DropNet, an iterative node/filter pruning approach with
reinitialization, which iteratively removes nodes/filters with the lowest
average post-activation value across all training samples (either layer-wise
or globally) and, hence, reduces network complexity. To the best of our
knowledge, our method is the first to prune both MLPs and CNNs based on the
average post-activation values of nodes/filters, which utilizes both the
information about the input weights as well as the input data to make an
informed selection of the nodes/filters to drop.

  * •

DropNet achieves a robust performance across a wide range of scenarios
compared to several benchmark metrics. DropNet achieves similar performance to
an oracle which greedily removes nodes/filters one at a time to minimise
training loss.

  * •

DropNet does not require special initialization of weights and biases (unlike
(Frankle & Carbin, 2018)). It is shown in subsequent experiments that a random
initialization of the pruned model will do just as well as original
initialization. This means the architecture pruned by DropNet can be readily
deployed using off-the-shelf machine learning libraries and hardware.

##  2 DropNet Algorithm

In this section, we describe the DropNet algorithm and discuss its properties.

Similar to the Lottery Ticket Hypothesis (Frankle & Carbin, 2018), which
iteratively drops weights with reinitialization, DropNet applies iterative
dropping for nodes/filters with reinitialization.

Model: Consider a dense feed-forward neural network $f(x;n)$ with initial
configuration of weights and biases $\theta=\theta_{0}$ and initial
configuration of nodes/filters $n$. Let $f$ reach minimum validation loss $l$
with test accuracy $a$ when optimizing with stochastic gradient descent (SGD)
on a training set. Consider also training $f(x;m\odot n)$ with a mask
$m\in\\{0,1\\}^{|n|}$ on its nodes/filters such that its initialization is
$f(x;m\odot n)$, where $m\odot n$ is an element-wise multiplication between
$m$ and $n$. Let $f$ with the mask reach minimum validation loss $l^{\prime}$
with with test accuracy $a^{\prime}$.

Problem: Find a subnetwork $f(x;m\odot n)$ such that $a^{\prime}\approx a$
(similar accuracy) and $||m||_{0}\ll n$ (fewer parameters).

Algorithm: The proposed iterative pruning algorithm is shown in Algorithm 1.
DropNet applies Algorithm 1 with the following metric: the lowest average
post-activation value across all training samples (either layer-wise or
globally). An example of the training cycle using Algorithm 1 is shown in Fig.
1.

Algorithm 1 Iterative Pruning Algorithm

Input: Neural network with initial state $\theta_{0}$, inital nodes/filters
$n$, pruning metric.

Hyperparameters: Training iterations $j$, pruning fraction $p\in(0,1]$,
maximum loss factor $\kappa\in(0,1]$

Initialize starting mask $m=\\{1\\}^{|n|}$

repeat

1\. Revert network to initial state $\theta_{0}$

2\. Apply mask to nodes/filters: $f(x;m\odot n)$

3\. Train network for at most $j$ iterations until early stopping

4\. Apply pruning metric to choose a fraction $p$ of nodes/filters to drop and
update $m$

until validation accuracy $a^{\prime}\leq\kappa a$

Run steps 1 to 3

Figure 1: Illustration of DropNet algorithm as described in Algorithm 1. The
mask is initially set to all 1s, meaning all nodes/filters are present. As the
training cycle progresses, more and more nodes/filters are dropped. The final
model at cycle 2 is then reset to initial state and retrained to give the
final parameters.

Expected Absolute Value of a Node: Unlike weights, which can be of the same
value for all training samples, nodes will change their post-activation values
according to the input training samples. Hence, we use a node’s expected
absolute value across all training samples ($x_{1},x_{2},...,x_{t}$) to
evaluate its importance. For each node $a_{i},i\in\mathbb{Z^{+}},1\leq i\leq
n$, out of all $n$ nodes in the network, we have its expected absolute value
as:

$E(a_{i})=\frac{1}{t}\sum_{j=1}^{t}|V(a_{i}|x_{j})|,$

where $V(a_{i}|x_{j})$ is the post-activation value of the node $a_{i}$ with
input sequence $x_{j}$.

Expected Absolute Value of a Filter: For CNNs, the filters are comprised of a
set of constituent nodes. Choosing which filter to drop is thus equivalent to
choosing a set of constituent nodes to drop. Hence, in order to evaluate the
importance of each filter $f_{i},i\in\mathbb{Z^{+}},1\leq i\leq n$, out of all
$n$ filters in the network, we take the average of the expected absolute value
of all its constituent nodes $a_{1},a_{2},...,a_{r}$, $r\in\mathbb{Z}^{+}$.
That is:

$E(f_{i})=\frac{1}{r}\sum_{j=1}^{r}|E(a_{j})|,$

where $E(f_{i})$ is the expected absolute value of the filter $f_{i}$ across
all training samples $x_{1},x_{2},...,x_{t}$.

Intuition: We propose two reasons for the competitiveness of dropping
nodes/filters with lowest average post-activation value across all training
samples. (i) Firstly, Rectified Linear Unit ($ReLU$) activation will lead to
inactive nodes which do not “fire” once the value of node reaches 0 or below.
Hence, if a node does not fire most of the time, it will have a low expected
absolute value and removing it will affect only a small amount of
classifications when its post-activation value is non-zero. (ii) Secondly,
during backpropagation, the input weights of a node with low expected absolute
value will be updated by only a small amount, which means that the node is
less adaptive to learning from the inputs. Removing these less adaptive nodes
should impact classification accuracy less than removing more adaptive ones.

##  3 Methodology

In order to evaluate the effectiveness of the DropNet algorithm, we test it
empirically using MLPs and CNNs on MNIST (LeCun et al., 2010), CIFAR-10
(Krizhevsky et al., 2009) and Tiny ImageNet (taken from https://tiny-
imagenet.herokuapp.com, results in Supplementary Material) datasets.

###  3.1 Pruning Metrics

The pruning metrics we consider are listed in Table 1. At the end of each
training cycle in Algorithm 1, we use the metric to evaluate the importance
score for each node/filter, and drop the nodes/filters with the lowest
importance scores. Note that ties are broken randomly.

Table 1: Pruning Metrics
Metric | Importance Score | Type
---|---|---
minimum |  $E(a_{i})$ or $E(f_{i})$ | Global
maximum |  $-E(a_{i})$ or $-E(f_{i})$ | Global
random | 0 | Global
minimum_layer |  $E(a_{i})$ or $E(f_{i})$ | Layer-wise
maximum_layer |  $-E(a_{i})$ or $-E(f_{i})$ | Layer-wise
random_layer | 0 | Layer-wise

The minimum, maximum and random metrics prune a fraction $p$ of nodes/filters
globally. The minimum metric drops a fraction $p$ the nodes/filters globally
with the lowest post-activation values. The maximum metric serves as a
comparison to the minimum metric to compare the effectiveness of the metric.
The random metric, which prunes a fraction $p$ of nodes randomly, serves as a
control.

We consider layer-wise pruning metrics. These metrics are termed
minimum_layer, maximum_layer and random_layer, and they prune a fraction $p$
of nodes/filters layer-wise.

DropNet utilizes Algorithm 1 with either the minimum or minimum_layer metrics.
As will be shown in our experimental results, these metrics prove to be quite
competitive for different scenarios.

###  3.2 Experiment Details

Train-Validation-Test Split: For MNIST, the dataset is split into 54000
training, 6000 validation and 10000 testing samples. For CIFAR-10, the dataset
is split into 45000 training, 5000 validation and 10000 testing samples.

Pre-processing: The input pixel values are scaled to be between 0 and 1.

Activation Function: The model activation functions are all ReLU, except the
final classification layer where it is softmax in order to choose one out of
multiple classes.

Optimization Function: The optimization function used is SGD with a learning
rate of 0.1.

Loss Function: The loss function used is cross entropy.

Training Runs: The experiments are repeated over 15 runs, each with a
different initial random seed. To serve as comparison between the various
metrics, the average accuracy against the fraction of nodes/filters remaining
across all 15 runs are plotted, together with the error bars denoting the 95%
confidence interval.

Training Cycles: The masks are applied at the start of each training cycle,
which comprises 100 epochs, with early stopping using validation loss with
patience of 5 epochs. Over each training cycle, a fraction $p=0.2$ of the
nodes are dropped.

Model A

Model B

Model C

Figure 2: Models of the neural network architectures considered. Varying
numbers of initial hidden nodes/filters are used in different experiments, but
the baseline architectures remain the same. The layers where the masks are
applied are written with a postfix ‘[Mask]’, and shown in orange. The CNN
models (Model B and C) are variants of the VGG architecture (Simonyan &
Zisserman, 2014). Model A: FC40 - FC40 (Left): This is a network with two
fully-connected (FC) hidden layers, each with 40 nodes. The mask is applied
after each hidden layer. Model B: Conv64 - Conv64 (Middle): This is a network
with a two 2D convolutional layers each comprising of 64 filters of size 3x3
with ’same’ padding. The mask is applied after the convolutional layer and
before the MaxPooling2D layer. Model C: Conv64 - Conv64 - Conv128 - Conv128
(Right): This is a network with a four 2D convolutional layers. The first two
convolutional layers have 64 filters, while the next two convolutional layers
have 128 filters. The filter is of size 3x3 with ’same’ padding. The mask is
applied after the convolutional layer and before the MaxPooling2D layer.

Figure 3: Test accuracy vs. fraction of nodes remaining for various metrics in
Model A: FC40 - FC40 on MNIST

Figure 4: Test accuracy vs. fraction of nodes remaining for various metrics in
Model A: FC20 - FC40 on MNIST

Figure 5: Test accuracy vs. fraction of nodes remaining for various metrics in
Model A: FC40 - FC20 on MNIST

Figure 6: Test accuracy vs. fraction of filters remaining for various metrics
in Model B: Conv64 - Conv64 on MNIST

Figure 7: Test accuracy vs. fraction of filters remaining for various metrics
in Model B: Conv32 - Conv64 on MNIST

Figure 8: Test accuracy vs. fraction of filters remaining for various metrics
in Model B: Conv64 - Conv32 on MNIST

###  3.3 Model Details

The experiments are performed on a variety of network models. We use three
types of feed-forward architectures: two fully-connected (FC) hidden layers of
nodes (Model A), two 2D convolutional (Conv) layers (Model B) and four 2D
convolutional layers (Model C). The model architectures are detailed in Fig.
2.

##  4 Results

###  4.1 MLP - MNIST

Q1. Can DropNet perform robustly well on MLPs of various starting
configurations?

To address this question, we conduct the experiments on different versions of
Model A on MNIST, listed as follows:

  1. 1.1)

Model A: FC40 - FC40. The plot of test accuracy against fraction of nodes
remaining for various metrics is shown in Fig. 5.

  2. 1.2)

Model A: FC20 - FC40. The plot of test accuracy against fraction of nodes
remaining for various metrics is shown in Fig. 5.

  3. 1.3)

Model A: FC40 - FC20. The plot of test accuracy against fraction of nodes
remaining for various metrics is shown in Fig. 5.

As the trends for training, validation, and test accuracies are similar, we
only show the plots for test accuracy.

For 1.1), it can be seen (Fig. 5) that the minimum_layer performs the best,
followed closely by minimum, then random_layer, maximum_layer, random and
lastly maximum. The maximum metric performs poorly when the fraction of nodes
remaining is 0.5 and below.

For 1.2), it can be seen (Fig. 5) that the minimum metric performs the best,
followed by the layer-wise and random metrics, and lastly the maximum metric.

For 1.3), it can be seen (Fig. 5) that the minimum and minimum_layer are both
competitive, followed by random_layer, random, maximum_layer, and lastly
maximum metric. minimum performs well for all fractions of nodes remaining
except between 0.1 and 0.3 where minimum_layer performs slightly better. The
maximum metric can be seen to be consistently poor when the fraction of nodes
remaining is 0.6 and below.

Evaluation: The results indicate that, for fully connected networks, when the
hidden layer sizes are equal (i.e., no bottleneck layer), the minimum_layer
metric is competitive. When the hidden layer sizes are unequal (i.e., there
potentially exists a bottleneck layer), the minimum metric is competitive.
This shows that minimum and minimum_layer are competitve metrics for Model A.
Using DropNet, we are able to reduce the number of nodes by 60% or more
without significantly affecting model accuracy, highlighting its effectiveness
in reducing network complexity.

###  4.2 CNN - MNIST

Q2. Can DropNet perform robustly well on CNNs of various starting
configurations?

To address this question, we conduct experiments on different versions of
Model B on MNIST, listed as follows:

  1. 2.1)

Model B: Conv64 - Conv64. The plot of test accuracy against fraction of
filters remaining for various metrics is shown in Fig. 8.

  2. 2.2)

Model B: Conv32 - Conv64. The plot of test accuracy against fraction of
filters remaining for various metrics is shown in Fig. 8.

  3. 2.3)

Model B: Conv64 - Conv32. The plot of test accuracy against fraction of
filters remaining for various metrics is shown in Fig. 8.

As the trends for training, validation, and test accuracies are similar, we
only show the plots for test accuracy.

For 2.1), it can be seen (Fig. 8) that the minimum_layer metric performs the
best, followed by random_layer, minimum, random, maximum_layer, and lastly
maximum metric. The maximum metric can be seen to be consistently poor when
the fraction of filters remaining is 0.3 and below.

For 2.2), it can be seen (Fig. 8) that the minimum metric performs the best,
followed by minimum_layer, random_layer, random, maximum_layer, and lastly
maximum metric. The maximum metric can be seen to be consistently poor when
the fraction of filters remaining is 0.5 and below. The random metric is in
between the performance of the minimum and maximum metrics.

For 2.3), it can be seen (Fig. 8) that the minimum metric performs the best,
followed by minimum_layer, random, random_layer, maximum_layer, and lastly
maximum metric. The maximum metric can be seen to be consistently poor when
the fraction of filters remaining is 0.2 and below. The performance of the
random metrics is in between the performance of the minimum and maximum
metrics.

Evaluation: The findings are similar to Section 4.1. The results indicate that
for convolutional layers, when the hidden layer sizes are equal (i.e., no
bottleneck layer), the minimum_layer metric is competitive. When the hidden
layer sizes are unequal (i.e., there potentially exists a bottleneck layer),
the minimum metric is competitive. This shows that minimum and minimum_layer
are competitive metrics for Model B. Using DropNet, we are able to reduce the
number of filters by 90% or more without significantly affecting model
accuracy, highlighting its effectiveness in reducing network complexity.

Figure 9: Test accuracy vs. fraction of filters remaining for various metrics
in Model C: Conv64 - Conv64 - Conv128 - Conv128 on CIFAR-10

Figure 10: Test accuracy vs. fraction of filters remaining for various metrics
in Model C: Conv128 - Conv128 - Conv256 - Conv256 on CIFAR-10

Figure 11: Test accuracy vs. fraction of filters remaining for various metrics
in ResNet18 on CIFAR-10

Figure 12: Test accuracy vs. fraction of filters remaining for various metrics
in VGG19 on CIFAR-10

###  4.3 CNN - CIFAR-10: Model C

Q3. Can DropNet perform well on a larger dataset like CIFAR-10?

To address this question, we conduct an experiment using Model C on CIFAR-10,
listed as follows:

  1. 3.1)

Model C: Conv64 - Conv64 - Conv128 - Conv128. The plot of test accuracy
against fraction of filters remaining for various metrics are shown in Fig A7.

  2. 3.2)

Model C: Conv128 - Conv128 - Conv256 - Conv256. The plot of training accuracy
and test accuracy against fraction of filters remaining for various metrics
are shown in Fig. 10.

As the trends for training, validation, and test accuracies are similar, we
only show the plots for test accuracy.

It can be seen (Figs. A7 and 10) that the minimum_layer metric performs the
best, followed by minimum, random_layer, random, and maximum_layer and lastly
maximum metric. The minimum and minimum_layer perform equally well when the
fraction of filters remaining is 0.5 and above. The maximum metric can be seen
to be consistently poor when the fraction of filters remaining is 0.5 and
below. The random metric is in between the minimum and maximum metrics.

Evaluation: The results show that minimum and minimum_layer are both
competitive when less than of half of the filters are dropped. Thereafter,
minimum_layer performs significantly better. Using DropNet, we can reduce the
number of filters by 50% or more without significantly affecting model
accuracy, highlighting its effectiveness in reducing network complexity.

The results indicate that, for larger convolutional models like Model C,
global pruning methods like the minimum metric may not be as competitive as
layer-wise pruning methods like the minimum_layer metric.

###  4.4 CNN - CIFAR-10: ResNet18/VGG19

Q4. Can DropNet perform robustly well on even larger models such as ResNet18
and VGG19?

To address this question, we conduct an experiment using Algorithm 1 for
ResNet18 and VGG19 on CIFAR-10, both following closely to the implementation
in their respective papers (Simonyan & Zisserman, 2014; He et al., 2016).
Details of the model are in the Supplementary Material. Of note, ResNet18 is
implemented without Batch Normalization, while VGG19 had a Batch Normalization
before every MaxPooling2D layer.

The plot of test accuracy against fraction of filters remaining for various
metrics for ResNet18 and VGG19 is shown in Figs. C9 and C11 respectively.

For ResNet18, it can be seen (Fig. C9) that the minimum_layer metric performs
the best, followed by minimum, then random_layer, random, maximum_layer and
and lastly maximum metric. The minimum_layer and minimum are both competitive.

For VGG19, it can be seen (Fig. C11) that the minimum_layer metric performs
the best, followed by random_layer, random, max_layer, minimum, and and lastly
maximum metric. The minimum_layer metric is the most competitive.

For both ResNet18 and VGG19, the maximum metric can be seen to be consistently
poor when the fraction of filters remaining is 0.5 and below. The
maximum_layer metric is consistently poor when the fraction of filters
remaining is 0.2 and below.

Evaluation: The results show that for larger models, minimum_layer is the most
competitive. It can also be seen that with the exception of minimum and
maximum_layer, the layer-wise metrics outperform the global metrics for larger
models. This shows that there may be significant statistical differences
between layers for larger models such that comparing magnitudes across layers
may not be a good way to prune nodes/filters. That said, minimum_layer can be
seen to perform very well and consistently performs better than random, which
shows promise that it is a good metric.

The minimum metric proves to be almost as competitive as minimum_layer for
ResNet18, but performs worse than random for VGG19. This shows that the skip
connections in ResNet18 help to alleviate some of the pitfalls of global
metrics. Interestingly, the minimum metric tends to prune out some skip
connections completely, which shows that certain skip connections are
unnecessary. This means that DropNet using the minimum metric is able to
automatically identify these redundant connections on its own.

Overall, using DropNet, we can reduce the number of filters by 80% or more
without significantly affecting model accuracy, highlighting its effectiveness
in reducing network complexity even in larger models.

Figure 13: Test accuracy vs. fraction of nodes remaining when compared to an
oracle in Model A: FC20 - FC20 on MNIST

Figure 14: Test accuracy vs. fraction of filters remaining when compared to an
oracle in Model B: Conv20 - Conv20 on MNIST

##  5 Empirical Analysis - Oracle Comparison

Q5. How competitive is the DropNet algorithm compared to an oracle?

There are numerous node/filter pruning methods and algorithms available,
hence, rather than comparing the performance of DropNet with these individual
methods and algorithms, we utilize an oracle in order to establish the
competitiveness of DropNet. We define the oracle as the algorithm which
greedily drops a node/filter out of all remaining node/filters available at
every iteration of Algorithm 1 such that the overall training loss is
minimized. In order to provide a fair comparison with the oracle,
nodes/filters are also pruned one at a time when using the various metrics.

In order to perform a “stress” test on the various metrics compared to the
oracle, we analyze their performance on a smaller scale model, listed as
follows:

  1. 5.1)

Model A: FC20 - FC20. The plot of test accuracy against fraction of nodes
remaining when compared against an oracle on MNIST is shown in Fig. 14.

  2. 5.2)

Model B: Conv20 - Conv20. The plot of test accuracy against fraction of
filters remaining when compared against an oracle on MNIST is shown in Fig.
14.

For 5.1), it can be seen (Fig. 14) that the oracle performs the best, followed
closely by minimum, then random and lastly maximum metrics. The random metric
used here is actually the random_layer metric, as it provides a stronger
baseline performance. Although not shown, minimum_layer has similar
performance to minimum.

For 5.2), it can be seen (Fig. 14) that the oracle, minimum and random perform
equally well. The worst performing is the maximum metric, with poor
performance with 0.4 or less fraction of filters remaining.

Evaluation: The results indicate that overall, the minimum metric is
competitive even to an oracle which minimizes training loss. This shows that
the minimum metric is indeed a competitive criterion to drop nodes/filters.

We note that Algorithm 1 with a pruning metric runs in linear time. The oracle
runs in polynomial time, as it has to iterate through all possible node/filter
selections at the end of each iteration of Algorithm 1.

Figure 15: Test accuracy vs. fraction of nodes remaining for original
initialization and random initialization in Model A: FC20 - FC20 on MNIST

Figure 16: Test accuracy vs. fraction of filters remaining for original
initialization and random initialization in Model B: Conv64 - Conv64 on MNIST

Figure 17: Test accuracy vs. fraction of nodes remaining for original init.
and random init. in Model C: Conv64 - Conv64 - Conv128 - Conv128 on CIFAR-10

Figure 18: Test accuracy vs. fraction of filters remaining for various pruning
fractions $p$ in Model A: FC20 - FC20 on MNIST

Figure 19: Test accuracy vs. fraction of filters remaining for various pruning
fractions $p$ in Model B: Conv64 - Conv64 on MNIST

Figure 20: Test accuracy vs. fraction of filters remaining for various pruning
fractions $p$ in Model C: Conv64 - Conv64 - Conv128 - Conv128 on CIFAR-10

##  6 Empirical Analysis - Random Initialization

Q6. Is the starting initialization of weights and biases important?

We compare the performance of a network retaining its initial weights and
biases $\theta_{0}$ when performing iterative node/filter pruning, as compared
to a network with the pruned architecture but with a random initialization
(randominit). We conduct the following experiments:

  1. 6.1)

Model A: FC20 - FC20. The plot of test accuracy against fraction of nodes
remaining for original and random initialization is shown in Fig. 17.

  2. 6.2)

Model B: Conv64 - Conv64. The plot of test accuracy against fraction of
filters remaining for original and random initialization is shown in Fig. 17.

  3. 6.3)

Model C: Conv64 - Conv64 - Conv128 - Conv128. The plot of test accuracy
against fraction of filters remaining for original and random initialization
is shown in Fig. 17.

It can be seen (Figs. 17, 17, and 17) that unlike the Lottery Ticket
Hypothesis (see Figure 4 in (Frankle & Carbin, 2018)), DropNet does not suffer
from loss of performance when randomly initialized for up to 70% to 80% of the
nodes/filters being dropped.

Evaluation: This means that for DropNet, only the final pruned network
architecture is important, and not the initial weights and biases of the
network. One reason that the original initialization is not important may be
because the learning rate is high enough (0.1) for the network to retrain
sufficiently given just the model architecture. This concurs with the finding
that the ’winning ticket’ in the Lottery Ticket Hypothesis does not confer
significant advantages over random reinitialization if a larger learning rate
of 0.1 is used instead of 0.01 (Liu et al., 2018).

Similar results are also obtained for larger models such as ResNet18 and VGG19
(details in Supplementary Material), which shows that this finding is a
general one.

##  7 Empirical Analysis - Percentage of nodes/filters to Drop

Q7. Can we drop more nodes/filters at a time to reduce number of training
cycles and prune the model faster without affecting accuracy?

We explore this question by analyzing the performance of the minimum metric.
We compare the performance of the model using different values of the pruning
fraction $p=0.2,0.3,0.4,0.5$ and $0.9$. The experiments performed are as
follows:

  1. 7.1)

Model A: FC20 - FC20. The plot of test accuracy against fraction of nodes
remaining for various pruning fractions $p$ on MNIST is shown in Fig. 20.

  2. 7.2)

Model B: Conv64 - Conv64. The plot of test accuracy against fraction of
filters remaining for various pruning fractions $p$ on MNIST is shown in Fig.
20.

  3. 7.3)

Model C: Conv64 - Conv64 - Conv128 - Conv128. The plot of test accuracy
against fraction of nodes remaining for various pruning fractions $p$ on
CIFAR-10 is shown in Fig. 20.

Evaluation: Overall, it can be seen (Figs. 20, 20, and 20) that dropping a
greater fraction $p$ of nodes/filters per training cycle leads to poorer
performance. For one-shot pruning methods which attempt to remove 90% of
nodes/filters, it is not ideal as it generally leads to worse performance.
This further reinforces the competitiveness of iterative pruning as compared
to one-shot pruning.

The results also show that larger $p$ has similar performance when dropping up
to 70% of the nodes/filters. Hence, it may be possible to iterate faster
through Algorithm 1 by adopting a larger $p$ under certain conditions. Further
experiments need to be done to determine the optimal pruning fraction $p$, but
$p=0.2$ seems to be competitive.

##  8 Concluding Remarks

Reflections: In this paper, we propose DropNet, which iteratively drops
nodes/filters and, hence, reduces network complexity. We illustrate how
DropNet can potentially reduce network size by up to 90% without any
significant loss of accuracy. Also, there does not need to be a particular
initialization required when dropping up to 70% of the nodes/filters. DropNet
also has similar performance to an oracle which greedily removes nodes/filters
one at a time to minimise training loss, which shows its competitiveness.

DropNet, utilizing either the minimum or minimum_layer metric, proves to be
competitive over a variety of model architectures. The minimum metric seems to
work robustly well for smaller models such as Models A and B, while the
minimum_layer metric appears to be better for larger models such as Model C,
ResNet18 and VGG19, in the configurations we considered.

We conjecture that one reason for the better performance of the minimum_layer
metric in larger models is that as the number of layers increases, the
statistical properties of the post-activation values of the nodes/filters may
be significantly different in each layer. Hence, using a single metric to
prune globally may not be as good as pruning layer-wise. The exception is when
using skip connections, as empirical results suggest that the minimum metric
is also competitive for larger models for ResNet18. This seems to suggest that
skip connections work well with DropNet.

Additional Experiments: In our Supplementary Material, we show that DropNet is
scalable and achieves similar results on Tiny ImageNet . Furthermore, we also
compare DropNet with another data-driven approach, APoZ, and show that DropNet
outperforms APoZ and can achieve better test accuracy for the same amount of
pruning.

Future Work: To further show DropNet’s generalizability, more experiments can
be done on i) alternative neural network architectures such as RNNs, as well
as ii) other domains such as NLP and reinforcement learning. DropNet has been
shown to work well empirically with $ReLU$ activation functions, and it
remains to be seen whether other metrics may be required for other activation
functions such as $sigmoid$, $tanh$ and $ReLU$ variants.

Source Code: To encourage further research on iterative pruning techniques,
the source code used for our experiments is publicly available at
https://github.com/tanchongmin/DropNet.

Acknowledgements

This research is supported by the National Research Foundation, Singapore
under its AI Singapore Programme (AISG Award No: AISG-GC-2019-002). Any
opinions, findings and conclusions or recommendations expressed in this
material are those of the author(s) and do not reflect the views of National
Research Foundation, Singapore.

## References

  * Alemu et al. (2019) Alemu, H. Z., Zhao, J., Li, F., and Wu, W.  Group $l_{1/2}$ regularization for pruning hidden layer nodes of feedforward neural networks.  _IEEE Access_ , 7:9540–9557, 2019.
  * Alvarez & Salzmann (2016) Alvarez, J. M. and Salzmann, M.  Learning the number of neurons in deep networks.  _Advances in Neural Information Processing Systems_ , (Nips):2270–2278, 2016.  ISSN 10495258.
  * Anwar et al. (2017) Anwar, S., Hwang, K., and Sung, W.  Structured pruning of deep convolutional neural networks.  _ACM Journal on Emerging Technologies in Computing Systems (JETC)_ , 13(3):1–18, 2017.
  * Castellano et al. (1997) Castellano, G., Fanelli, A. M., and Pelillo, M.  An iterative pruning algorithm for feedforward neural networks.  _IEEE transactions on Neural networks_ , 8(3):519–531, 1997.
  * Frankle & Carbin (2018) Frankle, J. and Carbin, M.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  _arXiv preprint arXiv:1803.03635_ , 2018.
  * Gupta et al. (2015) Gupta, S., Agrawal, A., Gopalakrishnan, K., and Narayanan, P.  Deep learning with limited numerical precision.  In _International Conference on Machine Learning_ , pp. 1737–1746, 2015.
  * Han et al. (2015) Han, S., Pool, J., Tran, J., and Dally, W.  Learning both weights and connections for efficient neural network.  In _Advances in neural information processing systems_ , pp. 1135–1143, 2015.
  * Hassibi et al. (1993) Hassibi, B., Stork, D. G., and Wolff, G. J.  Optimal brain surgeon and general network pruning.  In _IEEE international conference on neural networks_ , pp. 293–299. IEEE, 1993.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2017) He, Y., Zhang, X., and Sun, J.  Channel pruning for accelerating very deep neural networks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 1389–1397, 2017.
  * Hu et al. (2016) Hu, H., Peng, R., Tai, Y., Tang, C., and Trimming, N.  A data-driven neuron pruning approach towards efficient deep architectures.  _arXiv preprint arXiv:1607.03250_ , 2016.
  * Krizhevsky et al. (2009) Krizhevsky, A., Hinton, G., et al.  Learning multiple layers of features from tiny images.  2009\.
  * LeCun et al. (1990) LeCun, Y., Denker, J. S., and Solla, S. A.  Optimal brain damage.  In _Advances in neural information processing systems_ , pp. 598–605, 1990.
  * LeCun et al. (2010) LeCun, Y., Cortes, C., and Burges, C.  Mnist handwritten digit database.  _ATT Labs [Online]. Available: http://yann. lecun. com/exdb/mnist_ , 2, 2010.
  * Li et al. (2016) Li, H., Kadav, A., Durdanovic, I., Samet, H., and Graf, H. P.  Pruning filters for efficient convnets.  _arXiv preprint arXiv:1608.08710_ , 2016.
  * Lin et al. (2018) Lin, S., Ji, R., Li, Y., Wu, Y., Huang, F., and Zhang, B.  Accelerating convolutional networks via global & dynamic filter pruning.  In _IJCAI_ , pp. 2425–2432, 2018.
  * Liu et al. (2017) Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., and Zhang, C.  Learning efficient convolutional networks through network slimming.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 2736–2744, 2017.
  * Liu et al. (2018) Liu, Z., Sun, M., Zhou, T., Huang, G., and Darrell, T.  Rethinking the value of network pruning.  _arXiv preprint arXiv:1810.05270_ , 2018.
  * Molchanov et al. (2016) Molchanov, P., Tyree, S., Karras, T., Aila, T., and Kautz, J.  Pruning convolutional neural networks for resource efficient inference.  _arXiv preprint arXiv:1611.06440_ , 2016.
  * Polyak & Wolf (2015) Polyak, A. and Wolf, L.  Channel-level acceleration of deep face representations.  _IEEE Access_ , 3:2163–2175, 2015.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Simonyan & Zisserman (2014) Simonyan, K. and Zisserman, A.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Szegedy et al. (2015) Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1–9, 2015.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  In _Advances in neural information processing systems_ , pp. 5998–6008, 2017.
  * Wen et al. (2016) Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H.  Learning structured sparsity in deep neural networks.  In _Advances in neural information processing systems_ , pp. 2074–2082, 2016.
  * Zhang & Qiao (2010) Zhang, Z. and Qiao, J.  A node pruning algorithm for feedforward neural network based on neural complexity.  In _2010 international conference on intelligent control and information processing_ , pp. 406–410. IEEE, 2010.

##  Appendix A Summary (Supplementary Materials)

ResNet18

VGG19

Figure A1: Architecture of ResNet18 (He et al., 2016) and VGG19 (Simonyan &
Zisserman, 2014) used in the experiments. The layers where the masks are
applied are written with a postfix ‘[Mask]’, and shown in orange. ResNet18
(Left): The network architecture closely follows that of ResNet18. It consists
of several skip-connection blocks which are shown in square brackets. The
model consists of repeated residual blocks comprising two 2D convolutional
layers followed by a MaxPooling2D layer of stride 2. Each 2D convolutional
layer comprises either 64, 128, 256 or 512 filters, of size 3x3 with ’same’
padding. After the multiple residual blocks, the filters are averaged using
GlobalAveragePooling2D before passing into the final fully connected layer
with 10 nodes. The mask is applied after the convolutional layer and before
the MaxPooling2D layer. Batch Normalization is not applied between layers as
the model is found to work well even without it. VGG19 (Right): This is a
network with repeated blocks of 2/4 2D convolutional layers followed by a
MaxPooling2D layer. The 2D convolutional layer comprises either 64, 128, 256
or 512 filters, of size 3x3 with ’same’ padding. The mask is applied after the
convolutional layer and before the MaxPooling2D layer. Batch normalization is
applied right before every MaxPooling2D layer.

In this supplementary material, we present the following:

  1. 1.

Results using more variants of Model C (Figs. A4, A4, A4, A7, A7, A7) on the
CIFAR-10 dataset

  2. 2.

Results using ResNet18 (Figs. C9, C9) and VGG19 (Figs. C11, C11) on the
CIFAR-10 dataset

  3. 3.

Results of random initialization of ResNet18 (Figs. C13, C13) and VGG19 (Figs.
C15, C15) on the CIFAR-10 dataset

  4. 4.

Results using ResNet18 (Figs. C17, C17) and VGG19 (Figs. C19, C19) on the Tiny
ImageNet dataset

  5. 5.

Performance comparison to Average Percentage of Zeros (APoZ) (Hu et al., 2016)
for ResNet18 (Figs. C21, C21) and VGG19 (Figs. C23, C23) on the CIFAR-10
dataset

The results show that DropNet is robust for larger models, and the final
pruned model is able to achieve a similar performance even after
reinitialization. DropNet also has better empirical performance than prior
data-driven approach APoZ and is able to achieve better test accuracy for the
same amount of pruning.

Figure A2: Plot of training accuracy against fraction of filters remaining for
various metrics in Model C: Conv64 - Conv64 - Conv128 - Conv128 on CIFAR-10

Figure A3: Plot of training accuracy against fraction of filters remaining for
various metrics in Model C: Conv128 - Conv128 - Conv128 - Conv128 on CIFAR-10

Figure A4: Plot of training accuracy against fraction of filters remaining for
various metrics in Model C: Conv128 - Conv128 - Conv64 - Conv64 on CIFAR-10

Figure A5: Plot of test accuracy against fraction of filters remaining for
various metrics in Model C: Conv64 - Conv64 - Conv128 - Conv128 on CIFAR-10

Figure A6: Plot of test accuracy against fraction of filters remaining for
various metrics in Model C: Conv128 - Conv128 - Conv128 - Conv128 on CIFAR-10

Figure A7: Plot of test accuracy against fraction of filters remaining for
various metrics in Model C: Conv128 - Conv128 - Conv64 - Conv64 on CIFAR-10

##  Appendix B Methodology

The supplementary experiments performed use the same methodology as the main
paper. In addition, to show DropNet’s scalability, we also perform experiments
on the Tiny ImageNet dataset. We demonstrate how effective pruning using
DropNet can be done on larger models like Model C (Conv4), ResNet18 and VGG19.
For ResNet18 and VGG19, the model architecture follows closely from the
original papers (Simonyan & Zisserman, 2014; He et al., 2016) and are detailed
in Fig. A1.

Algorithm 1, which is used throughout the supplementary material, is detailed
in the main paper.

##  Appendix C Experiments

###  C.1 CNN - CIFAR-10: Model C (Conv4)

Q1. Can DropNet perform robustly well on larger CNNs of various starting
configurations?

To address this question, we conduct an experiment using Algorithm 1 for
various configurations of Model C on CIFAR-10, listed as follows:

  1. 1.1)

Model C: Conv64 - Conv64 - Conv128 - Conv128. The plot of training and test
accuracy against fraction of filters remaining for various metrics are shown
in Figs. A4 and A7 respectively.

  2. 1.2)

Model C: Conv128 - Conv128 - Conv128 - Conv128. The plot of training accuracy
and test accuracy against fraction of filters remaining for various metrics
are shown in Figs. A4 and A7 respectively.

  3. 1.3)

Model C: Conv128 - Conv128 - Conv64 - Conv64. The plot of training accuracy
and test accuracy against fraction of filters remaining for various metrics
are shown in Figs. A4 and A7 respectively.

For 1.1), it can be seen (Figs. A4 and A7) that the minimum_layer metric
performs the best, followed by minimum, random_layer, random, and
maximum_layer and lastly maximum metric. The minimum and minimum_layer perform
equally well when the fraction of filters remaining is 0.3 and above. The
maximum metric can be seen to be consistently poor when the fraction of
filters remaining is 0.5 and below. The random metric is in between the
performance of the minimum and maximum metrics.

For 1.2), it can be seen (Figs. A4 and A7) that the minimum_layer metric
performs the best, followed by random_layer, random, minimum, maximum_layer
and lastly maximum metric. The minimum and minimum_layer perform equally well
when the fraction of filters remaining is 0.3 and above. The maximum metric
can be seen to be consistently poor when the fraction of filters remaining is
0.6 and below.

For 1.3), it can be seen (Figs. A4 and A7) that the minimum_layer metric
performs the best, followed by minimum, random_layer, random, and
maximum_layer and lastly maximum metric. The minimum and minimum_layer perform
equally well when the fraction of filters remaining is 0.4 and above, with the
minimum metric performing significantly better when the fraction of filters
remaining is 0.6 and above, even outperforming the original model accuracy at
some instances. The maximum metric can be seen to be consistently poor when
the fraction of filters remaining is 0.8 and below.

Evaluation: The results show that minimum and minimum_layer are both
competitive when less than of half of the filters are dropped. Thereafter,
minimum_layer performs significantly better. Using DropNet, we can reduce the
number of filters by 50% or more without significantly affecting model
accuracy, highlighting its effectiveness in reducing network complexity.

The results indicate that, for larger convolutional models like Model C,
global pruning methods like the minimum metric are only good at the early
stages of pruning. In fact, for models with non-symmetric layers (see Figs. A4
and A7), minimum works the best when the fraction of filters remaining is 0.5
and above, and may even outperform the original model’s accuracy. We posit
that this is due to the flexibility of global pruning methods to avoid pruning
small layers which pose a bottleneck as compared to layer-wise pruning
methods. That said, not all global pruning methods can do that - maximum and
random do not display such a trend of avoiding bottlenecks.

One further observation is that the train and test data show similar accuracy
trends (the same applies for validation accuracy, although not shown here).
This shows that the train-test-validation split is done well and the general
distribution of the train dataset is similar to that of the test dataset.
Hence, a metric to prune based on the node’s post-activation value such as
DropNet works well using just the post-activation values from the training
data only.

Figure C8: Plot of training accuracy against fraction of filters remaining for
various metrics in ResNet18 on CIFAR-10

Figure C9: Plot of test accuracy against fraction of filters remaining for
various metrics in ResNet18 on CIFAR-10

Figure C10: Plot of training accuracy against fraction of filters remaining
for various metrics in VGG19 on CIFAR-10

Figure C11: Plot of test accuracy against fraction of filters remaining for
various metrics in VGG19 on CIFAR-10

Figure C12: Plot of test accuracy against fraction of filters remaining for
original initialization and random initialization in ResNet18 using pruned
model from minimum metric on CIFAR-10

Figure C13: Plot of test accuracy against fraction of filters remaining for
original initialization and random initialization in ResNet18 using pruned
model from mininum_layer metric on CIFAR-10

Figure C14: Plot of test accuracy against fraction of filters remaining for
original initialization and random initialization in VGG19 using pruned model
from minimum metric on CIFAR-10

Figure C15: Plot of test accuracy against fraction of filters remaining for
original initialization and random initialization in VGG19 using pruned model
from minimum_layer metric on CIFAR-10

###  C.2 CNN - CIFAR-10: ResNet18/VGG19

Q2. Can DropNet perform robustly well on even larger models such as ResNet18
and VGG19?

To address this question, we conduct an experiment using Algorithm 1 for
ResNet18 and VGG19 on CIFAR-10:

  1. 2.1)

ResNet18. The plot of training and test accuracy against fraction of filters
remaining for various metrics is shown in Figs. C9 and C9 respectively.

  2. 2.2)

VGG19. The plot of training and test accuracy against fraction of filters
remaining for various metrics is shown in Figs. C11 and C11 respectively.

For ResNet18, it can be seen (Figs. C9 and C9) that the minimum_layer metric
performs the best, followed by minimum, then random_layer, random,
maximum_layer and and lastly maximum metric. The minimum_layer and minimum are
both competitive.

For VGG19, it can be seen (Figs. C11 and C11) that the minimum_layer metric
performs the best, followed by random_layer, random, max_layer, minimum, and
and lastly maximum metric. The minimum_layer metric is the most competitive.

For both ResNet18 and VGG19, maximum can be seen to be consistently poor when
the fraction of filters remaining is 0.5 and below, while maximum_layer is
consistently poor when the fraction of filters remaining is 0.2 and below.

Evaluation: The results show that for larger models, minimum_layer is the most
competitive. It can also be seen that with the exception of minimum and
maximum_layer, the layer-wise metrics outperform the global metrics for larger
models. This shows that there may be significant statistical differences
between layers for larger models such that comparing magnitudes across layers
may not be a good way to prune nodes/filters. That said, the minimum_layer can
be seen to perform very well and consistently performs better than random,
which shows promise that it is a good metric.

The minimum metric proves to be almost as competitive as minimum_layer for
ResNet18, but performs worse than random for VGG19. This shows that the skip
connections in ResNet18 does help to alleviate some of the pitfalls of global
metrics. Interestingly, the minimum metric tends to prune out some skip
connections completely, which shows that certain skip connections are
unnecessary. This means that DropNet using the minimum metric is able to
automatically identify these redundant connections on its own.

In comparison, it can be seen that the maximum metric performs the worse in
all cases, and shows that filters with high expected absolute post-activate
values are generally important in classification and should not be removed.

The maximum_layer metric on the other hand, performs poorly in ResNet18, but
has comparable performance to the layer-wise metrics in VGG19. This may be due
to the fact that VGG19 in the experiments use a Batch Normalization after
every change of Conv2D filter size, which helps to normalize the post-
activation values and hence, the layer-wise pruning metrics do not differ much
in performance.

Using DropNet, we can reduce the number of filters by 80% or more without
significantly affecting model accuracy, highlighting its effectiveness in
reducing network complexity.

###  C.3 CNN - CIFAR-10: ResNet18/VGG19 (Random Initialization)

Q3. Is the starting initialization of weights and biases important for larger
models such as ResNet18 and VGG19?

We compare the performance of a network retaining its initial weights and
biases $\theta_{0}$ when performing iterative node/filter pruning, as compared
to a network with the pruned architecture but with a random initialization
(randominit). In our experiments, we focus on the pruned architecture produced
by DropNet metrics, namely minimum and minimum_layer. The experiments are
conducted on CIFAR-10, and are detailed as follows:

  1. 3.1)

ResNet18. The plot of test accuracy against fraction of nodes remaining for
original and random initialization using pruned model from minimum metric and
minimum_layer metric respectively are shown in Figs. C13 and C13.

  2. 3.2)

VGG19. The plot of test accuracy against fraction of nodes remaining for
original and random initialization using pruned model from minimum metric and
minimum_layer metric respectively are shown in Figs. C15 and C15.

It can be seen (Figs. C13, C13, C15, C15) that unlike the Lottery Ticket
Hypothesis (see Figure 4 in (Frankle & Carbin, 2018)), DropNet does not suffer
from loss of performance when randomly initialized.

Evaluation: This means that for DropNet, only the final pruned network
architecture is important, and not the initial weights and biases of the
network. This is a pleasant finding as it shows that DropNet can prune a model
down to an ideal structure, from which it can be readily deployed on modern
machine learning libraries.

Figure C16: Plot of training accuracy against fraction of filters remaining
for various metrics in ResNet18 on TinyImageNet

Figure C17: Plot of test accuracy against fraction of filters remaining for
various metrics in ResNet18 on TinyImageNet

Figure C18: Plot of training accuracy against fraction of filters remaining
for various metrics in VGG19 on TinyImageNet

Figure C19: Plot of test accuracy against fraction of filters remaining for
various metrics in VGG19 on TinyImageNet

###  C.4 CNN - Tiny ImageNet: ResNet18/VGG19

Q4. Can DropNet perform well for even larger datasets such as Tiny ImageNet?

In order to show the generalizability of the DropNet algorithm on larger
datasets, we utilize the Tiny ImageNet dataset, which is a smaller-resolution
parallel of the larger ImageNet dataset. This dataset was taken from
https://tiny-imagenet.herokuapp.com/.

Dataset Details: Tiny Imagenet has 200 classes. Each class has 500 training
images, 50 validation images, and 50 test images. Each image has a resolution
of 64 pixels by 64 pixels by 3 channels.

Changes to model parameters: We utilize a similar model for ResNet18 and VGG19
as in the earlier experiment with CIFAR-10. We only modify the models slightly
in order to cater for the 200 output classes of Tiny ImageNet, which is an
increase from the 10 output classes in CIFAR-10. As such, the last linear
layer for ResNet18 has 200 nodes (instead of 10 nodes for CIFAR-10), while the
last two linear layers for VGG19 has 1024 nodes and 200 nodes respectively
(instead of 256 nodes and 10 nodes respectively for CIFAR-10).

Due to the complexity of this dataset, in order to attain better train/test
accuracies, we apply image augmentation to each data sample per epoch. The
image augmentation parameters applied are shown in Table C1.

Table C1: Image Augmentation Parameters
Metric | Value
---|---
Rotation | 40 degree
Width shift range | 0.2
Height shift range | 0.2
Zoom range | 0.2
Shear range | 0.2
Flipping | Horizontal

In order to give the model more time to converge for this larger dataset, we
also increase the number of epochs before early stopping to 10 (instead of 5
in earlier experiments).

Also, in order to reduce experimental running time for Tiny ImageNet, we drop
at each pruning cycle a fraction 0.5 of the filters (instead of 0.2 for
CIFAR-10).

Experiments: We conduct an experiment using Algorithm 1 for ResNet18 and VGG19
on Tiny ImageNet:

  1. 4.1)

ResNet18. The plot of training and test accuracy against fraction of filters
remaining for various metrics is shown in Figs. C17 and C17 respectively.

  2. 4.2)

VGG19. The plot of training and test accuracy against fraction of filters
remaining for various metrics is shown in Figs. C19 and C19 respectively.

For ResNet18, it can be seen (Figs. C17 and C17) that minimum metric and
minimum_layer metrics have the most competitive performance, followed by
random_layer, random, maximum_layer and and lastly maximum metric. The minimum
metric is the most competitive.

For VGG19, it can be seen (Figs. C19 and C19) that the minimum_layer metric
performs the best, followed by random, random_layer, max_layer, aximum, and
and lastly minimum metric. The minimum_layer metric is the most competitive.

For both ResNet18 and VGG19, maximum can be seen to be consistently poor when
the fraction of filters remaining is 0.3 and below. Surprisingly, minimum has
the poorest performance for VGG19.

Evaluation: The results show that for ResNet18, minimum performs the best.
This is similar to the results obtained on the CIFAR-10 dataset (Refer to
Figs. C9 and C9). This could be the fact that the skip connections in ResNet18
allow the pruned model to perform well even if majority of the layer is
removed, and gives greater redundancy for pruning. With such redundancy, some
of the pitfalls of global pruning methods can be alleviated.

For VGG19, minimum_layer performs the best, while minimum performs the worst.
This could again be because there is no redundancy in the layers for VGG19 and
if one layer gets pruned aggressively by a global metric, it might affect
performance negatively. The drop in performance of minimum is worse in the
Tiny ImageNet dataset as compared to the CIFAR-10 dataset (Refer to Figs. C11
and C11) likely because the proportion of filters dropped per pruning cycle is
larger at 0.5 as compared to 0.2, hence there is a greater chance of a layer
getting pruned aggressively.

Similar to the CIFAR-10 experiments, the experiments on the larger Tiny
ImageNet dataset also suggest that the minimum and minimum_layer are both
competitive in ResNet18. For most models, the layer-wise minimum_layer metric
is a general all-round metric to be used.

Using DropNet, we can reduce the number of filters by 50% or more without
significantly affecting model accuracy, highlighting its effectiveness in
reducing network complexity.

Figure C20: Plot of training accuracy against fraction of filters remaining
for DropNet and APoZ in ResNet18 on CIFAR-10

Figure C21: Plot of test accuracy against fraction of filters remaining for
DropNet and APoZ in ResNet18 on CIFAR-10

Figure C22: Plot of training accuracy against fraction of filters remaining
for DropNet and APoZ in VGG19 on CIFAR-10

Figure C23: Plot of test accuracy against fraction of filters remaining for
DropNet and APoZ in VGG19 on CIFAR-10

##  Appendix D Benchmarking against APoZ

Q5. Does DropNet perform better than prior data-driven pruning methods?

While we utilize an oracle for smaller model sizes such as Model A and B, in
order to evaluate the effectiveness of DropNet for larger models, we compare
its performance to a similar data-driven metric known as Average Percentage of
Zeros (APoZ) (Hu et al., 2016).

APoZ measures the percentage of zero activations of a neuron after a ReLU
activation function. The neuron/filter with the highest percentage of zero
activations is considered least important and is pruned first.

In their original paper (Hu et al., 2016), APoZ used a variant of layer-wise
pruning, where they first prune “a few layers with high mean APoZ, and then
progressively trim its neighboring layers”. In order to keep the methodology
consistent to that of DropNet, we adapt the same APoZ metric of percentage of
zero activations of a neuron/filter after a ReLU activation, but use DropNet’s
layer-wise and global-wise iterative pruning approaches as depicted in
Algorithm 1. APoZ using layer-wise pruning is termed apoz_layer, while APoZ
using global pruning is termed apoz. We compare its performance to DropNet’s
layer-wise pruning minimum_layer and global pruning minimum.

We conduct an experiment to compare DropNet and APoZ using Algorithm 1 for
ResNet18 and VGG19 on CIFAR-10:

  1. 5.1)

ResNet18. The plot of training and test accuracy against fraction of filters
remaining for DropNet and APoZ is shown in Figs. C21 and C21 respectively.

  2. 5.2)

VGG19. The plot of training and test accuracy against fraction of filters
remaining for DropNet and APoZ is shown in Figs. C23 and C23 respectively.

For ResNet18, it can be seen (Figs. C21 and C21) that the minimum_layer and
apoz_layer both perform the best, followed closely by minimum, then the apoz.
After a fraction of 0.7 or more filters are pruned, the apoz metric suffers a
huge performance drop.

For VGG19, it can be seen (Figs. C23 and C23) that the minimum_layer performs
the best, followed by apoz_layer, then minimum and finally apoz. After a
fraction of 0.3 or more filters are pruned, the global metrics apoz and
minimum suffer a huge performance drop.

Evaluation: The results show that DropNet in general outperforms APoZ, both
layer-wise and globally. In general, for the same amount of filters pruned,
DropNet achieves higher test accuracy than APoZ. The minimum_layer metric is
consistently the best performing metric across both ResNet18 and VGG19 models,
outperforming apoz_layer. For global metrics, the minimum metric also
consistently outperforms apoz. Notably, while the minimum metric has good
performance for ResNet18, the same performance is not seen in apoz.

This shows that DropNet has merit as a data-driven pruning approach, as it
captures more information about the importance of a particular node/filter
through the use of the expected absolute value. This is an improvement from
APoZ, as it also takes into account the magnitudes of the post-activation
values, rather than just only relying on the percentage of zero activations of
a node/filter.

##  Appendix E Concluding Remarks

The results show that DropNet shows significantly better performance than
random pruning, even for larger models such as ResNet18 and VGG19. DropNet
also manages to achieve higher test accuracy for the same amount of pruning as
compared to prior work APoZ, highlighting its competency. DropNet is a highly-
effective general-purpose pruning algorithm able to work on datasets of
varying sizes such as MNIST, CIFAR-10 and Tiny ImageNet. Overall, DropNet is
able to prune up to 90% or more of nodes/filters without significant loss of
accuracy.

Global or layer-wise pruning: As shown in the main paper, if we are pruning
small models such as Model A or Model B, minimum works well. Furthermore, we
show here that in large models such as Model C, minimum shows promise in
avoiding pruning bottlenecks as compared to its layer-wise counterpart
minimum_layer, and gives significantly better performance if we are just
pruning a small fraction of the original model. However, when pruning even
larger models such as ResNet18 and VGG19, we show that it is better to use
minimum_layer instead. One reason for this may be that the statistical
properties of the post-activation values of each layer may differ
significantly as the model grows large, and a global metric for all the layers
may not work as well. That said, the empirical results of ResNet18 show that
minimum can be competitive as well for these larger models, which suggests
that skip connections may be able to alleviate the pitfalls of global metrics.