11institutetext: National University of Singapore 22institutetext: Huawei
Technologies Ltd.
22email<EMAIL_ADDRESS><EMAIL_ADDRESS>

# Isomorphic Pruning for Vision Models

Gongfan Fang 11 Xinyin Ma 11 Michael Bi Mi 22 Xinchao Wang 11

###### Abstract

Structured pruning reduces the computational overhead of deep neural networks
by removing redundant sub-structures. However, assessing the relative
importance of different sub-structures remains a significant challenge,
particularly in advanced vision models featuring novel mechanisms and
architectures like self-attention, depth-wise convolutions, or residual
connections. These heterogeneous substructures usually exhibit diverged
parameter scales, weight distributions, and computational topology,
introducing considerable difficulty to importance comparison. To overcome
this, we present _Isomorphic Pruning_ , a simple approach that demonstrates
effectiveness across a range of network architectures such as Vision
Transformers and CNNs, and delivers competitive performance across different
model sizes. Isomorphic Pruning originates from an observation that, when
evaluated under a pre-defined importance criterion, heterogeneous sub-
structures demonstrate significant divergence in their importance
distribution, as opposed to isomorphic structures that present similar
importance patterns. This inspires us to perform isolated ranking and
comparison on different types of sub-structures for more reliable pruning. Our
empirical results on ImageNet-1K demonstrate that Isomorphic Pruning surpasses
several pruning baselines dedicatedly designed for Transformers or CNNs. For
instance, we improve the accuracy of DeiT-Tiny from 74.52% to 77.50% by
pruning an off-the-shelf DeiT-Base model. And for ConvNext-Tiny, we enhanced
performance from 82.06% to 82.18%, while reducing the number of parameters and
memory usage. Code is available at https://github.com/VainF/Isomorphic-
Pruning.

###### Keywords:

Network Pruning Vision Transformers CNNs

##  1 Introduction

Figure 1: ImageNet Top-1 Accuracy vs. Multiply–Accumulate Operations (MACs) of
pruned DeiT [51] and pruned ConvNext [33]. The pruned models, marked as
“$\bigstar$” have comparable or better latency yet superior performance
compared to scratch training counterparts highlighted by “$\pentagon$”. The
MACs are in log scale for better visualization

Deep Neural Networks are typically over-parameterized, containing numerous
redundant sub-structures. Such redundancy can be effectively eliminated via
structured pruning, without significantly sacrificing the network’s
performance [25, 16, 22]. Pruning usually follows a “ranking-to-prune”
framework: To facilitate compression while retaining the learned capabilities
of networks, a carefully-designed criterion is utilized to assess the
importance score of different structures [22, 40, 11], followed by a ranking
process to identify and remove those relatively unimportant ones. After
pruning, the compressed model is fine-tuned to recover its performance [18,
11].

However, ranking-based structured pruning presupposes an underlying assumption
that the parameters from different structures are directly comparable. Classic
deep neural networks, such as VGG [47], are built by stacking similar
convolutional layers or blocks where the neuron connections share a similar
computational topology. Under the circumstances, a straightforward global
ranking is expected to reveal their relative importance and remove unimportant
parameters [22, 11]. However, with recent innovation in network structure
design, advanced vision models like Vision Transformers [10, 51, 31] and
ConvNext [33] incorporate more sophisticated designs, such as complicated
residual connections [19, 24], attention mechanisms [53] and depth-wise
convolutions [6]. These novel structures increase the network’s internal
complexity and heterogeneity, challenging the reliability of pruning
importance criteria. For instance, a vanilla Vision Transformer [10] is
composed of MLP, Multi-head Attention, and skip connections. Even in the same
Multi-head attention layer, there are two pruning schemes to removing either
entire attention heads [37, 54] or the within-head dimensions [70, 60]. These
structures vary by their computational topologies and, as will be illustrated
in the experiments, tend to exhibit diverged importance distributions. In this
case, directly comparing the importance of these heterogeneous might be
problematic.

To address the problem, we propose a simple pruning approach, termed
Isomorphic Pruning, to alleviate the incomparability issue arising from
heterogeneous sub-structures. The essence of this method lies in decomposing a
network into multiple isomorphic groups, as illustrated in Figure 2 (d), based
on the computational topology of the sub-structures. Each isomorphic group
contains elements with the same structure and tends to present a similar
distribution of importance, making it more rational to compare their relative
importance.

To validate the effectiveness of our method, we conducted experiments on a
variety of visual models, including DeiT [51], ConvNext [33], ResNet [19], and
MobileNet-V2 [45]. Pruning results, as shown in Figure 1, on ImageNet-1K [8]
demonstrate that Isomorphic Pruning achieves comparable or even superior
performance relative to pruning algorithms specifically designed for CNNs or
transformers. For instance, by pruning a pre-trained large DeiT model, we
enhanced the performance of the DeiT-T model from 74.52% to 77.50%, when
compared to the scratch training one. Similarly, our algorithm also yields
satisfactory results on CNNs. We constructed compact ConvNexts and ResNets via
pruning, which has fewer MACs and parameters while surpassing a conventionally
trained model. Additionally, we also studied the real-world latency and memory
footprint of the compressed models on both GPU and CPU devices. Our findings
indicate that models pruned with Isomorphic Pruning achieve significant actual
speed-ups and are competitive to carefully designed efficient architectures.

Therefore, Our contribution lies in a practical and general pruning approach,
which can handle various vision models with novel mechanisms and
architectures. The pruned models show superior performance to several pruning
competitors on ImageNet-1K benchmark.

Figure 2: For a pre-trained network (a), we show three pruning strategies: (b)
Local Pruning that compares parameter importance within current layers; (c)
Global Pruning that performs global ranking for all parameters; (d) the
proposed Isomorphic pruning that groups parameters by the computational
topology and applies importance ranking within groups. In each group, the
importance distributions are more comparable. Details about the distribution
can be found in Fig. 4 of the experiments section.

##  2 Related Works

####  2.0.1 Network Pruning.

Deep neural networks are typically over-parameterized and can be pruned
without significant performance drop [25, 16]. The exploration of network
pruning can be dated back to early works [25, 17, 1] about neural networks.
Over the past few years, a large number of works have been conducted on
convolutional networks to verify the effectiveness of pruning, which can be
roughly categorized into unstructured [16, 35, 29] and structured pruning [57,
18, 11, 22, 5]. Unstructured pruning compresses a pre-trained model by forcing
partial weights to be zero, leading sparse networks [29]. To accelerate the
sparse model, special patterns and devices will be used such as N:M sparsity
[43, 12, 49]. Structured pruning provides a hardware-friendly approach by
removing sub-structures such as channels [11] or blocks [14] in CNNs, which
improves the model efficiency without dedicated accelerators. In recent years,
a large number of works adapted the idea of network pruning to vision
transformers. It demonstrated that even the vanilla vision transformers can be
squeezed for a better efficiency-performance trade-off [72, 70, 66, 63, 67,
46, 60, 4, 2, 48, 62, 64]. In the literature, transformer pruning has been
studied from several perspectives, such as width reduction [72], depth pruning
[63], attention head removal [60], sparse training [4] and architecture
transformation [18].

####  2.0.2 Pruning by Global Ranking.

In the context of network pruning, global pruning is recognized as a pivotal
topic for exploration. This method entails a comprehensive ranking of all
network layers, assigning adaptive pruning ratios to each layer according to
the ranks [22, 5, 60, 11]. Different layers within a network often exhibit
varying levels of redundancy. To this end, the advantage of global pruning
lies in its ability to create a more parameter-efficient, non-uniformly
compressed model [60]. However, a key challenge remains in the ranking: how to
ensure the consistency and comparability across different layers? A widely-
used solution for this issue is normalizing scores on a per-layer basis to
even out the importance distribution across the network [30]. Alternatively,
some approaches use sparse training to learn an effective target structure
[11], which might be costly due to the additional optimization. In this work,
we introduce a more straightforward approach termed isomorphic pruning to
facilitate a reliable ranking for pruning.

##  3 Method

The core idea of Isomorphic Pruning is straightforward: we group sub-
structures in a network by their network topology and perform importance
ranking within groups. This imposes two questions 1) how to identify removable
sub-structures in networks and 2) how to compare the topology graph for
grouping and pruning.

###  3.1 Sub-structure in Network

Structured Pruning aims to remove sub-structures in networks, such as the
dimensions of hidden states or features. To make the formulation clear, we
primarily concentrate on stacked fully-connected networks, although the
principles established herein can be readily extended to CNNs and
transformers. Initially, we consider a single linear layer which is
parameterized by a matrix $W\in\mathbb{R}^{m\times n}$. To articulate the
pruning process, we introduce a pruning function $\mathrm{g}(W,d,k)$ defined
as follows:

$\hat{w}=\mathrm{g}(W,d,k)=\begin{cases}\mathrm{g}(W,0,k)=W_{[k,:]}\in\mathbb{R}^{1\times
n},&\text{if pruning output dimension},\\\
\mathrm{g}(W,1,k)=W_{[:,k]}\in\mathbb{R}^{m\times 1},&\text{if pruning input
dimension},\\\ \end{cases}$

(1)

where $\hat{w}$ refers to the weight column or row slated for removal,
acquired via a slicing operation denoted by subscript $[:,k]$ or $[k,:]$. The
pruning function $\mathrm{g}(W,d,k)$ is defined by three parameters, the
parameter matrix $W$, the axis $d\in\\{0,1\\}$ indicating the dimension to be
compressed, and the index $k$ identifying the $k$-th rows or columns in $W$.
In the context of a linear layer represented by a 2-D matrix
$W\in\mathbb{R}^{m\times n}$, the axis $d$ can assume values of either 0 or 1,
with $k$ having a permissible range of $[0,m]$ and $[0,n]$, respectively. This
formulation encapsulates the pruning of a single layer; to extend it to the
entire network, we also need to take the dependency [11, 5, 30] into account,
where parameters from different layers are coupled and should be pruned
simultaneously. As depicted in Figure 3, for two adjacent linear layers
parameterized by $W_{1}$ and $W_{2}$, the output dimension of the former is
integrally coupled to the input dimension of the latter, thereby constituting
a minimal sub-structure eligible for removal. If we treat each pruning
operation $\mathrm{g}(W,d,k)$ as vertices defined by a triplet $(W,d,k)$ and
the dependency as edges, then it is natural to model sub-structures as graphs.

###  3.2 Graph Modeling of Sub-structures

In this section, we delve deeper into the methodology for decomposing a
network into several independent and minimal sub-structures in pruning, and
model them as graphs. We can begin with the MLP network as illustrated in
Figure 3, where the activation functions are omitted. The network comprises a
sequence of 2-D parameter matrices $\\{W_{1},W_{2},...,W_{L}\\}$, with $L$
denoting the network depth. Initially, all parameters are not assigned to any
sub-structure. We randomly select one parameter $W$ as the seed to find its
associated substructures and build the corresponding graph. For instance,
selecting parameter $W_{1}$ and setting the axis $d=0$ allows us to establish
an initial graph $G=(V=\\{(W,0,k)\\},E=\\{\\})$, focusing on the elimination
of $k$-th the output dimension in $W_{1}$. Then we iterate through all
potential $(W^{\prime},d^{\prime},k^{\prime})\notin V$ to ascertain any
dependency to $(W,d,k)\in V$. Dependencies between layers can be ascertained
through either manually-crafted patterns or automatic algorithms. In this
work, we adhere to the approach of [11] to assess dependency based on two
rudimentary rules:

Dependency Modeling. For a pair of triplets
$(W^{\prime},d^{\prime},k^{\prime}),(W,d,k)$, a dependency is established if
either of the following criteria is met: 1) $W^{\prime}$ and $W$ are adjacent
layers with $d=d^{\prime}$ and $k=k^{\prime}$; 2) $W^{\prime}=W$ and
$\mathrm{g}(W^{\prime},d^{\prime},k^{\prime})=\mathrm{g}(W,d,k)$. The first
criterion addresses adjacent layers as exemplified in Figure 3, while the
second pertains to specialized layers, such as normalization layers, which
share the same pruning pattern for both outputs and inputs. Thereafter, we
update the vertex set $V$ and edge set $E$ with the discovered dependency
between $(W^{\prime},d^{\prime},k^{\prime})$ and $(W,d,k)$ accordingly:

$\begin{split}V&=V\cup\\{{(W^{\prime},d^{\prime},k^{\prime})}\\},\\\
E&=E\cup\\{((W,d,k),(W^{\prime},d^{\prime},k^{\prime}))\\}.\end{split}$ (2)

We can further repeat the above process until no additional vertex is
incorporated into the current graph. The resultant graph $G$ then serves as a
topological representation of a single sub-structure associated with the
initial seed parameter. To map all substructures onto different graphs, we
subsequently embark on a recursive procedure, commencing from those nodes not
yet allocated to any sub-structure. Figure 3 furnishes an exemplar of this
process, showcasing three distinct substructures labeled as sub-structures 1,
2, and 3.

Figure 3: Isomorphic Pruning models sub-structures as graphs, and apply
isolated ranking and pruning with isomorphic structures. We highlight three
removable substructures in an MLP and show their corresponding graphs below.
Substructures 1 & 3 are isomorphic but 2 & 3 are non-isomorphic due to the
additional residual connections.

###  3.3 Ranking with Graph Isomorphism

With the graph-based representation of network sub-structures, we advance to
the ranking phase. This stage involves estimating and ranking the significance
of each sub-structure to identify redundancies. A pivotal component of this
process is an importance criterion, which quantifies the relative significance
of various sub-structures. Previous works in the literature presents numerous
well-crafted criteria, including magnitude pruning and Taylor expansion. In
this work, we focus on the general importance function $I(\cdot)$ for a single
layer, represented as:

$I(W,d,k)=\begin{cases}\|\mathrm{g}(W,d,k))\|,&\quad\text{if Magnitude
Pruning}\\\
\|\frac{\partial\mathcal{L}}{\partial\mathrm{g}(W,d,k)}\;\mathrm{g}(W,d,k))\|_{2},&\quad\text{if
Taylor Pruning}\\\ \text{...}&\quad\text{Other importance Criteria.}\\\
\end{cases}$

(3)

In structural pruning, our objective is to remove sub-structures encompassing
parameters across multiple layers. Existing approaches [5, 11, 30] generally
extend the single-layer importance to a holistic measure by aggregating scores
across all constituents $(W,d,k)$ for the vertexes set, denoted as
$I^{*}(\cdot)$ as follows:

$I^{*}(G(V,E))=\sum\nolimits_{(W,d,k)\in V}I(W,d,k),$ (4)

thereby cumulating the importance of all elements within graph $G$. However,
the inherent diversity in parameters and computational topology among
different substructures $G$ introduces potential ambiguities in this metric.
Experimental evidence, particularly in the context of a vision transformer as
shown in Figure 4, elucidates this issue: the largest substructure encompasses
all embedding dimensions, containing a substantial number of parameters,
whereas the most minimal sub-structure comprises merely two linear layers.
Consequently, such aggregation may fail to accurately reflect the true
relative importance of these entities. To address the challenges posed by the
heterogeneity of substructures, we delineate a simple approach designed to
preserve the nuanced information encapsulated by the aggregated importance, as
outlined in Equation 4, whilst alleviating the detrimental repercussions
raised by the heterogeneity of substructures. Our principal strategy entails
the clustering of sub-structures into various isomorphic groups, predicated on
their topological logic and parameter configurations. In this case, applying
aggregated importance within groups can be meaningful, since isomorphic sub-
structures show one-to-one correspondence among their parameters, and how
different parameters participate in computations is entirely identical. To
facilitate this, the key step is to examine the isomorphism of two graphs
$G_{1}$ and $G_{2}$. Formally, we adopt the following definition to ascertain
graph isomorphism.

######  Definition 1 (Graph Isomorphism)

Two labeled graphs $G$ and $G^{\prime}$ are isomorphic if there exists a
bijection between their vertices so that, all mapped vertices preserve the
same labels, and are connected in the same way.

To develop a general indicator Isomorphic($\mathrm{g}_{1}$, $\mathrm{g}_{2}$),
we need to examine both the label and connectivity in two graphs.

####  3.3.1 Vertex Label and Connectivity.

In our settings, each vertex is labeled by the triplet $(W,d,k)$, encompassing
the parameter matrix $\mathbf{W}$, axis $d$, and index $k$. We follow a simple
principle for labling: Two vertices $(\mathbf{W},d,k)$ and
$(\mathbf{W}^{\prime},d^{\prime},k^{\prime})$ are labeled identically if and
only if the parameter $\mathbf{W}^{\prime}$ originates from the same layer
type such as Linear or Conv, with the same pruning dimension $d=d^{\prime}$.
We permit the index $k$ to differ, since within the same parameter matrix, the
weight vectors sliced along the $d$-th dimension are consistently homogeneous
and thus can be considered isomorphic. After determining the label for each
vertex, we further explore the connectivity in two graphs $G_{1}$ and $G_{2}$.
As illustrated in Figure 3, we show three substructures alongside their
respective graphs $G_{1}$, $G_{2}$, and $G_{3}$, where $G_{1}$ and $G_{2}$
incorporate a singular edge, whereas $G_{3}$ encompasses three edges. If we
check the edges following the execution order of the computational graphs,
it’s natural to observe the isomorphism of $G_{1}$ and $G_{3}$, since they
share the same topology. To make this process more formal, we assume the edges
in graph $G$ are already sorted according to their execution order in
forwarding, and check the isomorphism by

$\leavevmode\resizebox{399.2946pt}{}{$\text{Isomorphic}(G,G^{\prime}))=\mathbbm{1}\\{|G|=|G^{\prime}|\land\text{label}(V_{i})=\text{label}(V^{\prime}_{i});\forall
i\in\\{1,2,\ldots,|G^{\prime}|\\}\\}$},$ (5)

where $\mathbbm{1}$ is an indicator function. $|G^{\prime}|$ denotes the
number of edges in the graph and $\land$ signifies the logical AND operation.
The first condition ensures the equivalence of the number of vertices in the
graphs, while the second condition confirms the matching labels of vertices on
either side of the edge pairs $V_{i}$ and $V^{\prime}_{i}$.

Using Equation 5, it is easy to cluster substructures into different groups
and perform rankings within each group exclusively. Specifically, we examine a
collection of substructures $\\{G_{1},G_{2},\ldots,G_{N}\\}$ that share
identical graph topologies. We estimate the aggregated significance of each
graph independently using Equation 4, resulting in an $N$-dimensional
importance vector $[I^{*}(G_{1}),\ldots,I^{*}(G_{N})]$. This facilitates a
straightforward ranking of the importance vector, followed by the elimination
of the p% unimportant substructures based on their scores. This ranking method
is deemed reliable since all substructures within a group possess the same
architectural design, parameter scale, and computational graph. A similar
principle can be uniformly applied to other isomorphic groups for pruning, as
detailed in Algorithm 1.

Algorithm 1 Isomorphic Pruning

1:Input: Network with parameters $\\{W_{1},W_{2},\ldots,W_{n}\\}$, Pruning
ratio p%.

2:Output: Pruned Network
$\\{W^{\prime}_{1},W^{\prime}_{2},\ldots,W^{\prime}_{n}\\}$

3:procedure IdentifySubStructures($\\{W_{1},W_{2},\ldots,W_{n}\\}$)

4: Initialize empty list of sub-structures $\textbf{S}=\\{\\}$

5: for each layer $W_{i}$ in $\\{W_{1},W_{2},\ldots,W_{n}\\}$ do

6: if $W_{i}$ is not assigned to any substructure then

7: Initialize a graph $G(V=\\{W_{i}\\},E=\\{\\})$.

8: Find and add coupled $W_{j}$ to $G$ until there is no new dependency.

9: $\textbf{S}=\textbf{S}\cup\\{G\\}$

10: return S

11:procedure PruneByGroupedRanking(S, p)

12: for each substructure $G_{i}$ in $\\{G_{1},G_{2},\ldots,G_{n}\\}$ do

13: if $G_{i}$ is not assigned to any group then

14: Initialize a Group $\textbf{R}=\\{G_{i}\\}$.

15: Find and add $G_{j}$ that satisfies Isomorphism$(G_{i},G_{j})=1$ to R.

16: Calculate the importance scores
$\left\\{I^{*}(G)|G\in\textbf{R}\right\\}$.

17: Remove p% substructures in R by ranking the importance scores.

18: return S

19:$\textbf{S}\leftarrow\textsc{IdentifySubStructures}(\\{W_{1},W_{2},\ldots,W_{n}\\})$

20:$\textbf{S}\leftarrow\textsc{PruneByGroupedRanking}(\textbf{S},p)$

##  4 Experiments

This section reports our empirical results on the ImageNet-1K dataset [8],
which comprises 1,281,167 images for training and 50,000 for validation. We
deploy the proposed Isomorphic Pruning to compress several vision models,
including ConvNext [33], ResNet [36], MobileNetv2 [45], and Vision
Transformers [10, 51]. Our approach crafted a series of streamlined models
varying in size and Multiply–Accumulate Operations (MACs). We report the
parameter amount (#Params), MACs, and Top-1 validation accuracy in Table 1 and
2, and report the actual latency and peak memory consumption in Table 3.

###  4.1 Settings for ImageNet-1K

####  4.1.1 Pruning.

For simplicity, we adopt the Taylor-based pruning framework [39], which
applies a data-driven Taylor expansion to the loss function. For this purpose,
we randomly sampled 100 mini-batches from the ImageNet dataset, each
containing 64 images. These images were resized to 256x256 pixels and then
center-cropped to 224x224 pixels. We normalize all images using the mean and
standard deviation specific to ImageNet. Notably, while some methods, such as
ConvNext such as ConvNext [33] and DeiT [51] employ additional training
techniques like label smoothing and knowledge distillation, our approach only
leverages a straightforward cross-entropy loss for the Taylor expansion. We
accumulate gradients from all the batches for a more reliable importance
estimation and apply Isomorphic pruning in a one-shot manner without iterative
pruning and fine-tuning [40].

####  4.1.2 Finetuning.

During the fine-tuning process of pruned models, we adhere to the original
training protocol for a fair comparison. While maintaining most of the
original settings unchanged, we adjusted the learning rates and batch sizes to
make the training suitable for A5000 GPUs, following the linear LR scaling
rule proposed in [13]. For example, to compressing a pre-trained DeiT models
[51], we adopt the same augmentation strategies for training, including
RandAugmentation [7], Mixup [69], CutMix [68], Random Erasing [71], Repeated
Augumentation [23], label smoothing [50]. The pruned models are distilled with
a pre-trained RegNetY [51, 44] for 300 epochs using the AdamW [34] optimizer,
with a learning rate of 0.0005 and a total batch size of 2048. A weight decay
of 0.05 and a cosine annealing scheduler is deployed for training. For other
models like ConvNext [33], ResNet [19] and MobileNet-v2 [45], the same
principle will be applied. More details of hyper-parameters, pruning, and
finetuning can be found in the appendix.

####  4.1.3 Evaluation.

We report the optimal classification accuracy achieved on the ImageNet-1K
validation set, employing the standard resize-and-crop protocol [19, 33] for
evaluation. To further validate the efficiency of models, we profiled the
latency and the peak memory consumption of the pruned models on both GPU (a
single RTX A5000) and CPU. To guarantee a fair and standardized comparison, we
also tested the latency of several baseline models sourced from the Pytorch-
Image Models [58] and Torchvision [38], following the same testing procedure.
The latency was estimated with a batch size of 256 for GPU and 8 for CPU,
averaged on 100 repeats. Appropriate GPU warmup was deployed in our test.

###  4.2 Pruning Convolutional Neural Networks

Table 1: Pruning CNNs on ImageNet-1K.

Pruning Method | #Params (M) | MACs (G) | Base Acc. | Final Acc. |  $\Delta$ Acc.
---|---|---|---|---|---
ConvNext-B† [33] | 88.59 | 15.39 | - | 83.83 | -
ConvNext-S† [33] | 50.22 | 8.71 | - | 83.14 | -
ConvNext-S (Ours) | 47.36 | 8.48 | 83.83 | 83.17 | -0.66
ConvNext-T† [33] | 28.59 | 4.47 | - | 82.06 | -
ConvNext-T (Ours) | 25.32 | 4.19 | 83.83 | 82.19 | -1.64
ResNet-152† | 60.19 | 11.58 | - | 78.31 | -
AOFP-E2 [9] | - | 4.13 | 77.37 | 77.00 | -0.37
ABCPruner [28] | 24.07 | 4.31 | 78.31 | 77.12 | -1.19
EPruner-0.63 [27] | 21.56 | 4.08 | 78.31 | 76.83 | -1.48
ResNet-152-4.0G (Ours) | 23.11 | 4.05 | 78.31 | 77.84 | -0.47
ResNet-101† | 44.55 | 7.85 | - | 77.38 | -
IE [39] | 31.2 | 4.70 | 77.37 | 77.35 | -0.02
FPGM [21] | - | 4.51 | 77.37 | 77.32 | -0.05
SFP [20] | - | 4.51 | 77.37 | 77.51 | +0.14
ISTA [61] | - | 4.47 | 76.40 | 75.27 | -1.13
Res101-4.5G (Ours) | 29.14 | 4.48 | 77.38 | 77.56 | +0.16
GFP [30] | 28.02 | 3.90 | 78.29 | 78.33 | +0.04
AOFP [9] | - | 3.89 | 76.63 | 76.40 | -0.23
Res101-3.8G (Ours) | 24.87 | 3.85 | 77.38 | 77.43 | +0.05
ResNet-50† [19] | 25.56 | 4.13 | - | 76.13 | -
Taylor [39] | 14.20 | 2.25 | 76.18 | 74.50 | -1.68
CCP [42] | - | 2.11 | 76.15 | 75.50 | -0.65
GFP [30] | 19.42 | 2.04 | 76.79 | 76.42 | -0.37
AutoSlim [65] | 20.60 | 2.00 | 76.10 | 75.60 | -0.50
DepGraph [11] | - | 1.99 | 76.15 | 75.83 | -0.32
ResNet50-2G (Ours) | 15.05 | 2.06 | 76.13 | 75.91 | -0.22
Mobv2† | 3.50 | 0.32 | - | 71.88 | -
Meta [32] | - | 0.14 | 74.70 | 68.20 | -6.50
GFP [30] | - | 0.15 | 75.74 | 69.16 | -6.58
DepGraph [11] | - | 0.15 | 71.90 | 68.49 | -3.41
Mobv2-0.15G (Ours) | 2.19 | 0.15 | 71.88 | 68.91 | -2.97

Convolutional Neural Network is a critical focus in the area of network
pruning. In this section, we concentrate on pruning three popular ConvNets:
ConvNext [33], ResNet [19] and MobileNet-v2 [45].

####  4.2.1 ConvNext.

Our experiments in Table 1 begin with the evaluation on an advanced CNN,
ConvNext [33], which achieves impressive performance on several benchmarks. We
use the official ConvNext-Base [38] as the pre-trained model for pruning,
which achieves an accuracy of 83.83% on the ImageNet-1K validation set. The
pruning of ConvNext models is a relatively underexplored topic. Hence, we
establish our baselines using the official variants with fewer parameters,
ConvNext-S and ConvNext-T. For ConvNext, we also generalize our method to
depth pruning, which removes unimportant layers with the minimal average
scores on substructures, so that the pruned models have the same depth as
baselines. The pruned models are fine-tuned for 300 epochs following the same
training protocol as [33]. During fine-tuning, we observed that the pruned
ConvNext with half of the parameters removed, achieves an accuracy of 80.09 in
only 20 epochs, which effectively validates the efficiency of pruning.
Moreover, as illustrated in Table 1, the final model after 300 full fine-
tuning, achieves competitive performance (82.19% vs. 82.06%) to pre-trained
models with fewer parameters (25.32M vs. 28.59M) and MACs (4.19G vs. 4.47G).

####  4.2.2 ResNet.

In line with prior studies [22, 11, 5, 30], we demonstrate the efficacy of our
method by structurally pruning pre-trained ResNet MACs. ResNet-50/101/152
consists of four residual blocks, each formed by several Bottle-neck
structures and residual connections [19]. The proposed Isomorphic Pruning
identifies five kinds of distinct isomorphic groups within ResNet-50, four
targeting the block structures and one associated with bottleneck structures
[19]. It is worthwhile to note that different sub-structures vary in their
parameter and MAC counts. For example, residual blocks naturally contain more
parameters than the bottleneck structures which are only composed of a few
convolutional layers. The conventional global pruning, employing a naive
global ranking threshold, faces the potential risks of over-pruning certain
layers due to the biased importance distribution. Isomorphic Pruning, however,
employs a more stable scheme for pruning. As illustrated in Table 1, since
different baseline methods deploy pre-trained models with diverged accuracy,
we report both the accuracy and the accuracy drop compared to their base
models. Our method utilizes a simple Taylor criterion [39] and obtains a
series of lightweight ResNets. Notably, Isomorphic pruning achieves lossless
compression on ResNet-101, under both 4.5G and 3.8G settings.

####  4.2.3 MobileNet-v2.

We also conduct experiments on lightweight models, MobileNet-v2, by
compressing its MACs from 0.32 G to 0.15 G. In MobileNet-v2, we can also
discover 5 isomorphic groups in MobileNet. The pruned model achieves better
accuracy compared to DepGraph [11] which requires sparse training.

###  4.3 Pruning Vision Transformers.

Vision Transformers (ViT) has profoundly reshaped the paradigm of visual
modeling over the past few years. Nevertheless, their inherent over-
parameterization often imposes huge overheads to both training and inference.
Compared to CNNs that are composed of similar convolution layers or blocks,
Vision Transformers are usually built with MLP, Multi-head Attention, and skip
connections, which are more challenging for pruning due to the heterogeneous
internal structures.

Table 2: Pruning DeiTs on ImageNet-1K.

Method | #Params (M) | MACs (G) | Acc (%)
---|---|---|---
DeiT-B† [51] | 87.34 | 17.69 | 83.32
DeiT-S† [51] | 22.44 | 4.64 | 81.17
DeiT-S-600EP [60] | 22.44 | 4.64 | 81.80
VTP - 40% [72] | 48.00 | 10.00 | 80.70
WDPruning [63] | 55.30 | 9.90 | 80.76
UVC [67] | - | 8.00 | 80.57
X-Pruner [66] | - | 8.50 | 81.02
NViT-B [60] | 34.00 | 6.80 | 83.29
SAViT [70] | 25.40 | 5.30 | 81.66
UP-DeiT-S [64] | 22.10 | - | 81.56
NViT-S [60] | 21.00 | 4.20 | 82.19
Upop [46] | 19.90 | 4.10 | 81.10
DeiT-S (Ours) | 20.69 | 4.16 | 82.41
GOHSP [62] | 14.40 | 2.99 | 79.98
GOHSP [62] | 11.10 | 2.81 | 79.86
ViT-Slim [2] | 13.50 | 2.80 | 79.20
Upop [46] | 13.50 | 2.80 | 79.60
ViT-Slim [2] | 13.50 | 2.80 | 79.20
CP-ViT [48] | - | 2.66 | 79.08
DeiT-2.6G (Ours) | 13.07 | 2.62 | 81.13

Method | #Params (M) | MACs (G) | Acc (%)
---|---|---|---
DeiT-T† [51] | 5.91 | 1.27 | 74.52
DeiT-T-600EP [60] | 5.91 | 1.27 | 75.00
WDPruning [63] | 13.30 | 2.60 | 78.38
X-Pruner [66] | - | 2.40 | 78.93
UVC [67] | - | 2.32 | 78.82
ViT-Slim [72] | 11.40 | 2.30 | 77.94
ViT-Slim [72] | 11.40 | 2.30 | 77.94
NViT-T [60] | 6.90 | 1.30 | 76.21
SAViT+KD [70] | 6.60 | 1.30 | 76.95
UP-DeiT-T [64] | 5.70 | - | 75.79
DeiT-T (Ours) | 5.74 | 1.21 | 77.50
VTC-LFC [56] | 5.10 | - | 71.60
SViTE-T [70] | 4.21 | 0.99 | 70.12
GOHSP [62] | 4.00 | 0.91 | 70.24
SAViT [70] | 4.20 | 0.90 | 70.72
CP-ViT [48] | - | 0.74 | 71.24
WDPruning [63] | 3.50 | 0.70 | 70.34
X-Pruner [66] | - | 0.60 | 71.10
UVC [67] | - | 0.51 | 70.60
DeiT-0.6G (Ours) | 3.08 | 0.62 | 72.60

Table 3: Profiling the pruned models and other off-the-shelf vision models
from PyTorch-Image-Models [58] on a single RTX A5000 GPU and CPU. For NViT
[60], we report their best speed up in the original paper, tested on a single
V100 or RTX 3080.

Method | #Params $\downarrow$ | MACs $\downarrow$ | Peak Mem. $\downarrow$ | GPU Latency (256) $\downarrow$ | CPU Latency (8) $\downarrow$ | Acc. (%)
---|---|---|---|---|---|---
DeiT-B [51] | 86.40 M | 17.60 G | 2909.78 MB |  802.97 ms (1.00$\times$) |  197.73 ms (1.00$\times$) | 83.32
DeiT-S [51] | 22.44 M | 4.64 G | 1363.47 MB |  241.81 ms (3.32$\times$) |  80.64 ms (2.45$\times$) | 81.17
Swin-T [31] | 28.29 M | 4.51 G | 4257.98 MB |  375.45 ms (2.14$\times$) |  203.86 ms (0.97$\times$) | 81.19
NViT-S [60] | 21.00 M | 4.20 G | - |  \- (2.52$\times$) | - | 82.19
TNT-S [15] | 23.76 M | 4.85 G | 1457.56 MB |  631.02 ms (1.27$\times$) |  175.58 ms (1.13$\times$) | 81.50
CaiT-XS24 [52] | 26.56 M | 5.40 G | 1496.42 MB |  557.41 ms (1.44$\times$) |  342.28 ms (0.58$\times$) | 82.00
CrossViT-S [3] | 26.86 M | 5.64 G | 1546.94 MB |  315.02 ms (2.55$\times$) |  115.51 ms (1.71$\times$) | 81.00
EfficienctFormer-L3 [26] | 31.41 M | 3.94 G | 2681.75 MB |  249.05 ms (3.22$\times$) |  167.16 ms (1.18$\times$) | 82.40
PVTv2-B2 [55] | 25.36 M | 4.00 G | 4478.18 MB |  364.00 ms (2.21$\times$) |  212.44 ms (0.93$\times$) | 82.00
DeiT-S (Ours) | 20.69 M | 4.16 G | 1547.97 MB | 230.84 ms (3.48$\times$) | 44.10 ms (4.48$\times$) | 82.41
DeiT-2.6G (Ours) | 13.07 M | 2.62 G | 1203.07 MB | 189.68 ms (4.23$\times$) | 56.59 ms (3.49$\times$) | 81.13
DeiT-T [51] | 5.91 M | 1.27 G | 720.22 MB |  93.78 ms (8.56$\times$) |  46.83 ms (4.22$\times$) | 74.52
NViT-T [60] | 6.90 M | 1.30 G | - |  \- (4.97$\times$) | - | 76.21
DeiT-T (Ours) | 5.71 M | 1.20 G | 837.16 MB | 93.17 ms (8.62$\times$) | 47.89 ms (4.13$\times$) | 77.50
DeiT-0.6G (Ours) | 3.08 M | 0.62 G | 629.28 MB | 63.23 ms (12.70$\times$) | 40.98 ms (4.83$\times$) | 72.60
ConvNext-B [33] | 88.59 M | 15.36 G | 5394.10 MB |  789.24 ms (1.00$\times$) |  262.58 ms (1.00$\times$) | 83.83
ConvNext-S [33] | 50.22 M | 8.69 G | 3958.45 MB |  508.20 ms (1.57$\times$) |  188.07 ms (1.40$\times$) | 83.14
ConvNext-S (Ours) | 47.36 M | 8.48 G | 3788.65 MB | 535.95 ms (1.59$\times$) | 179.72 ms (1.46$\times$) | 83.17
ConvNext-T [33] | 28.59 M | 4.46 G | 3785.37 MB |  291.02 ms (2.82$\times$) |  126.17 ms (2.08$\times$) | 82.06
ConvNext-T (Ours) | 25.32 M | 4.19 G | 3607.06 MB | 292.77 ms (2.83$\times$) | 97.82 ms (2.68$\times$) | 82.19
ResNet152 [19] | 60.19 M | 11.58 G | 3320.24 MB |  386.65 ms (1.00$\times$) |  272.21 ms (1.00$\times$) | 78.31
ResNet152-4G (Ours) | 23.11 M | 4.06 G | 1938.91 MB | 238.12 ms (1.62$\times$) | 180.65 ms (1.51$\times$) | 77.84
ResNet101 [19] | 44.55 M | 7.85 G | 3196.77 MB |  271.34 ms (1.00$\times$) |  202.66 ms (1.00$\times$) | 77.38
ResNet101-4.5G (Ours) | 29.14 M | 4.48 G | 2639.40 MB | 220.72 ms (1.23$\times$) | 162.33 ms (1.25$\times$) | 77.56
ResNet101-3.8G (Ours) | 24.87 M | 3.85 G | 2286.36 MB | 194.39 ms (1.40$\times$) | 158.98 ms (1.28$\times$) | 77.43
ResNet50 [19] | 25.56 M | 4.13 G | 3035.84 MB |  171.54 ms (1.00$\times$) |  114.86 ms (1.00$\times$) | 76.13
ResNet50-2.0G (Ours) | 15.05 M | 2.06 G | 2238.15 MB | 78.75 ms (1.33$\times$) | 80.88 ms (1.42$\times$) | 75.91
Mobv2 [45] | 3.50 M | 0.32 G | 2855.44 MB |  71.42 ms (1.00$\times$) |  74.46 ms (1.00$\times$) | 71.88
Mobv2-0.15G (Ours) | 2.19 M | 0.15 G | 1354.82 MB | 43.48 ms (1.64$\times$) | 38.63 ms (1.93$\times$) | 68.91

Table 2 presents our pruning results on DeiT, a plain vision transformer
enhanced with knowledge distillation [51]. We benchmark our approach with a
range of innovative pruning baselines specifically tailored for vision
transformers [51, 60, 72, 56, 70, 48, 63, 67, 66, 46, 62]. A DeiT model
consists of various components, such as MLP and attention layers. Within the
attention layers, we can even find two feasible pruning schemes: pruning 1)
the number of heads or 2) within-head dimensions. We implement Isomorphic
Pruning to compress a pre-trained DeiT-Base model by pruning both heads, head
dimensions, and the embedding sizes following [60]. This is achieved through a
straightforward first-order Taylor expansion for importance estimation, and
clustering isomorphic sub-structures for ranking.

We developed four lightweight vision transformers: DeiT-S, DeiT-2.6G, DeiT-T
and DeiT-0.6G. The pruned DeiT-S and DeiT-T models achieve comparable
latencies comparable to the standard DeiT-S and DeiT-T with manually designed
architecture. The empirical results are reported in Table 2, which shows that
our method yields greater performance gains in complex models with extensive
heterogeneous internal structures, especially in comparison to CNNs.
Specifically, the optimized DeiT-T achieves an accuracy of 77.50%, with only
1.2G MACs, which is better than the results (74.52%) of the pre-trained DeiT-T
with uniform width across layers. Detailed configurations, such as base models
and pruning ratios, are available in the appendix.

(a)

(b)

(c)

Figure 4:  (a-b) Top-1 Accuracy and loss of pruning ResNet-50 on ImageNet-1K
validation set without finetuning. We report the mean and standard derivation
of 10 experiments. IsomorphicP Pruning consistently improves the performance
of pruned models. (c) Histogram of Taylor importance score of DeiT sub-
structures. Isomorphic structures are highlighted with the same color. Zoom in
for more details about each structure. We visualize the threshold of 50%
pruning ratio for naive global pruning (the black dash) and isomorphic pruning
(the colored dash).

###  4.4 Analysis on the Pruned Models

####  4.4.1 On-device Latency

Table 3 measures the actual latency and peak memory consumptions on a single
RTX A5000 GPU and CPU. We evaluate all models with a batch size of 256 for GPU
and 8 for CPU, respectively. The average latency over 100-round experiments
are reported, with a 20-step warmup process before evaluation. Firstly,
results show that the pruned DeiT models can achieve comparable performance
(82.41% vs. 82.40%) to some advanced transformers like EfficientFormer [26],
while preserving a competitive latency. Additionally, we observe that the peak
memory consumption is primarily determined by the largest layer width.
Therefore, under the same budget, a uniform DeiT may require less peak memory
(1363 MB for DeiT-S) for inference. In contrast, a non-uniform architecture
may require slightly more memory (1547 MB for the DeiT-S Pruned) under the
given budget. But non-unfirom pruning can achieve better accuracy, which
reveals an important trade-off between accuracy and memory consumption for
transformers. In addition, for CNNs like ResNets, the pruned model requires
much less memory due to the compressed feature dimensions. Furthermore, we
note that the actual speed-up may not be linearly correlated with the MAC
reductions. This is particularly evident in ResNet-50, where only a
1.33$\times$ actual speed-up is achieved despite the removal of half of the
MACs. In contrast, compression on Mobv2 is more effective, achieving a
1.64$\times$ acceleration with a similar compression ratio on MACs.

####  4.4.2 Distribution of Importance Scores.

To further explore how Isomorphic Pruning affects the pruning behavior, we
study the distribution of importance score in a DeiT-Base model [51]. As shown
in Figure 4 (c), the importance histogram of different sub-structures is
highlighted with different colors, each corresponding to one kind of
isomorphic structure. It is natural to find that, the MLP layer and the
attention layer have diverged importance distributions due to their different
computation process. Besides, when we apply a simple 50% global pruning on the
DeiT model, most parameters in Attentions #5, #9, and #13 will be removed,
since the importance scores in attentions are centered around 0. In contrast,
the embedding number as #1 will not be even been pruned in this case, due to
the relatively large importance scores. This inevitably leads to an imbalanced
pruning of different structures. Notably, the embedding structure, as
illustrated in the appendix, contains massive parameters in the DeiT, where
biased pruning may lead to very limited acceleration.

####  4.4.3 Importance Criteria.

Note that Isomorphic Pruning can be combined with different pruning criteria
like Taylor [39], Hessian [25] and even L1-norm [22]. We quantify the
performance gain of Isomorphic Pruning by visualizing the accuracy and loss
curve with different pruning ratios in Figure 4 (a) and (b). Specifically, we
gradually increase the pruning ratio for each criterion and report the average
accuracy and loss without finetuning over 10 repeated runs with random inputs
for Taylor expansion [39, 25]. Note that L1 pruning is deterministic and thus
presents 0 std. We find the Isomorphic Pruning, marked as solid lines,
consistently improves the counterpart baselines highlighted in dashed lines.

Table 4: Transfer learning on pruned models.

Models | CIFAR-10 | CIFAR-100
---|---|---
DeiT-S [39] | 98.52 | 87.07
NViT-S [60] | 98.78 | 87.90
DeiT-S (Ours) | 99.02 | 90.09
DeiT-T | 93.93 | 85.66
NViT-T [60] | 98.31 | 85.88
DeiT-S (Ours) | 98.28 | 87.08
ConvNext-S | 98.77 | 89.15
ConvNext-S (Ours) | 98.78 | 89.36

####  4.4.4 Transfer Learning.

We also evaluate the performance of transfer learning based on our pruned
models. Table 4 reports the accuracy of transferring different models to
CIFAR-10 and CIFAR-100. We resize all images to 224$\times$224 and apply the
same training recipe for transfer. For the DeiT models, we did not include the
supervision from the teacher models. It is observed that our method achieves
better transfer accuracy compared to the scratch training model and baseline
methods.

##  5 Conclusion

In this paper, we introduce Isomorphic Pruning, a practical approach designed
to compress vision networks with novel mechanisms and architectures. This
method focuses on mitigating the challenges posed by heterogeneous sub-
structures in networks, thereby enhancing the reliability of ranking and
pruning processes. Our empirical results on the ImageNet-1k dataset, showcase
the effectiveness of Isomorphic Pruning across several vision models like
ConvNext, ResNet, MobileNet-v2, and Vision Transformers.

## References

  * [1] Baum, E., Haussler, D.: What size net gives valid generalization? Advances in neural information processing systems 1 (1988)
  * [2] Chavan, A., Shen, Z., Liu, Z., Liu, Z., Cheng, K.T., Xing, E.P.: Vision transformer slimming: Multi-dimension searching in continuous optimization space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4931–4941 (2022)
  * [3] Chen, C.F.R., Fan, Q., Panda, R.: Crossvit: Cross-attention multi-scale vision transformer for image classification. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 357–366 (2021)
  * [4] Chen, T., Cheng, Y., Gan, Z., Yuan, L., Zhang, L., Wang, Z.: Chasing sparsity in vision transformers: An end-to-end exploration. Advances in Neural Information Processing Systems 34, 19974–19988 (2021)
  * [5] Chen, T., Liang, L., Ding, T., Zhu, Z., Zharkov, I.: Otov2: Automatic, generic, user-friendly. arXiv preprint arXiv:2303.06862 (2023)
  * [6] Chollet, F.: Xception: Deep learning with depthwise separable convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1251–1258 (2017)
  * [7] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 702–703 (2020)
  * [8] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [9] Ding, X., Ding, G., Guo, Y., Han, J., Yan, C.: Approximated oracle filter pruning for destructive cnn width optimization. In: International Conference on Machine Learning. pp. 1607–1616. PMLR (2019)
  * [10] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [11] Fang, G., Ma, X., Song, M., Mi, M.B., Wang, X.: Depgraph: Towards any structural pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16091–16101 (2023)
  * [12] Frantar, E., Alistarh, D.: Sparsegpt: Massive language models can be accurately pruned in one-shot (2023)
  * [13] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017)
  * [14] Guo, J., Ouyang, W., Xu, D.: Multi-dimensional pruning: A unified framework for model compression. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 1508–1517 (2020)
  * [15] Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., Wang, Y.: Transformer in transformer. Advances in Neural Information Processing Systems 34, 15908–15919 (2021)
  * [16] Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations (ICLR) (2016)
  * [17] Hassibi, B., Stork, D.G., Wolff, G.J.: Optimal brain surgeon and general network pruning. In: IEEE international conference on neural networks. pp. 293–299. IEEE (1993)
  * [18] He, H., Liu, J., Pan, Z., Cai, J., Zhang, J., Tao, D., Zhuang, B.: Pruning self-attentions into convolutional layers in single path. arXiv preprint arXiv:2111.11802 (2021)
  * [19] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [20] He, Y., Kang, G., Dong, X., Fu, Y., Yang, Y.: Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866 (2018)
  * [21] He, Y., Liu, P., Wang, Z., Hu, Z., Yang, Y.: Filter pruning via geometric median for deep convolutional neural networks acceleration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4340–4349 (2019)
  * [22] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: Proceedings of the IEEE international conference on computer vision. pp. 1389–1397 (2017)
  * [23] Hoffer, E., Ben-Nun, T., Hubara, I., Giladi, N., Hoefler, T., Soudry, D.: Augment your batch: Improving generalization through instance repetition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8129–8138 (2020)
  * [24] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [25] LeCun, Y., Denker, J., Solla, S.: Optimal brain damage. Advances in neural information processing systems 2 (1989)
  * [26] Li, Y., Yuan, G., Wen, Y., Hu, J., Evangelidis, G., Tulyakov, S., Wang, Y., Ren, J.: Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems 35, 12934–12949 (2022)
  * [27] Lin, M., Ji, R., Li, S., Wang, Y., Wu, Y., Huang, F., Ye, Q.: Network pruning using adaptive exemplar filters. IEEE Transactions on Neural Networks and Learning Systems 33(12), 7357–7366 (2021)
  * [28] Lin, M., Ji, R., Zhang, Y., Zhang, B., Wu, Y., Tian, Y.: Channel pruning via automatic structure search. arXiv preprint arXiv:2001.08565 (2020)
  * [29] Liu, B., Wang, M., Foroosh, H., Tappen, M., Pensky, M.: Sparse convolutional neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 806–814 (2015)
  * [30] Liu, L., Zhang, S., Kuang, Z., Zhou, A., Xue, J.H., Wang, X., Chen, Y., Yang, W., Liao, Q., Zhang, W.: Group fisher pruning for practical network compression. In: International Conference on Machine Learning. pp. 7021–7032. PMLR (2021)
  * [31] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
  * [32] Liu, Z., Mu, H., Zhang, X., Guo, Z., Yang, X., Cheng, K.T., Sun, J.: Metapruning: Meta learning for automatic neural network channel pruning. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 3296–3305 (2019)
  * [33] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11976–11986 (2022)
  * [34] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [35] Louizos, C., Welling, M., Kingma, D.P.: Learning sparse neural networks through $l\\_0$ regularization. arXiv preprint arXiv:1712.01312 (2017)
  * [36] Ma, X., Yuan, G., Lin, S., Li, Z., Sun, H., Wang, Y.: Resnet can be pruned 60$\times$: Introducing network purification and unused path removal (p-rm) after weight pruning. In: 2019 IEEE/ACM International Symposium on Nanoscale Architectures (NANOARCH). pp. 1–2. IEEE (2019)
  * [37] Ma, X., Fang, G., Wang, X.: Llm-pruner: On the structural pruning of large language models. In: Advances in Neural Information Processing Systems (2023)
  * [38] maintainers, T., contributors: Torchvision: Pytorch’s computer vision library. GitHub repository (2016)
  * [39] Molchanov, P., Mallya, A., Tyree, S., Frosio, I., Kautz, J.: Importance estimation for neural network pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11264–11272 (2019)
  * [40] Molchanov, P., Tyree, S., Karras, T., Aila, T., Kautz, J.: Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440 (2016)
  * [41] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
  * [42] Peng, H., Wu, J., Chen, S., Huang, J.: Collaborative channel pruning for deep networks. In: International Conference on Machine Learning. pp. 5113–5122. PMLR (2019)
  * [43] Pool, J., Sawarkar, A., Rodge, J.: Accelerating inference with sparsity using the nvidia ampere architecture and nvidia tensorrt. NVIDIA Developer Technical Blog, https://developer. nvidia. com/blog/accelerating-inference-with-sparsityusing-ampere-and-tensorrt (2021)
  * [44] Radosavovic, I., Kosaraju, R.P., Girshick, R., He, K., Dollár, P.: Designing network design spaces. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10428–10436 (2020)
  * [45] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)
  * [46] Shi, D., Tao, C., Jin, Y., Yang, Z., Yuan, C., Wang, J.: UPop: Unified and progressive pruning for compressing vision-language transformers. In: Proceedings of the 40th International Conference on Machine Learning. vol. 202, pp. 31292–31311. PMLR (2023)
  * [47] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [48] Song, Z., Xu, Y., He, Z., Jiang, L., Jing, N., Liang, X.: Cp-vit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint arXiv:2203.04570 (2022)
  * [49] Sun, M., Liu, Z., Bair, A., Kolter, J.Z.: A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695 (2023)
  * [50] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [51] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jegou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning. vol. 139, pp. 10347–10357 (July 2021)
  * [52] Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., Jégou, H.: Going deeper with image transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 32–42 (2021)
  * [53] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [54] Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 (2019)
  * [55] Wang, W., Xie, E., Li, X., Fan, D.P., Song, K., Liang, D., Lu, T., Luo, P., Shao, L.: Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media 8(3), 415–424 (2022)
  * [56] Wang, Z., Luo, H., Wang, P., Ding, F., Wang, F., Li, H.: Vtc-lfc: Vision transformer compression with low-frequency components. Advances in Neural Information Processing Systems 35, 13974–13988 (2022)
  * [57] Wen, W., Wu, C., Wang, Y., Chen, Y., Li, H.: Learning structured sparsity in deep neural networks. Advances in neural information processing systems 29 (2016)
  * [58] Wightman, R.: Pytorch-image-models. https://github.com/huggingface/pytorch-image-models
  * [59] Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, I.S., Xie, S.: Convnext v2: Co-designing and scaling convnets with masked autoencoders. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16133–16142 (2023)
  * [60] Yang, H., Yin, H., Shen, M., Molchanov, P., Li, H., Kautz, J.: Global vision transformer pruning with hessian-aware saliency. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 18547–18557 (2023)
  * [61] Ye, J., Lu, X., Lin, Z., Wang, J.Z.: Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124 (2018)
  * [62] Yin, M., Uzkent, B., Shen, Y., Jin, H., Yuan, B.: Gohsp: A unified framework of graph and optimization-based heterogeneous structured pruning for vision transformer. arXiv preprint arXiv:2301.05345 (2023)
  * [63] Yu, F., Huang, K., Wang, M., Cheng, Y., Chu, W., Cui, L.: Width & depth pruning for vision transformers. In: AAAI Conference on Artificial Intelligence (AAAI). vol. 2022 (2022)
  * [64] Yu, H., Wu, J.: A unified pruning framework for vision transformers. Science China Information Sciences 66(7), 1–2 (2023)
  * [65] Yu, J., Huang, T.: Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 (2019)
  * [66] Yu, L., Xiang, W.: X-pruner: explainable pruning for vision transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24355–24363 (2023)
  * [67] Yu, S., Chen, T., Shen, J., Yuan, H., Tan, J., Yang, S., Liu, J., Wang, Z.: Unified visual transformer compression. arXiv preprint arXiv:2203.08243 (2022)
  * [68] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 6023–6032 (2019)
  * [69] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)
  * [70] Zheng, C., Zhang, K., Yang, Z., Tan, W., Xiao, J., Ren, Y., Pu, S., et al.: Savit: Structure-aware vision transformer pruning via collaborative optimization. Advances in Neural Information Processing Systems 35, 9010–9023 (2022)
  * [71] Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 34, pp. 13001–13008 (2020)
  * [72] Zhu, M., Tang, Y., Han, K.: Vision transformer pruning. arXiv preprint arXiv:2104.08500 (2021)

##  A Details of Vision Transformer Pruning

Vision transformers, in contrast to Convolutional Neural Networks, encompass a
more diverse composition of substructures within their network architecture.
This section presents a detailed case study on vision transformers,
elucidating the isomorphic pruning process. As depicted in Figure 5, a
fundamental block of vision transformers, as described in [10], comprises a
multi-head attention layer and a Multi-Layer Perceptron (MLP) layer. We
annotate the dimensions of intermediate features and demarcate their
isomorphic groups using different colors. Owing to their heterogeneous
composition, vision transformers naturally form several groups:

##### The Embedding Group:

This group encompasses parameters responsible for generating intermediate
features between modules, of witch the dimension is marked as $E$. In the ViT-
Base model, as specified in [10], the embedding size is typically 768. The
presence of residual connections mandates uniformity in embedding sizes across
different blocks, necessitating simultaneous pruning. Consequently, the
embedding group in a ViT-Base model comprises $E=768$ substructures.

##### The MLP Group:

A vision transformer includes several MLP layers, each with an identical
structure. This group maps $N\times E$ embeddings to $N\times M$ intermediate
results before transforming them back into $E$-dimensional features. Dimension
$M$ is pruned within this group to effectively reduce the model size.

##### Head Dimension Group:

Central to the vision transformer is the self-attention module, which
aggregates information across tokens. A typical self-attention module maps
embeddings to Query, Key, and Value, with the dimension such as $N\times
H\times Q$ for the Query. The dimensions of $Q$ and $K$ must be identical,
while the dimension of $V$ can be set variably. However, many implementations,
such as Pytorch-Image-Models [58], require identical QKV dimensions.
Therefore, this work prunes $Q$, $K$, and $V$ concurrently. Additionally, the
input dimension of the subsequent projection layer is adjusted accordingly.

##### Head Group:

In addition to the aforementioned groups for width pruning, the pruning of the
attention head is also considered for further acceleration. This involves
compressing the $H$ dimension as shown in Figure 5.

The above analysis of substructures within vision transformers presents 4
unique isomorphic groups, associated with the dimensions $E,QKV,H$, and $M$,
in which all elements have the same architecture and computational topology.
In isomorphic pruning, ranking is applied within each isomorphic group for a
reliable comparison. For vision transformers, the above analysis is feasible
since all sub-structures are aligned with the modular design. However, for
CNNs like ConvNext, ResNet, the substructures can be more complicated. Thus,
our implementation automate the identification of substructures with
dependency analysis [11, 5, 30] as discussed in the main paper.

Figure 5: The isomorphic groups in a vision transformer block. There are three groups for width pruning, which reduces the dimensions of embedding, MLP and attention. One special group works in the head level, which removes entire heads for acceleration. The shapes of intermediate features are hightlighted. Architecture | Base Model | Emb% | Head% | Dim%
---|---|---|---|---
DeiT-S | DeiT-B | 50% | 50% | 25%
DeiT-2.6G | DeiT-B | 60% | 60% | 30%
DeiT-T | DeiT-S | 50% | 50% | 10%
DeiT-0.6G | DeiT-T | 25% | 30% | 30%
Table 5: The target architecture and corresponding configurations for pruning.
Specifically, “Emb”, “Head”, and “Dim” refer to the pruning ratios for the
number of heads, the size of embeddings, and the size of head dimensions.

Figure 6: The number of pruning functions in each isomorphic group. The types of pruning functions are highlighted with different colors. In isomorphic pruning, we perform ranking and pruning within each isomorphic group separately. Method | #Params (M) | MACs (G) | Acc (%)
---|---|---|---
Swin-B† | 87.77 | 15.48 | 83.42
Swin-T† | 28.29 | 4.51 | 81.19
Swin-T (Ours) | 24.66 | 4.47 | 81.32
X-Pruner [66] | N/A | 3.20 | 80.70
Table 6: Pruning results for Swin Transformers pre-trained on ImageNet-1K.

##### Pruning Ratios for Each Isomorphic Group:

The established grouping facilitates the allocation of different pruning
ratios to different isomorphic groups. In our experiments, we employed the
pruning ratios detailed in Table 5 to accelerate transformer models. For the
DeiT-S and DeiT-T models, we keep the number of heads and embedding dimensions
after pruning the same as the official models [51], while modifying the head
dimensions to achieve further model compression. DeiT-2.6G uses scaled pruning
ratios based on the DeiT-S configurations.

##### Swin Transformers

Our method can be directly applied to Swin Transformers [31], which introduces
local attention for better performance. We prune the Swin-Base to obtain 6G
and 4.5G models and compare them to several baselines such as WDPruning [63],
NViT-H [60] and X-Pruner [66]. The proposed method achieves better results
compared to existing methods and pre-trained baselines.

##  B Details of CNN Pruning

This section elaborates on the pruning strategies applied to ConvNext, ResNet,
and MobileNet-v2 networks. Compared to Transformers, Convolutional Neural
Networks used in our experiments are more irregular, with intricated internal
connections. We model the substructures are modeled as graphs, which
facilitates automation of the pruning process, thereby obviating the need for
cumbersome manual analysis. Figure 6 visually illustrates the statistics of
the detected isomorphic groups. In isomorphic pruning, we seperately eliminate
parameters from each distinct group. These groups encompass multiple sub-
structures, varying in the number of layers. As elucidated in the main paper,
each layer can be subject to two types of pruning functions. Assessing the
relative significance of these substructures presents a challenge. Figure 6
illustrates the number of different pruning functions across various
isomorphic groups. The x-axis represents the group ID, denoting isomorphic
groups with identical graph structures. The y-axis quantifies the total number
of pruning functions for each group, corresponding to the number of total
nodes in the graph modeling. Different pruning functions are colored for
better illustration. For instance, pruning functions targeting the input and
output dimensions of a linear layer are distinguished by unique colors, since
the pruning is performed on different dimensions.

Observations in Figure 6 reveal significant variability in the composition of
each isomorphic group, potentially leading to unreliable rankings if a
straightforward global pruning approach is employed. Additionally, the varying
sizes of the isomorphic groups suggest that independent pruning within each
group achieves a more balanced acceleration across different substructures,
akin to local pruning. This approach also retains the flexibility to
dynamically adjust pruning ratios for different layers.

Training Configs | DeiT | ConvNext | ResNet-50 | MobileNet-v2 | Swin
---|---|---|---|---|---
optimizer | AdamW | AdamW | SGD | SGD | AdamW
base learning rate | 0.0005 | 0.001 | 0.08 | 0.036 | 0.0005
weight decay | 0.05 | 0.05 | 1e-4 | 4e-5 | 0.05
optimizer momentum | (0.9, 0.999) | (0.9, 0.999) | 0.9 | 0.9 | (0.9, 0.999)
batch size | 2048 | 1024 | 1024 | 4096 | 2048
training epochs | 300 | 300 | 100 | 300 | 300
learning rate schedule | cosine | cosine | 30,60,90 | cosine | cosine
warmup epochs | 0 | 0 | 0 | 0 | 0
layer-wise lr decay | 0 | 0 | 0 | 0 | 0
randaugment | $\checkmark$ | $\checkmark$ | None | None | $\checkmark$
mixup | 0.2 | 0.2 | None | None | 0.2
cutmix | 1.0 | 1.0 | None | None | 1.0
random erasing | 0.25 | 0.25 | None | None | 0.25
label smoothing | 0.1 | 0.1 | None | None | 0.1
stochastic depth | 0 | 0.1 (S) / 0.4 (T) | None | None | 0.1
layer scale | None | None | None | None | None
gradient clip | 5 | None | None | None | 5
exp. mov. avg. (EMA) | None | 0.9999 | None | None | None

Table 7: Pruning Plain Vision Transformers. All pruned models are only fine-
tuned on ImageNet-1K.

##  C Experimental Details

##### Training.

This section further details the training process and hyper-parameters in our
experiments. We report the training configurations including optimizer,
learning rate, and augmentation in Table 7. All models are fine-tuned with 8
RTX A5000 GPUs, with Automatic Mixed Precision implemented by PyTorch [41].
For DeiT and ConvNext, we use strong augmentations as mentioned in the
original paper [51, 59], but did not deploy warmup, layer-wise lr decay and
layer scale for training. ResNet and MobileNet-v2 were trained with weak
augmentation described in [19, 45].

##### Latency Test

For the Latency test on GPU, we forward the model a batch size of 256 for
20-step warmup and 100-step experiments. We report the average execution time
of the 100 rounds. For CPU test, we deploy a batch size of 8 and follow the
same principle as GPU testing.

##  D Limitations

In this study, we empirically examine the impact of isomorphic sub-structures
on pruning. Structural similarity might not be the sole determinant of
importance distribution. Factors such as the training methodology,
regularization techniques, and network depth can also play significant roles
in shaping different distributions. While our experiments demonstrate the
effectiveness of isomorphic pruning, further investigation in the future into
these additional factors is necessary for a more comprehensive framework.