# Adaptive Depth Networks with Skippable Sub-Paths

Woochul Kang
Deparment of Embedded Systems
Incheon National University
119 Academy-Ro, Yeonsu-Gu, Incheon, South Korea
<EMAIL_ADDRESS>
###### Abstract

Systematic adaptation of network depths at runtime can be an effective way to
control inference latency and meet the resource condition of various devices.
However, previous depth adaptive networks do not provide general principles
and a formal explanation on why and which layers can be skipped, and, hence,
their approaches are hard to be generalized and require long and complex
training steps. In this paper, we present an architectural pattern and
training method for adaptive depth networks that can provide flexible
accuracy-efficiency trade-offs in a single network. In our approach, every
residual stage is divided into 2 consecutive sub-paths with different
properties. While the first sub-path is mandatory for hierarchical feature
learning, the other is optimized to incur minimal performance degradation even
if it is skipped. Unlike previous adaptive networks, our approach does not
iteratively self-distill a fixed set of sub-networks, resulting in
significantly shorter training time. However, once deployed on devices, it can
instantly construct sub-networks of varying depths to provide various
accuracy-efficiency trade-offs in a single model. We provide a formal
rationale for why the proposed architectural pattern and training method can
reduce overall prediction errors while minimizing the impact of skipping
selected sub-paths. We also demonstrate the generality and effectiveness of
our approach with various residual networks, both from convolutional neural
networks and vision transformers.

##  1 Introduction

Modern deep neural networks such as convolutional neural networks (CNNs) and
transformers [44] provide state-of-the-art performance at high computational
costs, and, hence, lots of efforts have been made to leverage those inference
capabilities in various resource-constrained devices. Those efforts include
compact architectures [21, 13], network pruning [14, 33], weight/activation
quantization [26], knowledge distillation [19], to name a few. However, those
approaches provide static accuracy-efficiency trade-offs, and, hence, it is
infeasible to deploy one single model to meet devices with all kinds of
resource-constraints.

There have been some attempts to provide predictable adaptability to neural
networks by exploiting the redundancy in either network depths [24, 6], widths
[59, 48], or both [47, 20]. However, one major difficulty with prior adaptive
networks is that they are hard to train and require significantly longer
training time than non-adaptive networks. For example, most adaptive networks
select a fixed number of sub-networks of varying depths or width, and train
them iteratively, mostly through self-distilling knowledge from the largest
sub-network (also referred to as the _super-net_) [59, 48, 20, 43]. However,
this iterative self-distillation takes long time and can generate conflicting
training objectives for different parameter-sharing sub-networks, potentially
resulting in worse performance [30, 10]. Further, unlike width-adaptation
networks, no general principle has been proposed for how to select sub-
networks for depth adaptation, since the effect of skipping individual layers
has not been formally specified.

Figure 1: (a) At training time, the last half layers of every residual stage
is optimized to have minimal performance degradation even if they are skipped.
No explicit self-distillation is required for sub-networks, except the
smallest base-net. (b) At test time, $2^{N_{r}}$ sub-networks become available
in a single network by systematically skipping last half layers of $N_{r}$
residual stages. (c) Sub-networks instantly selected from a single network
outperform counterpart individually trained networks due to regularization
effect.

In this work, we introduce an architectural pattern and training method for
adaptive depth networks that is generally applicable to residual networks,
e.g., CNNs and transformers. In the proposed adaptive depth networks, every
residual stage is divided into 2 consecutive sub-paths and they are trained to
have different properties. While the first sub-paths are mandatory for
hierarchical feature learning, the second sub-paths are optimized to incur
minimal performance degradation even if they are skipped. More specifically,
the second sub-path of every residual stage is optimized to preserve the
feature distribution from its previous mandatory sub-path in order to minimize
the performance degradation when it is skipped. During training, this property
of the second sub-paths is enforced through _skip-aware self-distillation_ ,
in which only one smallest sub-network, also referred as the _base-net_ , is
jointly trained in order to self-distill intermediate feature distributions at
every residual stage, as shown in Figure-1-(a). This skip-aware self-
distillation requires no explicit self-distillation or additional fine-tuning
of individual sub-networks except the base-net, resulting in significantly
shorter training time than previous adaptive networks. However, once trained,
sub-networks with various depths can be selected instantly from a single
network to meet the resource condition of devices, as shown in Figure 1-(b).
Further, these sub-networks with various depths outperform individually
trained non-adaptive networks due to the regularization effect, as shown in
Figure 1-(c).

In Section 3, we discuss the details of our architectural pattern and training
algorithm, and show formally that the chosen sub-paths trained with our skip-
aware self-distillation are optimized to reduce prediction errors while
minimally changing the level of input features. In Section 4, we empirically
demonstrate that our adaptive depth networks with skippable sub-paths
outperform counterpart individual networks, both in CNNs and vision
transformers, and achieve actual inference acceleration and energy-saving.

To authors’ best knowledge, this work is the first general approach to
adaptive depth networks, providing a general principle for depth adaptation
and a formal explanation on why layers can be skipped with minimal performance
degradation.

Figure 2: Illustration of a residual stage with 2 sub-paths. While the first
sub-path is mandatory for hierarchical feature learning, the second (yellow)
sub-path can be skipped for efficiency. The layers in the skippable sub-path
are trained to preserve the feature distribution from $\mathbf{h}^{s}_{base}$
to $\mathbf{h}^{s}_{super}$ using _skip-aware self-distillation_. Having
similar feature distributions, either $\mathbf{h}^{s}_{base}$ or
$\mathbf{h}^{s}_{super}$ can be provided as an input to the next residual
stage. In the mandatory sub-path, another set of batch normalization
operators, called _skip-aware BNs_ , is exploited when the second sub-path is
skipped. This architectural pattern repeats in every residual stage throughout
the network.

##  2 Related Work

Adaptive Networks: In most adaptive networks, parameter-sharing sub-networks
are selected by adjusting either widths, depths, or resolutions [24, 59, 48,
22, 20, 30, 61, 47, 3]. For example, slimmable neural networks adjust channel
widths of CNN models on the fly for accuracy-efficiency trade-offs and they
exploit switchable batch normalization to handle multiple sub-networks [59,
58, 48]. Transformer-based adaptive depth networks have been proposed for
language models to dynamically skip some of the layers during inference [20,
6]. However, in these adaptive networks, every target sub-network with varying
widths or depths need to be trained explicitly, incurring significant training
overheads and potential conflicts between sub-networks.

_Dynamic networks_ [15] are another class of adaptive networks that exploit
additional control networks or decision gates for input-dependent adaption of
CNN models [51, 29, 12, 31, 53] and transformers [35, 55, 8, 18]. In
particular, most dynamic networks for depth-adaptation have some kinds of
decision gates at every layers (or blocks) that determine if the layers can be
skipped [9, 51, 45, 50]. These approaches are based on the thought that some
layers can be skipped on ‘easy’ inputs. However, the learned policy for
skipping layers is opaque to users and does not provide a formal description
of when and which layers can be skipped for a given input. Therefore, the
network depths cannot be adapted in a predictable manner to meet the resource
condition of target devices.

Residual Blocks with Shortcuts: Since the introduction of ResNets [16],
residual blocks with shortcuts have received extensive attention because of
their ability to train very deep networks, and have been chosen by many CNNs
[38, 41] and transformers [44, 5, 34]. In [46], Veit et al. argue that
identity shortcuts make exponential paths and results in an ensemble of
shallower sub-networks. This thought is supported by the fact that removing
individual residual blocks at test time does not significantly affect
performance, and it has been further exploited to train deep networks [23,
52]. Other works argue that identity shortcuts enable residual blocks to
perform iterative feature refinement, where each block improves slightly but
keeps the semantic of the representation of the previous layer [11, 27]. Our
work build upon those views on residual blocks with shortcuts and further
extend them for adaptive depth networks by introducing an architectural
pattern and training method that exploits the properties of residual blocks
more explicitly for selected sub-paths.

##  3 Adaptive Depth Networks

We first present an architectural pattern and training details to build
adaptive depth networks. And, then, we discuss the theoretic rationale for how
depth adaption can be achieved with minimal performance degradation.

###  3.1 Architectural Pattern for Depth Adaptation

In typical hierarchical residual networks such as ResNets [16] and Swin
transformers [34], the $s$-th residual stage is consisted of $L$ identical
residual blocks, which transform input features $\mathbf{h}^{s}_{1}$
additively to produce the output features $\mathbf{h}^{s}$, as follows:

$\displaystyle\underbrace{\underbrace{\mathbf{h}^{s}_{1}+\underbrace{F_{1}(\mathbf{h}^{s}_{1})+...+{F}^{s}_{L/2}(\mathbf{h}^{s}_{L/2})}_{\mathbf{F}^{s}_{base}}}_{=\mathbf{h}^{s}_{base}}+\underbrace{...+F_{L}(\mathbf{h}^{s}_{L})}_{\mathbf{F}^{s}_{skippable}}}_{=\mathbf{h}^{s}_{super}}=\mathbf{h}^{s}$
(1)

While a block with a residual function $F_{\ell}$ ($\ell=1,...,L$) learns
hierarchical features as traditional compositional networks [40], previous
literature [27, 11] demonstrates that a residual function also tend to learn a
function that refines already learned features at the same feature level. If a
residual block mostly performs feature refinement while not changing the level
of input features, the performance of the residual network is not
significantly affected by dropping the block at test time [23, 52]. However,
in typical residual networks, most residual blocks tend to refine features
while learning new level features as well, and, hence, random dropping of
residual blocks at test time degrades the performance significantly.
Therefore, we hypothesize that if some designated residual blocks can be
encouraged during training to focus more on feature refinement, then these
blocks can be skipped to save computation at marginal loss of prediction
accuracy at test time.

To this end, we propose an architectural pattern for adaptive depth networks,
in which every residual stage is divided into two consecutive sub-paths, or
$\mathbf{F}^{s}_{base}$ and $\mathbf{F}^{s}_{skippable}$ as in Equation 1 and
Figure 2. While $\mathbf{F}^{s}_{base}$ learns feature representation
$\mathbf{h}^{s}_{base}$ (= $\mathbf{h}^{s}_{L/2+1}$) with no constraint, the
second sub-path $\mathbf{F}^{s}_{skippable}$ is constrained to preserve the
feature level of $\mathbf{h}^{s}_{base}$ and only refine it to produce
$\mathbf{h}^{s}_{super}$. Since layers in $\mathbf{F}^{s}_{base}$ perform
essential transformations for hierarchical feature learning, they cannot be
bypassed during inference. However, since layers in
$\mathbf{F}^{s}_{skippable}$ only refine $\mathbf{h}^{s}_{base}$, they can be
skipped to save computation. If $\mathbf{F}^{s}_{skippable}$ is skipped, then
intermediate features $\mathbf{h}^{s}_{base}$ becomes the input to the next
residual stage. Therefore, the overall network depth can be adjusted by
choosing whether to skip $\mathbf{F}^{s}_{skippable}(s=1,...,N_{s})$ or not.

In Section 4.1, we show that this architectural pattern for adaptive depth
networks is generally applicable to a wide range of residual networks.

###  3.2 Skip-Aware Self-Distillation

Preserving the feature level of $\mathbf{h}^{s}_{base}$ in
$\mathbf{F}^{s}_{skippable}$ implies, more specifically, that two feature
representations $\mathbf{h}^{s}_{base}$ and $\mathbf{h}^{s}_{super}$ have
similar distributions over training input $\mathbf{X}$. If
$\mathbf{h}^{s}_{super}$ and $\mathbf{h}^{s}_{base}$ have similar
distributions, skipping the sub-path $\mathbf{F}^{s}_{skippable}$ during
inference results in minimal internal covariate shifts [25] to the following
network layers. Kullback-Leibler (KL) divergence measures how different two
distributions are over the same random variable, and, hence, we use Equation 2
to measure the similarity of two distributions, $\mathbf{h}^{s}_{base}$ and
$\mathbf{h}^{s}_{super}$, over input $\mathbf{X}$:

$\displaystyle\underset{x\in\mathbf{X}}{D_{KL}}(\mathbf{h}^{s}_{super}||\mathbf{h}^{s}_{base})$
(2)

Algorithm 1 shows our training method, called _skip-aware self-distillation_ ,
in which Equation 2 is included in the loss function while the largest and the
smallest sub-networks of $\mathbf{M}$, which are called _super-net_ and the
_base-net_ , respectively, are jointly trained 111Some frameworks, such as
Pytorch’s distributed data parallel (DDP), do not support two consecutive
forwards passes followed by a single backward pass: $super.forward\rightarrow
base.fowrard\rightarrow loss.backward$. For such frameworks, we adapt the
algorithm to $super.forward\rightarrow loss_{super}.backward\rightarrow
base.forward\rightarrow loss_{sub\\_path}.backward$. The latter method has
similar results and is more memory-efficient. . In Algorithm 1, the forward
function of the proposed adaptive depth networks $\mathbf{M}$ accepts an extra
argument, ’skip’, that controls which residual stages skip their skippable
sub-paths. For example, if $\mathbf{M}$ has 4 residual stages, its _base-net_
is selected by passing ‘skip=[True, True, True, True]’.

Algorithm 1 Training algorithm for an adaptive depth network $\mathbf{M}$. Two
parameter-sharing sub-networks, or _super-net_ and _base-net_ , are jointly
trained using _skip-aware self-distillation._ $N_{r}$ denotes the number of
residual stages.

1:Initialize an adaptive depth network $\mathbf{M}$

2:for $i=1$ to $n_{iters}$ do

3: Get next mini-batch of data $\mathbf{x}$ and label $\mathbf{y}$

4: $optimizer.zero\\_grad()$

5:
$\mathbf{\hat{y}}_{super}=\mathbf{M}.forward(\mathbf{x},\textrm{skip=[False,...,
False]})$

6: $loss_{super}=criterion(\mathbf{y},\mathbf{\hat{y}}_{super})$

7: $\mathbf{\hat{y}}_{base}=\mathbf{M}.forward(\mathbf{x},\textrm{skip=[True,
... , True]})$

8:
$loss_{base}=\sum_{s=1}^{N_{r}}D_{KL}(\mathbf{h}^{s}_{super}\|\mathbf{h}^{s}_{base})\newline
\hskip 50.00008pt+D_{KL}(\mathbf{\hat{y}}_{super}\|\mathbf{\hat{y}}_{base})$

9: $loss=\alpha loss_{super}+(1-\alpha)loss_{base}$

10: $loss.backward()$

11: $optimizer.step()$

12:end for

In steps 5 and 7, the forward passes of the super-net and the base-net are
executed for the same input $\mathbf{x}$. Intermediate features
$\mathbf{h_{super}}$ and $\mathbf{h_{base}}$ are also obtained during the
forward passes of $\mathbf{M}$. In step 8,
$D_{KL}(\mathbf{h}^{s}_{super}||\mathbf{h}^{s}_{base})$ in Equation 2 is
included in the loss function $loss_{base}$. By minimizing $loss_{base}$, two
feature representations $\mathbf{h}^{s}_{super}$ and $\mathbf{h}^{s}_{base}$,
are explicitly enforced to have similar distributions for the same input
$\mathbf{x}$. Further, this step gives the effect of transferring the
knowledge from $\mathbf{h}^{s}_{super}$ to $\mathbf{h}^{s}_{base}$ at every
residual stage. Therefore, $\mathbf{h}^{s}_{base}$ is expected to learn more
compact representation from $\mathbf{h}^{s}_{super}$. The hyperparameter
$\alpha$ controls the strength of this skip-aware self-distillation. In step
8, $D_{KL}(\mathbf{\hat{y}}_{super}\|\mathbf{\hat{y}}_{base})$ is included in
the loss function for further distillation effect from the super-net to the
sub-networks. Due to the architectural pattern of interleaving the mandatory
and the skippable sub-paths, minimizing
$D_{KL}(\mathbf{\hat{y}}_{super}\|\mathbf{\hat{y}}_{base})$ also minimizes
$D_{KL}(\mathbf{h}_{super}\|\mathbf{h}_{base})$ implicitly. Experimental
result shows that similar results can be achieved when only
$D_{KL}(\mathbf{\hat{y}}_{super}\|\mathbf{\hat{y}}_{base})$ is used in
$loss_{base}$ (Section 4.5). This implicit approach is useful when the
extraction of intermediate features is tricky.

In Algorithm 1, only two sub-networks are involved for the training of
$\mathbf{M}$, and, hence, the total training time is no greater than training
two sub-networks individually. However, at test time, sub-networks with
various depths can be selected on the fly by systematically skipping sub-paths
in residual stages. For example, in a network with 4 residual stages, $16\
(=2^{4})$ parameter-sharing sub-networks can be selected by varying the _skip_
argument. In contrast, prior adaptive networks supporting 16 parameter-sharing
sub-networks need to perform 16 explicit self-distillation from the super-net
to the sub-networks [48, 59, 20]. We demonstrate this result in Section 4.3.

###  3.3 Formal Analysis of Skippable Sub-Paths

$D_{KL}(\mathbf{h}^{s}_{super}||\mathbf{h}^{s}_{base})$ in Equation 2 can be
trivially minimized if residual blocks in $\mathbf{F}^{s}_{skippable}$ learn
identity functions, or
$\mathbf{h}^{s}_{base}+\mathbf{F}^{s}_{skippable}(\mathbf{h}^{s}_{base})=\mathbf{h}^{s}_{base}$.
However, since the super-net is jointly trained with the loss function
$loss_{super}$, the residual functions in $\mathbf{F}^{s}_{skippable}$ cannot
simply be an identity function. Then, what do the residual functions in
$\mathbf{F}^{s}_{skippable}$ learn during training? This can be further
investigated through Taylor expansion [27]. For our adaptive depth networks, a
loss function $\mathcal{L}$ used for training the super-net can be
approximated with Taylor expansion as follows:

$\displaystyle\mathcal{L}(\mathbf{h}^{s}_{super})=\mathcal{L}\\{\mathbf{h}^{s}_{base}+\mathbf{F}^{s}_{skippable}(\mathbf{h}^{s}_{base})\\}$
(3)
$\displaystyle=\mathcal{L}\\{\mathbf{h}^{s}_{base}+...+F_{L-1}(\mathbf{h}^{s}_{L-1})+F_{L}(\mathbf{h}^{s}_{L})\\}$
(4)
$\displaystyle\approx\mathcal{L}\\{\mathbf{h}^{s}_{base}+...+F_{L-1}(\mathbf{h}^{s}_{L-1})\\}$
$\displaystyle\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
+F_{L}(\mathbf{h}^{s}_{L})\cdot\frac{\partial\mathcal{L}(\mathbf{h}^{s}_{L})}{\partial\mathbf{h}^{s}_{L}}+\mathcal{O}(F_{L}(\mathbf{h}_{L}^{s}))$
(5)

In Equation 5, the loss function is expanded around $\mathbf{h}^{s}_{L}$, or
$\mathbf{h}^{s}_{base}+...+F_{L-1}(\mathbf{h}^{s}_{L-1})$. Only the first
order term is left and all high order terms, such as
$F_{L}(\mathbf{h}_{L}^{s})^{2}\cdot\frac{{\partial^{2}\mathcal{L}(\mathbf{h}^{s}_{L})}}{2{\partial(\mathbf{h}^{s}_{L})^{2}}}$,
are absorbed in $\mathcal{O}(F_{L}(\mathbf{h}_{L}^{s}))$.

The terms in $\mathcal{O}(F_{L}(\mathbf{h}_{L}^{s}))$ can be ignored if
$F_{L}(\mathbf{h}_{L}^{s})$ has a small magnitude. In typical residual
networks, however, every layer is trained to learn new features with no
constraint, and, hence, there is no guarantee that $F_{L}(\mathbf{h}^{s}_{L})$
have small magnitude. In contrast, in our adaptive depth networks, the
residuals in $\mathbf{F}^{s}_{skippable}$ are explicitly enforced to have
small magnitude through the skip-aware self-distillation, and, hence, the
terms in $\mathcal{O}(F_{L}(\mathbf{h}_{L}^{s}))$ can be ignored for the
approximation. If we similarly keep expanding the loss function around
$\mathbf{h}_{j}$ ($j=L/2+1,...,L$) while ignoring high order terms, we obtain
the following approximation:

$\displaystyle\mathcal{L}(\mathbf{h}^{s}_{super})\approx\mathcal{L}(\mathbf{h}^{s}_{base})+\sum_{j={L/2+1}}^{L}F_{j}(\mathbf{h}^{s}_{j})\cdot\frac{\partial\mathcal{L}(\mathbf{h}^{s}_{j})}{\partial\mathbf{h}^{s}_{j}}$
(6)

In Equation 6, minimizing the loss $\mathcal{L}(\mathbf{h}^{s}_{super})$
during training drives $F_{j}(\mathbf{h}^{s}_{j})$ $(j=L/2+1,...,L)$ in the
negative half space of $\textrm{
}\frac{\partial\mathcal{L}(\mathbf{h}^{s}_{j})}{\partial\mathbf{h}^{s}_{j}}$
to minimize the dot product between $F_{j}(\mathbf{h}^{s}_{j})$ and
$\frac{\partial\mathcal{L}(\mathbf{h}^{s}_{j})}{\partial\mathbf{h}^{s}_{j}}$.
This implies that every residual function in $\mathbf{F}^{s}_{skippable}$ is
optimized to learn a function that has a similar effect to gradient descent:

$\displaystyle F_{j}(\mathbf{h}^{s}_{j})\simeq\mathbf{-}\textrm{
}\frac{\partial\mathcal{L}(\mathbf{h}^{s}_{j})}{\partial\mathbf{h}^{s}_{j}}\textrm{
}(j=L/2+1,...,L)$ (7)

In other words, the residual functions in the skippable sub-paths reduce the
loss $\mathcal{L}(\mathbf{h}^{s}_{base})$ iteratively during inference while
preserving the feature distribution of $\mathbf{h}^{s}_{base}$.

Considering this result, we can conjecture that, with our architectural
pattern and training method, layers in $\mathbf{F}^{s}_{skippable}$ learn
functions that refine input features $\mathbf{h}^{s}_{base}$ iteratively for
better inference accuracy while minimally changing the distribution of
$\mathbf{h}^{s}_{base}$. Therefore, skipping $\mathbf{F}^{s}_{skippable}$ only
slightly reduces prediction accuracy.

###  3.4 Skip-aware Batch Normalization

Originally, batch normalization (BN) [25] was proposed to handle internal
covariate shift during training non-adaptive networks by normalizing features.
In our adaptive depth networks, however, internal covariate shifts can occur
during inference in mandatory sub-paths if different sub-networks are
selected. To handle potential internal covariate shifts, switchable BN
operators, called _skip-aware BNs_ , are used in mandatory sub-paths. For
example, at each residual stage, two sets of BNs are available for the
mandatory sub-path, and they are switched depending on whether its skippable
sub-path is skipped or not.

The effectiveness of switchable BNs has been demonstrated in networks with
adaptive widths [57, 48] and adaptive resolutions [62]. However, in previous
adaptive networks, $N$ sets of switchable BNs are required in every layer to
support $N$ parameter-sharing sub-networks. Such a large number of switchable
BNs not only requires more parameters, but also makes the training process
complicated since $N$ sets of switchable BNs need to be trained iteratively
during training. In contrast, in our adaptive depth networks, every mandatory
sub-path needs only two sets of switchable BNs, regardless of the number of
supported sub-networks. This reduced number of switchable BNs significantly
simplifies the training process as shown in Algorithm 1. Furthermore, the
amount of parameters for skip-aware BNs is negligible. For instance, in
ResNet50, skip-aware BNs increase the parameters by 0.07%.

Transformers [44, 34] exploit layer normalization (LN) instead of BNs and
naive replacement of LNs to BNs incurs instability during training [54].
Therefore, for our adaptive depth transformers, we apply switchable LN
operators in mandatory sub-paths instead of switchable BNs.

##  4 Experiments

To demonstrate the generality and effectiveness of our approach for adaptive
depth networks, we conduct experiments on various networks and vision tasks.

###  4.1 Networks

We use four representative residual networks as base models to apply the
proposed architecture pattern in Section 3.1: MobileNet V2 [38] is a
lightweight CNN model, ResNet [16] is a larger CNN model, and ViT [5] and
Swin-T [34] are representative vision transformers. All base models except ViT
have 4 residual stages, each with 2 $\sim$ 6 (residual or encoder) blocks. So,
according to the proposed architectural pattern, every residual stage is
evenly divided into 2 sub-paths for depth adaptation. Since ViT does not
define residual stages, we divide 12 encoder blocks into 4 groups, resembling
other residual networks, and desiginate the last encoder block of each group
as a skippable sub-path.

We use the suffix _‘-ADN’_ to denote our adaptive depth networks. Since our
adaptive depth networks have many parameter-sharing sub-networks in a single
network, we indicate which sub-network is used for evaluation in parenthesis,
e.g., ResNet50-ADN (super-net). For our sub-networks, boolean values are also
used to indicate in which residual stages the sub-paths are skipped. For
example, ResNet50-ADN (base-net) is equivalent to ResNet50-ADN (TTTT).

Table 1:  Results on ImageNet. Individual networks with the suffix ‘-Base’ have the same depths as the base-nets of corresponding adaptive depth networks. Latency is measured on an RTX 3090 (batch size = 64).  Model | Params | FLOPs | Acc@1 | Latency
---|---|---|---|---
(M) | (G) | (%) | (msec)
ResNet50-ADN (super-net) | 25.58 | 4.11 | 77.6 | 43.2
ResNet50-ADN (base-net) | 2.58 | 76.1 | 27.6
ResNet50 (individual) | 25.56 | 4.11 | 76.7 | 43.2
ResNet50-Base (individual) | 17.11 | 2.58 | 75.0 | 27.6
MV2-ADN (super-net) | 3.72 | 0.32 | 72.7 | 15.5
MV2-ADN (base-net) | 0.25 | 70.7 | 13.2
MV2 (individual) | 3.50 | 0.32 | 72.1 | 15.5
MV2-Base (individual) | 2.99 | 0.25 | 70.7 | 13.2
ViT-b/16-ADN (super-net) | 86.59 | 17.58 | 81.3 | 138.6
ViT-b/16-ADN (base-net) | 11.76 | 79.4 | 90.7
ViT-b/16 (individual) | 86.57 | 17.58 | 81.1 | 138.6
ViT-b/16-Base (individual) | 67.70 | 11.76 | 78.7 | 90.7
Swin-T-ADN (super-net) | 28.30 | 4.49 | 81.8 | 74.4
Swin-T-ADN (base-net) | 2.69 | 78.7 | 39.9
Swin-T (individual) | 28.29 | 4.49 | 81.5 | 74.4
Swin-T-Base (individual) | 15.63 | 2.69 | 78.6 | 39.9

###  4.2 ImageNet Classification

We evaluate our method on ILSVRC2012 dataset [37] that has 1000 classes. The
dataset consists of 1.28M training and 50K validation images. For CNN models,
we follow most training settings in the original papers [16, 38], except that
ResNet models are trained for 150 epochs. ViT and Swin-T are trained for 300
epochs, following DeiT’s training recipe [42, 1]. However, in Swin-T-ADN, we
disable stochastic depths [23] for the base-net since the strategy of random
dropping of residual blocks conflicts with our approach to skipping sub-paths.
For fair comparison, our adaptive depth networks and corresponding individual
networks are trained in the same training settings. The hyperparameter
$\alpha$ in Algorithm 1 is set to $0.5$ for all networks.

The results in Table 1 show that our adaptive depth networks outperform
individual counterpart networks even though our sub-networks share parameters
in a single network. Further, our results with vision transformers demonstrate
that our approach is generally applicable to residual networks and compatible
with their state-of-the-art training techniques such as DeiT’s training recipe
[42]. We conjecture that this performance improvement results from effective
distillation of knowledge from $\textbf{h}^{s}_{super}$ to
$\textbf{h}^{s}_{base}$ at each residual stage and the iterative feature
refinement at skippable sub-paths, shown in Equation 7.

In Table 2 and Figure 3, several state-of-the-art efficient inference methods
and dynamic networks are compared with our sub-networks of adaptive depth
networks. The result demonstrates that our adaptive depth networks match or
outperform many state-of-the-art static and dynamic networks across varying
depth ranges.

In Figure 3, sub-networks of ResNet50-ADN are compared to the counterpart
individual ResNets of equivalent depths. In particular, it should be noted
that ResNets trained with knowledge distillation in the same training settings
has worse performance than individual ResNets trained without knowledge
distillation. As reported in previous works, successful knowledge distillation
requires a patient and long training [2], and straightforward knowledge
distillation using ImageNet does not improve the performance of student models
[60, 4]. In contrast, our ResNet50-ADN trained with skip-aware self-
distillation achieves better performance than counterpart ResNets. This
demonstrates that the high performance of adaptive depth networks does not
simply come from the distillation effect, but from the effective combination
of the proposed architectural pattern and the skip-aware self-distillation
strategy.

Figure 3:  Comparison with various approaches on ImageNet. Sub-networks of our adaptive depth networks match or outperform other approaches on ImageNet. ResNet50 (individual) shows individual networks that have equivalent network depths as the sub-networks of ResNet50-ADN. ResNet50 (KD individual) shows individual networks trained using knowledge distillation from pretrained Pytorch ResNet50. In legend, $\dagger$ denotes static compression methods, and $\star$ denotes input-dependent dynamic networks.  Table 2: Our base-nets are compared with state-of-the-art efficient inference methods for various networks. $\dagger$ denotes static compression methods, $\ast$ denotes width-adaptation networks, and $\star$ denotes input-dependent dynamic networks. Note that many of these state-of-the-art methods exploit various non-canonical training techniques, such as iterative retraining and post-processing, while our base-nets are selected from adaptive depth networks without any fine-tuning.  Model | FLOPs |  $\downarrow$FLOPs | Acc@1
---|---|---|---
GReg-1 [49]$\dagger$ | 2.8G | 33% | 76.1%
DR-ResNet50 ($\alpha$=2.0) [62]$\star$ | 2.3G | 44% | 75.3%
DepGraph [7]$\dagger$ | 2.0G | 51% | 75.8%
ResNet50-ADN (base-net) | 2.6G | 37% | 76.1%
AlphaNet-0.75x [48] $\ast$ | 0.21G | 34% | 70.5%
MV2-ADN (base-net) | 0.25G | 22% | 70.7%
Ada-DeiT-S [35]$\star$ | 2.3G | - | 77.3%
WDPruning-0.3-11 [56]$\dagger$ | 2.6G | - | 78.3%
DeiT-S+ATS (not fine-tuned) [8]$\star$ | 3.0G | - | 78.5%
SPViT-Swin-Ti [28]$\star$ | 3.3G | 27% | 80.1%
Swin-T-ADN (base-net) | 2.7G | 40% | 78.7%

###  4.3 Training Time

Table 3: Training time (1 epoch), measured on a Nvidia RTX 4090 (batch size = 128). All compared models are based on ResNet50 or have similar FLOPs and depths.  Model | Training time (min) | # of sub-networks
---|---|---
ResNet50 | 32.9 | 1
ResNet50-Base | 22.7 | 1
ResNet50-ADN (ours) | 54.7 |  $2^{4}$ (varying depths)
MSDNet [24] | 67.1 | 9 (early-exit branches)
S-ResNet50 [59] | 82.5 | 4 (varying widths)
SkipNet [50] | 90.6 | dynamic depths

One important advantage of our approach is that our adaptive depth networks
require significantly shorter training time than other adaptive networks.
Table 3 shows that training our ResNet50-ADN takes similar time to train two
individual networks combined. In contrast, the compared adaptive networks
requires much longer training time than ours. MSDNet [24] represents adaptive
depth networks with multiple early-exiting branches and classifiers. SkipNet
[50] represents input-dependent dynamic networks that drop layers for ‘easy’
inputs. In S-ResNet50 [59], 4 sub-networks with varying widths are trained
through iterative self-distillation from its super-net.

Despite significantly shorter training time is required, our adaptive depth
networks can support various sub-networks at test time. Figure 4-(left) shows
the performance of ResNet50-ADN when its depth is varied at test time. Among
them, only the super-net and the base-net are explicitly trained in Algorithm
1. Other sub-networks are selected on the fly at test time by skipping sub-
paths in varying number of residual stages. Although these sub-networks are
not trained explicitly, they show graceful degradation of performance as the
depth of sub-networks becomes gradually shallower. However, although
ResNet50-ADN support $2^{4}$ sub-networks in theory, Figure 4-(right) shows
that some sub-networks are more useful than the others. For instance, even
though ResNet50-ADN(TFFF) and ResNet50-ADN(FFFT) have the same network depths,
ResNet50-ADN(TFFF) has about $0.58\%$ higher top-1 accuracy. This result shows
that skipping sub-paths in later stages is more detrimental to the performance
in general.

Figure 4: Validation accuracy of our sub-networks of varying depths. The
boolean values below the bars indicate the residual stages where the sub-paths
are skipped.

###  4.4 Performance on Devices

Figure 5 shows the performance of adaptive networks in an actual device. The
inference latency and energy consumption of ResNet50-ADN is compared to
S-ResNet50 [59], a representative width-adaptation network. The result shows
that depth-adaptation of ResNet50-ADN is highly effective in accelerating
inference speeds and reducing energy consumption. For example, in
ResNet50-ADN, reducing FLOPs by 38% through depth adaptation reduces both
inference latency and energy consumption by 35%. In contrast, even though
S-ResNet50 can reduce FLOPs by up to 93% by adjusting its width, it only
achieves up to 9% acceleration in practice.

Figure 5: Inference latency and energy consumption of adaptive networks, measured on Nvidia Jetson Orin Nano (batch size = 1). Although adjusting the network width can support a much wider FLOPs adaptation range, depth adaptation of ResNet50-ADN is more effective for actual inference acceleration and saving energy.  Table 4: Ablation analysis with ResNet50-ADN and ViT-b/32-ADN. Applied components are checked. In the last row, only $D_{KL}$ with softmax outputs is used for the skip-aware self-distillation. $\downarrow$ and $\uparrow$ in parentheses are comparisons to individual networks.  Skip-aware | Skip-aware | ResNet50 Acc@1 | ViT-b/32 Acc@1
---|---|---|---
self-distillation | BNs / LNs | Super-net | Base-net | Super-net | Base-net
|  |  75.2% ($\downarrow$ 1.5%) |  72.2% ($\downarrow$ 2.8%) |  75.7% ($\downarrow$0.2%) |  74.1% ($\uparrow$ 0.3%)
O |  |  76.1% ($\downarrow$ 0.6%) |  74.9 % ($\downarrow$ 0.1%) |  76.4% ($\uparrow$0.5%) |  74.3% ($\uparrow$0.5%)
| O |  76.6% ($\downarrow$ 0.1%) |  75.1% ($\uparrow$ 0.1%) |  76.0% ($\uparrow$0.1%) |  74.3% ($\uparrow$0.5%)
O | O |  77.6% ($\uparrow$ 0.9%) |  76.1% ($\uparrow$ 1.1%) |  76.6% ($\uparrow$ 0.8%) |  74.3% ($\uparrow$0.5%)
$\triangle$ | O |  77.6% ($\uparrow$ 0.9%) |  75.9% ($\uparrow$ 0.9%) |  |

###  4.5 Ablation Study

We conduct ablation study on ImageNet classification to investigate the
influence of two key components of the proposed adaptive depth networks: (1)
skip-aware self-distillation and (2) skip-aware BN/LNs. When our skip-aware
self-distillation is not applied, the loss in Algorithm 1 is modified to
$loss=\frac{1}{2}\\{criterion(\mathbf{y},\mathbf{\hat{y}}_{super})+criterion(\mathbf{y},\mathbf{\hat{y}}_{base})\\}$
for joint training of the super-net and the base-net. Table 4 shows the
results. For ResNet50, when neither of them is applied, the inference accuracy
of the super-net and the base-net is significantly lower than individual
networks by 1.5% and 2.8%, respectively. This result shows the difficulty of
joint training sub-networks for adaptive networks. When one of the two
components is applied individually, the performance is still slightly worse
than individual networks’. In the third row, when both skip-aware self-
distillation and skip-aware BNs are applied together, ResNet50-ADN achieves
significantly better performance than individual networks, both in the super-
net and the base-net. Finally, the last row shows the result when only
$D_{KL}(\mathbf{\hat{y}}_{super}\|\mathbf{\hat{y}}_{base})$ is used for skip-
aware self-distillation. Without explicit distillation of intermediate
features, slightly lower performance is observed in the base-net. The results
with ViT-b/32 show that switchable layer normalization has similar effect in
vision transformers.

(a) Original ResNet50

(b) ResNet50-ADN (super-net)

(c) ResNet50-ADN (base-net)

Figure 6: Class Activation Maps of the 3rd residual stages of ResNet50s. (a)
Original ResNet50’s activation regions change gradually across all blocks. (b)
In ResNet50-ADN (super-net), the first 3 blocks have extensive hot activation
regions, implying active learning of new level features. In contrast, the
skippable last 3 blocks have far less activation regions and they are
gradually refined around the target. (c) The base-net’s final activation map
is very similar to super-net’s final activation map, implying that they are at
the same feature level.

###  4.6 Visual Analysis of Sub-Paths

To investigate how our training method affects feature representations in the
mandatory and the skippable sub-paths, we visualize the activation of 3rd
residual stage of ResNet50-ADN using Grad-CAM [39]. The 3rd residual stage of
ResNet50-ADN has 6 residual blocks and the last three blocks are skippable. In
Figure 6-(a), the activation regions of original ResNet50 changes gradually
across all consecutive blocks. In contrast, in Figure 6-(b), ResNet50-ADN
(super-net) manifests very different activation regions in two sub-paths. In
the first three residual blocks, we can observe lots of hot activation regions
in wide areas, suggesting active learning of new level features. In contrast,
significantly less activation regions are found in the skippable last three
blocks and they are gradually concentrated around the target object,
demonstrating the refinement of learned features. Further, in Figure 6-(c), we
can observe that the final activation map of the ResNet50 (base-net) is very
similar to the super-net’s final activation map in Figure 6-(b). This implies
that they have similar distributions for the same inputs, as suggested in
Section 3.3.

###  4.7 Object Detection and Instance Segmentation

Table 5: Object detection and instance segmentation results on MS COCO dataset. Detector | Backbone | FLOPs | Individual Networks |  | ResNet50-ADN (ours)
---|---|---|---|---|---
Box AP | Mask AP |  | Box AP | Mask AP
Faster-RCNN | ResNet50 | 207.07G | 36.4 |  |  | 37.8 |
[36] | ResNet50-Base | 175.66G | 32.4 |  |  | 34.0 |
Mask-RCNN | ResNet50 | 260.14G | 37.2 | 34.1 |  | 38.3 | 34.1
[17] | ResNet50-Base | 228.73G | 32.7 | 29.9 |  | 34.1 | 31.2

In order to investigate the generalization ability of our approach, we use MS
COCO 2017 datasets on object detection and instance segmentation tasks using
representative detectors. We compare individual ResNet50 and our adaptive
depth ResNet50-ADN as backbone networks of the detectors. For training of
detectors, we use Algorithm 1 with slight adaptation. For object detection,
the intermediate features $\textbf{h}_{base}^{s}$ and
$\textbf{h}_{super}^{s}(s=1..N_{r})$ can be obtained directly from backbone
network’s feature pyramid networks (FPN) [32], and, hence, a wrapper function
is not required to extract intermediate features. All networks are trained on
train2017 for 12 epochs from ImageNet pretrained weights, following the
training settings suggested in [32]. Table 5 shows the results on val2017
containing 5000 images. Our adaptive depth backbone networks still outperform
individual static backbone networks in terms of COCO’s standard metric AP.

##  5 Conclusions

We propose an architectural pattern for adaptive depth networks and its
training method that renders selected sub-paths of the network to incur
minimal performance degradation even if they are skipped during inference. Our
approach does not train a fixed set of sub-networks iteratively, and, hence
significantly shorter training time is required than previous adaptive
networks. However, once deployed on devices, it can instantly construct sub-
networks of various depths to provide various accuracy-efficiency trade-offs
in a single network. We show formally that the proposed architectural pattern
and training method reduces overall prediction errors while minimizing the
impact of skipping sub-paths. We also empirically demonstrate the generality
and effectiveness of our approach using representative convolutional neural
networks and vision transformers.

## References

  * pyt [2023] Image classification reference training scripts in PyTorch, https://github.com/pytorch/vision/tree/main/references/classification, 2023.
  * Beyer et al. [2022] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10925–10934, 2022.
  * Beyer et al. [2023] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic.  Flexivit: One model for all patch sizes.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Cho and Hariharan [2019] Jang Hyun Cho and Bharath Hariharan.  On the efficacy of knowledge distillation.  In _International Conference on Computer Vision (ICCV)_ , pages 4794–4802, 2019.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Fan et al. [2020] Angela Fan, Edouard Grave, and Armand Joulin.  Reducing transformer depth on demand with structured dropout.  In _International Conference on Learning Representations (ICLR)_ , 2020.
  * Fang et al. [2023] Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang.  Depgraph: Towards any structural pruning.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 16091–16101, 2023.
  * Fayyaz et al. [2022] Mohsen Fayyaz, Soroush Abbasi Koohpayegani, Farnoush Rezaei Jafari, Sunando Sengupta, Hamid Reza Vaezi Joze, Eric Sommerlade, Hamed Pirsiavash, and Jürgen Gall.  Adaptive token sampling for efficient vision transformers.  In _European Conference on Computer Vision (ECCV)_ , pages 396–414, 2022.
  * Figurnov et al. [2017] Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov.  Spatially adaptive computation time for residual networks.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1039–1048, 2017.
  * Gong et al. [2022] Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra.  NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training.  In _International Conference on Learning Representations (ICLR)_ , 2022.
  * Greff et al. [2016] Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber.  Highway and residual networks learn unrolled iterative estimation.  In _International Conference on Learning Representations (ICLR)_ , 2016.
  * Guo et al. [2019] Q. Guo, Z. Yu, Y. Wu, D. Liang, H. Qin, and J. Yan.  Dynamic recursive neural network.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5142–5151, 2019.
  * Han et al. [2020] Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu.  Ghostnet: More features from cheap operations.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Han et al. [2016] Song Han, Huizi Mao, and William J. Dally.  Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding.  In _International Conference on Learning Representations (ICLR)_ , 2016.
  * Han et al. [2022] Y. Han, G. Huang, S. Song, L. Yang, H. Wang, and Y. Wang.  Dynamic neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(11):7436–7456, 2022.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In _International Conference on Computer Vision (ICCV)_ , pages 2961–2969, 2017.
  * Heo et al. [2021] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.  Rethinking spatial dimensions of vision transformers.  In _International Conference on Computer Vision (ICCV)_ , 2021.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hou et al. [2020] Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu.  Dynabert: Dynamic bert with adaptive width and depth.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 33, 2020.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Hu et al. [2019] Hanzhang Hu, Debadeepta Dey, Martial Hebert, and J Andrew Bagnell.  Learning anytime predictions in neural networks via adaptive loss balancing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3812–3821, 2019.
  * Huang et al. [2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger.  Deep networks with stochastic depth.  In _European Conference on Computer Vision (ECCV)_ , pages 646–661. Springer, 2016.
  * Huang et al. [2018] Gao Huang, Danlu Chen, Tianhong Li, Felix Wu, Laurens van der Maaten, and Kilian Weinberger.  Multi-scale dense networks for resource efficient image classification.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International Conference on Machine Learning (ICML)_ , pages 448–456. PMLR, 2015.
  * Jacob et al. [2018] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko.  Quantization and training of neural networks for efficient integer-arithmetic-only inference.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Jastrzebski et al. [2018] Stanisław Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio.  Residual connections encourage iterative inference.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Kong et al. [2022] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, Minghai Qin, and Yanzhi Wang.  Spvit: Enabling faster vision transformers via latency-aware soft token pruning.  In _European Conference on Computer Vision (ECCV)_ , pages 620–640, 2022.
  * Li et al. [2021] Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang.  Dynamic slimmable network.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8607–8617, 2021.
  * Li et al. [2019] Hao Li, Hong Zhang, Xiaojuan Qi, Ruigang Yang, and Gao Huang.  Improved techniques for training adaptive deep networks.  In _International Conference on Computer Vision (ICCV)_ , pages 1891–1900, 2019.
  * Li et al. [2020] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, and Jian Sun.  Learning dynamic routing for semantic segmentation.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Liu et al. [2019] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun.  Metapruning: Meta learning for automatic neural network channel pruning.  In _International Conference on Computer Vision (ICCV)_ , pages 3296–3305, 2019.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _International Conference on Computer Vision (ICCV)_ , 2021.
  * Meng et al. [2022] L. Meng, H. Li, B. Chen, S. Lan, Z. Wu, Y. Jiang, and S. Lim.  Adavit: Adaptive vision transformers for efficient image recognition.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 12299–12308, 2022.
  * Ren et al. [2017] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2017.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Sandler et al. [2018] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4510–4520, 2018.
  * Selvaraju et al. [2017] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _International Conference on Computer Vision (ICCV)_ , pages 618–626, 2017.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3th International Conference on Learning Representations (ICLR)_ , 2015.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International Conference on Machine Learning (ICML)_ , pages 6105–6114. PMLR, 2019.
  * Touvron et al. [2020] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  _arXiv preprint_ , 2020.
  * Touvron et al. [2023] Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, and Hervé Jégou.  Co-training 2l submodels for visual recognition.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Veit and Belongie [2018] Andreas Veit and Serge Belongie.  Convolutional networks with adaptive inference graphs.  In _European Conference on Computer Vision (ECCV)_ , pages 3–18, 2018.
  * Veit et al. [2016] Andreas Veit, Michael J Wilber, and Serge Belongie.  Residual networks behave like ensembles of relatively shallow networks.  _Conference on Neural Information Processing Systems (NeurIPS)_ , 29:550–558, 2016.
  * Wan et al. [2020] Chengcheng Wan, Henry Hoffmann, Shan Lu, and Michael Maire.  Orthogonalized sgd and nested architectures for anytime neural networks.  In _International Conference on Machine Learning (ICML)_ , pages 9807–9817. PMLR, 2020.
  * Wang et al. [2021a] Dilin Wang, Chengyue Gong, Meng Li, Qiang Liu, and Vikas Chandra.  Alphanet: Improved training of supernets with alpha-divergence.  In _International Conference on Machine Learning (ICML)_ , pages 10760–10771. PMLR, 2021a.
  * Wang et al. [2021b] Huan Wang, Can Qin, Yulun Zhang, and Yun Fu.  Neural pruning via growing regularization.  In _International Conference on Learning Representations (ICLR)_ , 2021b.
  * Wang et al. [2018] Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez.  Skipnet: Learning dynamic routing in convolutional networks.  In _European Conference on Computer Vision (ECCV)_ , pages 409–424, 2018.
  * Wu et al. [2018] Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris.  Blockdrop: Dynamic inference paths in residual networks.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8817–8826, 2018.
  * Xie et al. [2020] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10687–10698, 2020.
  * Yang et al. [2020] Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, and Gao Huang.  Resolution adaptive networks for efficient inference.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2020.
  * Yao et al. [2021] Zhuliang Yao, Yue Cao, Yutong Lin, Ze Liu, Zheng Zhang, and Han Hu.  Leveraging batch normalization for vision transformers.  In _2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW)_ , pages 413–422, 2021.
  * Yin et al. [2022] Hongxu Yin, Arash Vahdat, Jose M. Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov.  A-ViT: Adaptive tokens for efficient vision transformer.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10809–10818, 2022.
  * Yu et al. [2022] Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui.  Width & depth pruning for vision transformers.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(3):3143–3151, 2022.
  * Yu and Huang [2019a] Jiahui Yu and Thomas Huang.  Autoslim: Towards one-shot architecture search for channel numbers.  _arXiv preprint_ , 2019a.
  * Yu and Huang [2019b] Jiahui Yu and Thomas S Huang.  Universally slimmable networks and improved training techniques.  In _International Conference on Computer Vision (ICCV)_ , pages 1803–1811, 2019b.
  * Yu et al. [2018] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang.  Slimmable neural networks.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Zagoruyko and Komodakis [2017] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  In _5th International Conference on Learning Representations, ICLR,_ , 2017.
  * Zhang et al. [2019] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In _International Conference on Computer Vision (ICCV)_ , pages 3713–3722, 2019.
  * Zhu et al. [2021] Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang, Ying Nie, Zhenzhong Lan, and Yunhe Wang.  Dynamic resolution network.  In _Conference on Neural Information Processing Systems (NeurIPS)_ , pages 10985–10998, 2021.