#  NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of
Deep Giants ††thanks:  This work was supported by the NSF SCH program (Award
number: 1838873) and NSF NIH program (Award number: R01HL144683).

Zhongzhi Yu1, Yonggan Fu1, Jiayi Yuan2, Haoran You1, Yingyan (Celine) Lin1
1Georgia Institute of Technology, 2Rice University {zyu401, yfu314, hyou37,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Tiny deep learning has attracted increasing attention driven by the
substantial demand for deploying deep learning on numerous intelligent
Internet-of-Things devices. However, it is still challenging to unleash tiny
deep learning’s full potential on both large-scale datasets and downstream
tasks due to the under-fitting issues caused by the limited model capacity of
tiny neural networks (TNNs). To this end, we propose a framework called
NetBooster to empower tiny deep learning by augmenting the architectures of
TNNs via an expansion-then-contraction strategy. Extensive experiments show
that NetBooster consistently outperforms state-of-the-art tiny deep learning
solutions.

###### Index Terms:

Tiny Neural Networks, Efficient Deep Learning

##  I Introduction

Tiny deep learning, which aims to develop tiny neural networks (TNNs)
featuring much-reduced network sizes along with lower memory and computational
costs, has emerged as a promising direction to enable deep learning’s wider
real-world applications in resource-constrained Internet-of-Things (IoT)
devices [17] and has attracted an increasingly growing interest from both
industry and academia [17, 3]. In particular, existing tiny deep learning
works strive to improve the achievable accuracy-efficiency trade-off of TNNs
by either designing novel efficient network architectures [17, 26] or
compressing a large deep neural network (DNN) to reduce their network
redundancy [9]. However, the achieved accuracy-efficiency trade-off of
existing TNN works is still far from satisfactory for many IoT emerging
applications [3]. Specifically, we summarize that the constraints to the
achievable accuracy-efficiency trade-offs of TNNs stem from two factors:
Constraint 1: It is challenging for TNNs to learn complex but representative
features [3] and achieve satisfactory accuracy on commonly used large-scale
datasets (e.g., ImageNet), and Constraint 2: TNNs’ limited accuracy on large-
scale datasets further hinders TNN-based solutions from leveraging the widely
adopted pretrain-then-finetune paradigm for real-world downstream tasks.

Figure 1: (a) Constraint 1: TNN training suffers from under-fitting issues.
When training MobileNetV2 [26] on ImageNet, regularization techniques (e.g.,
DropBlock [10]) even lead to inferior accuracy compared with vanilla training.
Our proposed NetBooster can boost TNNs’ accuracy by increasing its capacity
during training. (b) Constraint 2: Inadequately trained TNNs cannot learn
complex features and thus suffer from limited downstream task accuracy.
Finetuning ImageNet pretrained vanilla MobileNetV2-35 with a resolution of
$224\times 224$ and $144\times 144$, respectively, on the CIFAR-100 dataset
for even four times more epochs (i.e., 600 epochs) still cannot improve the
achievable accuracy. Our proposed NetBooster can boost TNNs’ accuracy by
inheriting pretrained deep giants’ learned complex features.

In parallel, it has recently been recognized that a dedicated training recipe
can boost the accuracy of TNNs [3], although this area remains still under-
explored. Unlike DNN training, which requires techniques like data
augmentation [4] and/or regularization [10, 27] to alleviate the over-fitting
issue, a recent study [3] has shown that the small network capacity of TNNs
makes them more prone to the under-fitting issue. Specifically, due to TNNs’
limited ability to learn complex features, extensively augmented training data
or a heavily regularized training process can hurt the achievable accuracy of
TNNs on large-scale datasets (i.e., Constraint 1), as shown in Fig. 1 (a). The
lack of learned complex and representative features in the pretrained TNNs
further limits the achievable accuracy of downstream tasks, which cannot be
mitigated by additional training epochs (i.e., Constraint 2), as shown in Fig.
1 (b).

To narrow the gap between the increasing demand for more powerful TNNs in
real-world applications and the lack of effective TNN training schemes, we aim
to develop a technique that can boost the achievable task accuracy of TNNs,
while preserving their appealing efficiency, by empowering TNNs’ learned
features. In particular, this work makes the following contributions:

  * $\bullet$

To the best of our knowledge, we are the first to discover and promote a new
paradigm of training TNNs to boost their achievable accuracy via constructing
a competent deep giant using compound network augmentation (i.e., augmenting
both width and depth dimensions of the given TNNs), which is simple,
effective, and generally applicable.

  * $\bullet$

By leveraging the above discovery, we propose a TNN training framework, dubbed
NetBooster, that alleviates TNNs’ under-fitting issue during training,
boosting their achievable accuracy while preserving their original network
complexity and thus inference efficiency. Specifically, NetBooster
incorporates a two-step expansion-then-contraction training strategy: Step-1:
Network Expansion constructs an expanded deep giant by converting some layers
of the original TNN into multi-layer blocks, facilitating the learning of more
complex features by leveraging the corresponding deep giant counterpart, which
equips the original TNN with an initial state already possessing sufficient
knowledge, and Step-2: Progressive Linearization Tuning (PLT) then reverts the
deep giant back to the original TNN’s structure by removing the non-linear
layers from the expanded blocks and then contracting them.

  * $\bullet$

We make heuristic efforts to empirically investigate the optimal setting for
effectively boosting the achievable accuracy of TNNs when implementing Network
Expansion in NetBooster. Specifically, we address the following questions:
$\mathcal{Q}1$. What kind of block to use for expansion, $\mathcal{Q}2$. Where
to expand within a TNN, and $\mathcal{Q}3$. How to determine the expansion
ratio.

  * $\bullet$

Extensive experiments and ablation studies on two tasks, four networks, and
seven datasets demonstrate that NetBooster consistently achieves a non-trivial
accuracy boost (e.g., 1.3% $\sim$ 2.6%) compared to state-of-the-art (SOTA)
TNNs on the ImageNet dataset and up to 4.7% higher accuracy on various
downstream tasks, while still maintaining the original TNNs’ inference
efficiency.

##  II Related Works

###  II-A Tiny Neural Network

Tiny deep learning aims to develop TNNs with reduced network sizes, lower
memory and computational costs, and acceptable accuracy, enabling deep
learning-powered solutions in resource-constrained IoT devices. Existing
techniques towards fulfilling the goal of tiny deep learning can mostly be
categorized into two trends. One trend is to design novel TNN architectures by
leveraging either human expertise [26] or automated tools, e.g., neural
architecture search [2]. The other trend is to make use of compression
techniques, including pruning [19], quantization [9], dynamic inference [29],
to further reduce network complexity on top of existing TNN architectures.

In this work, we propose to pursue a tiny deep learning solution that boosts
TNNs’ accuracy-efficiency trade-off from an underexplored and orthogonal
direction: how to train TNNs to unleash their achievable accuracy more
effectively. To the best of our knowledge, the only pioneering work that
focuses on a similar direction is NetAug [3], which proposes to augment TNNs
from the width dimension by introducing a wider supernet to assist training
and then directly remove the supernet during inference. In contrast,
NetBooster proposes to first expand a TNN from both the depth and width
dimensions to create a competent deep giant during TNN training, and then
gradually contract it back to the original structure, instead of directly
removing expanded parts, to avoid unrecoverable information loss that could
result in nontrivial accuracy drops.

###  II-B Data Augmentation and Regularization

Data augmentation and regularization techniques have been proposed to
alleviate the over-fitting issue associated with large-scale DNNs in order to
boost their network generalization ability and thus the achievable accuracy.
Specifically, data augmentation techniques focus on manipulating the input
data samples [5], while regularization techniques focus on the network aspect
and randomly drop different components from the network [27] during training.

However, a recent study [3] and Fig. 1 (a) have shown that TNN training
suffers from under-fitting instead of over-fitting. As a result, existing data
augmentation and regularization techniques are unable to fully unleash the
potential of TNNs.

###  II-C Knowledge Distillation

Knowledge distillation (KD) aims to transfer the already learned knowledge
from a larger teacher network to a smaller student network [12]. Instead of
relying on a teacher network to provide guidance during training, NetBooster
aims to inherit the learned features from the expanded deep giants to achieve
higher accuracy than the original TNNs. As such, our proposed NetBooster is
orthogonal to KD, and when combined with KD, it is expected to further enhance
the performance of TNNs.

###  II-D Transfer Learning

Motivated by DNNs’ strong feature extraction capability, transfer learning [7]
has become a widely used paradigm for transferring knowledge across different
domains [21]. Despite the extensive efforts to boost transferability, larger
pretrained networks are commonly believed to have better transferability due
to their representative and generalizable features. Our proposed NetBooster
aims to empower TNNs with high-quality features and thus better
transferability by leveraging TNNs’ corresponding deep giants through compound
network augmentation.

##  III The Proposed NetBooster Framework

###  III-A Motivations and Inspirations

The key challenge for TNN training. Due to the lack of sufficient network
capacity, TNNs tend to suffer from severe under-fitting issues when being
trained on large-scale datasets (e.g., ImageNet), limiting their ability to
learn complex but representative features [3] and further hindering their
achievable accuracy on downstream tasks.

Inspirations for our works. Recent works have demonstrated that
overparameterization during training can lead to improved achievable accuracy,
while network complexity during inference can be reduced without adversely
affecting accuracy [18]. Various techniques for compressing DNNs have also
supported this observation. For example, the pruning method [19] preserves the
original dense network during training and then removes redundant neurons for
inference. Nevertheless, the aforementioned methods only focus on
overparameterization from the width dimension. At the same time, existing work
[22] shows that DNNs with varying depth and width tend to learn different
features, urging the need to introduce overparameterization into both
dimensions. Thus, if we can equip a given TNN (e.g., original TNN) with
comprehensive overparameterization during training, and then restore the
original TNN’s structure during inference, the under-fitting issue can be
effectively mitigated to achieve more accurate yet efficient TNN inference.
This has inspired us to design a principled expansion-then-contraction
methodology by first expanding the original TNN to a more overparameterized
network (e.g., deep giant) for better feature learning and then contracting it
back to the original structure to preserve its efficiency.

Figure 2: An overview of the proposed NetBooster framework. In NetBooster, we
augment the original TNN from both depth and width dimensions. Specifically,
we uniformly select layers from the original TNN and expand them into inverted
residual blocks [26] to formulate the deep giant, helping to learn complex
features. Then in PLT, we progressively decay the non-linear activation
functions within the expanded inverted residual blocks to an identity mapping
function and contract the expanded blocks back to the corresponding layers to
maintain the original TNN’s structure and inference efficiency.

###  III-B Overview

Implementation of expansion-then-contraction. Given the expansion-then-
contraction principle, there are several potential implementations. Inspired
by the success of RepVGG [6], which shows that parallel branches can be merged
thanks to their linearity, we hypothesize that if we can properly remove the
non-linear activation functions between layers, the consecutive layers can
also be merged via a linear combination. Fortunately, recent works show that
some of the activation functions can be safely removed from the network
without hurting the task accuracy [13]. This motivates us to propose our
expansion-then-contraction-based NetBooster training framework, which adopts
two steps: Step-1: Network Expansion, where we augment the original TNN from
both depth and width dimensions during training to construct the corresponding
deep giant by replacing some layers in the original TNN with multi-layer
blocks, aiming to increase the original TNN’s capacity and alleviate its
under-fitting issue during training and thus enabling a better feature
learning on the large-scale training dataset, and Step-2: Progressive
Linearization Tuning (PLT), where we progressively remove the non-linearity
inside the expanded blocks on the target dataset. After the non-linear layers
inside the blocks are removed, we contract the expanded deep giant back to the
original TNN at the end of training to inherit the learned features while
ensuring the boosted accuracy does not come with additional inference
overhead.

Technical challenges to achieve NetBooster. While the aforementioned principle
sounds straightforward, implementing such a training pipeline is non-trivial.
In Step-1, naively expanding all layers with a high expansion ratio can lead
to an excessively large network which is difficult to train. To enable a
practical and effective expansion strategy, at least the following three
questions need to be addressed: $\mathcal{Q}1$: what blocks to insert? What
kind of blocks we should use to expand the original TNN, $\mathcal{Q}2$: where
to expand? How to find a set of target layers to be expanded, and
$\mathcal{Q}3$: how to determine the expansion ratio? To what extent we should
expand the target layers. In Step-2, how to contract the deep giant expanded
from both width and depth dimensions while preserving the learned knowledge of
the deep giant is still an open question. Despite the method proposed in [6]
can merge the parallel convolution layers into one single layer via linear
combination, the non-linear activation layers between sequentially connected
convolution layers make it impossible to merge convolution layers along the
depth dimension directly. We next elaborate on our proposed solutions to
tackle the above challenges and the design of each step.

###  III-C Step 1: Network Expansion

The network expansion step aims to increase the capacity of the original TNN,
transforming them into a more powerful deep giant. This boosts their ability
to learn complex and representative features from large-scale training
datasets, improving task accuracy and transferability. To answer the questions
raised in Sec. III-B, we propose the following criteria when expanding the
network:

  1. a.

Structural consistency: To guarantee that applying NetBooster does not change
the original TNN’s structure for inference, each expansion block needs to be
able to be contracted back to the original single layer via linear combination
in the PLT step. Thus, for the network expansion step, the receptive field of
each expansion block should be equal to that of the original layer.

  2. b.

Sufficient capacity: Motivated by the findings in [3], we aim to alleviate the
under-fitting issue and ease the learning process by creating a deep giant
with increased capacity. Thus, we should sufficiently expand the original TNN
from multiple positions and dimensions (i.e., expand width with increased
expansion ratios and depth by inserting multiple layers).

  3. c.

Effective feature inheritance: In addition to having sufficient capacity, it
is equally important to effectively inherit the learned features of the deep
giant. As suggested in [20], excessively large networks tend to learn
significantly different feature distribution from that of small networks,
which can not only forbid small networks from inheriting but even hurt small
networks’ task accuracy. Thus, (1) the complexity gap between the original TNN
and its expanded deep giant should not be too large and (2) the selected
layers to be expanded should contain sufficient parameters to ensure an
effective knowledge inheritance from the deep giant.

Based on the above criteria, we answer the questions raised in Sec. III-B
below:

$\mathcal{Q}1$. What kind of block to use? We select the type of inserted
blocks from a pool of well-established DNN building blocks (e.g., the basic
and bottleneck blocks in ResNet [11] and the inverted residual block in
MobileNetV2 [26]). To maintain structure consistency (criteria a.), we
eliminate the basic block as it stacks two layers with large convolution
kernels, leading to a receptive field larger than that of the original layer.
To narrow down the complexity gap for effective feature inheritance (criteria
c.), we select the inverted residual block over the bottleneck block.

$\mathcal{Q}2$. Where to expand? The achievable accuracy of NetBooster is
limited by a trade-off between increasing the network capacity by constructing
a larger deep giant (criteria b.) and improving the feature inheritance
effectiveness by narrowing down the capacity gap between the deep giant and
original TNN (criteria c.). A simple but effective way to push the
aforementioned trade-off further is to consider the knowledge inheritance
effectiveness from a more fine-grained granularity (i.e., layer-wise instead
of model-wise). Specifically, multiple layers can have a better representation
ability than a single layer. Thus, the expanded block’s learned complex
features can be more effectively inherited by distributing them to multiple
adjacent layers in the original TNN. To this end, we propose to uniformly
select layers to be expanded from the original TNN, which can guarantee that
there are sufficient layers to inherit learned features from each of the
expanded blocks.

$\mathcal{Q}3$. How to determine the expansion ratio? Similar to
$\mathcal{Q}2$, the selection of the adopted expansion ratio has to trade-off
between the network capacity (criteria b.) and the effectiveness of knowledge
inheritance (criteria c.). However, thanks to the proposed uniform expansion
strategy in $\mathcal{Q}2$, we empirically find that the commonly used
expansion ratio, 6 [26] in the inserted inverted residual blocks works well on
balancing the aforementioned two criteria.

###  III-D Step 2: Progressive Linearization Tuning (PLT)

The next step is to recover the original TNN’s structure on the target dataset
while inheriting the knowledge learned by the deep giant. Inspired by [6],
which proposes to merge parallel convolution layers into one single layer, we
find that sequentially connected layers can also be merged via linear
combinations by properly removing the non-linear operations between them. To
achieve this, we propose PLT to progressively remove the non-linearity from
the expanded deep giant and then contract it back to the original TNN during
finetuning on the target dataset.

Motivating observation.  Non-linearity has been considered a key enabler for
the promising performance of DNNs and most existing works use the combination
of convolution and non-linear activation layers as a basic design unit. In
parallel, recent works [13] have shown that non-linearity within DNNs can be
highly redundant for inference, a large portion of element-wise non-linear
activation functions can be removed from a DNN, and the complex features
learned from the original TNN during training can be largely preserved.
Inspired by the revolution from element-wise pruning to structure pruning, we
aim to step further and remove the non-linearity in a structured manner (i.e.,
layer-wisely).

Non-linearity removal. We propose to transform the expanded deep giant back to
the original TNN meanwhile preserve the learned features by slowly decaying
the non-linear activation functions.

Without loss of generality, we take the ReLU activation function as an example
as it is the most commonly adopted activation function in TNNs, and the
following discussion can also be extended to other activation functions like
ReLU6. Here the ReLU function is defined as:

$Y_{l}=\max(0,X_{l}),$ (1)

where $X_{l}$ and $Y_{l}$ are the input and output of layer $l$, respectively.
We change the formulation of ReLU to the following format,

$Y_{l}=\max(\alpha_{l}X_{l},X_{l}),$ (2)

where $0<\alpha_{l}<1$ is the slope parameter to manipulate the non-linearity
of the corresponding activation layer. When $\alpha_{l}=0$, it is exactly the
ReLU function. When $\alpha_{l}=1$, the activation function is decayed to an
identity mapping.

Given a list $L$ of non-linear activation layers to be removed, we gradually
increase $\alpha_{l^{\prime}}$ for $l^{\prime}\in L$ from 0 to 1 in $E_{d}$
epochs, the value of $\alpha_{l^{\prime}}$ is uniformly increased in each
iteration. When $\alpha_{l^{\prime}}=1$, Eq. 2 is an identity mapping, and
thus the non-linearity is removed.

Expanded block contraction. With the non-linear activation layers removed, the
remaining layers can be contracted into one layer via simple linear
combinations.

Formulation: Without loss of generality, we take two convolution layers as an
example. Given the input to the first layer $X\in\mathcal{R}^{h_{1}\times
w_{1}\times c_{1}}$, the output $Y\in\mathcal{R}^{h_{3}\times w_{3}\times
c_{3}}$, as well as the kernels of two layers
$K^{1}\in\mathcal{R}^{k_{1}\times k_{1}\times c_{1}\times c_{2}}$ and
$K^{2}\in\mathcal{R}^{k_{2}\times k_{2}\times c_{2}\times c_{3}}$, where
$k_{1/2}$ are the kernel sizes, $h_{1/2/3},w_{1/2/3}$ are the heights and
widths of corresponding feature maps, $c_{1/2/3}$ are the channel numbers. The
overall functionality of the two convolution layers can be formulated as

$\displaystyle\begin{split}&Y_{p,q,o}=\sum^{k-1}_{i=0}\sum^{k-1}_{j=0}\sum^{c_{1}-1}_{m=0}X_{p-i,q-j,m}K_{i,j,m,o},\\\
\end{split}$ (3)
$\displaystyle\begin{split}where&\,\,K_{i,j,m,o}=\sum_{s=s_{l}}^{s_{h}}\sum_{t=t_{l}}^{t_{h}}\sum_{n=0}^{c_{2}-1}K^{1}_{i-s,j-t,m,n}K^{2}_{s,t,n,o},\end{split}$
(4)

where $k=k_{1}+k_{2}-1$, $s_{l}=\max(0,i-k_{1}+1)$, $s_{h}=\min(k_{2}-1,i)$,
$t_{l}=\max(0,j-k_{2}+1)$ and $t_{h}=\min(k_{2}-1,j)$.

Remark: It is worth noting that different expansion ratios of the inserted
inverted residual block will result in the same computational cost after
contraction since the input and output channels after contraction are always
equal to the input channel of the first layer and the output channel of the
last layer, respectively, regardless of intermediate channel numbers (i.e.,
$c_{2}$).

TABLE I: Benchmarking on ImageNet. ’r’ is the input resolution.

Network | FLOPs | Params | Training Method | Accuracy
---|---|---|---|---
MobileNetV2-Tiny (r=144) | 23.5M | 0.75M | Vanilla | 51.2
RocketLaunch [31] | 51.8
tf-KD [30] | 51.9
RCO-KD [14] | 52.6
NetAug [3] | 53.0
NetBooster | 53.7
MCUNet (r=176) | 81.8M | 0.74M | Vanilla | 61.4
NetAug [3] | 62.5
NetBooster | 62.8
MobileNetV2-50 (r=160) | 50.2M | 1.95M | Vanilla | 61.4
NetAug [3] | 62.5
NetBooster | 62.7
MobileNetV2-100 (r=160) | 154.1M | 3.47M | Vanilla | 69.6
NetAug [3] | 70.5
NetBooster | 70.9

##  IV Experiments

###  IV-A Experiments Setup

Tasks, datasets, and networks We consider two tasks, including image
classification and object detection, with seven datasets to provide a thorough
evaluation of NetBooster. Specifically, to evaluate NetBooster’s performance
in alleviating the under-fitting issue to achieve a higher accuracy on the
large-scale dataset, we consider the ImageNet dataset. To evaluate how deep
giant’s learned representation helps with downstream tasks, we consider image
classification tasks on five datasets, including CIFAR-100, Cars [15],
Flowers102 [23], Food101 [1], and Pets [24]. We also evaluate NetBooster on
the downstream object detection task with the Pascal VOC dataset [8]. We
consider four networks, including MobileNetV2-100/50/Tiny [26], and a neural
architecture searched hardware friendly network MCUNet [17].

Baselines. We benchmark the proposed NetBooster over five baselines including
networks trained with standard vanilla training, a series of SOTA KD
algorithms (i.e., tf-KD [30], RCO-KD [14], and RocketLaunch [31]), and NetAug
[3], which is a pioneering work in boosting TNN training performance.

Expansion strategy. We uniformly expand 50% of blocks in the original TNN. To
expand each block, we replace the first pointwise convolution layer with an
inverted residual block with an expansion ratio of 6. The kernel size of the
depthwise convolution layer in the inserted inverted residual block is set to
1 to make the inserted block’s receptive field the same as the pointwise
convolution.

Training settings. We develop our training settings based on the commonly
adopted settings. Specifically, when evaluating NetBooster’s performance in
improving TNNs’ accuracy on ImageNet dataset, we follow [3] to train the deep
giant for 160 epochs using SGD optimizer with a batch size of 1024, an initial
learning rate of 0.2 and cosine anneal learning rate schedule. In PLT, we set
$E_{d}=40$, and further finetune for 110 epochs. For downstream tasks, we use
the ImageNet pretrained deep giant as the starting point and develop our
training recipe on CIFAR-100 based on [28], on Cars, Flowers102, Food101, and
Pets based on [25], and on Pascal VOC based on [3]. In all experiments on
downstream tasks, we assign $E_{d}$ to be 20% of the total tuning epochs in
PLT.

###  IV-B Easing Constraint 1: Benchmarking on Large-scale Dataset

To evaluate whether the proposed NetBooster can help TNNs to learn the complex
features of the large-scale dataset and thus improve accuracy, we benchmark
NetBooster on the ImageNet dataset with vanilla training, NetAug, and various
KD algorithms. As shown in Table I, NetBooster achieves 1.3% $\sim$ 2.5%
accuracy improvements over the vanilla training, showing its strong ability in
boosting the TNNs’ accuracy by standing on the shoulder of deep giants
generated by NetBooster.

Compared with the KD baselines, our proposed NetBooster achieves 0.9% $\sim$
1.1% accuracy improvement without guidance from the teacher network (Assemble-
ResNet50 [16]), suggesting that Network Expansion in NetBooster can equip the
deep giant with sufficient capacity to learn complex features at least
comparable with the large teacher DNN used in the KD baselines and the learned
features can be effectively inherited from the PLT step.

Compared with NetAug, which is a pioneering work focusing on a similar
scenario as NetBooster, NetBooster also achieves superior accuracy over
NetAug, suggesting the multi-dimensional Network Expansion and the PTL for
features inheritance is more effective than the network width expansion and
directly dropping augmented neuron adopted in NetAug.

TABLE II: Benchmarking on downstream image classification datasets. ’r’ is the
input resolution.

Network | Training Method | CIFAR-100 | Cars | Flowers102 | Food101 | Pets
---|---|---|---|---|---|---
MobileNetV2-Tiny (r=144) | Vanilla | 74.07 | 76.18 | 90.01 | 75.43 | 78.30
NetBooster | 75.46 | 80.93 | 90.53 | 75.96 | 78.90
MobileNetV2-35 (r=160) | Vanilla | 76.08 | 78.36 | 90.63 | 76.80 | 80.64
Vanilla + KD | 76.38 | 77.47 | 91.41 | 77.02 | 82.44
NetBooster | 76.66 | 80.91 | 91.16 | 77.26 | 80.92
NetBooster + KD | 77.15 | 83.36 | 92.68 | 77.81 | 83.37

TABLE III: Benchmarking on object detection tasks with Pascal VOC dataset with
MobileNetV2-35 at 416 resolution.

Method | Vanilla | NetAug | NetBooster
---|---|---|---
AP50 | 60.8 | 62.4 | 62.6

###  IV-C Easing Constraint 2: Benchmarking on Downstream Tasks

To evaluate whether the learned complex and representative features in the
deep giant from the large-scale dataset can further help the original TNN to
achieve better accuracy on downstream tasks, we first evaluate NetBooster’s
performance when transferring the ImageNet pretrained deep giant to five
representative downstream image classification datasets with PLT. As shown in
Table II, compared with vanilla training, NetBooster achieves 0.46% $\sim$
4.75% accuracy improvement, showing that the features learned by the deep
giant are effectively inherited after PLT. It is worth noting that NetBooster
is also orthogonal to KD, applying KD on top of NetBooster can lead to another
0.49% $\sim$ 2.45% accuracy boost over using NetBooster alone.

We further evaluate NetBooster’s performance when transferring to the Pascal
VOC object detection task. As shown in Table III, NetBooster achieves 1.8 and
0.2 higher AP50 compared with vanilla training and NetAug, respectively. This
proves that NetBooster can be considered as a general method to boost TNNs’
performance across various tasks.

TABLE IV: Ablation study on what kind of block to insert.

Inserted Block Type | Expanded Acc. | Final Acc.
---|---|---
Vanilla | - | 51.20
Inverted Residual | 54.90 | 53.70
Basic Block | 54.52 | 53.41
Bottleneck Block | 55.23 | 53.62

###  IV-D Validating Expansion Strategy

We validate our answer to each question in Sec. III-C by validating the impact
of replacing our proposed strategy with alternatives when training MobileNet-
Tiny on the ImageNet dataset with an input resolution of 144.

$\mathcal{Q}1$. What kind of block to use: We ablate the impact of expanding
with different kinds of blocks and report the results in Table IV. Expanding
with inverted residual blocks leads to slightly better results (0.08% $\sim$
0.29%). It shows (1) the NetBooster framework can robustly boost TNNs’
performance, and (2) inserting inverted residual blocks is an effective
choice.

$\mathcal{Q}2$. Where to expand: We ablate different expansion locations’
impacts and report our findings in Table V. We observe that uniformly
expanding the model achieves a 1.08% $\sim$ 2.20% higher accuracy compared
with excessively expanding the first/middle/last part of the network, proving
the necessity to expand the model uniformly.

$\mathcal{Q}3$. How to determine expansion ratio: We ablate different
selections of expansion ratios in the inserted inverted residual blocks and
report the results in Table. VI. We observe that NetBooster with commonly used
expansion ratios (i.e., 4 $\sim$ 6) consistently improves the TNNs’ accuracy,
further proving NetBooster’s robustness to hyperparameter selection.

TABLE V: Ablation study on which block to expand.

Expansion | Expanded | Expanded Acc. | Final Acc.
---|---|---|---
FLOPs | Params
Vanilla | 29.4M | 0.75M | - | 51.20
Expand First 8 | 65.0M | 0.83M | 51.46 | 51.50
Expand Middle 8 | 49.6M | 0.93M | 52.98 | 52.62
Expand Last 8 | 51.2M | 1.25M | 53.90 | 52.47
Uniform Expand 8 | 63.9M | 0.99M | 54.90 | 53.70

TABLE VI: Ablation study on the expansion ratio.

Expansion ratio | 2 | 4 | 6 | 8
---|---|---|---|---
Final Acc. | 52.94 | 53.52 | 53.70 | 52.56

##  V Conclusion

In this paper, we discover and promote a new paradigm for training TNNs to
empower their achievable accuracy via augmenting both dimensions of TNNs
(i.e., depth and width) during training. Furthermore, we propose a framework
dubbed NetBooster, which is dedicated to boosting the accuracy of SOTA TNNs by
using an expand-then-contract training strategy to alleviate TNNs’ under-
fitting issues. Finally, we make heuristic efforts to empirically explore
what/when/where to augment when training TNNs with our proposed NetBooster.
Extensive experiments show that NetBooster consistently leads to a nontrivial
accuracy boost (e.g., 1.3% $\sim$ 2.5%) on top of SOTA TNNs on ImageNet and as
much as 4.7% higher accuracy on various downstream tasks, while maintaining
their inference efficiency.

## References

  * [1] L. Bossard _et al._ , “Food-101–mining discriminative components with random forests,” in _ECCV_. Springer, 2014, pp. 446–461.
  * [2] H. Cai, “Proxylessnas: Direct neural architecture search on target task and hardware,” _arXiv preprint_ , 2018.
  * [3] H. Cai _et al._ , “Network augmentation for tiny deep learning,” _arXiv preprint_ , 2021.
  * [4] E. D. Cubuk, “Randaugment: Practical automated data augmentation with a reduced search space,” in _CVPR Workshops_ , 2020, pp. 702–703.
  * [5] E. D. Cubuk _et al._ , “Autoaugment: Learning augmentation policies from data,” _arXiv preprint_ , 2018.
  * [6] X. Ding _et al._ , “Repvgg: Making vgg-style convnets great again,” in _CVPR_ , 2021, pp. 13 733–13 742.
  * [7] J. Donahue _et al._ , “Decaf: A deep convolutional activation feature for generic visual recognition,” in _ICML_. PMLR, 2014, pp. 647–655.
  * [8] M. Everingham _et al._ , “The pascal visual object classes (voc) challenge,” _IJCV_ , vol. 88, no. 2, pp. 303–338, 2010.
  * [9] Y. Fu _et al._ , “Double-win quant: Aggressively winning robustness of quantized deep neural networks via random precision training and inference,” in _ICML_. PMLR, 2021, pp. 3492–3504.
  * [10] G. Ghiasi _et al._ , “Dropblock: A regularization method for convolutional networks,” _arXiv preprint_ , 2018.
  * [11] K. He _et al._ , “Deep residual learning for image recognition,” in _CVPR_ , 2016, pp. 770–778.
  * [12] G. Hinton _et al._ , “Distilling the knowledge in a neural network,” _arXiv preprint_ , 2015.
  * [13] N. K. Jha _et al._ , “Deepreduce: Relu reduction for fast private inference,” in _ICML_. PMLR, 2021, pp. 4839–4849.
  * [14] X. Jin _et al._ , “Knowledge distillation via route constrained optimization,” in _ICCV_ , 2019, pp. 1345–1354.
  * [15] J. Krause _et al._ , “3d object representations for fine-grained categorization,” in _ICCV workshops_ , 2013, pp. 554–561.
  * [16] J. Lee _et al._ , “Compounding the performance improvements of assembled techniques in a convolutional neural network,” _arXiv preprint_ , 2020.
  * [17] J. Lin _et al._ , “Mcunet: Tiny deep learning on iot devices,” _arXiv preprint_ , 2020.
  * [18] S. Liu _et al._ , “Do we actually need dense over-parameterization? in-time over-parameterization in sparse training,” in _ICML_ , 2021.
  * [19] Z. Liu _et al._ , “Learning efficient convolutional networks through network slimming,” in _ICCV_ , 2017, pp. 2736–2744.
  * [20] S. I. Mirzadeh _et al._ , “Improved knowledge distillation via teacher assistant,” in _AAAI_ , vol. 34, no. 04, 2020, pp. 5191–5198.
  * [21] R. Mormont _et al._ , “Comparison of deep transfer learning strategies for digital pathology,” in _CVPR workshops_ , 2018, pp. 2262–2271.
  * [22] T. Nguyen _et al._ , “Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth,” _arXiv preprint_ , 2020.
  * [23] M.-E. Nilsback _et al._ , “Automated flower classification over a large number of classes,” in _ICVGIP_. IEEE, 2008, pp. 722–729.
  * [24] O. M. Parkhi _et al._ , “Cats and dogs,” in _CVPR_. IEEE, 2012, pp. 3498–3505.
  * [25] H. Salman _et al._ , “Do adversarially robust imagenet models transfer better?” _NeurIPS_ , vol. 33, pp. 3533–3545, 2020.
  * [26] M. Sandler _et al._ , “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _CVPR_ , 2018, pp. 4510–4520.
  * [27] N. Srivastava _et al._ , “Dropout: a simple way to prevent neural networks from overfitting,” _JMLR_ , vol. 15, no. 1, pp. 1929–1958, 2014.
  * [28] Y. Tian _et al._ , “Contrastive representation distillation,” _arXiv preprint_ , 2019.
  * [29] Z. Yu _et al._ , “Mia-former: Efficient and robust vision transformers via multi-grained input-adaptation,” _arXiv preprint_ , 2021.
  * [30] L. Yuan _et al._ , “Revisiting knowledge distillation via label smoothing regularization,” in _CVPR_ , 2020, pp. 3903–3911.
  * [31] G. Zhou _et al._ , “Rocket launching: A universal and efficient framework for training well-performing light net,” in _AAAI_ , 2018.


* [57] O. V. Tarasov, ”Connection between Feynman integrals having different values of the space-time dimension,”Phys. Rev. D54, 6479-6490 (1996).
  * [58] S. L. Cacciatori and P. Mastrolia, ”Intersection Numbers in Quantum Mechanics and Field Theory,” [arXiv:2211.03729 [hep-th]].
  * [59] G. Brunello, V. Chestnov, G. Crisanti, H. Frellesvig, M. K. Mandal and P. Mastrolia, ”Intersection Numbers, Polynomial Division and Relative Cohomology,” [arXiv:2401.01897 [hep-th]].
  * [60] G. Brunello, G. Crisanti, M. Giroux, P. Mastrolia and S. Smith, ”Fourier Calculus from Intersection Theory,” [arXiv:2311.14432 [hep-th]].
  * [61] K. Bönisch, C. Duhr, F. Fischbach, A. Klemm, C. Nega, ”Feynman integrals in dimensional regularization and extensions of Calabi-Yau motives,” J. High Energy Phys. 09 (2022) 156.
  * [62] C. Duhr, A. Klemm, C. Nega, L. Tancredi, ”The ice cone family and iterated integrals for Calabi-Yau varieties,” J. High Energ. Phys. 2023, 228 (2023).
  * [63] A. Klemm, C. Nega and R. Safari, ”The $l$-loop Banana Amplitude from GKZ Systems and relative Calabi-Yau Periods,” JHEP 04 (2020), 088.
  * [64] J. L. Bourjaily, A. J. McLeod, M. von Hippel and M. Wilhelm, ”Bounded Collection of Feynman Integral Calabi-Yau Geometries,” Phys. Rev. Lett. 122 (2019) no.3, 031601
  * [65] J. L. Bourjaily, A. J. McLeod, C. Vergu, M. Volk, M. Von Hippel and M. Wilhelm, ”Embedding Feynman Integral (Calabi-Yau) Geometries in Weighted Projective Space,” JHEP 01 (2020), 078
  * [66] S. Pögel, X. Wang and S. Weinzierl, ”Taming Calabi-Yau Feynman Integrals: The Four-Loop Equal-Mass Banana Integral,” Phys. Rev. Lett. 130 (2023) no.10, 101601
  * [67] H. Frellesvig, R. Morales and M. Wilhelm, ”Calabi-Yau meets Gravity: A Calabi-Yau three-fold at fifth post-Minkowskian order,” [arXiv:2312.11371 [hep-th]].
  * [68] D. Broadhurst, D.P. Roberts, ”Quadratic relations between Feynman integrals,” PoS LL2018 (2018) 053.
  * [69] D. Broadhurst, A. Mellit, ”Perturbative quantum field theory informs algebraic geometry,” PoS LL2016 (2016) 079.
  * [70] S. Bloch, M. Kerr and P. Vanhove, ”A Feynman integral via higher normal functions,” Compos. Math. 151 (2015) no.12, 2329-2375
  * [71] Y. Zhou, ”Wronskian factorizations and Broadhurst-Mellit determinant formulae,” Commun. Number Theory Phys. 12 (2018) 355-407.
  * [72] D. Broadhurst, ”Feynman integrals, L-series and Kloosterman moments,” Commun. Number Theory Phys. 10 (2016) 527-569.
  * [73] Y. Zhou, ”Wronskian algebra and Broadhurst-Roberts quadratic relations,” Commun. Number Theory Phys. 15 (4) (2021) 651-741.
  * [74] E. Remiddi, L. Tancredi, Schouten identities for Feynman graph amplitudes; the Master Integrals for the two-loop massive sunrise graph, Nucl. Phys. B 880 (2014) 343-377.
  * [75] D. Zagier, The dilogarithm function, in: P. Cartier, P. Moussa, B. Julia, P. Vanhove (Eds.), Frontiers in Number Theory, Physics, and Geometry II, Springer, Berlin, Heidelberg, 2007.
  * [76] S. Bloch, P. Vanhove, ”The elliptic dilogarithm for the sunset graph,” J. Number Theory 148 (2015) 328-364.
  * [77] V. Mishnyakov, A. Morozov, P. Suprun, ”Position space equations for banana Feynman diagrams,” Nuclear Physics B Volume 992, July 2023, 116245.
  * [78] J. Chen and B. Feng, ”Towards Systematic Evaluation of de Sitter Correlators via Generalized Integration-By-Parts Relations,” [arXiv:2401.00129 [hep-th]].
  * [79] F. W. J. Olver: Asymptotics and Special Functions, A K Peters, Wellesley, Massachusetts, 1997.