# ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks

Yang Sui Miao Yin Yu Gong Jinqi Xiao Huy Phan Bo Yuan

Rutgers University
Piscataway, New Jersey, USA

###### Abstract

Low-rank compression, a popular model compression technique that produces
compact convolutional neural networks (CNNs) with low rankness, has been well-
studied in the literature. On the other hand, low-rank training, as an
alternative way to train low-rank CNNs from scratch, has been exploited little
yet. Unlike low-rank compression, low-rank training does not need pre-trained
full-rank models, and the entire training phase is always performed on the
low-rank structure, bringing attractive benefits for practical applications.
However, the existing low-rank training solutions still face several
challenges, such as a considerable accuracy drop and/or still needing to
update full-size models during the training. In this paper, we perform a
systematic investigation on low-rank CNN training. By identifying the proper
low-rank format and performance-improving strategy, we propose ELRT, an
efficient low-rank training solution for high-accuracy, high-compactness, low-
rank CNN models. Our extensive evaluation results for training various CNNs on
different datasets demonstrate the effectiveness of ELRT.

##  1 Introduction

Convolutional neural networks (CNNs) have obtained widespread adoption in
numerous real-world computer vision applications, such as image
classification, video recognition, and object detection. However, modern CNN
models are typically storage-intensive and computation-intensive, potentially
hindering their efficient deployment in many resource-constrained scenarios,
especially at the edge and embedded computing platforms. To address this
challenge, many prior efforts have been proposed and conducted to produce low-
cost, compact CNN models. Among them, low-rank compression is a popular model
compression solution. By leveraging matrix or tensor decomposition techniques,
low-rank compression aims to explore the potential low-rankness exhibited in
the full-rank CNN models, enabling simultaneous reductions in both memory
footprint and computational cost. To date, numerous low-rank CNN compression
solutions have been reported in the literature (Phan et al., 2020; Kossaifi et
al., 2020; Li et al., 2021b; Liebenwein et al., 2021).

Low-rank Training: A Promising Alternative Towards Low-rank CNNs. From the
perspective of model production, performing low-rank compression on the full-
rank networks is not the only approach to obtaining low-rank CNNs. In
principle, we can also adopt a low-rank training strategy to directly train a
low-rank model from scratch. As illustrated in Fig. 1, low-rank training
starts from a low-rank initialization and keeps the desired low-rank structure
in the entire training phase. Compared with low-rank compression that is built
on a two-stage pipeline (“pre-training-then-compressing”), the single-stage
low-rank training enjoys two attractive benefits: relaxed operational
requirement and reduced training cost. More specifically, first, the
underlying training-from-scratch scheme, by its nature, completely eliminates
the need for pre-trained full-rank high-accuracy models, thereby lowering the
barrier to obtaining low-rank CNNs. In other words, producing low-rank
networks becomes more feasible and accessible. Second, the overall
computational cost for the entire low-rank CNN production pipeline is
significantly reduced. This is because: 1) the removal of the pre-training
phase completely saves the incurred computations that were originally needed
for pre-training the full-rank models; and 2) directly training on the compact
low-rank CNNs naturally consumes much fewer floating point operations (FLOPs)
than full-rank pre-training.

Figure 1: Different paths towards producing low-rank CNN models.

Existing Works and Limitations. Despite the above-analyzed benefits, low-rank
training is currently little exploited in the literature. Unlike the
prosperity of studying low-rank compression, to date, very few research
efforts have been conducted towards efficient low-rank training. (Ioannou et
al., 2015; Tai et al., 2015) are the pioneering works in this research
direction; however, the obtained low-rank models suffer a considerable
accuracy drop. In addition, the corresponding training methods are not
evaluated on modern CNNs such as ResNet. Recently, (Gural et al., 2020;
Hawkins et al., 2022) propose emerging memory-aware and Bayesian estimation-
based low-rank training, respectively; however, these two methods are either
built on costly repeated SVD operations (Gural et al., 2020) or Bayesian
estimation (Hawkins et al., 2022), which are very computation intensive and
potentially not scalable for practical deployment. (Hayashi et al., 2019;
Khodak et al., 2020) propose to learn the suitable low-rank format and/or
apply spectral initialization during the training. However, the trained models
with the new format have a considerable accuracy drop, even on the CIFAR-10
dataset. (Waleffe & Rekatsinas, 2020; Wang et al., 2021) perform several
epochs of full-size training to mitigate this issue. However, with the cost of
increasing memory and computational overhead, this hybrid training strategy
still brings a considerable accuracy drop. And it is essentially not training
a low-rank model from scratch. Therefore, a satisfactory answer to the
following fundamental question is still missing:

Fundamental Question for Low-rank Training: What is the proper training-from-
scratch solution that can produce modern low-rank CNN models with high
accuracy on the large-scale dataset, even outperforming the state-of-the-art
low-rank compression methods?

Technical Preview and Contributions. To answer this question and put the low-
rank training technique into practice, in this paper, we perform a systematic
investigation on training low-rank CNN models from scratch. By identifying the
proper low-rank format and performance-improving strategy, we propose ELRT, an
efficient low-rank training solution for high-accuracy high-compactness CNN
models. Compared with the state-of-the-art low-rank compression approaches,
ELRT demonstrates superior performance for various CNNs on different datasets,
demonstrating the promising potential of low-rank training in practical
applications. Overall, the contributions of this paper are summarized as
follows:

  * •

We systematically investigate the important design knobs of low-rank CNN
training from scratch, such as the suitable low-rank format and the potential
performance-improving strategy, to understand the key factors for building a
high-performance low-rank CNN training framework.

  * •

Based on the study and analysis of these design knobs, we develop ELRT, an
orthogonality-aware low-rank training approach that can train high-accuracy
high-compactness low-tensor-rank CNN models from scratch. By enforcing and
imposing the desired orthogonality on the low-rank model during the training
process, significant performance improvement with low computational overhead
can be obtained.

  * •

We conduct empirical evaluations for various CNN models to demonstrate the
effectiveness of ELRT. On the CIFAR-10 dataset, ELRT can train low-rank
ResNet-20, ResNet-56 and MobileNetV2 from scratch by providing 1.98$\times$,
2.05$\times$ and 1.71$\times$ FLOPs reduction, respectively; and meanwhile the
trained compact models enjoy 0.48$\%$, 0.70$\%$ and 0.29$\%$ accuracy increase
over the baseline. On ImageNet dataset, compared with the state-of-the-art
approaches that generate compact ResNet-50 models, ELRT achieves 0.49% higher
accuracy with the same or even higher inference and training FLOPs reduction,
respectively.

##  2 Related Works

Low-rank Compression. As an important type of model compression strategy, low-
rank compression aims to leverage low-rank decomposition techniques to
factorize the original full-rank neural network model into a set of small
matrices or tensors, leading to storage and computational savings. Based on
the adopted factorization methods, the existing low-rank CNN compression works
can be categorized into 2-D matrix decomposition based (Tai et al., 2015; Li &
Shi, 2018; Xu et al., 2020; Idelbayev & Carreira-Perpinán, 2020; Yang et al.,
2020; Liebenwein et al., 2021) and high-order tensor decomposition based
(Denton et al., 2014; Kim et al., 2015; Novikov et al., 2015; Yang et al.,
2017; Wang et al., 2018; Kossaifi et al., 2019; Phan et al., 2020; Kossaifi et
al., 2020; Yin et al., 2021; Li et al., 2021a).

Low-rank Training. Similar to low-rank compression, the goal of low-rank
training is also to produce compact neural network models with low-rankness;
while the key difference is that low-rank training initializes and updates the
low-rank CNNs during the entire training process. In other words, the pre-
trained full-rank models are not required in this scenario, and the CNN models
being updated are always kept in the low-rank format. To date, efficient low-
rank training approaches are still little exploited. More specifically, the
existing works either have considerable accuracy loss (Ioannou et al., 2015;
Tai et al., 2015; Hayashi et al., 2019; Khodak et al., 2020) or suffer high
computational overhead because of the use of costly SVD operations (Gural et
al., 2020), Bayesian estimation (Hawkins et al., 2022) or only performing
partially low-rank training (Waleffe & Rekatsinas, 2020; Wang et al., 2021),
limiting their effectiveness in the practical scenarios.

Unstructured $\&$ Structured Sparse Training. Low-rank training is essentially
a type of compression-aware training solutions, which include another related
strategy as sparse training. Sparse training can be performed in the
unstructured (Lee et al., 2018; Wang et al., 2019; Evci et al., 2020; Mostafa
& Wang, 2019; Liu et al., 2020; Mocanu et al., 2018; Bellec et al., 2018) and
structured (Yuan et al., 2021; Zhou et al., 2021) ways, corresponding to
training/obtaining unstructured and structured sparse CNN models,
respectively. From the perspective of model deployment, structured sparse
training is more practical and important than its unstructured counterpart
because it can produce structured sparse CNN models that exhibit considerable
speedup on the off-the-shelf GPU/CPU platforms.

Orthogonality in CNN Training. Another line of works that is related to this
paper is training orthogonal full-size full-rank CNNs (Rodriguez et al., 2017;
Huang et al., 2018; Miyato et al., 2018; Bansal et al., 2018; Wang et al.,
2020). Based on the observation that orthogonality in CNN weights can
stabilize the distribution of activations and bring efficient optimizations,
these prior efforts explore different methods to enforce orthogonality on the
convolutional layer in both the initialization and training phases. Compared
with them, ELRT has two key differences. First, the existing orthogonal CNN
training works are performed on full-rank full-size models, and they cannot
bring any memory or FLOPs reduction. Instead, ELRT aims to train compact low-
rank CNN models from scratch, naturally bringing reduced inference and
training costs. Second, orthogonal CNN training directly enforces
orthogonality on the entire convolutional layer. The underlying motivation is
mainly based on experimental observation (e.g., stabilizing the distribution
of activations). On the other hand, ELRT focuses on exploring the
orthogonality of the decomposed components (e.g., factor matrices) of the
weight tensors. Such strategy is naturally consistent with the requirement of
low-rank theory – the decomposed matrix/tensor should exhibit self-
orthogonality after SVD or tensor decomposition.

##  3 Preliminaries

Notation. Throughout this paper, the $d$-order tensor, matrix and vector are
represented by boldface calligraphic script letter
$\bm{\mathcal{X}}\in\mathbb{R}^{n_{1}\times n_{2}\times\cdots\times n_{d}}$,
boldface capital letters $\bm{X}\in\mathbb{R}^{n_{1}\times n_{2}}$, and
boldface lower-case letters $\bm{x}\in\mathbb{R}^{n_{1}}$, respectively. Also,
$\bm{\mathcal{X}}_{i_{1},\cdots,i_{d}}$ and $\bm{X}_{i,j}$ denote the entry of
tensor $\bm{\mathcal{X}}$ and matrix $\bm{X}$, respectively.

Tucker-2 Format for Convolutional Layer. As will be analyzed in Section 4, in
this paper we choose to use low-tensor-rank format (e.g., Tucker-2) to form
efficient low-rank training approach. In general, given a convolutional layer
$\bm{\mathcal{W}}\in\mathbb{R}^{C_{\mathrm{in}}\times C_{\mathrm{out}}\times
K\times K}$, it can be represented in a Tucker-2 format as follows:

$\displaystyle\bm{\mathcal{W}}_{p,q,i,j}=\sum_{r_{1}=1}^{\Phi_{1}}\sum_{r_{2}=1}^{\Phi_{2}}\bm{\mathcal{G}}_{r_{1},r_{2},i,j}\mathbf{U}_{r_{1},p}^{(1)}\mathbf{U}_{r_{2},q}^{(2)},$
(1)

where $\bm{\mathcal{G}}$ is the 4-D core tensor of size
$\Phi_{1}\times\Phi_{2}\times K\times K$, and
$\mathbf{U}^{(1)}\in\mathbb{R}^{\Phi_{1}\times C_{\mathrm{in}}}$ and
$\mathbf{U}^{(2)}\in\mathbb{R}^{\Phi_{2}\times C_{\mathrm{out}}}$ are the
factor matrices. In addition, $\Phi_{1}$ and $\Phi_{2}$ are the tensor ranks
that determine the complexity of compact convolutional layer.

Computation on the Tucker-2 Convolutional Layer. Given the above-described
Tucker-2 format representation, the corresponding execution on this compact
convolutional layer can be performed via consecutive computations as:

$\displaystyle\bm{\mathcal{T}}_{r_{1},h,w}^{1}=\sum_{p=1}^{C_{\mathrm{in}}}\textbf{U}_{r_{1},p}^{(1)}\bm{\mathcal{X}}_{p,h,w}$
$\displaystyle,\quad\bm{\mathcal{T}}^{2}_{r_{2},h^{\prime},w^{\prime}}=\sum_{i=1}^{K}\sum_{j=1}^{K}\sum_{r_{1}=1}^{\Phi_{1}}\bm{\mathcal{G}}_{r_{1},r_{2},i,j}\bm{\mathcal{T}}_{r_{1},h_{i},w_{j}}^{1},$
(2) $\displaystyle\bm{\mathcal{Y}}_{q,h^{\prime},w^{\prime}}$
$\displaystyle=\sum_{r_{2}=1}^{\Phi_{2}}\textbf{U}_{r_{2},q}^{(2)}\bm{\mathcal{T}}_{r_{2},h^{\prime},w^{\prime}}^{2},$

where $\bm{\mathcal{X}}\in\mathbb{R}^{C_{\mathrm{in}}\times H\times W}$ and
$\bm{\mathcal{Y}}\in\mathbb{R}^{C_{\mathrm{out}}\times H^{\prime}\times
W^{\prime}}$ are the input and output tensor of the convolutional layer,
respectively. For indices of $\bm{\mathcal{T}}^{1}$,
$h_{i}=\texttt{stride}\times(h^{\prime}-1)+i-\texttt{padding},w_{j}=\texttt{stride}\times(w^{\prime}-1)+j-\texttt{padding}$.
In addition, $\bm{\mathcal{T}}^{1}$ and $\bm{\mathcal{T}}^{2}$ are the
incurred intermediate results.

##  4 Proposed Low-rank Training Solution

As outlined in Section 1, to date, the efficient training for high-accuracy
low-rank CNN models from scratch is still largely under-explored. In this
section, we propose to systematically explore several important design knobs
and factors when training low-rank CNN models from scratch. Based on the
outcomes from these analytic and empirical studies, we will then further
develop efficient solutions for low-rank CNN training from scratch.

Questions to be Answered. To be specific, in order to obtain a better
understanding of low-rank training and improve its performance, we explore the
answers to the following three questions.

Question1: Which type of low-rank format is more suitable for efficient
training from scratch, 2-D matrix or high-order tensor?

Analysis. In general, when training a compact CNN model from scratch, there
are two types

Figure 2: A low-rank CONV layer can either exhibit low-matrix-rankness (Top)
or low-tensor-rankness (Bottom).

of low-rank formats that can be considered. The 4-D weight tensor of the
trained convolutional layer can be either in the format of a low-matrix-rank
2-D matrix or a high-order low-tensor-rank tensor. As illustrated in Fig. 2,
the low-matrix-rankness means the flattened and matricized 4-D weight tensor
exhibits low-rankness; while the low-tensor-rankness means the trained
convolutional layer can be directly represented and constructed via multiple
small-size factorized matrices/tensors without any flattening operations.

Our Proposal. Currently, most of the existing low-rank training works (Yang et
al., 2020; Ioannou et al., 2015; Tai et al., 2015) conduct and keep 4-D
convolutional layer training in the format of low-rank 2-D matrix. Instead, we
propose to perform low-rank CNN training directly in the high-order tensor
format. In other words, each convolutional layer always stays in the low-rank
tensor decomposition format, e.g., Tucker (Tucker, 1966) or CP (Hitchcock,
1927),

(a) The 15-th convolutional layer.

(b) The 16-th convolutional layer.

Figure 3: Approximation error (Mean Square Error (MSE)) of low-matrix-rank and
low-tensor-rank methods for approximating ResNet-20 layers. Notice that MSE
measurement is our analysis and exploration to identify the suitable low-rank
format. It is not actually executed during training.

during the entire training phase. Our rationale for this proposal is that
unlike the low-rank matrix format, which may lose the important spatial weight
correlation incurred by the inevitable flattening operation; the low-rank
tensor format is a more natural way to represent the 4-D weight tensor of the
convolutional layer; therefore it can better extract and preserve the weight
information and correlation existed in the 4-D space (Liu et al., 2012). For
instance, as illustrated in Fig. 3, when representing the same weight tensors
of layers in the ResNet-20 model, the low-rank Tucker format enjoys a smaller
approximation error than the low-rank matrix format with the same number of
weight parameters. Encouraged by this better representation capability for the
high-order weight tensors, we choose to train low-rank tensor format CNN from
scratch.

Question2: Considering low-rankness implies potentially low model capacity,
what is the proper strategy to improve the performance of low-rank training?

Analysis. As analyzed in Section 1, a key challenge of the existing low-rank
training works (Tai et al., 2015; Ioannou et al., 2015) is their inferior
model accuracy as compared to model compression approaches. We hypothesize
that this phenomenon is caused by two reasons: 1) low-rank training starts
from a low-accuracy random initialization with low-rank constraints; while
model compression is built on a pre-trained high-accuracy model; and 2) the
entire procedure of low-rank training is constrained in the low-rank space,
thereby limiting the growing space for model capability. Notice that a
possible solution is that we can reconstruct and train a full-rank model in
some epochs, and then decompose the model to low-rank format again during the
training procedure. However, such “train-decompose-train” scheme is very
costly since it needs to perform computation-intensive tensor decomposition
many times.

Our Proposal. To improve the performance of low-rank training while preserving
low computational and memory costs, we propose to perform orthogonality-aware
low-tensor-rank training.

(a) Training loss.

(b) Test accuracy.

Figure 4: Training loss (left) and test accuracy (right) for low-tensor-rank
ResNet-20 on CIFAR-10 with/without SO regularization. The same ranks are used
for different experiments. Ranks are selected to provide 2$\times$ FLOPs
reduction.

Our key idea is to impose and enforce the orthogonality on the factor matrices
$\textbf{U}^{(1)}$ and $\textbf{U}^{(2)}$ during the entire training process,
and such training philosophy lies in the following rationale: It is well known
that in principle using orthogonal-format basis can maximize the capacity of
information representation. For instance, when we aim to approximate a full-
rank matrix with low-rank SVD factorization, the decomposed U and V always
exhibit self-orthogonality (as unitary matrix) with the smallest approximation
error (Eckart & Young, 1936). A similar phenomenon also exists for tensor
decomposition, where the factor matrices $\textbf{U}^{(1)}$ and
$\textbf{U}^{(2)}$ are also unitary matrices after performing Tucker
decomposition on the original full-rank tensor. Inspired by the above
observations, when we aim to approach a high-capacity full-rank model using
the low-rank format, the orthogonality-based representation is a very suitable
solution. However, such orthogonality cannot be automatically obtained and
ensured during the low-rank training. Therefore, an efficient mechanism of
enforcing and ensuring orthogonality is very desired.

To verify the potential effectiveness of the proposed orthogonality-aware
idea, we perform a preliminary ablation study of low-tensor-rank training with
and without enforcing orthogonality. Here, a ResNet-20 model is trained from
scratch on the CIFAR-10 dataset, and a simple soft orthogonal regularization
term (Xie et al., 2017; Bansal et al., 2018) is used to impose orthogonality
on $\textbf{U}^{(1)}$ and $\textbf{U}^{(2)}$ during the training. As shown in
Fig. 4, enforcing orthogonality during the training can significantly improve
training and testing performance.

Question3: How should we properly impose the orthogonality during the low-
tensor-rank training?

Analysis. The preliminary experiment in Fig. 4 shows the encouraging benefits
of orthogonality-aware low-tensor-rank training. To fully unlock its promising
potential, an efficient scheme for imposing the desired orthogonality should
be properly explored. To be specific, there are four commonly used
regularization approaches (Bansal et al., 2018) that can introduce the
orthogonality on the target factor matrices:

Figure 5: The mechanism of different approaches to impose orthogonality on
$\textbf{U}^{(2)}$. From top to bottom: (a) Soft Orthogonal Regularization,
(b) Double Soft Orthogonal Regularization, (c) Spectral Restricted Isometry
Property Regularization, (d) Mutual Coherence Regularization.

Soft Orthogonal (SO) Regularization:

$\displaystyle\mathcal{R}_{s}(\textbf{A})=\frac{\rho}{\Phi^{2}}\|\textbf{A}^{\text{T}}\textbf{A}-\textbf{I}\|^{2}_{F},$
(3)

where $\|\cdot\|_{F}$ is the Frobenius norm, $\rho$ is regularization
strength, A is the matrix to be enforced with orthogonality, and $\Phi$ is the
rank of A. Notice the experiment in Fig. 4 adopts this approach.

Double Soft Orthogonal (DSO) Regularization:

$\displaystyle\mathcal{R}_{d}(\textbf{A})=\frac{\rho}{\Phi^{2}}(\|\textbf{A}^{\text{T}}\textbf{A}-\textbf{I}\|^{2}_{F}+\|\textbf{A}\textbf{A}^{\text{T}}-\textbf{I}\|^{2}_{F}).$
(4)

Here, compared with SO regularization, this DSO scheme can always impose the
proper orthogonality on $\mathbf{A}$ no matter if it is over-complete or
under-complete.

Mutual Coherence (MC) Regularization:

$\displaystyle\mathcal{R}_{mc}(\textbf{A})=\rho(\|\textbf{A}^{\text{T}}\textbf{A}-\textbf{I}\|_{\infty}),$
(5)

where $\|\cdot\|_{\infty}$ is the matrix norm induced by the
$\ell_{\infty}$-norm.

Spectral Restricted Isometry Property (SRIP) Regularization:

$\displaystyle\mathcal{R}_{sp}(\textbf{A})={\rho}\cdot\sigma(\textbf{A}^{\text{T}}\textbf{A}-\textbf{I}),$
(6)

where $\sigma(\mathbf{R})=\sup_{z\in{\mathbf{R}^{n}},z\neq
0}\frac{\|\mathbf{R}z\|^{2}}{\|z\|^{2}}$ is the spectral norm of
$\textbf{A}^{\text{T}}\textbf{A}-\textbf{I}$ (Yoshida & Miyato, 2017) based on
RIP condition (Candes & Tao, 2005).

Our Proposal. We propose to develop a DSO regularized scheme to impose the
desired orthogonality on the factor matrices. Here our main rationale is that
DSO regularization can better provide the desired orthogonality. To be
specific, as illustrated in Fig. 5, the orthogonality of the factor matrix
$\textbf{{U}}^{(2)}$ can be measured by its corresponding residual matrix
$\mathbf{R}={\textbf{U}^{(2)}}^{\text{T}}{\textbf{U}^{(2)}}-\textbf{I}$. Here,
when $\mathbf{R}$ has lower energy, it is more likely that $\textbf{U}^{(2)}$
will exhibit more self-orthogonality. Therefore, the essential

Input: Dataset $\mathcal{D}$, pre-set ranks {${\Phi}$}, Tucker-2-format
weights in each layer $\textbf{U}^{(1)}\in\mathbb{R}^{{\Phi}_{1}\times
C_{in}},\bm{\mathcal{G}}\in\mathbb{R}^{\Phi_{1}\times\Phi_{2}\times K\times
K},\textbf{U}^{(2)}\in\mathbb{R}^{\Phi_{2}\times C_{out}}$, orthogonal
parameters $\rho,\lambda_{d}$, training epochs $T$.

Output: Trained
$\\{\textbf{{U}}^{(1)},\textbf{{U}}^{(2)},\bm{\mathcal{G}}\\}$.

Initialize:
xavier_uniform($\\{\textbf{{U}}^{(1)},\textbf{{U}}^{(2)},\bm{\mathcal{G}}\\}$).

for  _$t=1$ to $T$_ do

$\bm{\mathcal{X}},\bm{\mathcal{Y}}\leftarrow\texttt{sample\\_batch}(\mathcal{D})$;

$\hat{\bm{\mathcal{Y}}}\leftarrow\texttt{forward}(\bm{\mathcal{X}},\\{\textbf{{U}}^{(1)},\textbf{{U}}^{(2)},\bm{\mathcal{G}}\\})$
via Eq. 2;

$\triangleright$ Orthogonal regularization

$\mathcal{R}_{d}(\textbf{U}^{(1)})\leftarrow\frac{\rho}{\Phi_{1}^{2}}(\|{\textbf{U}^{(1)}}^{\text{T}}\textbf{U}^{(1)}-\textbf{I}\|^{2}_{F}+\|\textbf{U}^{(1)}{\textbf{U}^{(1)}}^{\text{T}}-\textbf{I}\|^{2}_{F})$;

$\mathcal{R}_{d}(\textbf{U}^{(2)})\leftarrow\frac{\rho}{\Phi_{2}^{2}}(\|{\textbf{U}^{(2)}}^{\text{T}}{\textbf{U}^{(2)}}-\textbf{I}\|^{2}_{F}+\|{\textbf{U}^{(2)}}{\textbf{U}^{(2)}}^{\text{T}}-\textbf{I}\|^{2}_{F})$;

$\texttt{loss}\leftarrow\mathcal{L}({\bm{\mathcal{Y}},\hat{\bm{\mathcal{Y}}}})+\lambda_{d}(\mathcal{R}_{d}(\textbf{U}^{(1)})+\mathcal{R}_{d}(\textbf{U}^{(2)}))$;

update
($\\{\textbf{{U}}^{(1)},\textbf{{U}}^{(2)},\bm{\mathcal{G}}\\},\texttt{loss}$);

end for

Algorithm 1 Overall ELRT training procedure

mechanism of SRIP and MC is to minimize the maximum singular value of
$\mathbf{R}$ and the maximum row sum norm of $\mathbf{R}$, respectively, to
push $\mathbf{R}$ close to the all-zero matrix. Evidently, such constraint
posed by SRIP and MC schemes inherently targets to the local components of
$\mathbf{R}$, thereby degrading the overall effect of orthogonality. On the
other hand, DSO and SO schemes aim to push all the singular values of
$\mathbf{R}$ to zero in a global way, thereby, in principle, bringing stronger
orthogonality for $\textbf{U}^{(2)}$. In addition, since $\textbf{U}^{(1)}$
and $\textbf{U}^{(2)}$ may not be a square matrix in practice, the DSO scheme,
as an approach that simultaneously considers the potential under-completeness
and over-completeness of matrix, is a more general and better solution than SO
scheme. To verify this hypothesis, we also perform an ablation study for low-
tensor-rank training using different orthogonal regularization schemes. As
reported in the Appendix B, DSO scheme shows consistently better performance
than other schemes, and hence it is adopted in our proposed low-rank training
procedure.

Overall Training Procedure. Based on the above analysis and proposals, we
summarize them and develop the corresponding efficient low-rank training
(ELRT) algorithm to train high-performance high-compactness low-tensor rank
CNN model from scratch. Algorithm 1 describes the details.

##  5 Experiments

Datasets and Baselines. We evaluate our approach on CIFAR-10 and ImageNet
datasets with different inference and training FLOPs reduction ratios (e.g.,
2$\times$ FLOPs reduction ratios refer to 50% FLOPs reduction). For
experiments on CIFAR-10 dataset, the performance of ELRT for VGG-16,
ResNet-20, ResNet-56 and MobileNetV2 are evaluated and compared with the
results using compression and structured sparse training methods. For
experiments on ImageNet dataset, we evaluate our approach for training
ResNet-50.

Calculation of Inference and Training FLOPs Reduction. A very important
benefit provided by compression-aware training, e.g., low-rank training and
sparse training, is simultaneously achieving both the “Inference FLOPs
Reduction” and “Training FLOPs Reduction”. The details of the calculation
mechanism for these two metrics are described in the Appendix C.

Hyperparameter. We use SGD optimizer for training with batch size, momentum,
and weight decay as 128, 0.9 and 0.0001, respectively. The learning rates are
set as 0.1 on the CIFAR-10 dataset and 0.05 on the ImageNet dataset,
respectively, with the cosine scheduler. All experiments are performed using
PyTorch 1.12 and following the PyTorch official training strategy. The
detailed configurations for the tensor rank values in each layer are reported
in the Appendix D.

Table 1: Results for VGG-16, ResNet-20, ResNet-56 and MobileNetV2 on CIFAR-10
dataset. “$*$” denotes compression ratio since the corresponding work does not
report FLOPs reduction.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
VGG-16
Original VGG-16 | ✗ | ✓ | Dense | 93.96 | 1.00$\times$ | 1$\times$
LREL (Idelbayev & Carreira-Perpinán, 2020) | ✗ | ✗ | Low-rank | 92.72 | 6.88$\times$ |  $<$1$\times$
ALDS (Liebenwein et al., 2021) | ✓ | ✗ | Low-rank | 92.67 | 7.26$\times$ |  $<$1$\times$
TETD (Yin et al., 2021) | ✗ | ✗ | Low-rank | 93.11 | 7.37$\times$ |  $<$1$\times$
GrowEfficient (Yuan et al., 2021) | ✓ | ✓ | Sparse | 92.50 | 7.35$\times$ | 1.22$\times$
BackSparse (Zhou et al., 2021) | ✓ | ✓ | Sparse | 92.50 | 11.5$\times$ | 8.69$\times$
HRank (Lin et al., 2020a) | ✓ | ✗ | Sparse | 92.34 | 2.24$\times$ |  $<$1$\times$
CHIP (Sui et al., 2021) | ✓ | ✗ | Sparse | 93.18 | 4.67$\times$ |  $<$1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 93.30 | 9.16$\times$ | 9.16$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 92.99 | 12.4$\times$ | 12.4$\times$
ResNet-20
Original ResNet-20 | ✗ | ✓ | Dense | 91.25 | 1.00$\times$ | 1$\times$
SVDT (Yang et al., 2020) | ✓ | ✗ | Low-rank | 90.39 | 2.94$\times$ |  $<$1$\times$
PSTRN-S (Li et al., 2021a) | ✓ | ✗ | Low-rank | 90.80 | 2.25$\times$* |  $<$1$\times$
ALDS (Liebenwein et al., 2021) | ✓ | ✗ | Low-rank | 90.92 | 3.11$\times$ |  $<$1$\times$
LREL (Idelbayev & Carreira-Perpinán, 2020) | ✗ | ✗ | Low-rank | 90.20 | 3.00$\times$ |  $<$1$\times$
GrowEfficient (Yuan et al., 2021) | ✓ | ✓ | Sparse | 90.91 | 2.00$\times$ | 1.13$\times$
BackSparse (Zhou et al., 2021) | ✓ | ✓ | Sparse | 90.93 | 2.77$\times$ | 2.09$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.73 | 1.98$\times$ | 1.98$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.26 | 2.51$\times$ | 2.51$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 90.95 | 3.02$\times$ | 3.02$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 89.64 | 6.01$\times$* | 3.87$\times$
ResNet-56
Original ResNet-56 | ✗ | ✓ | Dense | 93.26 | 1.00$\times$ | 1$\times$
SVDT (Yang et al., 2020) | ✓ | ✗ | Low-rank | 93.17 | 3.75$\times$ |  $<$1$\times$
TRP (Xu et al., 2020) | ✗ | ✗ | Low-rank | 92.63 | 2.43$\times$ |  $<$1$\times$
ENC-Inf (Kim et al., 2019) | ✓ | ✗ | Low-rank | 93.00 | 2.00$\times$ |  $<$1$\times$
CaP (Minnehan & Savakis, 2019) | ✗ | ✗ | Low-rank | 93.22 | 2.00$\times$ |  $<$1$\times$
CC (Li et al., 2021b) | ✓ | ✗ | Low-rank | 93.64 | 2.08$\times$ |  $<$1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 93.96 | 2.05$\times$ | 2.05$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 94.01 | 2.52$\times$ | 2.52$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 93.67 | 3.01$\times$ | 3.01$\times$
MobileNetV2
Original MobileNetV2 | ✗ | ✓ | Dense | 94.48 | 1.00$\times$ | 1$\times$
GAL (Lin et al., 2019) | ✗ | ✗ | Sparse | 93.07 | 1.69$\times$ |  $<$1$\times$
DCP (Zhuang et al., 2018) | ✗ | ✗ | Sparse | 94.25 | 1.36$\times$ |  $<$1$\times$
SCOP (Tang et al., 2020) | ✓ | ✗ | Sparse | 94.24 | 1.67$\times$ |  $<$1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 94.87 | 1.47$\times$ | 1.47$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 94.77 | 1.71$\times$ | 1.71$\times$

Table 2: Results for ResNet-50 on ImageNet dataset.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Top-5 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---|---
ResNet-50
Original ResNet-50 | ✗ | ✓ | Dense | 76.15 | 92.87 | 1.00$\times$ | 1$\times$
AutoP (Luo & Wu, 2020) | ✗ | ✗ | Sparse | 74.76 | 92.15 | 1.95$\times$ |  $<$1$\times$
PruneTrain (Lym et al., 2019) | ✗ | ✓ | Sparse | 74.62 | N/A | 2.13$\times$ | 1.67$\times$
HRank (Lin et al., 2020a) | ✓ | ✗ | Sparse | 74.98 | 92.33 | 1.78$\times$ |  $<$1$\times$
SCOP (Tang et al., 2020) | ✓ | ✗ | Sparse | 75.26 | 92.53 | 2.21$\times$ |  $<$1$\times$
CHIP (Sui et al., 2021) | ✓ | ✗ | Sparse | 75.26 | 92.53 | 2.68$\times$ |  $<$1$\times$
ISP (Ganjdanesh et al., 2022) | ✗ | ✗ | Sparse | 75.97 | 92.74 | 2.31$\times$ |  $<$1$\times$
SVDT (Yang et al., 2020) | ✓ | ✗ | Low-rank | N/A | 91.81 | 1.79$\times$ |  $<$1$\times$
TRPNu (Xu et al., 2020) | ✗ | ✗ | Low-rank | 74.06 | 92.07 | 1.80$\times$ |  $<$1$\times$
Stable (Phan et al., 2020) | ✓ | ✗ | Low-rank | 74.68 | 92.16 | 2.64$\times$ |  $<$1$\times$
Hinge (Li et al., 2020) | ✗ | ✗ | Low-rank | 74.70 | N/A | 2.15$\times$ |  $<$1$\times$
CC (Li et al., 2021b) | ✓ | ✗ | Low-rank | 75.59 | 92.64 | 2.12$\times$ |  $<$1$\times$
CC (Li et al., 2021b) | ✓ | ✗ | Low-rank | 74.54 | 92.25 | 2.68$\times$ |  $<$1$\times$
GrowEfficient (Yuan et al., 2021) | ✓ | ✓ | Sparse | 75.20 | N/A | 1.99$\times$ | 1.10$\times$
BackSparse (Zhou et al., 2021) | ✓ | ✓ | Sparse | 76.00 | N/A | 2.13$\times$ | 1.60$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 76.49 | 93.28 | 2.19$\times$ | 2.19$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 76.11 | 92.88 | 2.49$\times$ | 2.49$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 75.32 | 92.57 | 2.91$\times$ | 2.91$\times$

Table 3: Results for ConvNext and ViT on ImageNet dataset.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
ConvNext
ConvNext-Tiny (Liu et al., 2022) | ✗ | ✓ | Dense | 82.1 | 1.00$\times$ | 1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 81.63 | 1.30$\times$ | 1.30 $\times$
ViT-Base-16
ViT-B-16 (Dosovitskiy et al., 2020) | ✗ | ✓ | Dense | 77.9 | 1.00$\times$ | 1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 78.4 | 1.26$\times$ | 1.26$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 78.0 | 1.68$\times$ | 1.68$\times$

Table 4: Runtime (per image) for the low-rank ResNet-50 trained via our
proposed ELRT.

|  FLOPs Reduction |  ASIC Eyeriss 65nm |  FPGA Xilinx PYNQZ1 |  Desktop GPU Nvidia V100 |  Embedded GPU Nvidia Jetson TX2
---|---|---|---|---|---
Original ResNet-50 | 1$\times$ | 38.10ms (1$\times$) | 172.0ms (1$\times$) | 0.8145ms (1$\times$) | 29.46ms (1$\times$)
ELRT (Ours) | 2.19$\times$ | 19.24ms (1.98$\times$) | 85.62ms (2.01$\times$) | 0.6934ms (1.17$\times$) | 21.47ms (1.37$\times$)
ELRT (Ours) | 2.49$\times$ | 15.96ms (2.39$\times$) | 71.08ms (2.42$\times$) | 0.6241ms (1.31$\times$) | 19.24ms (1.53$\times$)
ELRT (Ours) | 2.91$\times$ | 13.85ms (2.75$\times$) | 61.02ms (2.82$\times$) | 0.5963ms (1.37$\times$) | 18.29ms (1.61$\times$)

VGG-16, ResNet-20, ResNet-56, MobileNetV2 on CIFAR-10. As shown in Table 1,
for ResNet-20 model, the proposed ELRT can bring 0.48% accuracy increase over
baseline model with 1.98$\times$ inference and training FLOPs reduction. For
the ResNet-56 model, the ELRT method can bring 0.75% accuracy increase over
the baseline model with 2.52$\times$ inference and training FLOPs reduction.
We also compare ELRT with several compression approaches on the MobileNetV2
model. ELRT brings 1.71$\times$ inference and training FLOPs reduction with
0.29$\%$ accuracy increase over the baseline. Notice that similar to TETD,
HRank and CHIP, we add BN layers in VGG-16 to stabilize the training process.

ResNet-50, and ViT on ImageNet. Table 2 summarizes the performance of
different methods for generating the compact ResNet-50 model on ImageNet.
Compared with the other approaches, ELRT achieves at least 0.49$\%$ accuracy
increase with 2.19$\times$ inference and training FLOPs reduction. Table 3
indicates the performance of the proposed method for ViT-Base-16 model
(Dosovitskiy et al., 2020) on ImageNet. ELRT achieves at least 0.01$\%$
accuracy increase with 1.68$\times$ inference and training FLOPs reduction.

Remark: Benefits of ELRT on Training FLOPs Reduction. From Table 1 and 2, it
is seen that ELRT enjoys two benefits on low training cost. First, unlike low-
rank compression or pruning, ELRT can reduce overall training costs because it
eliminates the need of the pre-training phase. Second, unlike structured
sparse training, e.g., GrowEfficient and BackSparse, ELRT has the same FLOPs
reduction in both inference and training. This is because structured sparse
training consumes extra computation to calculate the channel mask during
backward propagation, which is not needed in low-rank training. More details
of the calculation of Inference and Training FLOPs reduction for different
methods are reported in the Appendix C.

Comparison with Existing Low-rank Training and Directly Training Small Model.
Because the existing low-rank training works (Tai et al., 2015; Ioannou et
al., 2015; Hawkins et al., 2022) are evaluated on a small dataset (MNIST)
and/or non-popularly used models, we report their comparison with ELRT in the
Appendix A.3. Also, we compare ELRT with directly training a small dense model
with the same target model size, and the results are reported in the Appendix
A.4.

Ablation Study. We perform an ablation study on the effect of imposing
orthogonality and different orthogonal regulation schemes. The details are
reported in the Appendix B.

Practical Speedup of Low-Tensor-Rank CNNs. To demonstrate the practical
effectiveness of low-tensor-rank models, we measure the inference time of the
models trained by using ELRT. Here, we evaluate the speedup on four hardware
platforms: Nvidia V100 (desktop GPU), Nvidia Jetson TX2 (embedded GPU), Xilinx
PYNQZ1 (FPGA), and Eyeriss (ASIC). The evaluated models are low-rank ResNet-50
for ImageNet with three target FLOPs reduction settings. As summarized in
Table 4, the low-rank CNNs obtained from ELRT enjoy considerable speedup
across different hardware platforms.

##  6 Conclusion

This paper proposes ELRT, an efficient low-rank training approach for training
CNN models from scratch. ELRT is essentially an orthogonality-aware training
solution that can impose and ensure the important orthogonality on the low-
rank models during the training process. Evaluation results demonstrate the
effectiveness of ELRT as compared to state-of-the-art model compression and
other compression-aware training approaches.

## References

  * Bansal et al. (2018) Nitin Bansal, Xiaohan Chen, and Zhangyang Wang.  Can we gain more from orthogonality regularizations in training deep networks?  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Baydin et al. (2018) Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind.  Automatic differentiation in machine learning: a survey.  _Journal of Marchine Learning Research_ , 18:1–43, 2018\.
  * Bellec et al. (2018) Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein.  Deep rewiring: Training very sparse deep networks.  In _International Conference on Learning Representations_ , 2018.
  * Candes & Tao (2005) Emmanuel J Candes and Terence Tao.  Decoding by linear programming.  _IEEE transactions on information theory_ , 51(12):4203–4215, 2005.
  * Denton et al. (2014) Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus.  Exploiting linear structure within convolutional networks for efficient evaluation.  In _Advances in Neural Information Processing Systems_ , pp. 1269–1277, 2014.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Eckart & Young (1936) Carl Eckart and Gale Young.  The approximation of one matrix by another of lower rank.  _Psychometrika_ , 1(3):211–218, 1936.
  * Evci et al. (2020) Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen.  Rigging the lottery: Making all tickets winners.  In _International Conference on Machine Learning_ , pp. 2943–2952. PMLR, 2020.
  * Ganjdanesh et al. (2022) Alireza Ganjdanesh, Shangqian Gao, and Heng Huang.  Interpretations steered network pruning via amortized inferred saliency maps.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2022.
  * Gural et al. (2020) Albert Gural, Phillip Nadeau, Mehul Tikekar, and Boris Murmann.  Low-rank training of deep neural networks for emerging memory technology.  _arXiv preprint arXiv:2009.03887_ , 2020.
  * Hawkins et al. (2022) Cole Hawkins, Xing Liu, and Zheng Zhang.  Towards compact neural networks via end-to-end training: A bayesian tensor approach with automatic rank determination.  _SIAM Journal on Mathematics of Data Science_ , 4(1):46–71, 2022.
  * Hayashi et al. (2019) Kohei Hayashi, Taiki Yamaguchi, Yohei Sugawara, and Shin-ichi Maeda.  Exploring unexplored tensor network decompositions for convolutional neural networks.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * He et al. (2018) Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_ , pp. 2234–2240, 2018.
  * He et al. (2019) Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang.  Filter pruning via geometric median for deep convolutional neural networks acceleration.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4340–4349, 2019.
  * Hitchcock (1927) Frank L Hitchcock.  The expression of a tensor or a polyadic as a sum of products.  _Journal of Mathematics and Physics_ , 6(1-4):164–189, 1927.
  * Huang et al. (2018) Lei Huang, Xianglong Liu, Bo Lang, Adams Yu, Yongliang Wang, and Bo Li.  Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Idelbayev & Carreira-Perpinán (2020) Yerlan Idelbayev and Miguel A Carreira-Perpinán.  Low-rank compression of neural nets: Learning the rank of each layer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8049–8059, 2020.
  * Ioannou et al. (2015) Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, and Antonio Criminisi.  Training cnns with low-rank filters for efficient image classification.  _arXiv preprint arXiv:1511.06744_ , 2015.
  * Khodak et al. (2020) Mikhail Khodak, Neil A Tenenholtz, Lester Mackey, and Nicolo Fusi.  Initialization and regularization of factorized neural layers.  In _International Conference on Learning Representations_ , 2020.
  * Kim et al. (2019) Hyeji Kim, Muhammad Umar Karim Khan, and Chong-Min Kyung.  Efficient neural network compression.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12569–12577, 2019.
  * Kim et al. (2015) Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin.  Compression of deep convolutional neural networks for fast and low power mobile applications.  _arXiv preprint arXiv:1511.06530_ , 2015.
  * Kossaifi et al. (2019) Jean Kossaifi, Adrian Bulat, Georgios Tzimiropoulos, and Maja Pantic.  T-net: Parametrizing fully convolutional nets with a single high-order tensor.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7822–7831, 2019.
  * Kossaifi et al. (2020) Jean Kossaifi, Antoine Toisoul, Adrian Bulat, Yannis Panagakis, Timothy M Hospedales, and Maja Pantic.  Factorized higher-order cnns with an application to spatio-temporal emotion estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6060–6069, 2020.
  * Lee et al. (2018) Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr.  Snip: Single-shot network pruning based on connection sensitivity.  In _International Conference on Learning Representations_ , 2018.
  * Li & Shi (2018) Chong Li and CJ Shi.  Constrained optimization based low-rank approximation of deep neural networks.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pp. 732–747, 2018.
  * Li et al. (2021a) Nannan Li, Yu Pan, Yaran Chen, Zixiang Ding, Dongbin Zhao, and Zenglin Xu.  Heuristic rank selection with progressively searching tensor ring network.  _Complex & Intelligent Systems_, pp. 1–15, 2021a.
  * Li et al. (2020) Yawei Li, Shuhang Gu, Christoph Mayer, Luc Van Gool, and Radu Timofte.  Group sparsity: The hinge between filter pruning and decomposition for network compression.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8018–8027, 2020.
  * Li et al. (2021b) Yuchao Li, Shaohui Lin, Jianzhuang Liu, Qixiang Ye, Mengdi Wang, Fei Chao, Fan Yang, Jincheng Ma, Qi Tian, and Rongrong Ji.  Towards compact cnns via collaborative compression.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6438–6447, 2021b.
  * Liebenwein et al. (2021) Lucas Liebenwein, Alaa Maalouf, Dan Feldman, and Daniela Rus.  Compressing neural networks: Towards determining the optimal layer-wise decomposition.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=BvJkwMhyInm.
  * Lin et al. (2013) Min Lin, Qiang Chen, and Shuicheng Yan.  Network in network.  _arXiv preprint arXiv:1312.4400_ , 2013.
  * Lin et al. (2020a) Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao.  Hrank: Filter pruning using high-rank feature map.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1529–1538, 2020a.
  * Lin et al. (2019) Shaohui Lin, Rongrong Ji, Chenqian Yan, Baochang Zhang, Liujuan Cao, Qixiang Ye, Feiyue Huang, and David Doermann.  Towards optimal structured cnn pruning via generative adversarial learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2790–2799, 2019.
  * Lin et al. (2020b) Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi.  Dynamic model pruning with feedback.  _arXiv preprint arXiv:2006.07253_ , 2020b.
  * Liu et al. (2012) Ji Liu, Przemyslaw Musialski, Peter Wonka, and Jieping Ye.  Tensor completion for estimating missing values in visual data.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(1):208–220, 2012.
  * Liu et al. (2020) Junjie Liu, Zhe Xu, Runbin Shi, Ray CC Cheung, and Hayden KH So.  Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers.  _arXiv preprint arXiv:2005.06870_ , 2020.
  * Liu et al. (2022) Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 11976–11986, 2022.
  * Luo & Wu (2020) Jian-Hao Luo and Jianxin Wu.  Autopruner: An end-to-end trainable filter pruning method for efficient deep model inference.  _Pattern Recognition_ , 107:107461, 2020.
  * Lym et al. (2019) Sangkug Lym, Esha Choukse, Siavash Zangeneh, Wei Wen, Sujay Sanghavi, and Mattan Erez.  Prunetrain: fast neural network training by dynamic sparse model reconfiguration.  In _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_ , pp. 1–13, 2019.
  * Minnehan & Savakis (2019) Breton Minnehan and Andreas Savakis.  Cascaded projection: End-to-end network compression and acceleration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10715–10724, 2019.
  * Miyato et al. (2018) Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida.  Spectral normalization for generative adversarial networks.  In _International Conference on Learning Representations_ , 2018.
  * Mocanu et al. (2018) Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta.  Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science.  _Nature communications_ , 9(1):1–12, 2018.
  * Mostafa & Wang (2019) Hesham Mostafa and Xin Wang.  Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization.  In _International Conference on Machine Learning_ , pp. 4646–4655. PMLR, 2019.
  * Ning et al. (2020) Xuefei Ning, Tianchen Zhao, Wenshuo Li, Peng Lei, Yu Wang, and Huazhong Yang.  Dsa: More efficient budgeted pruning via differentiable sparsity allocation.  In _The European Conference on Computer Vision (ECCV)_ , 2020.
  * Novikov et al. (2015) Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov.  Tensorizing neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 442–450, 2015.
  * Phan et al. (2020) Anh-Huy Phan, Konstantin Sobolev, Konstantin Sozykin, Dmitry Ermilov, Julia Gusak, Petr Tichavskỳ, Valeriy Glukhov, Ivan Oseledets, and Andrzej Cichocki.  Stable low-rank tensor decomposition for compression of convolutional neural network.  In _European Conference on Computer Vision_ , pp. 522–539. Springer, 2020.
  * Rodriguez et al. (2017) Pau Rodriguez, Jordi Gonzalez Sabate, Guillem Cucurull, Josep Maria Gonfaus Sitjes, Francesc Xavier, and Roca Marvà.  Regularizing cnns with locally constrained decorrelations.  In _International Conference on Learning Representations (ICLR)_ , 2017.
  * Sui et al. (2021) Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Zonouz, and Bo Yuan.  Chip: Channel independence-based pruning for compact neural networks.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Tai et al. (2015) Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al.  Convolutional neural networks with low-rank regularization.  _arXiv preprint arXiv:1511.06067_ , 2015.
  * Tang et al. (2020) Yehui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing XU, Chao Xu, and Chang Xu.  Scop: Scientific control for reliable neural network pruning.  In _Advances in Neural Information Processing Systems_ , pp. 10936–10947, 2020.  URL https://proceedings.neurips.cc/paper/2020/file/7bcdf75ad237b8e02e301f4091fb6bc8-Paper.pdf.
  * Tucker (1966) Ledyard R Tucker.  Some mathematical notes on three-mode factor analysis.  _Psychometrika_ , 31(3):279–311, 1966.
  * Waleffe & Rekatsinas (2020) Roger Waleffe and Theodoros Rekatsinas.  Principal component networks: Parameter reduction early in training.  _arXiv preprint arXiv:2006.13347_ , 2020.
  * Wang et al. (2019) Chaoqi Wang, Guodong Zhang, and Roger Grosse.  Picking winning tickets before training by preserving gradient flow.  In _International Conference on Learning Representations_ , 2019.
  * Wang et al. (2021) Hongyi Wang, Saurabh Agarwal, and Dimitris Papailiopoulos.  Pufferfish: communication-efficient models at no extra cost.  _Proceedings of Machine Learning and Systems_ , 3:365–386, 2021.
  * Wang et al. (2020) Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu.  Orthogonal convolutional neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 11505–11515, 2020.
  * Wang et al. (2018) Wenqi Wang, Yifan Sun, Brian Eriksson, Wenlin Wang, and Vaneet Aggarwal.  Wide compression: Tensor ring nets.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9329–9338, 2018.
  * Xie et al. (2017) Di Xie, Jiang Xiong, and Shiliang Pu.  All you need is beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6176–6185, 2017.
  * Xu et al. (2020) Yuhui Xu, Yuxi Li, Shuai Zhang, Wei Wen, Botao Wang, Yingyong Qi, Yiran Chen, Weiyao Lin, and Hongkai Xiong.  Trp: Trained rank pruning for efficient deep neural networks.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , pp. 977–983, 2020.
  * Yang et al. (2020) Huanrui Yang, Minxue Tang, Wei Wen, Feng Yan, Daniel Hu, Ang Li, Hai Li, and Yiran Chen.  Learning low-rank deep neural networks via singular vector orthogonality regularization and singular value sparsification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pp. 678–679, 2020.
  * Yang et al. (2017) Yinchong Yang, Denis Krompass, and Volker Tresp.  Tensor-train recurrent neural networks for video classification.  In _International Conference on Machine Learning_ , pp. 3891–3900, 2017.
  * Yin et al. (2021) Miao Yin, Yang Sui, Siyu Liao, and Bo Yuan.  Towards efficient tensor decomposition-based dnn model compression with optimization framework.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10674–10683, 2021.
  * Yoshida & Miyato (2017) Yuichi Yoshida and Takeru Miyato.  Spectral norm regularization for improving the generalizability of deep learning.  _arXiv preprint arXiv:1705.10941_ , 2017.
  * Yuan et al. (2021) Xin Yuan, Pedro Henrique Pamplona Savarese, and Michael Maire.  Growing efficient deep networks by structured continuous sparsification.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=wb3wxCObbRT.
  * Zhou et al. (2021) Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang.  Efficient neural network training via forward and backward propagation sparsification.  _Advances in Neural Information Processing Systems 34_ , 2021.
  * Zhuang et al. (2018) Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu.  Discrimination-aware channel pruning for deep neural networks.  In _Advances in Neural Information Processing Systems_ , pp. 875–886, 2018.

## Appendix

Appendix for Submission #4375 ”ELRT: Towards Efficient Low-Rank Training for
Compact Neural Networks”

The entire Appendix consists of four sections.

  * •

Section A lists the additional experiment results with more model types and
more comparisons.

  * •

Section B reports the ablation study for the orthogonality-imposing strategy.

  * •

Section C presents the calculation schemes for two important performance
metrics: “inference FLOPs reduction” and “training FLOPs reduction”.

  * •

Section D shows the layer-wise rank distribution in the experiments, thereby
demonstrating the convenience of rank setting.

##  Appendix A Additional Experiments

###  A.1 ResNet-32 and WRN-28-8 on CIFAR-10

Table 5 shows the evaluation results of low-rank training for ResNet-32 and
WideResNet-28-8 model on CIFAR-10 dataset. Ror ResNet-32 mode, it is seen that
the proposed ELRT method can bring 0.23% accuracy increase over the baseline
model with 2.31$\times$ inference and training FLOPs reduction. Compared with
the existing methods, the proposed low-rank training from scratch approach
shows better performance with respect to FLOPs reduction and accuracy.

Table 5: Results for ResNet-32 and WRN-28-8 on CIFAR-10 dataset. “$*$” denotes
compression ratio since the corresponding work does not report FLOPs
reduction.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
ResNet-32
Original ResNet-32 | ✗ | ✓ | Dense | 92.49 | 1.00$\times$ | 1$\times$
FPGM(He et al., 2019) | ✓ | ✗ | Sparse | 91.93 | 2.12$\times$ |  $<$1$\times$
SCOP(Tang et al., 2020) | ✓ | ✗ | Sparse | 92.13 | 2.27$\times$ |  $<$1$\times$
SVDT(Yang et al., 2020) | ✓ | ✗ | Low-rank | 90.55 | 3.93$\times$ | N/A
PSTRN-S(Li et al., 2021a) | ✓ | ✗ | Low-rank | 91.44 | 2.60$\times$* |  $<$1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 92.67 | 2.05$\times$ | 2.05$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 92.72 | 2.31$\times$ | 2.31$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 92.03 | 3.01$\times$ | 3.01$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.21 | 5.40$\times$* | 3.34$\times$
WRN-28-8
Original WRN-28-8 | ✗ | ✓ | Dense | 95.60 | 1.00$\times$ | 1$\times$
DST (Liu et al., 2020) | ✓ | ✓ | Sparse | 94.80 | 10.0$\times$* | N/A
SFP (He et al., 2018) | ✓ | ✓ | Sparse | 94.22 | 5.00$\times$* |  $<$1$\times$
DPF (Lin et al., 2020b) | ✓ | ✓ | Sparse | 95.15 | 5.00$\times$* | N/A
ELRT (Ours) | ✓ | ✓ | Low-rank | 95.65 | 5.36$\times$* | 3.14$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 95.22 | 6.44$\times$* | 3.32$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 94.87 | 11.8$\times$* | 5.76$\times$

###  A.2 ResNet-18 on ImageNet

Table 6 shows the evaluation results of low-rank training for ResNet-18 on
ImageNet dataset. It is seen that the proposed ELRT method can increase
0.22$\%$ accuracy over the baseline model with 1.40$\times$ inference and
training FLOPs reduction.

Table 6: Results for ResNet-18 on ImageNet dataset.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Top-5 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---|---
ResNet-18
Original ResNet-18 | ✗ | ✓ | Dense | 69.79 | 89.08 | 1.00$\times$ | 1$\times$
FPGM (He et al., 2019) | ✓ | ✗ | Sparse | 68.34 | 88.53 | 1.72$\times$ |  $<$1$\times$
DSA (Ning et al., 2020) | ✓ | ✗ | Sparse | 68.61 | 88.35 | 1.67$\times$ |  $<$1$\times$
SCOP (Tang et al., 2020) | ✓ | ✗ | Sparse | 68.62 | 88.45 | 1.81$\times$ |  $<$1$\times$
SVDT (Yang et al., 2020) | ✓ | ✗ | Low-rank | N/A | 87.26 | 2.03$\times$ |  $<$1$\times$
TRP (Xu et al., 2020) | ✗ | ✗ | Low-rank | 65.46 | 86.48 | 1.81$\times$ |  $<$1$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 70.01 | 89.46 | 1.40$\times$ | 1.40$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 68.91 | 88.72 | 1.84$\times$ | 1.84$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 68.65 | 88.38 | 2.17$\times$ | 2.17$\times$

###  A.3 Comparison with Existing Low-rank Training

We also compare the performance of ELRT with the existing low-rank training
methods (Tai et al., 2015; Ioannou et al., 2015; Hawkins et al., 2022). Table
7 shows the experimental results of training NIN model (Lin et al., 2013) from
scratch on the CIFAR-10 dataset. It is seen that compared with (Tai et al.,
2015; Ioannou et al., 2015), ELRT can achieve higher FLOPs reduction while
providing better accuracy performance. In addition, we also compare our
proposed method with another Tucker-format work (Hawkins et al., 2022) for
low-rank training from scratch. As shown in Table 8, ELRT can achieve higher
accuracy and FLOPs reduction than (Hawkins et al., 2022). In addition,
considering ELRT does not require computation-intensive Bayesian estimation
that is used in (Hawkins et al., 2022), ELRT is more attractive for practical
applications.

Table 7: Comparison with existing low-rank training works for NIN on CIFAR-10
dataset.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
NIN
Original NIN | ✗ | ✓ | Dense | 91.88 | 1.00$\times$ | 1$\times$
LRR (Tai et al., 2015) | ✓ | ✓ | Low-rank | 93.02 | 1.50$\times$ | N/A
LRF (Ioannou et al., 2015) | ✓ | ✓ | Low-rank | 91.78 | 1.86$\times$ | N/A
ELRT (Ours) | ✓ | ✓ | Low-rank | 93.16 | 1.62$\times$ | 1.62$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 92.23 | 2.03$\times$ | 2.03$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.86 | 2.52$\times$ | 2.52$\times$

Table 8: Results for low-rank training works on MNIST dataset. The model used
here can be referred to (Hawkins et al., 2022).

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
Customized Model from (Hawkins et al., 2022)
Original Customized Model | ✗ | ✓ | Dense | 98.09 | 1.00$\times$ | 1$\times$
ARD-LU (Hawkins et al., 2022) | ✓ | ✓ | Low-rank | 98.30 | 4.00$\times$ | N/A
ARD-HC (Hawkins et al., 2022) | ✓ | ✓ | Low-rank | 98.30 | 4.50$\times$ | N/A
ELRT (Ours) | ✓ | ✓ | Low-rank | 98.41 | 4.74$\times$ | 4.74$\times$

###  A.4 Comparison with Other Compact Models Trained From Scratch

We also compare the performance of ELRT with directly training small dense
models from scratch with the similar target model sizes. Table 9 shows the
experimental results of training ResNet-20 model from scratch on CIFAR-10
dataset. It is seen that compared with models by uniformly removing 25% and
37.5% filters, ELRT can achieve better FLOPs reduction with providing higher
accuracy.

Table 9: Comparison with directly training small dense models with the similar
target model sizes on CIFAR-10 dataset.

Method |  Always Train Compact Model |  Direct Train (No Pre-train) |  Post-Train Model |  Top-1 (%) |  Inference FLOPs $\downarrow$ |  Training FLOPs $\downarrow$
---|---|---|---|---|---|---
ResNet-20
Original ResNet-20 | ✗ | ✓ | Dense | 91.25 | 1.00$\times$ | 1$\times$
Uniform-0.75 | ✓ | ✓ | Dense | 91.08 | 1.70$\times$ | 1.70$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.73 | 1.98$\times$ | 1.98$\times$
Uniform-0.625 | ✓ | ✓ | Dense | 90.11 | 2.49$\times$ | 2.49$\times$
ELRT (Ours) | ✓ | ✓ | Low-rank | 91.26 | 2.51$\times$ | 2.51$\times$

###  A.5 Comparison with Other Related Works

We also compare the performance of ELRT with (1) the works that employ tensor
networks straightforwardly (Hayashi et al., 2019); (2) other advanced low-rank
schemes (Khodak et al., 2020; Wang et al., 2021; Waleffe & Rekatsinas, 2020).

(Hayashi et al., 2019) explores to train low-rank CNNs with newly discovered
decomposition formats via using architecture search. A key drawback is its
very high training cost. As reported in (Hayashi et al., 2019), it needs 829
GPU days to train ResNet-50 on the CIFAR-10 dataset, limiting the practicality
of this approach, especially on large-scale datasets. Instead, ELRT provides
an efficient low-rank training solution with much fewer costs and higher model
performance. As shown in the table below, ELRT shows higher 0.5% higher
accuracy than (Hayashi et al., 2019) with two times fewer model parameters for
training low-rank ResNet-50 on the CIFAR-10 dataset.

Table 10: Comparison with (Hayashi et al., 2019) on CIFAR-10 dataset.

Method |  Remaining Parameters |  Top-1 (%)
---|---|---
ResNet-50
(Hayashi et al., 2019) | 22.2M | 92.2
ELRT (Ours) | 10.6M | 92.73

(Khodak et al., 2020) applies spectral initialization (SI) and Frobenius decay
(FD) for low-rank training. As shown in the following table, ELRT outperforms
different variants of [R5] when training VGG on the CIFAR-10 dataset with a
higher compression ratio and at least 1.4% accuracy increase.

Table 11: Comparison with (Khodak et al., 2020) on CIFAR-10 dataset.

Method |  Compression Ratio |  Top-1 (%)
---|---|---
VGGNet
Low-rank (Khodak et al., 2020) | 10.0$\times$ | 90.71
Low-rank+FI (Khodak et al., 2020) | 10.0$\times$ | 90.99
Low-rank+FD (Khodak et al., 2020) | 10.0$\times$ | 91.57
Low-rank+SI+FD (Khodak et al., 2020) | 10.0$\times$ | 91.58
ELRT (Ours) | 12.4$\times$ | 92.99

(Wang et al., 2021) needs a warm-up phase to train a full-rank model in the
first epochs, so it does not reduce the overall memory requirement, which is
measured by peak memory usage. Instead, ELRT trains a low-rank model from
scratch and always keeps the low-rank format during the training. As shown in
the following table, ELRT brings higher model accuracy with fewer FLOPs than
(Wang et al., 2021) for training ResNet-50 on ImageNet.

Table 12: Comparison with (Wang et al., 2021) on ImageNet dataset.

Method |  FLOPs Reduction $\downarrow$ |  Top-1 (%)
---|---|---
ResNet-50
FP32 (Wang et al., 2021) | 1.14$\times$ | 76.43
AMP (Wang et al., 2021) | 1.14$\times$ | 76.35
ELRT (Ours) | 2.2$\times$ | 76.49

Similar to (Wang et al., 2021), (Waleffe & Rekatsinas, 2020) is initialized
with a wide and full-rank model and then decomposed in low-rank format after a
few epochs. Instead, ELRT performs low-rank training from scratch and always
keeps the model in the low-rank format during the entire training procedure.
As shown in the following table, ELRT shows a 0.51% accuracy increase over
(Waleffe & Rekatsinas, 2020) with two times model size reduction for training
ResNet-50 on the ImageNet dataset.

Table 13: Comparison with (Waleffe & Rekatsinas, 2020) on ImageNet dataset.

Method |  Remaining Parameters |  Top-1 (%)
---|---|---
ResNet-50
(Waleffe & Rekatsinas, 2020) | 25.5M | 75.6
ELRT (Ours) | 11.4M | 76.11

##  Appendix B Ablation Studies

Effect of Imposing Orthogonality. We conduct experiments to study the effect
of imposing the orthogonality on the factor matrices via using DSO
regularization. As shown in Fig. 6, our proposed orthogonality-aware low-rank
training shows very significant performance improvement compared to the
standard low-rank training with the same FLOPs reduction and same low-rank
format, thereby demonstrating the importance of enforcing orthogonality on the
components of the low-rank model.

Figure 6: Performance of low-tensor-rank training with and without using DSO
for ResNet-20/32/56 on CIFAR-10 dataset.

Different Orthogonal Regularization Schemes. We also conduct the ablation
study to explore the best-suited scheme to impose the orthogonality. As shown
in Table 14, DSO regularization demonstrates consistently better performance
than the other schemes. Such empirical results also coincide with our analysis
in Question 3. Therefore, we adopt the DSO scheme for all our experiments.

Table 14: Performance of using different orthogonal regularization schemes for
training low-rank ResNet-20, ResNet-32 and ResNet-56 on CIFAR-10 dataset.

FLOPs $\downarrow$ |  SO |  SRIP |  MC |  DSO
---|---|---|---|---
ResNet-20
2$\times$ | 91.43 | 91.32 | 91.36 | 91.73
2.5$\times$ | 91.05 | 90.92 | 90.91 | 91.26
3$\times$ | 90.72 | 90.59 | 90.43 | 90.95
ResNet-32
2$\times$ | 92.75 | 92.38 | 92.41 | 92.67
3$\times$ | 91.79 | 91.72 | 91.76 | 92.03
ResNet-56
2$\times$ | 93.52 | 93.44 | 93.45 | 93.96
3$\times$ | 93.35 | 93.26 | 93.28 | 93.67

Effect of Different Orthogonality Penalty Parameters $\lambda_{d}$. As shown
in Table 15, 16 and Figure 7, we evaluate the effect of different
orthogonality penalty parameters $\lambda_{d}$.

Table 15: ResNet-20 with different orthogonality penalty parameters
($\lambda_{d}$).

$\lambda_{d}$ |  FLOPs $\downarrow$ |  Accuracy (%) |  FLOPs $\downarrow$ |  Accuracy (%) |  FLOPs $\downarrow$ |  Accuracy (%)
---|---|---|---|---|---|---
1e-6 | 1.98$\times$ | 91.48 | 2.51$\times$ | 91.11 | 3.02$\times$ | 90.68
5e-4 | 91.64 | 91.16 | 90.80
1e-3 | 91.73 | 91.26 | 90.95
5e-3 | 91.61 | 91.29 | 90.96
1e-2 | 91.66 | 91.22 | 90.88
5e-2 | 91.55 | 91.37 | 91.28
1 | 91.78 | 91.12 | 91.09

Table 16: ResNet-56 with different orthogonality penalty parameters
($\lambda_{d}$).

$\lambda_{d}$ |  FLOPs $\downarrow$ |  Accuracy (%) |  FLOPs $\downarrow$ |  Accuracy (%) |  FLOPs $\downarrow$ |  Accuracy (%)
---|---|---|---|---|---|---
1e-6 | 2.05$\times$ | 92.82 | 2.52$\times$ | 93.24 | 3.01$\times$ | 93.28
5e-4 | 93.62 | 93.86 | 93.51
1e-3 | 93.96 | 94.01 | 93.67
5e-3 | 93.95 | 93.87 | 93.64
1e-2 | 93.87 | 93.86 | 93.59
5e-2 | 94.03 | 93.71 | 93.55
1 | 93.87 | 93.75 | 93.64

Figure 7: Performance of ELRT for ResNet-20/56 on CIFAR-10 dataset with
different orthogonal penalty parameters $\lambda_{d}$.

##  Appendix C Calculation Schemes for Inference and Training FLOPs Reduction

###  C.1 Calculation of Inference FLOPs Reduction

To calculate the inference FLOPs reduction of one convolution layer after
using Tucker-2 decomposition, we adopt the scheme used in (Kim et al., 2015):

$\displaystyle
E=\frac{D^{2}STH^{\prime}W^{\prime}}{SR_{1}H^{\prime}W^{\prime}+D^{2}R_{1}R_{2}H^{\prime}W^{\prime}+TR_{2}H^{\prime}W^{\prime}},$
(7)

where $D$ is the kernel height and width, $S$ is the number of input channels,
$T$ is the number of output channels, $H^{\prime},W^{\prime}$ are output
height and width, and $R_{1},R_{2}$ are the ranks of the Tucker decomposition.

With the above notation, the inference FLOPs reduction can be calculated as
$\frac{\sum_{\text{all layers}}A}{\sum_{\text{all layers}}B}$, where
$A=D^{2}STH^{\prime}W^{\prime}$ represents the number of multiplication-
addition operations of a layer in the oiginal model, and
$B=SR_{1}H^{\prime}W^{\prime}+D^{2}R_{1}R_{2}H^{\prime}W^{\prime}+TR_{2}H^{\prime}W^{\prime}$
represents the number of multiplication-addition operations of a layer in the
Tucker decomposed low-rank model.

###  C.2 Calculation of Training FLOPs Reduction

As defined in (Zhou et al., 2021; Evci et al., 2020), “training FLOPs
reduction”, also noted as “training-cost saving”, is the ratio of the average
FLOPs of the dense network over that of the compact network. Here, the total
FLOPs of training a network consist of the parts in forward pass and backward
pass. As indicated in (Evci et al., 2020; Zhou et al., 2021; Baydin et al.,
2018), the FLOPs of backward propagation can be roughly counted as about two
times that is consumed in the forward propagation. Next, we denote the FLOPs
of the dense network, the sparse network, and the low-rank network during
forward propagation as $f_{D},f_{S}$ and $f_{L}$, respectively.

####  C.2.1 Dense Network.

The FLOPs of the forward propagation is $f_{D}$. The FLOPs of the backward
propagation is $2f_{D}$. The training FLOPs reduction is
$\frac{f_{D}+2f_{D}}{f_{D}+2f_{D}}=1$, which means there is no training FLOPs
reduction.

####  C.2.2 Pruning.

Assume $T$ epochs are needed to obtain a pre-trained model. After pruning,
another $K$ epochs are needed to re-train the model for fine-tuning. In the
pre-training phase, the total computation cost of the forward propagation and
backward propagation are $f_{D}*T$ and $2f_{D}*T$, respectively. In the re-
training phase, the total computation cost of the forward propagation and
backward propagation are $f_{S}*K$ and $2f_{S}*K$, respectively. So overall
the training FLOPs reduction is $\frac{3f_{D}*T}{3f_{D}*T+3f_{S}*K}<1$, which
means there is no training FLOPs reduction.

####  C.2.3 Low-rank Compression.

Assume $T$ epochs are needed to obtain a pre-trained model. After low-rank
decomposition, another $K$ epochs are needed to re-train the model for fine-
tuning. In the pre-training phase, the total computation cost of the forward
propagation and backward propagation are $f_{D}*T$ and $2f_{D}*T$,
respectively. In the re-training phase, the total computation cost of the
forward propagation and backward propagation are $f_{L}*K$ and $2f_{L}*K$,
respectively. So overall the training FLOPs reduction is
$\frac{3f_{D}*T}{3f_{D}*T+3f_{L}*K}<1$, which means there is no training FLOPs
reduction.

####  C.2.4 Structured Sparse Training.

  * •

GrowEfficient(Yuan et al., 2021). For training a sparse model via
GrowEfficient, the FLOPs of the forward propagation is $f_{S}$. Since the
channel masks and scores, which determine to keep or remove the corresponding
channels/filters, are updated by the Straight-Through gradient Estimation
(STE), the backward propagation has to go through all the channels/filters,
leading to dense computation. Therefore, the FLOPs of the backward propagation
is $2f_{D}$. For training a dense model, the computation cost of the forward
propagation and backward propagation are $f_{D}$ and $2f_{D}$, respectively.
Therefore, the training FLOPs reduction is
$\frac{f_{D}+2f_{D}}{f_{S}+2f_{D}}=\frac{3f_{D}}{f_{S}+2f_{D}}$.

  * •

SparseBackward(Zhou et al., 2021). For training a sparse model via
SparseBackward, the FLOPs of the forward propagation is $f_{S}$. The backward
propagation keeps sparse with $2f_{S}$ computation cost. Unlike updating
masks/scores via dense backward propagation in GrowEfficient, SparseBackward
updates them via Variance Reduced Policy Gradient Estimator (VR-PGE), which
only requires an extra one-time forward propagation with training cost
$f_{S}$. For training a dense model, the computation cost of the forward
propagation and backward propagation are $f_{D}$ and $2f_{D}$, respectively.
Therefore, the training FLOPs reduction is
$\frac{f_{D}+2f_{D}}{2f_{S}+2f_{S}}=\frac{3f_{D}}{4f_{S}}$.

####  C.2.5 ELRT (Ours).

Recall that ELRT is built on Tucker-2 decomposition that converts one
convolutional layer into two factor matrices, which can be viewed as $1\times
1$ convolutional layers, and one core tensor that is viewed as $3\times 3$
convolutional layer. Therefore, our low-rank model can be viewed as a compact
dense network with more convolutional layers but fewer FLOPs and parameters.
Therefore, assume the FLOPs of the forward propagation is $f_{L}$. The FLOPs
of the backward propagation is $2f_{L}$. The training FLOPs reduction is
calculated as $\frac{f_{D}+2f_{D}}{f_{L}+2f_{L}}=\frac{f_{D}}{f_{L}}$, which
is identical to the inference FLOPs reduction as calculated in Eq. 7. In other
words, the inference and training FLOPs reductions brought by ELRT are the
same. Notice that the FLOPs of calculating the orthogonality loss term is
ignored here since it only occurs per batch, while $f_{L}$ and $f_{D}$ are
consumed per data. Considering the batch size is typically large (e.g., 128 in
our experiment), the FLOPs contribution of calculating the orthogonality loss
term is negligible.

##  Appendix D Rank Settings

Table 17, 18 and 19 list the layer-wise rank settings of ResNet-20 model on
CIFAR-10 dataset under 1.98$\times$, 3.02$\times$ FLOPs and 6.01$\times$
parameters reduction, respectively. Table 20, 21 list the layer-wise rank
settings of ResNet-56 model on CIFAR-10 dataset under 2.05$\times$ and
2.52$\times$ FLOPs reduction. Table 22 lists the layer-wise rank settings of
ResNet-50 model on ImageNet dataset under 2.49$\times$ FLOPs reduction.

From these tables, it is seen that many adjacent layers share the same rank
value, and the entire model only needs a few rank values to be assigned. For
instance, for ResNet-56 model under 2.05$\times$ FLOPs reduction on CIFAR-10
dataset, only three numbers: 12, 18, 26, are selected as the ranks to be
assigned to all layers, where the first/second/last eighteen layers share the
rank value 12/18/26, respectively. Such a rank-sharing phenomenon
significantly simplifies the rank-selection process.

Table 17: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 1.98$\times$ FLOPs reduction. Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 1.98$\times$
layer1.0.conv1 | (14, 14) | layer2.0.conv1 | (14, 14) | layer3.0.conv1 | (28, 28)
layer1.0.conv2 | (12, 12) | layer2.0.conv2 | (14, 14) | layer3.0.conv1 | (28, 28)
layer1.1.conv1 | (12, 12) | layer2.1.conv1 | (14, 14) | layer3.1.conv1 | (28, 28)
layer1.1.conv2 | (12, 12) | layer2.1.conv2 | (14, 14) | layer3.1.conv1 | (28, 28)
layer1.2.conv1 | (12, 12) | layer2.2.conv1 | (16, 16) | layer3.2.conv1 | (28, 28)
layer1.2.conv2 | (12, 12) | layer2.2.conv2 | (16, 16) | layer3.2.conv1 | (28, 28)
Table 18: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 3.02$\times$ FLOPs reduction. Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 3.02$\times$
layer1.0.conv1 | (10, 10) | layer2.0.conv1 | (12, 12) | layer3.0.conv1 | (20, 20)
layer1.0.conv2 | (10, 10) | layer2.0.conv2 | (12, 12) | layer3.0.conv1 | (20, 20)
layer1.1.conv1 | (10, 10) | layer2.1.conv1 | (12, 12) | layer3.1.conv1 | (20, 20)
layer1.1.conv2 | (8, 8) | layer2.1.conv2 | (14, 14) | layer3.1.conv1 | (22, 22)
layer1.2.conv1 | (8, 8) | layer2.2.conv1 | (14, 14) | layer3.2.conv1 | (22, 22)
layer1.2.conv2 | (8, 8) | layer2.2.conv2 | (14, 14) | layer3.2.conv1 | (22, 22)
Table 19: Layer-wise rank settings of the compressed ResNet-20 model on CIFAR-10 dataset with 6.01$\times$ parameters reduction. Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 6.01$\times$
layer1.0.conv1 | (9, 9) | layer2.0.conv1 | (12, 12) | layer3.0.conv1 | (16, 16)
layer1.0.conv2 | (9, 9) | layer2.0.conv2 | (12, 12) | layer3.0.conv1 | (16, 16)
layer1.1.conv1 | (9, 9) | layer2.1.conv1 | (12, 12) | layer3.1.conv1 | (16, 16)
layer1.1.conv2 | (9, 9) | layer2.1.conv2 | (12, 12) | layer3.1.conv1 | (16, 16)
layer1.2.conv1 | (9, 9) | layer2.2.conv1 | (12, 12) | layer3.2.conv1 | (16, 16)
layer1.2.conv2 | (9, 9) | layer2.2.conv2 | (12, 12) | layer3.2.conv1 | (16, 16)
Table 20: Layer-wise rank settings of the compressed ResNet-56 model on CIFAR-10 dataset with 2.05$\times$ FLOPs reduction. Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 2.05$\times$
layer1.0.conv1 | (12, 12) | layer2.0.conv1 | (18, 18) | layer3.0.conv1 | (26, 26)
layer1.0.conv2 | (12, 12) | layer2.0.conv2 | (18, 18) | layer3.0.conv1 | (26, 26)
layer1.1.conv1 | (12, 12) | layer2.1.conv1 | (18, 18) | layer3.1.conv1 | (26, 26)
layer1.1.conv2 | (12, 12) | layer2.1.conv2 | (18, 18) | layer3.1.conv1 | (26, 26)
layer1.2.conv1 | (12, 12) | layer2.2.conv1 | (18, 18) | layer3.2.conv1 | (26, 26)
layer1.2.conv2 | (12, 12) | layer2.2.conv2 | (18, 18) | layer3.2.conv1 | (26, 26)
layer1.3.conv1 | (12, 12) | layer2.3.conv1 | (18, 18) | layer3.3.conv1 | (26, 26)
layer1.3.conv2 | (12, 12) | layer2.3.conv2 | (18, 18) | layer3.3.conv1 | (26, 26)
layer1.4.conv1 | (12, 12) | layer2.4.conv1 | (18, 18) | layer3.4.conv1 | (26, 26)
layer1.4.conv2 | (12, 12) | layer2.4.conv2 | (18, 18) | layer3.4.conv1 | (26, 26)
layer1.5.conv1 | (12, 12) | layer2.5.conv1 | (18, 18) | layer3.5.conv1 | (26, 26)
layer1.5.conv2 | (12, 12) | layer2.5.conv2 | (18, 18) | layer3.5.conv1 | (26, 26)
layer1.6.conv1 | (12, 12) | layer2.6.conv1 | (18, 18) | layer3.6.conv1 | (26, 26)
layer1.6.conv2 | (12, 12) | layer2.6.conv2 | (18, 18) | layer3.6.conv1 | (26, 26)
layer1.7.conv1 | (12, 12) | layer2.7.conv1 | (18, 18) | layer3.7.conv1 | (26, 26)
layer1.7.conv2 | (12, 12) | layer2.7.conv2 | (18, 18) | layer3.7.conv1 | (26, 26)
layer1.8.conv1 | (12, 12) | layer2.8.conv1 | (18, 18) | layer3.8.conv1 | (26, 26)
layer1.8.conv2 | (12, 12) | layer2.8.conv2 | (18, 18) | layer3.8.conv1 | (26, 26)
Table 21: Layer-wise rank settings of the compressed ResNet-56 model on CIFAR-10 dataset with 2.52$\times$ FLOPs reduction. Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 2.52$\times$
layer1.0.conv1 | (10, 10) | layer2.0.conv1 | (15, 15) | layer3.0.conv1 | (28, 28)
layer1.0.conv2 | (10, 10) | layer2.0.conv2 | (15, 15) | layer3.0.conv1 | (28, 28)
layer1.1.conv1 | (10, 10) | layer2.1.conv1 | (15, 15) | layer3.1.conv1 | (28, 28)
layer1.1.conv2 | (10, 10) | layer2.1.conv2 | (15, 15) | layer3.1.conv1 | (28, 28)
layer1.2.conv1 | (10, 10) | layer2.2.conv1 | (15, 15) | layer3.2.conv1 | (28, 28)
layer1.2.conv2 | (10, 10) | layer2.2.conv2 | (15, 15) | layer3.2.conv1 | (28, 28)
layer1.3.conv1 | (10, 10) | layer2.3.conv1 | (15, 15) | layer3.3.conv1 | (28, 28)
layer1.3.conv2 | (10, 10) | layer2.3.conv2 | (15, 15) | layer3.3.conv1 | (28, 28)
layer1.4.conv1 | (10, 10) | layer2.4.conv1 | (15, 15) | layer3.4.conv1 | (28, 28)
layer1.4.conv2 | (10, 10) | layer2.4.conv2 | (15, 15) | layer3.4.conv1 | (28, 28)
layer1.5.conv1 | (10, 10) | layer2.5.conv1 | (15, 15) | layer3.5.conv1 | (28, 28)
layer1.5.conv2 | (10, 10) | layer2.5.conv2 | (15, 15) | layer3.5.conv1 | (28, 28)
layer1.6.conv1 | (10, 10) | layer2.6.conv1 | (15, 15) | layer3.6.conv1 | (28, 28)
layer1.6.conv2 | (10, 10) | layer2.6.conv2 | (15, 15) | layer3.6.conv1 | (28, 28)
layer1.7.conv1 | (10, 10) | layer2.7.conv1 | (15, 15) | layer3.7.conv1 | (28, 28)
layer1.7.conv2 | (10, 10) | layer2.7.conv2 | (15, 15) | layer3.7.conv1 | (28, 28)
layer1.8.conv1 | (10, 10) | layer2.8.conv1 | (15, 15) | layer3.8.conv1 | (28, 28)
layer1.8.conv2 | (10, 10) | layer2.8.conv2 | (15, 15) | layer3.8.conv1 | (28, 28)
Table 22: Layer-wise rank settings of the compressed ResNet-50 model on ImageNet dataset with 2.49$\times$ FLOPs reduction. “N/A” denotes we do not compress that layer.  Layer Name | Rank | Layer Name | Rank | Layer Name | Rank
---|---|---|---|---|---
FLOPs Reduction = 2.49$\times$
layer1.0.conv1 | N/A | layer2.3.conv1 | (48) | layer3.5.conv1 | (64)
layer1.0.conv2 | (32, 32) | layer2.3.conv2 | (48, 48) | layer3.5.conv2 | (64, 64)
layer1.0.conv3 | (32) | layer2.3.conv3 | (48) | layer3.5.conv3 | (72)
layer1.1.conv1 | N/A | layer3.0.conv1 | (64) | layer4.0.conv1 | (96)
layer1.1.conv2 | (32, 32) | layer3.0.conv2 | (64, 64) | layer4.0.conv2 | (96, 96)
layer1.1.conv3 | (32) | layer3.0.conv3 | (72) | layer4.0.conv3 | (96)
layer1.2.conv1 | N/A | layer3.1.conv1 | (64) | layer4.1.conv1 | (96)
layer1.2.conv2 | (32, 32) | layer3.1.conv2 | (64, 64) | layer4.1.conv2 | (96, 96)
layer1.2.conv3 | (32) | layer3.1.conv3 | (72) | layer4.1.conv3 | (96)
layer2.0.conv1 | (48) | layer3.2.conv1 | (64) | layer4.2.conv1 | (96)
layer2.0.conv2 | (48, 48) | layer3.2.conv2 | (64, 64) | layer4.2.conv2 | (96, 96)
layer2.0.conv3 | (48) | layer3.2.conv3 | (72) | layer4.2.conv3 | (96)
layer2.1.conv1 | (48) | layer3.3.conv1 | (64) |  |
layer2.1.conv2 | (48, 48) | layer3.3.conv2 | (64, 64) |  |
layer2.1.conv3 | (48) | layer3.3.conv3 | (72) |  |
layer2.2.conv1 | (48) | layer3.4.conv1 | (64) |  |
layer2.2.conv2 | (48, 48) | layer3.4.conv2 | (64, 64) |  |
layer2.2.conv3 | (48) | layer3.4.conv3 | (72) |  |