#  Recurrent Parameter Generators

Jiayun Wang 1 Yubei Chen∗ 2 Stella X. Yu1 Brian Cheung4 Yann LeCun2,3
1 UC Berkeley / ICSI 2 Facebook AI Research
3 New York University 4 MIT CSAIL & BCS
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
indicates equal contribution.

###### Abstract

We present a generic method for recurrently using the same parameters for many
different convolution layers to build a deep network. Specifically, for a
network, we create a recurrent parameter generator (RPG), from which the
parameters of each convolution layer are generated. Though using recurrent
models to build a deep convolutional neural network (CNN) is not entirely new,
our method achieves significant performance gain compared to the existing
works. We demonstrate how to build a one-layer neural network to achieve
similar performance compared to other traditional CNN models on various
applications and datasets. Such a method allows us to build an arbitrarily
complex neural network with any amount of parameters. For example, we build a
ResNet34 with model parameters reduced by more than $400$ times, which still
achieves $41.6\%$ ImageNet top-1 accuracy. Furthermore, we demonstrate the RPG
can be applied at different scales, such as layers, blocks, or even sub-
networks. Specifically, we use the RPG to build a ResNet18 network with the
number of weights equivalent to one convolutional layer of a conventional
ResNet and show this model can achieve $67.2\%$ ImageNet top-1 accuracy. The
proposed method can be viewed as an inverse approach to model compression.
Rather than removing the unused parameters from a large model, it aims to
squeeze more information into a small number of parameters. Extensive
experiment results are provided to demonstrate the power of the proposed
recurrent parameter generator.

##  1 Introduction

Deep learning has achieved great success with increasingly more training data
and deeper & larger neural networks: A recently developed NLP model, GPT-3
brown2020language , has astonishingly 175 billion parameters! While the model
performance generally scales with the number of parameters henighan2020scaling
, with parameters outnumbering training data, the model is significantly over-
parameterized. Tremendous effort has been made to reduce the parameter
redundancy from different perspectives, including neural network pruning
lecun1990optimal ; han2015deep ; liu2018rethinking ; frankle2018lottery ,
efficient network design spaces howard2017mobilenets ; iandola2016squeezenet ;
tan2019efficientnet ; sandler2018mobilenetv2 , parameter regularization
wan2013regularization ; wang2020orthogonal ; srivastava2014dropout , model
quantization hubara2017quantized ; rastegari2016xnor ; louizos2018relaxed ,
neural architecture search zoph2016neural ; cai2018proxylessnas ;
wan2020fbnetv2 , recurrent models bai2019deep ; bai2020multiscale ;
wei2016convolutional , multi-task feature encoding ramamonjisoa2019sharpnet ;
hao2020multi , etc.

One of the most prominent approaches in this direction is the pruning-based
model compression, which dates back to the late 80s or early 90s
mozer1989using ; lecun1990optimal  and has enjoyed a resurgence han2015deep ;
blalock2020state  recently. These pruning methods seek to remove the
unimportant parameters from a pre-trained large neural network and can
frequently achieve an enormous model-compression ratio.

Figure 1:  We propose a recurrent parameter generator (RPG) that shares a
fixed set of parameters in a ring and use them to generate parameters of
different parts of a neural network, whereas in the standard neural network,
all the parameters are independent of each other, so the model gets bigger as
it gets deeper. Left: The third section of the model starts to overlap with
the first section in the model ring, and all later layers share generating
parameters for possibly multiple times. Right: Employing the Recurrent
Parameter Generator (RPG) for ResNet could reduce the model parameters to any
size. Specifically, with only half ResNet34 backbone parameters, we achieve
the same ImageNet top-1 accuracy. We also outperforms model pruning methods
such as Knapsack aflalo2020knapsack  and PruneNet khetan2020prunenet .

Though sharing a similar motivation to reduce the parameter redundancy, we
explore an entirely different territory to model compression: rather than
compressing a large model, we define an arbitrarily large model based on a
fixed set of parameters to maximize the model capacity. In this work, we
propose to define many different layers in a deep neural network based on a
fixed amount of parameters, which we call recurrent parameter generator (RPG).
We show that the excess parameter capacity can be exploited with almost no
assumptions of how the parameters are assigned to the neural network
architecture. In other words, there is excess capacity in neural network
models independent of how and where the parameters are used in the network.
Even at the level of individual scalar values, parameters can be reused in
another arbitrary location of the deep network architecture without
significantly impacting model performance. Surprisingly, backpropagation
training of a deep network is able to cope with that the same parameter can be
assigned to multiple random locations in the network without significantly
impacting model performance. Through extensive experiments, we show that a
large neural network does not need to be over overparameterized to achieve
competitive performance. Particularly, a Resnet18 model can be implemented
with the number of weights equivalent to one convolution layer in a
conventional Resnet ($4.72\times$ parameter reduction) and still achieve
$67.2\%$ ImageNet top-1 accuracy. The proposed method is also extremely
flexible in reducing the model parameters. In some sense, the proposed RPG
method can be viewed as an automatic model pruning technique, which explores
the optimal accuracy-parameter trade-off. When we reduce the model parameter,
RPG shows graceful performance degradation, and its compression results are
frequently on par or even better than the SOTA pruning methods besides the
flexibility. Even if we reduce the Resnet18 backbone parameters to $36$K,
which is about $300\times$ reduction, Resnet18 can still achieve $40.0\%$
ImageNet top-1 accuracy.

Notably, we choose a destructive parameter sharing method CheungPSP19  for RPG
in this work, which discourages any potential representation sharing from
layer to layer. Compared to other recurrent weight-sharing methods, e.g.,
convolutional pose machine (CPM) or multi-scale deep equilibrium models
(MDEQ), our method achieves competitive or even better performance on various
benchmarks. This makes RPG a strong baseline for probing whether there is
nontrivial representation sharing within any recurrent network.

To summarize, we make the following contributions:

  1. 1.

We propose the recurrent parameter generator (RPG), which decouples network
architecture and the number of parameters. Given a certain neural network
architecture, we can flexibly choose any number of parameters to construct the
network.

  2. 2.

Given a compression ratio, RPG achieves on-par or even better performance
compared to the state-of-the-art model pruning methods. This may provide an
entirely new perspective for model compression.

  3. 3.

With destructive weight sharing, RPG achieves competitive performance compared
to several recurrent weight-sharing models. This makes RPG a strong baseline
for probing the representation sharing for recurrent models.

Figure 2:  We demonstrate the effectiveness of RPGs on various applications
including image classification (Left), human pose estimation (Middle), and
multitask regression (Right). A network can either have a global RPG or
multiple local RPGs that are shared within blocks or sub-networks.

##  2 Related Work

There are many important efforts to reduce the redundancy in neural network
parameters. We discuss each of the approaches and their relationship to our
work.

Model Pruning, Neural Architecture Search, and Quantization. As we discussed
earlier, model pruning seeks to remove the unimportant parameters in a trained
model. Recently, it’s proposed to use neural architecture search as a coarse-
grained model pruning yu2018slimmable ; dong2019network . Another related
effort is neural network quantization hubara2017quantized ; rastegari2016xnor
; louizos2018relaxed , which seeks to reduce the bits used for each parameter
and can frequently reduce the model size by $4\times$ with minimal accuracy
drop. A more recent work dollar2021fast  presents a framework for analyzing
model scaling strategies that takes into account network properties such as
FLOPs and activations.

Parameter Regularization and Priors Another highly related direction to this
work is parameter regularization. Regularization has been widely used to
reduce model redundancy krogh1992simple , alleviate model overfitting
srivastava2014dropout ; wan2013regularization , and ensure desired
mathematical regularity wang2020orthogonal . The RPG can be viewed as a
parameter regularization in the sense that weight sharing poses many equality
constraints to weights and regularizes weights to a low-dimensional space.
HyperNeat stanley2009hypercube  and CPPNsstanley2007compositional  use neural
networks to determine the weight between two neurons as a function of their
positions. Similarly, karaletsos2018probabilistic ; KaraletsosB20  introduced
a similar idea by providing a hierarchical prior for the neural network
weights.

Recurrent Networks and Deep Equilibrium Models. Recurrence and feedback have
been shown in psychology and neuroscience to act as modulators or competitive
inhibitors to aid feature grouping gilbert2007brain , figure-ground
segregation hupe1998cortical  and object recognition wyatte2012limits .
Recurrence-inspired mechanisms also achieve success in feed-forward models.
There are two main types of employing recurrence based on if weights are
shared across recurrent modules. ResNet he2016deep , a representative of
reusing similar structures without weight sharing, introduces parallel
residual connections and achieves better performance by going deeper in
networks. Similarly, some works szegedy2015going ; srivastava2015highway  also
suggest iteratively injecting thus-far representations to the feed-forward
network useful. Stacked inference methods ramakrishna2014pose ;
wolpert1992stacked ; weiss2010structured  are also related while they consider
each output in isolation. Several works find sharing weights across recurrent
modules beneficial. They demonstrate applications in temporal modelling
weiss2010structured ; shi2015convolutional ; karpathy2015deep , spatial
attention mnih2014recurrent ; butko2009optimal , pose estimation
wei2016convolutional ; carreira2016human , and so on li2016iterative ;
zamir2017feedback . Such methods usually shine in modeling long-term
dependencies. In this work, we recurrently share weights across different
layers of a feedback network to reduce network redundancy.

Given stacking weight-shared modules improve the performance, researchers
consider running even infinite depth of such modules by making the sequential
modules converge to a fixed point lecun1988theoretical ; bai2019deep .
Employing such equilibrium models to existing networks, they show improved
performance in many natural language processing bai2019deep  and computer
vision tasks bai2020multiscale ; wang2020implicit . One issue with deep
equilibrium models is that the forward and backward propagation usually takes
much more iterations than explicit feed-forward networks. Some work
fung2021fixed  improves the efficiency by making the backward propagation
Jacobian free. Another issue is that infinite depth and fixed point may not be
necessary or even too strict for some tasks. Instead of achieving infinite
depth, our model shares parameters to a certain level. We empirically compare
with equilibrium models in Section 5.

Efficient Network Space and Matrix Factorization. Convolution is an efficient
and structured matrix-vector multiplication. Arguably, the most fundamental
idea in building efficient linear systems is matrix factorization. Given the
redundancy in deep convolutional neural network parameters, one can leverage
the matrix factorization concept, e.g., factorized convolutions, and design
more efficient network classes howard2017mobilenets ; iandola2016squeezenet ;
tan2019efficientnet ; sandler2018mobilenetv2 .

##  3 Recurrent Parameter Generator

We define recurrent parameter generators (RPG) and show a certain kind of
generating matrices that leads to destructive weight sharing. In order to
achieve better parameter capacity, we introduce an even sampling strategy.

Recurrent Parameter Generator. Assuming we are constructing a deep
convolutional neural network, which contains $L$ different convolution layers.
Let $\mathbf{K}_{1},\mathbf{K}_{2},\dots,\mathbf{K}_{L}$ be the corresponding
$L$ convolutional kernels 111In this paper, we treat each convolutional kernel
as a vector. When the kernel is used to do the convolution, it will be
reshaped into the corresponding shape.. Rather than using separate sets of
parameters for different convolution layers, we create a single set of
parameters $\mathbf{W}\in\Re^{N}$ and use it to generate the corresponding
parameters for For each convolution layer:

$\mathbf{K}_{i}=\boldsymbol{R}_{i}\cdot\mathbf{W},i\in\\{1,\dots,L\\}$ (1)

where $\boldsymbol{R}_{i}$ is a fixed predefined generating matrix, which is
used to generate $\mathbf{K}_{i}$ from $\mathbf{W}$. We call
$\\{\boldsymbol{R}_{i}\\}$ and $\mathbf{W}$ the recurrent parameter generator
(RPG). In this work, we always assume that the size of $\mathbf{W}$ is smaller
than the total parameters of the model, i.e.,
$|\mathbf{W}|\leq\sum_{i}{|\mathbf{K}_{i}|}$. This means an element of
$\mathbf{W}$ will generally be used in more than one layer of a neural
network. Additionally, the gradient of $\mathbf{W}$ is a linear superposition
of the gradients from each convolution layer. During the neural network
training, let’s assume convolution kernel $\mathbf{K}_{i}$ receives gradient
$\frac{\partial\ell}{\partial\mathbf{K}_{i}}$, where $\ell$ is the loss
function. Based on the chain rule, it is clear that the gradient of
$\mathbf{W}$ is:

$\frac{\partial\ell}{\partial\mathbf{W}}=\sum_{i=1}^{L}{\boldsymbol{R}^{T}_{i}\cdot\frac{\partial\ell}{\partial\mathbf{K}_{i}}}$
(2)

Generating Matrices. There are many different ways to create the generating
matrices $\\{\boldsymbol{R}_{i}\\}$. In this work, we primarily explore the
destructive generating matrices, which tend to prevent different kernels from
sharing the representation during weight sharing.

Destructive Weight Sharing. For easier discussion, let us first look at a
special case, where all of the convolutional kernels have the same size and
are used in the same shape in the corresponding convolution layers. In other
words, $\\{\boldsymbol{R}_{i}\\}$ are square matrices, and the spatial sizes
of all of the convolutional kernels have the same size, $d_{in}\times
d_{out}\times w\times h$, and the input channel dimension $d_{in}$ is always
equal to the output channel dimension $d_{out}$. In this case, a filter
$\mathbf{f}$ in a kernel can be treated as a vector in $\Re^{dwh}$. Further,
we choose $\boldsymbol{R}_{i}$ to be a block-diagonal matrix
$\boldsymbol{R}_{i}=\textit{block-
diag}\\{\boldsymbol{A}_{i},\boldsymbol{A}_{i},\dots,\boldsymbol{A}_{i}\\}$,
where $\boldsymbol{A}_{i}\in O(dwh)$ is an orthogonal matrix that rotates each
filter from the kernel $\mathbf{K}_{i}$ in the same fashion. Similar to the
Proposition 2 in CheungPSP19 , we show in the Appendix B that: if
$\boldsymbol{A}_{i}$, $\boldsymbol{A}_{j}$ are sampled from the $O(M)$ Haar
distribution and $\mathbf{f}_{i}$, $\mathbf{f}_{j}$ are the same filter
(generated by $\boldsymbol{R}_{i}$, $\boldsymbol{R}_{j}$ respectively from
$\mathbf{W}$) from $\mathbf{K}_{i}$, $\mathbf{K}_{j}$ respectively, then we
have ${\rm E}\,\left[\langle\mathbf{f}_{i},\mathbf{f}_{j}\rangle\right]=0$ and
${\rm
E}\,\left[\langle\frac{\mathbf{f}_{i}}{\|\mathbf{f}_{i}\|},\frac{\mathbf{f}_{j}}{\|\mathbf{f}_{j}\|}\rangle^{2}\right]=\frac{1}{M}$.
Since $M$ is usually large, the same filter from $\mathbf{K}_{i}$,
$\mathbf{K}_{j}$ are close to orthogonal and generally dissimilar. This shows
that even $\\{\mathbf{K}_{i}\\}$ are generated from the same $\mathbf{W}$,
they are not sharing the representation.

Even though $\\{\boldsymbol{A}_{i}\\}$ are not updated during the training,
the size of $\boldsymbol{A}_{i}$ can be quite large in general. In practice,
we can use permutation $p\in P(M)$ and element-wise random sign reflection
$b\in B(M)$ to construct a subset of the orthogonal group $O(M)$, i.e. we
choose $\boldsymbol{A}_{i}\in\\{b\circ p|\ b\in B(M),p\in
P(M)\\}$.222Permutations and element-wise random sign reflection conceptually
are subgroups from the orthogonal group, but we shall never use them in the
matrix form for the obvious efficiency purpose. Since pseudo-random numbers
are used, it takes only two random seeds to store a random permutation and an
element-wise random sign reflection.

In this we work, we generalize the usage of $\boldsymbol{R}_{i}$ beyond the
block-diagonal generating matrices described above. $\\{\mathbf{K}_{i}\\}$ may
have different sizes, which can be chosen even larger than the size of
$\mathbf{W}$. When $\mathbf{K}_{i}\in\Re^{N_{i}}$ is larger than
$\mathbf{W}\in\Re^{N}$, the corresponding generating matrix
$\boldsymbol{R}_{i}$ is a tall matrix. There are many ways to efficiently
create the generating matrices. We use random permutations $P(N_{i})$ and
element-wise random sign reflections $B(N_{i})$ to create
$\boldsymbol{R}_{i}$:

$R_{i}\in\\{b\circ p|\ b\in B(N_{i}),p\in P(N_{i})\\},\ i=1,\dots,L$ (3)

$\\{\boldsymbol{R}_{i}\\}$ tend to lead to destructive weight sharing and lead
to better utilization of the parameter capacity.

Even Parameter Distribution for Different Layers. While it is easy to randomly
sample elements from the $\mathbf{W}$ when generating parameters for each
layer, it may not be optimal as some elements in $\mathbf{W}$ may never be
used, and some elements may be used more than average. We use an equalization
technique to guarantee all elements of $\mathbf{W}$ are evenly sampled.
Suppose the size of the $\mathbf{W}$ is $N$, and the total size of parameters
of layers to be generated is $M$, $M>N$. We first generate
$\left\lfloor\frac{M}{N}\right\rfloor$ arrays $\\{x|x=1,..,N\\}$ and
concatenate them with $(M\mod N)$ elements randomly sampled from array
$\\{x|x=1,..,M\\}$. We call the concatenated array index array $u\in\Re^{M}$.
We randomly shuffle all elements in $u$. When initializing each layer’s
parameter, we sequentially get indices of chosen elements from the shuffled
index array $u$. In this way, each layer’s parameters are randomly and evenly
sampled from $\mathbf{W}$. We refer to $\mathbf{W}$ as model rings since
elements are recurrently used in a loop. For data saving efficiency, we just
need to save two random seed numbers (one for sampling $(M\mod N)$ elements
and one for shuffling) instead of saving the large index array $u$.

Batch Normalization. We find the model performance is relatively sensitive to
the batch normalization parameters. To achieve better performance, each of the
convolution layers needs to have its own batch normalization parameters. But
in general, batch normalization is relatively negligible. We show later that
when $\mathbf{W}$ is extremely small (e.g., $36$K parameters), the size of
batch normalization should be considered.

Figure 3:  Recurrent parameter generators at multiple scales. Upper: A global
RPG is used for generating convolution kernels for the entire ResNet18. Lower
: Four local RPGs are each responsible for generating convolution kernels
within each corresponding building block of the ResNet18.

##  4 Recurrent Parameter Generator at Multiple Scales

In the previous section, we discuss the general idea of superposition where
only one RPG is created and shared globally across all layers. We could also
create several local RPGs, and each of them is shared at certain scales, such
as blocks and sub-networks. Such super-positions may be useful for certain
applications such as recurrent modeling.

RPGs at Block-Level. Researchers propose network architectures that reuse the
same design of network blocks multiple times for higher learning capacity, as
discussed in the related work. Instead of using one global RPG for the entire
network, we could alternatively create several RPGs that are shared within
certain network blocks. We take ResNet18 he2016deep  as a concrete example
(Fig.3). ResNet18 has four building blocks. Every block has 2 residual
convolution modules. To superpose ResNet18 at block scale, we create four
local RPGs. Each RPG is shared within the corresponding building block, where
the size of the RPG is flexible and can be determined by users.

RPGs at Sub-Network-Level. Reusing sub-networks, or recurrent networks have
achieved success in many tasks as they iteratively refine and improves the
prediction. Usually, weights are shared when reusing the sub-networks. This
may not be optimal as sub-networks at different stages iteratively improve the
prediction, and shared weights may limit the learning capacity to adapt for
different stages. On the other hand, not sharing weights at all greatly
increases the model size. We superpose different sub-networks with one or more
RPGs. Superposition sub-networks could have a much smaller model size, while
parameters for different sub-networks undergo destructive changes instead of
directly copy-paste. We show applications of superpose sub-networks for pose
estimation and multitask regression (Section 5.3 and 5.4).

##  5 Experimental Results

We conduct 3 sets of experiments to evaluate the proposed superposition method
(with tasks illustrated in Fig.2). The first set benchmarks the performance of
RPG networks on image classification datasets including CIFAR100 and ImageNet.
We compare with deep equilibrium models, a strong baseline on exploiting model
capacity with limited model size. The second set benchmarks the performance of
RPG at block and sub-network level on human pose estimation and multi-task
regression datasets. We compare with recurrent models as baselines. The last
set provide additional analysis and ablation studies of RPG networks
Experiments are conducted on NVIDIA GeForce GTX 2080Ti GPUs.

###  5.1 CIFAR Classification

We benchmark the performance of the proposed RPG network on CIFAR10 and
CIFAR100 datasets.

Implementation Details. All CIFAR experiments use batchsize of 128, weight
decay of 5e-4, and initial learning rate of 0.1 with gamma of 0.1 at epoch 60,
120 and 160.

Table 1:  Our method compared with multiscale deep equilibrium models
bai2020multiscale  on CIFAR10 and CIFAR100 classification. At the same number
of model parameters, we achieve 3% - 6% improvement with 15 - 25x less
inference time. Inference time is measured by milliseconds per image.

|  | Our RPG
---|---|---
| MDEQ bai2020multiscale  | 2x MS blk | 3x MS blk | 4x MS blk
CIFAR10 acc. (%) | 85.1 | 88.5 | 90.1 | 90.9
CIFAR100 acc. (%) | 59.8 | 62.8 | 64.7 | 65.7
Inference time | 3.15 | 0.12 | 0.18 | 0.22

Table 2:  At the same number of backbone parameters on CIFAR100, using local
RPGs at block-level improves accuracy. Using a global RPG further improves the
accuracy.

| #Param | Acc. (%)
---|---|---
Res18 | 11M | 77.5
Res34-RPG.blk | 11M | 78.5
Res34-RPG | 11M | 78.9
Res34 | 21M | 79.1

Compared to Deep Equilibrium Models. As a representative of implicit models,
deep equilibrium models bai2019deep  can reduce model redundancy by finding
fix points via additional optimizations. We compare the image classification
accuracy on CIFAR10 and CIFAR100 datasets, as well as the inference time on
CIFAR100 (Table 2). Following the settings of MDEQ bai2020multiscale , an
image was sequentially fed into the initial convolutional block, the multi-
scale deep equilibrium block (dubbed as MS block), and the classification
head. MDEQ bai2020multiscale  achieves infinite MS blocks by finding the fixed
point of the MS block. We reuse the MS block two to four times without
increasing the number of parameters. Our superposition method achieves 3.4% -
5.8% gain on CIFAR10 and 3% - 5.9% gain on CIFAR100. Our inference time is 15
- 25 times smaller than MDEQ since MDEQ needs additional time to solve
equilibrium during training.

Figure 4:  CIFAR100 accuracy versus the number of backbone parameters for
plain ResNet and ours. We only have a 0.2% drop with 50% Res34 backbone
parameters.

Figure 5:  Ablation studies of permutation and reflection matrices. Having
both matrices gives the highest performance.

Global RPG with Varying # Parameters. We create one global RPG to generate
parameters for convolution layers of ResNet he2016deep  and refer to it as
ResNet-RPG. We report CIFAR100 top-1 accuracy of ResNet-RPG18 and ResNet-RPG34
at different number of parameters (Fig.5 and Table 3). Compared to ResNet,
ResNet-RPG achieves higher accuracy at the same parameter size. Specifically,
we achieve 36% CIFAR100 accuracy with only 8K backbone parameters.
Furthermore, ResNet34-RPG achieves higher accuracy than ResNet18-RPG,
indicating increasing time complexity gives performance gain.

Local RPGs at the Block-Level. In the previous ResNet-RPG experiments, we use
one global RPG (Fig.3-Upper). We also evaluate the performance when RPGs are
shared locally at a block level, as shown in Fig.3-Lower. In Table 2, compared
to plain ResNet18 at the same number of parameters, our block superposition
network gives 1.0% gain. In contrast, our ResNet-RPG (parameters are evenly
distributed) gives a 1.4% gain. Using one global RPG where parameters of each
layer are evenly distributed is 0.4% higher than multiple RPGs.

###  5.2 ImageNet Classification

Table 3:  ImageNet and CIFAR100 top-1 classification accuracy versus the
number of back-bone parameters for our ResNet-RPG and plain ResNet. Our
ResNet-RPG consistently achieves higher performance at the same number of
parameters.

| ResNet18 | ResNet34 | Our Res18-RPG | Our Res34-RPG
---|---|---|---|---
# Parameters | 11M | 21M | 45K | 2M | 45K | 2M | 11M
ImageNet Acc. (%) | 69.8 | 73.4 | 40.0 | 67.2 | 41.6 | 69.1 | 73.4
CIFAR100 Acc. (%) | 77.6 | 79.1 | 60.2 | 75.6 | 61.7 | 76.5 | 78.9

Implementation Details. All ImageNet experiments use batch size of 256, weight
decay of 3e-5, and an initial learning rate of 0.3 with gamma of 0.1 every 75
epochs.

Superposition with Varying # Parameters. We use one RPG with different number
of parameters for ResNet and report the top-1 accuracy (Table 3 and
Fig.1(Right)). ResNet-RPGs consistently achieve higher performance compared to
ResNets under the same number of parameters. Specifically, ResNet-RPG34
achieves the same accuracy 73.4% as ResNet34 with only half of ResNet34
backbone parameters. ResNet-RPG18 also achieves the same accuracy as ResNet18
with only half of ResNet18 backbone parameters. Further, we find RPG networks
have higher generalizability (Section 5.5).

Power Law. Empirically, accuracy and number of parameters follow a power law,
when RPG model size is lower than 50% original plain ResNet model size. The
exponents of the power laws are the same for ResNet18-RPG and ResNet34-RPG on
ImageNet, when comparing with ResNet34 accuracies. The scaling law may be
useful for estimating the network performance without training the network.
Similarly, henighan2020scaling  also identifies a power law for performance
and model size of transformers. The proposed RPG enables under-parameterized
models for large-scale datasets such as ImageNet, which may unleash more new
studies and findings.

###  5.3 Pose Estimation

Table 4:  Pose estimation performance (number of parameters) on MPII human
pose dataset andriluka20142d . The metric is<EMAIL_ADDRESS>
| CPM wei2016convolutional  | Ours | No shared w.
---|---|---|---
1x sub-net | 84.7 (3.3M)
2x sub-nets | 86.1 (3.3M) | 86.5 (3.3M) | 87.1 (6.7M)
4x sub-nets | 86.5 (3.3M) | 87.3 (3.3M) | 88.0 (13.3M)

Table 5:  Multitask regression errors on S3DIS armeni2017joint . Lower is
better. Number of parameters for different methods are the same.

| Depth RMSE | Normal RMSE
---|---|---
No reusing the sub-net ramamonjisoa2019sharpnet  | 25.5% | 41.0%
Reuse sub-net | 24.7% | 40.3%
Reuse & new BN | 24.0% | 39.4%
Reuse & new BN & perm. and reflect. | 22.8% | 39.1%

Implementation Details. We superpose sub-networks for pose estimation with a
globally shared RPG (Fig.7). We use hourglass networks newell2016stacked  as
the building backbone. The input image is first fed to an initial convolution
block to obtain a feature map. The feature map is then fed to multiple stacked
pose estimation sub-networks. Each sub-network outputs a pose estimation
prediction, which is penalized by the pose estimation loss. Convolutional pose
machine (CPM) wei2016convolutional  share all the weights for different sub-
networks. We create one global RPG and generate parameters for each sub-
network. Our model size is set to be the same as CPM. We also compare with
larger models where parameters of sub-networks are not shared.

We evaluate on MPII Human Pose dataset andriluka20142d , a benchmark for
evaluation of articulated human pose estimation, which consists of more than
28K training samples over 40K people with annotated body joints. We use the
hourglass network as backbone newell2016stacked  and follow all their
settings.

Results and Analysis. We report the Percentage of Correct Key-points at 50%
threshold<EMAIL_ADDRESS>of different methods in Table 5. CPM wei2016convolutional
share all parameters for different sub-networks. We use one RPG that is shared
globally at the same size as CPM. For reference, we also compare with the no-
sharing model as the performance ceiling. Adding the number of recurrences
leads to performance gain for all methods. At the same model size, our
superposition models achieve higher<EMAIL_ADDRESS>compared to CPM. Increasing the
number of parameters by not sharing sub-network parameters also leads to some
performance gain.

Figure 6:  Pose estimation with sub-network superposition. An input image is
fed to an initial convolutional block and $N$ stacked pose estimation
networks. Each sub-network predicts the pose that are penalized by
corresponding loss. Parameters of each sub-network are generated from a global
RPG.

Figure 7:  Normalized histogram of feature similarity for Res34-RPG with
different model parameters. Feature similarity increases as the model size
decreases, indicating our RPG networks can be further pruned with methods like
wang2019cop .

###  5.4 Multi-Task Regression

Implementation Details. We superpose sub-networks for multi-task regression
with multiple RPGs at the building-block level. We focus on predicting depth
and normal maps from a given image. We stack multiple SharpNet
ramamonjisoa2019sharpnet , a network for monocular depth and normal
estimation. Specifically, we create multiple RPGs at the SharpNet building-
block level. In other words, parameters of corresponding blocks of different
sub-networks are generated from the same RPG.

We evaluate the monocular depth and normal prediction performance on Standford
3D indoor scene dataset armeni2017joint . It contains over 70K images with
corresponding depths and normals covering over 6,000 $\text{m}^{2}$ indoor
area. We follow all settings of SharpNet ramamonjisoa2019sharpnet , a state-
of-the-art monocular depth and normal estimation network.

Results and Analysis. We report the mean square errors for depth and normal
estimation in Table 5. Compared to one-time inference without recurrence, our
RPG network gives 3% and 2% gain for depth and normal estimation,
respectively. Directly sharing weights but using new batch normalization
layers decrease the performance by 1.2% and 0.3% for depth and normal. Sharing
weights and normalization layers further decrease the performance by 0.7% and
0.9% for depth and normal.

###  5.5 Analysis

Table 6: The proposed RPG increases the model generalizability. (a) ResNet
with RPG has the lower gap between training and validation set on ImageNet
classification. The metric is training accuracy minus validation accuracy.
Lower is better. (b) Using RPG for pose estimation also decreases the training
and validation performance GAP. The metric is training<EMAIL_ADDRESS>minus
validation<EMAIL_ADDRESS>Lower is better. (c) ResNet with RPG has higher
performance on out-of-distribution dataset ObjectNet barbu2019objectnet . The
model is trained on ImageNet only and directly evaluated on ObjectNet.

(a)

| ResNet | ours
---|---|---
18 conv | -0.7 | -2.7
34 conv | 1.1 | -2.3

(b)

| no shared w | shared w wei2016convolutional  | ours
---|---|---|---
2x sub-nets | 1.15 | 1.13 | 0.64
4x sub-nets | 1.98 | 1.70 | 1.15

(c)

| R18 | R34-RPG | R34
---|---|---|---
# Params | 11M | 11M | 21M
Acc. (%) | 13.4 | 16.5 | 16.0

Comparison to Pruning Methods. We report our ResNet18-RPG performance with
different number of parameters on ImageNet and some baseline pruning methods
in Fig.1(Right). Our RPG networks outperform SOTA pruning methods such as
aflalo2020knapsack ; dong2019network ; he2019filter ; he2018soft ;
dong2017more ; khetan2020prunenet . Specifically, at the same number of
parameters, our RPG network has 0.6% gain over the knapsack pruning
aflalo2020knapsack , a method that achieves the best ImageNet pruning
accuracy.

Generalizability. To understand generalizability of proposed RPG networks, we
report the performance gap between training and validation set on ImageNet
(Table 6(a)) and MPII pose estimation (Table 6(b)). Superposition models
consistently achieve lower gaps between training and validation set,
indicating the superposition model suffers less from over-fitting.

We also report the out-of-distribution performance of superposition models.
ObjectNet barbu2019objectnet  contain 50k images with 113 classes overlapping
with ImageNet. Previous models are reported to have a large performance drop
on ObjectNet. We directly evaluate the performance of ImageNet-trained model
on ObjectNet without any fine-tuning (Table 6(c)). With the same number of
backbone parameters, our ResNet-RPG achieves a 3.1% gain compared to ResNet18.
With the same network architecture design, our ResNet-RPG achieves 0.5% gain
compared to ResNet34. This indicates our RPG networks have higher out-of-
distribution performance even with smaller model sizes.

Feature Similarity. RPG networks are small in parameter size, are they still
possible to prune? Some work wang2019cop  achieves satisfactory pruning
results by calculating the correlation matrix of a feature map from the
network, and prune channels with high correlations. Following their settings,
we report feature similarity of ResNet34-RPG after residual building block 3
(Fig.7). From our small ResNet34-RPG with 45k backbone parameters to plain
ResNet34 with 21M backbone parameters, the feature similarity gradually
increases. This indicates our RPG networks can still be pruned, and the FLOPs
and run time can be further reduced. We provide empirical pruning results in
Appendix A.

Security. Permutation matrices generated by the random seed can be considered
as security keys to decode the model. In addition, only random seeds to
generate transformation matrices need to be saved and transferred, which is
efficient in terms of size.

###  5.6 Ablation Studies

We conduct ablation studies to understand the functions of permutation and
reflection matrices (Fig.5). We evaluate ResNet-RPG34 with 2M backbone
parameters. With permutation and reflection matrices leads to 76.5% accuracy,
with permutation matrices only leads to 75.8%, with reflection matrices only
leads to 71.1%, and with no permutation and reflection matrices leads to
70.7%. This suggests permutation and reflection matrices are useful for RPGs.

##  6 Discussion

In this work, we limit our scope to linear destructive weight sharing for
different convolutional layers. However, in general, there might also exist
nonlinear RPGs and efficient nonlinear generation functions to create
convolutional kernels from a shared parameter $\mathbf{W}$. Additionally,
although we focus on reducing model parameters, results in Fig.7 indicate RPG
networks are compressible, and it is possible to further reduce the FLOPs and
run time.

To sum up, we develop an efficient approach to build an arbitrarily complex
neural network with any amount of parameters via a recurrent parameter
generator. On a wide range of applications, including image classification,
pose estimation and multitask regression, we show RPG networks consistently
achieve higher performance at the same model size. Further, analysis shows
that such networks are less possible to overfit and have higher performance on
out-of-distribution data.

RPG can be added to any existing network flexibly with any amount of total
parameters at the user’s discretion. It may provide new perspectives for
recurrent models, equilibrium models, and model compression.

## References

  * [1] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [2] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al.  Scaling laws for autoregressive generative modeling.  arXiv preprint arXiv:2010.14701, 2020.
  * [3] Yann LeCun, John S Denker, and Sara A Solla.  Optimal brain damage.  In Advances in neural information processing systems, pages 598–605, 1990.
  * [4] Song Han, Huizi Mao, and William J Dally.  Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding.  In Proceedings of the International Conference on Learning Representations, 2016.
  * [5] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell.  Rethinking the value of network pruning.  In International Conference on Learning Representations, 2018.
  * [6] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  In International Conference on Learning Representations, 2018.
  * [7] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  arXiv preprint arXiv:1704.04861, 2017.
  * [8] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [9] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International Conference on Machine Learning, pages 6105–6114, 2019.
  * [10] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510–4520, 2018.
  * [11] Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus.  Regularization of neural networks using dropconnect.  In International conference on machine learning, pages 1058–1066. PMLR, 2013.
  * [12] Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, and Stella X Yu.  Orthogonal convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11505–11515, 2020.
  * [13] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [14] Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio.  Quantized neural networks: Training neural networks with low precision weights and activations.  The Journal of Machine Learning Research, 18(1):6869–6898, 2017\.
  * [15] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi.  Xnor-net: Imagenet classification using binary convolutional neural networks.  In European conference on computer vision, pages 525–542. Springer, 2016.
  * [16] C Louizos, M Reisser, T Blankevoort, E Gavves, and M Welling.  Relaxed quantization for discretized neural networks.  In International Conference on Learning Representations. International Conference on Learning Representations, ICLR, 2019.
  * [17] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  arXiv preprint arXiv:1611.01578, 2016.
  * [18] Han Cai, Ligeng Zhu, and Song Han.  Proxylessnas: Direct neural architecture search on target task and hardware.  In International Conference on Learning Representations, 2018.
  * [19] Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al.  Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12965–12974, 2020.
  * [20] Shaojie Bai, J Zico Kolter, and Vladlen Koltun.  Deep equilibrium models.  Advances in Neural Information Processing Systems, 32:690–701, 2019\.
  * [21] Shaojie Bai, Vladlen Koltun, and J Zico Kolter.  Multiscale deep equilibrium models.  Advances in Neural Information Processing Systems, 33, 2020.
  * [22] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional pose machines.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4724–4732, 2016.
  * [23] Michael Ramamonjisoa and Vincent Lepetit.  Sharpnet: Fast and accurate recovery of occluding contours in monocular depth estimation.  The IEEE International Conference on Computer Vision (ICCV) Workshops, 2019.
  * [24] Yongchang Hao, Shilin He, Wenxiang Jiao, Zhaopeng Tu, Michael Lyu, and Xing Wang.  Multi-task learning with shared encoder for non-autoregressive machine translation.  arXiv preprint arXiv:2010.12868, 2020.
  * [25] Michael C Mozer and Paul Smolensky.  Using relevance to reduce network size automatically.  Connection Science, 1(1):3–16, 1989.
  * [26] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag.  What is the state of neural network pruning?  arXiv preprint arXiv:2003.03033, 2020.
  * [27] Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik.  Knapsack pruning with inner distillation.  arXiv preprint arXiv:2002.08258, 2020.
  * [28] Ashish Khetan and Zohar Karnin.  Prunenet: Channel pruning via global importance.  arXiv preprint arXiv:2005.11282, 2020.
  * [29] Brian Cheung, Alex Terekhov, Yubei Chen, Pulkit Agrawal, and Bruno Olshausen.  Superposition of many models into one.  In Advances in neural information processing systems, 2019.
  * [30] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang.  Slimmable neural networks.  arXiv preprint arXiv:1812.08928, 2018.
  * [31] Xuanyi Dong and Yi Yang.  Network pruning via transformable architecture search.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [32] Piotr Dollár, Mannat Singh, and Ross Girshick.  Fast and accurate model scaling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 924–932, 2021.
  * [33] Anders Krogh and John A Hertz.  A simple weight decay can improve generalization.  In Advances in neural information processing systems, pages 950–957, 1992.
  * [34] Kenneth O Stanley, David B D’Ambrosio, and Jason Gauci.  A hypercube-based encoding for evolving large-scale neural networks.  Artificial life, 15(2):185–212, 2009.
  * [35] Kenneth O Stanley.  Compositional pattern producing networks: A novel abstraction of development.  Genetic programming and evolvable machines, 8(2):131–162, 2007\.
  * [36] Theofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani.  Probabilistic meta-representations of neural networks.  arXiv preprint arXiv:1810.00555, 2018.
  * [37] Theofanis Karaletsos and Thang D. Bui.  Hierarchical gaussian process priors for bayesian neural network weights.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems (NeurIPS), 2020.
  * [38] Charles D Gilbert and Mariano Sigman.  Brain states: top-down influences in sensory processing.  Neuron, 54(5):677–696, 2007.
  * [39] JM Hupé, AC James, BR Payne, SG Lomber, P Girard, and J Bullier.  Cortical feedback improves discrimination between figure and background by v1, v2 and v3 neurons.  Nature, 394(6695):784–787, 1998.
  * [40] Dean Wyatte, Tim Curran, and Randall O’Reilly.  The limits of feedforward vision: Recurrent processing promotes robust object recognition when objects are degraded.  Journal of Cognitive Neuroscience, 24(11):2248–2261, 2012.
  * [41] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [42] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [43] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber.  Highway networks.  arXiv preprint arXiv:1505.00387, 2015.
  * [44] Varun Ramakrishna, Daniel Munoz, Martial Hebert, James Andrew Bagnell, and Yaser Sheikh.  Pose machines: Articulated pose estimation via inference machines.  In European Conference on Computer Vision, pages 33–47. Springer, 2014.
  * [45] David H Wolpert.  Stacked generalization.  Neural networks, 5(2):241–259, 1992.
  * [46] David Weiss and Benjamin Taskar.  Structured prediction cascades.  In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 916–923. JMLR Workshop and Conference Proceedings, 2010.
  * [47] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  arXiv preprint arXiv:1506.04214, 2015.
  * [48] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128–3137, 2015.
  * [49] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu.  Recurrent models of visual attention.  In Advances in Neural Information Processing Systems, 2014.
  * [50] Nicholas J Butko and Javier R Movellan.  Optimal scanning for faster object detection.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2751–2758. IEEE, 2009.
  * [51] Joao Carreira, Pulkit Agrawal, Katerina Fragkiadaki, and Jitendra Malik.  Human pose estimation with iterative error feedback.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4733–4742, 2016.
  * [52] Ke Li, Bharath Hariharan, and Jitendra Malik.  Iterative instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3659–3667, 2016.
  * [53] Amir R Zamir, Te-Lin Wu, Lin Sun, William B Shen, Bertram E Shi, Jitendra Malik, and Silvio Savarese.  Feedback networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1308–1317, 2017.
  * [54] Yann LeCun, D Touresky, G Hinton, and T Sejnowski.  A theoretical framework for back-propagation.  In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21–28, 1988.
  * [55] Tiancai Wang, Xiangyu Zhang, and Jian Sun.  Implicit feature pyramid network for object detection.  arXiv preprint arXiv:2012.13563, 2020.
  * [56] Samy Wu Fung, Howard Heaton, Qiuwei Li, Daniel McKenzie, Stanley Osher, and Wotao Yin.  Fixed point networks: Implicit depth models with jacobian-free backprop.  arXiv preprint arXiv:2103.12803, 2021.
  * [57] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2d human pose estimation: New benchmark and state of the art analysis.  In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pages 3686–3693, 2014.
  * [58] Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese.  Joint 2d-3d-semantic data for indoor scene understanding.  arXiv preprint arXiv:1702.01105, 2017.
  * [59] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European conference on computer vision, pages 483–499. Springer, 2016.
  * [60] Wenxiao Wang, Cong Fu, Jishun Guo, Deng Cai, and Xiaofei He.  Cop: Customized deep model compression via regularized correlation-based filter-level pruning.  In International Joint Conference on Artificial Intelligence, 2019\.
  * [61] Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz.  Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models.  Advances in neural information processing systems, 32:9453–9463, 2019.
  * [62] Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang.  Filter pruning via geometric median for deep convolutional neural networks acceleration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4340–4349, 2019.
  * [63] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang.  Soft filter pruning for accelerating deep convolutional neural networks.  In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 2234–2240, 2018.
  * [64] Xuanyi Dong, Junshi Huang, Yi Yang, and Shuicheng Yan.  More is less: A more complicated network with less inference complexity.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5840–5848, 2017.

## Appendices

##  Appendix A Pruning RPG Networks for Lower Inference Time

Can networks with recurrent parameter generators be pruned, so the inference
time can be further reduced? As discussed in Section 5.5 of the main paper,
there is room for RPG networks to be further pruned. We prune 30% backbone
parameters by dropping weights with lowest $\ell_{1}$ norm for ResNet18-RPG
and ResNet34-RPG. Number of backbone parameters are 11M and 5.6M, respectively
(Table 7). With 50% number of parameters and 70% FLOPS, the ImageNet top-1
accuracy only drops 1.2% and 0.9% for ResNet18 and ResNet34, respectively.
This indicates RPG networks can be pruned to further reduce the inference
time.

Table 7:  Pruning ResNet-RPG on ImageNet. Pruned RPG networks have 50% number
of parameters and 70% FLOPs as the plain ResNet. ResNet18-RPG and ResNet34-RPG
both only have 1% performance drop.

| Acc. before (%) | # Param ↓ | # FLOPs ↓ | Acc. After (%)
---|---|---|---|---
Res18-RPG | 70.2 | 50% | 30% |  69.0 ($-$1.2 )
Res34-RPG | 73.4 | 50% | 30% |  72.5 ($-$0.9 )

##  Appendix B Proof to the Orthogonal Proposition

We provide proofs to the orthogonal proposition mentioned in Section 3 of the
main paper. Suppose we have two vectors
$\mathbf{f}_{i}=\boldsymbol{A}_{i}\mathbf{f},\mathbf{f}_{j}=\boldsymbol{A}_{i}\mathbf{f}$,
where $\boldsymbol{A}_{i}$, $\boldsymbol{A}_{j}$ are sampled from the $O(M)$
Haar distribution.

######  Proposition 1.

${\rm E}\,\left[\langle\mathbf{f}_{i},\mathbf{f}_{j}\rangle\right]=0$.

###### Proof.

$\displaystyle{\rm
E}\,\left[\langle\mathbf{f}_{i},\mathbf{f}_{j}\rangle\right]$
$\displaystyle={\rm
E}\,\left[\langle\mathbf{f}_{i},\mathbf{f}_{j}\rangle\right]$
$\displaystyle={\rm
E}\,\left[\langle\boldsymbol{A}_{i}\mathbf{f},\boldsymbol{A}_{j}\mathbf{f}\rangle\right]$
$\displaystyle={\rm
E}\,\left[\langle\mathbf{f},\boldsymbol{A}_{i}^{T}\boldsymbol{A}_{j}\mathbf{f}\rangle\right]$
$\displaystyle=\mathbf{f}^{T}{\rm
E}\,\left[\boldsymbol{A}_{i}^{T}\boldsymbol{A}_{j}\right]\mathbf{f}$
$\displaystyle=0$

where $\boldsymbol{A}_{i}^{T}\boldsymbol{A}_{j}$ is equivalently a random
sample from $O(M)$ Haar distribution and its expectation is clearly 0. ∎

######  Proposition 2.

${\rm
E}\,\left[\langle\frac{\mathbf{f}_{i}}{\|\mathbf{f}_{i}\|},\frac{\mathbf{f}_{j}}{\|\mathbf{f}_{j}\|}\rangle^{2}\right]=\frac{1}{M}$.

###### Proof.

$\displaystyle{\rm
E}\,\left[\langle\frac{\mathbf{f}_{i}}{\|\mathbf{f}_{i}\|},\frac{\mathbf{f}_{j}}{\|\mathbf{f}_{j}\|}\rangle^{2}\right]$
$\displaystyle=\frac{{\rm
E}\,\left[\langle\boldsymbol{A}_{i}\mathbf{f},\boldsymbol{A}_{j}\mathbf{f}\rangle^{2}\right]}{\|\mathbf{f}\|_{2}^{2}\|\mathbf{f}\|_{2}^{2}}$
$\displaystyle={\rm
E}\,\left[\langle\boldsymbol{A}\frac{\mathbf{f}}{\|\mathbf{f}\|},\frac{\mathbf{f}}{\|\mathbf{f}\|}\rangle^{2}\right],\text{where
}\boldsymbol{A}=\boldsymbol{A}_{i}^{T}\boldsymbol{A}_{j}\sim O(M)\text{ Haar
distribution}$ Due to the symmetr y, $\displaystyle={\rm
E}\,\left[\langle\boldsymbol{A}\frac{\mathbf{f}}{\|\mathbf{f}\|},(1,0,0,\dots,0)^{T}\rangle^{2}\right]$
$\displaystyle\text{Let
}\mathbf{g}=\boldsymbol{A}\frac{\mathbf{f}}{\|\mathbf{f}\|},\ \ \ \ \ \ $
$\displaystyle={\rm E}\,\left[g_{1}^{2}\right]$ $\displaystyle=\frac{1}{M}$

since $\mathbf{g}$ is a random unit vector and ${\rm
E}\,\left[\sum_{k=1}^{M}{g_{k}^{2}}\right]=\sum_{k=1}^{M}{{\rm
E}\,\left[g_{k}^{2}\right]}=1$. ∎


* [Tana 19] G. Tanaka, T. Yamane, J. B. Héroux, R. Nakane, N. Kanazawa, S. Takeda, H. Numata, D. Nakano, and A. Hirose.  “Recent advances in physical reservoir computing: A review”.  Neural Networks, Vol. 115, pp. 100–123, 2019.
  * [Verz 20] P. Verzelli, C. Alippi, L. Livi, and P. Tino.  “Input representation in recurrent neural networks dynamics”.  mar 2020.
  * [Verz 21] P. Verzelli, C. Alippi, and L. Livi.  “Learn to synchronize, synchronize to learn”.  Chaos: An Interdisciplinary Journal of Nonlinear Science, Vol. 31, p. 083119, 2021.
  * [Vian 20] M. Viana.  “(Dis) continuity of Lyapunov exponents”.  Ergodic Theory and Dynamical Systems, Vol. 40, No. 3, pp. 577–611, 2020.
  * [Walt 00] P. Walters.  An Introduction to Ergodic Theory.  Graduate Texts in Mathematics, Springer New York, 2000.
  * [Wyff 10] F. Wyffels and B. Schrauwen.  “A comparative study of Reservoir Computing strategies for monthly time series prediction”.  Neurocomputing, Vol. 73, No. 10, pp. 1958–1964, 2010.