# When Does Re-initialization Work?

Sheheryar Zaidi1∗ Tudor Berariu2∗ Hyunjik Kim3
Jörg Bornschein3 Claudia Clopath2,3 Yee Whye Teh1,3 Razvan Pascanu3
1University of Oxford, 2Imperial College London, 3DeepMind
∗Equal contribution.
Correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Re-initializing a neural network during training has been observed to improve
generalization in recent works. Yet it is neither widely adopted in deep
learning practice nor is it often used in state-of-the-art training protocols.
This raises the question of when re-initialization works, and whether it
should be used together with regularization techniques such as data
augmentation, weight decay and learning rate schedules. In this work, we
conduct an extensive empirical comparison of standard training with a
selection of re-initialization methods to answer this question, training over
15,000 models on a variety of image classification benchmarks. We first
establish that such methods are consistently beneficial for generalization in
the absence of any other regularization. However, when deployed alongside
other carefully tuned regularization techniques, re-initialization methods
offer little to no added benefit for generalization, although optimal
generalization performance becomes less sensitive to the choice of learning
rate and weight decay hyperparameters. To investigate the impact of re-
initialization methods on noisy data, we also consider learning under label
noise. Surprisingly, in this case, re-initialization significantly improves
upon standard training, even in the presence of other carefully tuned
regularization techniques.

##  1 Introduction

Figure 1: Comparison of standard training (i.e. no re-initialization) with re-
initialization using Shrink & Perturb in three scenarios on Tiny ImageNet with
PreAct-ResNet-18. Full regularization refers to the usage of carefully tuned
data augmentation, weight decay and a learning rate schedule, whereas no
regularization is without. Without label noise, Shrink & Perturb helps in the
absence of other regularization but has limited benefit otherwise. With label
noise, Shrink & Perturb helps significantly even alongside other
regularization. Adding distillation further improves performance. All training
protocols have approximately equal computational cost.

Recent works [e.g. 1, 2] have proposed a set of techniques for training neural
networks based on re-initialization. These methods, which we collectively
refer to as re-initialization methods, involve re-initializing and
transforming a part or all of the parameters of a neural network periodically
throughout learning. Studies have shown that re-initialization can help in
certain settings, such as small-data regimes [1, 29] and online learning [2].
However, despite having no overhead in terms of computation cost and small
implementation overhead, such re-initialization techniques have not yet been
adopted as common deep learning practice. Indeed, most state-of-the-art (SOTA)
training protocols do not incorporate re-initialization techniques, relying
instead on advances in e.g. optimization [10], architectures [24, 30, 5], data
augmentation [8, 9] and pre-training [7].

However, prior work suggests that re-initialization can improve the
generalization performance of neural networks. For example, in the context of
online learning, Ash & Adams [2] studied a scenario in which training data
arrives sequentially in “chunks” over time such that at any point in time the
training dataset consists of the union of all chunks arrived so far. They
compared training the network from scratch each time a new chunk arrives to
warm-starting, where we continuously train (i.e. fine-tune) the model, finding
that warm-starting significantly underperformed. In order to remedy this, they
proposed Shrink & Perturb (defined in Section 3.2), a re-initialization
technique applied each time a new chunk arrives. Interestingly, applying
Shrink & Perturb not only closed the performance gap between warm-starting and
training from scratch, but improved upon the model trained from scratch. Does
this mean that the standard approach to training neural networks without re-
initialization is sub-optimal?

The motivation of our work is to systematically understand the benefits and
limitations of re-initialization methods in training neural networks. To that
end, we trained over 15,000 models to identify the settings under which re-
initialization methods are helpful. We study the interaction between re-
initialization and other widely used regularization and optimization
techniques, including data augmentation, weight decay and learning rate
schedules. We seek to answer a fundamental, practical question: are the
benefits of re-initialization additive with those of common, existing
techniques for improving generalization? Should re-initialization be present
in the arsenal of deep learning practices? Our experiments therefore include
careful ablations in which we consider settings ranging from vanilla training
to SOTA protocols which yield top performance for a given architecture. In
addition to varying the training protocol, we also investigate the impact of
re-initialization methods when learning on noisy data by studying what happens
under the presence of label noise, leading to some surprising results.

Our empirical study centers around two re-initialization methods. First, as
our primary focus, we consider Shrink & Perturb, which was proposed by Ash &
Adams [2] in the context of online learning and warm-starting. To the best of
our knowledge, the regularization benefits of Shrink & Perturb as a re-
initialization method for standard i.i.d. supervised learning have not been
studied before, and we find that an adaptation of Shrink & Perturb can
sometimes lead to significant gains in this setting. Second, we consider the
recently proposed Layer-wise Re-initialization [1] that has been shown to
outperform prior re-initialization schemes on a variety of datasets.
Furthermore, we show that re-initialization methods can naturally be adapted
to incorporate self-distillation [11] which typically leads to improvement, at
negligible cost. Figure 1 summarizes our results on Tiny ImageNet. From here
on, we will use standard training to refer to training without re-
initialization.

Our contributions and findings are as follows:

  * •

Shrink & Perturb can benefit i.i.d. learning. We investigate the benefits of
Shrink & Perturb, previously proposed for online learning, and its variant
with distillation as re-initialization methods that can be used for i.i.d
supervised learning. We show that they are helpful and outperform techniques
such as Layer-wise Re-initialization in certain settings.

  * •

Re-initialization improves generalization in the absence of other
regularization (Section 4). There is a consistent advantage in periodically
re-initializing a neural network—even up to 25 times—during training, pointing
to the inherent regularization benefit of re-initialization.

  * •

Re-initialization has limited advantage over standard training in a SOTA
setting (Section 5). When data augmentation, weight decay and learning rate
schedules are carefully tuned, re-initialization performs at par with standard
training. However, the optimal performance becomes more robust to the choice
of learning rate and weight decay, which is desirable in practice.

  * •

Under label noise, re-initialization methods lead to significant improvement
in generalization (Section 6). This improvement appears on top of other well-
tuned regularization, revealing a setting where the effects of re-
initialization and other regularization techniques do not overlap.

##  2 Related Work

Various re-initialization methods have been proposed in the literature,
differing usually by motivation and choice of weights that are re-initialized.
For example, motivated by sparsity, Han et al. [12] introduced dense-sparse-
dense training, where after an initial stage of training, the smallest weights
are pruned to induce sparsity, the network is trained and in the third and
last stage, the pruned weights are re-initialized from zero. Taha et al. [29]
also proposed “knowledge evolution” which partitions the weights of a network
into two parts, one of which is continuously training and the other repeatedly
re-initialized. For transfer learning, Li et al. [21] proposed to periodically
re-initialize only the final layer, optionally with ensembling [31]. Recently,
Alabdulmohsin et al. [1] compared a number of these methods with their
proposed Layer-wise Re-initialization. They found it outperformed prior
approaches, which is why we chose to study it in this work. Their empirical
evaluation focused on the small-data regime, where one set of hyperparameters
common across multiple datasets and architectures are used. They also did not
study how re-initialization methods interact with other regularizers. Also, in
the spirit of re-initialization, various approaches for “resetting” or
“restarting” training can also be found in the literature [25, 16, 28, 18].

Re-initialization also recently appeared in the context of online learning and
warm-starting neural networks, where data arrives sequentially [2, 6]. Ash &
Adams [2] showed that warm-starting neural network training worsens
generalization. They proposed Shrink & Perturb, a simple technique to improve
performance in their online learning setup. The benefits of Shrink & Perturb
as a generic re-initialization method for i.i.d. learning have not been
studied before. Caccia et al. [6] studied techniques for expanding capacity by
initializing new parameters as more data arrives. Igl et al. [17] observed a
similar phenomenon in reinforcement learning and proposed distillation as a
remedy. Generally, self-distillation [11, 15] itself closely relates to re-
initialization (see Section 3.1).

##  3 Background on Re-initialization Methods

Let $f_{\boldsymbol{\theta}}$ be a neural network with parameters
${\boldsymbol{\theta}}\in\mathbb{R}^{p}$. We will assume
$f_{\boldsymbol{\theta}}$ is trained to minimize some loss function
$L({\boldsymbol{\theta}})$. Moreover, let $p_{\text{init}}$ be a probability
distribution over $\mathbb{R}^{p}$ from which we sample the initialization
parameters ${\boldsymbol{\theta}}_{0}\sim p_{\text{init}}$, where optimization
begins. Then, we define a re-initialization of $f_{\boldsymbol{\theta}}$ to be
the function $f_{{\boldsymbol{\theta}}_{\text{RI}}}$, where we have replaced
the current parameters ${\boldsymbol{\theta}}$ with re-initialized parameters
${\boldsymbol{\theta}}_{\text{RI}}=R({\boldsymbol{\theta}}_{\text{init}},{\boldsymbol{\theta}})$.
The function $R$ computes the re-initialized parameters using freshly
initialized parameters ${\boldsymbol{\theta}}_{\text{init}}\sim
p_{\text{init}}$ and the current parameters ${\boldsymbol{\theta}}$. Different
re-initialization methods will differ in terms of $R$.

Re-initialization methods will modify standard training as follows. Assuming a
computational budget corresponding to $N$ epochs over the training data, we
train the model in $T$ stages. Each stage will consist of training the network
for $\lfloor N/T\rfloor$ epochs. Between any two stages, we apply re-
initialization as described earlier. This ensures that the computational cost
in terms of the total number of gradient steps is approximately equal to the
cost of standard training for $N$ epochs. This is key to ensure that the
performance of different methods can be compared fairly. Explicitly, let
${\boldsymbol{\theta}}_{t-1}^{\text{end}}$ denote the parameters at the end of
the previous stage $t-1$. Stage $t$ then consists of training the model
starting from initial parameters
${\boldsymbol{\theta}}_{t}^{\text{start}}=R({\boldsymbol{\theta}}_{\text{init}},{\boldsymbol{\theta}}_{t-1}^{\text{end}})$.
The initial parameters for stage 1 are sampled directly from
$p_{\text{init}}$. See Algorithm 1 for a pseudo-code description.

###  3.1 Incorporating Self-Distillation With Re-initialization

Our experiments will also consider the effect of incorporating self-
distillation, which is a natural extension of re-initialization methods
because training is split into multiple sequential stages. Specifically, when
using distillation, we train the model in stage $t$ by minimizing a
combination of the training loss and a distillation loss between the current
model (the student) and the model at the end of the previous stage $t-1$ (the
teacher), that is, we minimize the loss

$L({\boldsymbol{\theta}})+{\beta_{\text{distill}}}\,\text{KL}(f_{{\boldsymbol{\theta}}_{t-1}^{\text{end}}}\;\|\;f_{\boldsymbol{\theta}}),$
(1)

with respect to ${\boldsymbol{\theta}}$. The second term measures the KL-
divergence between the predicted class probabilities of the teacher and the
student on the training data, and ${\beta_{\text{distill}}}$ is a
hyperparameter determining the strength of distillation. In stage $t=1$, we
only optimize the loss $L({\boldsymbol{\theta}})$. Note that this requires
generating the predictions of the teacher network at the end of the previous
stage and caching them for use during the optimization of the student in the
current stage. The added computational cost is negligible, and we found that
the training time with and without distillation did not vary much. We also
remark that the setting in which re-initialization simply consists of
replacing all the old model parameters with a new initialization, i.e.
${\boldsymbol{\theta}}_{\text{RI}}={\boldsymbol{\theta}}_{\text{init}}$,
combined with distillation is equivalent to Born-Again Networks (BANs) [11]
supervised by the true and teacher labels, with a fixed computational
budget.111In the original BANs, the computational cost scaled proportionally
with the number self-distillation iterations. For a fair comparison, we
consider BANs under a fixed total budget.

###  3.2 Shrink & Perturb

Figure 2: Example test accuracy curves on the CIFAR datasets with ResNet-18 in
setting $\varnothing$. Shrink & Perturb involves 10 stages. For each method
and dataset, the learning rate is tuned separately – the optimal learning
rates are on average 10 times smaller for re-initialization than standard
training here.

Our primary focus in this work will be Shrink & Perturb, a method proposed by
Ash & Adams [2] in the context of online learning and warm-starting neural
network training. Ash & Adams [2] studied a scenario in which training data
arrives sequentially in “chunks” over time such that at any point in time the
training dataset consists of the union of all chunks arrived so far. They
compared re-training the network from scratch each time a new chunk arrives to
warm-starting, where we continuously train (i.e. fine-tune) the model. They
found that the warm-started network significantly underperformed. In order to
remedy this, they proposed Shrink & Perturb, a re-initialization technique
applied to the previous model each time a new chunk arrives. Shrink & Perturb
consists of defining the re-initialized parameters to be:

${\boldsymbol{\theta}}_{\text{RI}}=\lambda{\boldsymbol{\theta}}+\gamma{\boldsymbol{\theta}}_{\text{init}},$

where $\lambda,\gamma\in[0,1]$ are hyperparameters. We shrink the current
parameters by a multiplicative factor of $\lambda$ and perturb them using a
fresh initialization scaled by $\gamma$. In our initial experiments, we tuned
these hyperparameters which then remain fixed at $\lambda=0.4,\gamma=0.1$
across all architectures and datasets and are similar to the optimal values in
the setup of Ash & Adams [2]. The number of stages $T$ is a hyperparameter we
vary in our experiments.

Shrink & Perturb was demonstrated to improve upon re-training from scratch in
the online learning setup. Ash & Adams [2] suggested that it may have a
helpful regularization effect, but this effect has not been studied in any
detail in the usual i.i.d. learning setting. Contrary to their setup where
Shrink & Perturb is applied for online learning, we will view Shrink & Perturb
as a re-initialization method for training on a stationary dataset as
described earlier.

###  3.3 Layer-wise Re-initialization

Alabdulmohsin et al. [1] recently proposed Layer-wise Re-initialization, which
re-initializes the architecture block-by-block during training. Assuming that
there are $K$ “blocks” in the architecture, we have a total of $T=K$ stages
during training. At the end of stage $t$, we re-initialize all layers after
the $t$-th block, that is, we set:

${\boldsymbol{\theta}}_{\text{RI}}={\boldsymbol{\theta}}\odot{\boldsymbol{m}}^{(t)}+{\boldsymbol{\theta}}_{\text{init}}\odot(1-{\boldsymbol{m}}^{(t)}).$

Here ${\boldsymbol{m}}^{(t)}\in\\{0,1\\}^{p}$ is a vector which masks all
layers after the first $t$ blocks, i.e. ${\boldsymbol{m}}^{(t)}_{i}=1$ if the
$i$-th parameters belongs to the first $t$ blocks and is otherwise zero.
$\odot$ denotes an element-wise product. More generally, note that we can have
$T=KM$ total stages, where at the end of each of the first $M$ stages
$t=1,\dots,M$, we re-initialize all layers after the first block. At the end
of each of the second $M$ stages $t=M+1,\dots,2M$, we re-initialize all layers
after the second block and so forth. We note that Alabdulmohsin et al. [1] set
$M=1$ in their main experiments.

Moreover, in addition to re-initializing all blocks after the $t$-th block,
Layer-wise Re-initialization also (1) re-scales the norm of the first $t$
blocks to their norm at initialization and (2) adds a normalization layer
(without trainable parameters) after block $t$ whose mean and standard
deviation are computed by forward passing a batch of training data. Note that
unlike Shrink & Perturb, Layer-wise Re-initialization has a somewhat
architecture-specific definition, since it requires picking a priori what a
block corresponds to in the architecture. In our experiments with a ResNet-18
architecture, we considered four cases: $T=K=2$ where the network is split
between the second and third residual blocks, $T=K=5$ where we split by each
residual block and lastly $T=10$ and $20$ by letting $M=2$ and $4$
respectively in the previous case.

##  4 The Regularizing Effect of Re-initialization

Table 1: Test accuracy (%) of different methods in settings ranging from basic
to SOTA protocols on CIFAR-10 with ResNet-18.

Setting Abbrev. |  Data Aug. |  Cosine Anneal. |  Weight Decay |  No Re-initialization (standard training) |  Self-distillation (fixed-budget BAN) |  | Layer-wise
---
Re-initialization
Shrink & Perturb
w/o dist. | w/ dist. | w/o dist. | w/ dist.
$\varnothing$ | ✗ | ✗ | ✗ | $83.8\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $84.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.4}$ | $87.6\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.4}$ | $87.3\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{88.8}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{88.6}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.4}$
d | ✓ | ✗ | ✗ | $92.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $92.6\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $92.6\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $93.2\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $93.1\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{94.1}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$
dc | ✓ | ✓ | ✗ | $92.8\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $92.6\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $93.4\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $93.4\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{94.1}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{94.2}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$
dcw | ✓ | ✓ | ✓ | $95.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.0}$ | $94.8\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $94.7\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $95.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $94.7\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $94.7\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$

Table 2: Test accuracy (%) of different methods in settings ranging from basic
to SOTA protocols on CIFAR-100 with ResNet-18.

Setting Abbrev. |  Data Aug. |  Cosine Anneal. |  Weight Decay |  No Re-initialization (standard training) |  Self-distillation (fixed-budget BAN) |  SGDR |  | Layer-wise
---
Re-initialization
Shrink & Perturb
w/o dist. | w/ dist. | w/o dist. | w/ dist.
$\varnothing$ | ✗ | ✗ | ✗ | $55.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.6}$ | $56.4\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.5}$ | n/a | $61.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.6}$ | $62.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $63.1\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.6}$ | $\mathbf{63.5}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$
d | ✓ | ✗ | ✗ | $70.8\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $70.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.5}$ | n/a | $72.1\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $\mathbf{74.7}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $71.9\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $74.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.6}$
dc | ✓ | ✓ | ✗ | $71.2\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $70.9\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.4}$ | $71.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.6}$ | $74.6\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.5}$ | $\mathbf{75.4}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $\mathbf{75.4}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $\mathbf{75.4}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.4}$
dcw | ✓ | ✓ | ✓ | $\mathbf{77.9}\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $77.2\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $77.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $77.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.1}$ | $77.3\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$ | $77.5\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.2}$ | $77.0\,{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.3}$

Table 3: Test accuracy (%) of standard training and Shrink & Perturb on Tiny
ImageNet with PreAct-ResNet-18.

Setting Abbrev. |  Data Aug. |  Cosine Anneal. |  Weight Decay |  No Re-initialization (standard training) | Shrink & Perturb
---|---|---|---|---|---
w/o dist. | w/ dist.
$\varnothing$ | ✗ | ✗ | ✗ | $39.55{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.69}$ | $41.47{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.48}$ | $44.81{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.46}$
dcw | ✓ | ✓ | ✓ | $59.12{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.33}$ | $58.95{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.56}$ | $61.03{\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color<EMAIL_ADDRESS>0.38}$

In this section, we investigate the effect of re-initialization when no other
regularization is deployed. We show that re-intiliazation provides a
consistent and considerable advantage over standard training, even beyond the
small-data regime [1, 29], as is the case of Tiny ImageNet. This highlights
their effectiveness as a regularization technique. Additionally, we find that
Shrink & Perturb outperforms Layer-wise Re-initialization in this setting.

###  4.1 Experimental Setup

For each architecture (ResNet-18, PreAct-ResNet-18, MobileNetV2) [14, 13, 27]
and dataset (CIFAR-10, CIFAR-100, Tiny ImageNet), we used SGD with momentum
$0.9$ for training with early stopping by validation accuracy. For the CIFAR
datasets, we separately tuned the learning rate and weight decay
hyperparameters for each method by cross validation. Note that this is
different to the setup of Alabdulmohsin et al. [1] where the learning rate and
weight decay hyperparameters were fixed for all combinations of datasets and
architectures in their image classification experiments. For the larger and
more computationally costly Tiny ImageNet dataset, we tuned the learning rate
and weight decay hyperparameters for standard training (that is, no re-
initialization) and kept them fixed for the re-initialization methods. This
simulates a practical scenario: given tuned hyperparameters for standard
training on an expensive and large dataset, does it help to use re-
initialization without re-tuning the hyperparameters?

For re-initialization methods, we note that the number of stages $T$ is an
additional hyperparameter which we tune (cf. Section 5.5). Moreover, all
models are optimized for the same number of gradient steps for all methods.
All other hyperparameters such as batch size and number of epochs are the same
for all methods and specified in Tables 5 and 6 of Appendix B. Our code will
be open-sourced.

###  4.2 Re-initialization Improves Performance in the Absence of Other
Regularization

The different settings we consider are labeled $\varnothing$, d, dc and dcw,
as described on the left of Table 1. Setting $\varnothing$ is the basic
setting where we use a constant learning rate, no weight decay and no data
augmentation. As shown in the first rows of Tables 1 and 2, both Shrink &
Perturb and Layer-wise Re-initialization improve generalization performance
over standard training for both CIFAR datasets. Moreover, Shrink & Perturb
outperforms Layer-wise Re-initialization by a margin of 1-2 percentage points
in both cases. Both re-initialization methods also benefit from incorporating
distillation for CIFAR-10, although distillation matters less for CIFAR-100.
Next, we consider training in setting $\varnothing$ for Tiny ImageNet,
focusing only on Shrink & Perturb with and without distillation in this case,
as shown in Table 3. As before, re-initialization is beneficial compared to
standard training, particularly when we include distillation. Indeed, these
results indicate that re-initialization yields a notable generalization boost
in setting $\varnothing$ without additional computational cost.

Examples of the test accuracy learning curves are shown in Figure 2 for Shrink
& Perturb with 10 stages. Notice that each re-initialization causes a sudden
drop in performance from which there is a quick recovery. Whereas the test
accuracy for standard training stagnates early on, training with Shrink &
Perturb keeps improving test accuracy with each successive re-initialization
and already outperforms standard training by epoch 50. All models shown in
this plot achieved approximately 100% training accuracy and close to zero
training negative log-likelihood. See Figures 10 and 11 for examples of
training curves.

##  5 Re-initialization Alongside Other Regularization

Section 4 demonstrates that Re-initialization methods have a regularizing
effect on learning that improves generalization. Next, we consider how
beneficial Re-initialization is when deployed alongside other regularization
techniques that are commonly used to achieve high performance in SOTA training
protocols. In particular, we consider three common regularization techniques:
data augmentation, cosine annealing and weight decay. Note that while cosine
annealing is not an explicit regularizer, the choice of learning rates affects
regularization [22, 23] and is an important ingredient for well-performing
models. For re-initialization methods, we apply cosine annealing per stage,
yielding a cyclical learning rate [25]. Our motivation is to explore whether
the effect of these different techniques is additive with the benefit of Re-
initialization; indeed, regularizers that are helpful each on their own are
not necessarily helpful in the presence of one another.

###  5.1 Re-initialization Offers Little Benefit in SOTA Settings

Using a similar setup as before, we compare standard training with various Re-
initialization methods across these settings on the CIFAR datasets in Tables 1
and 2. Beginning with the simple setting explored in Section 4, we gradually
add data augmentation (setting d), cosine annealing (setting dc) and weight
decay (setting dcw), achieving around SOTA performance in setting dcw. First,
we observe that all Re-initialization methods except fixed-budget BAN
noticeably improve upon standard training in settings $\varnothing$, d and dc.
In most cases, the best performing method is Shrink & Perturb with
distillation, improving accuracy over standard training by upto 5 percentage
points on CIFAR-10 and 8 percentage points on CIFAR-100.

Figure 3: Weight norms during training vary across methods more in setting
$\varnothing$ than setting dcw. Curves show how the norm of ResNet-18 weights
varies through training for re-initialization methods (with 5 stages) and
standard training. In setting $\varnothing$, Shrink & Perturb yields a model
with smaller weight norm than standard training and Layer-wise Re-
initialization, but in setting dcw, all methods yield models with similar
weight norms.

However, a key finding here is that when all regularization techniques are
used and carefully tuned (setting dcw), the best Re-initialization methods do
not improve generalization performance over standard training. Therefore,
stacking Re-initialization on top of other regularization techniques results
in a sub-additive effect on performance. Note that this setting is arguably
more important and common in practice than previous settings. We speculate
that Shrink & Perturb affects the norm of the weights during training which
may overlap with the effect of weight decay. As shown in Figure 3, in the
absence of weight decay, the norm of the weights tends to increase
monotonically without re-initialization, whereas Shrink & Perturb periodically
reduces this weight norm. In setting $\varnothing$, this results in the
learned models’ weight norms varying across methods, while, in setting dcw,
all methods learn models with similar final norms due to weight decay.
Moreover, in setting $\varnothing$, Shrink & Perturb results in a model with
norm similar to that of standard training in setting dcw. Therefore, both
weight decay and Shrink & Perturb encourage the final learned model to have a
small weight norm, a quantity that is known to be linked to generalization
performance [26, 4]. Nevertheless, this does not fully explain why Shrink &
Perturb works, as shown later in Section 6. We also show the impact of re-
initialization in setting dcw for Tiny ImageNet in the last row of Table 3.
Similar to before, the improvement in performance from using Shrink & Perturb
is much smaller here than in setting $\varnothing$. Shrink & Perturb has no
impact on performance, whereas Shrink & Perturb with distillation slightly
improves performance over standard training.

###  5.2 The Role of Self-Distillation

Another observation from Tables 1 and 2 is that fixed-budget BANs do not
improve performance compared to standard training in most cases, sometimes
leading to worse test accuracies, and are outperformed by Shrink & Perturb and
Layer-wise Re-initialization in all settings except dcw. Recall from Section
3.1 that BANs can be viewed as a simple re-initialization method combined with
distillation: we re-initialize the full network from scratch in each stage.
Therefore, BANs constitute an important baseline for more sophisticated re-
initialization methods. Our results indicate that while distillation can boost
performance, it is computationally sub-optimal to completely re-initialize
each network in the sequence. Under a fixed-budget setting as shown here, too
many stages will then lead to too few gradient steps per stage, whereas too
few stages do not reap the full benefits of distillation, both of which can
negatively impact performance. On the other hand, Layer-wise Re-initialization
and Shrink & Perturb partly “re-use” the model from the previous stage,
circumventing the need for a large number of gradient steps per stage and
improving learning efficiency. Therefore, re-initialization is itself crucial,
and distillation does not suffice on its own. Figure 9 shows how complete re-
initialization in BANs leads to large drops in performance from which it is
more difficult to recover quickly.

###  5.3 The Role of Cosine Annealing

Recall that cosine annealing is applied per stage for re-initialization
methods, yielding a cyclical learning rate schedule as used in SGDR [25]. Such
a learning rate schedule can have its own benefits, therefore we compared re-
initialization in settings dc and dcw with SGDR for CIFAR-100. As shown in
Table 2, SGDR does not match the performance of the re-initialization methods
in setting dc and performs similar in setting dcw. This indicates that when
re-initialization is helpful, it is not an effect of the cyclical schedule.

###  5.4 Re-initialization Makes Optimal Performance More Robust to
Hyperparameters

Figure 4: Re-initialization can make performance less sensitive to the choice
of learning rate and weight decay. Performance of different methods over a
grid search of learning rate and weight decay combinations on CIFAR-100 with
ResNet-18 in setting dcw. All re-initialization methods use 5 stages. Shrink &
Perturb stands out most, as its performance varies much less over the grid
than other methods, especially standard training (no re-initialization).
Figure 5: Test accuracy as a function of the number of re-initialization
stages in settings $\varnothing$ and dcw. In setting $\varnothing$, any number
of stages improves upon standard training for Shrink & Perturb and Layer-wise
Re-initialization, though the optimal number tends to be 5-20 stages. In
setting dcw, adding more stages monotonically worsens generalization. Recall
that for Layer-wise Re-initialization, the number of stages is tied to the
architecture itself, and $T\in\\{2,5,10,20\\}$.

Although re-initialization methods do not offer much benefit in terms of
generalization for optimal hyperparameters in setting dcw, we found that they
can make performance less sensitive to the choice of learning rate and weight
decay hyperparameters – particularly Shrink & Perturb. Figure 4 shows the test
accuracy achieved by different methods over a grid of learning rate and weight
decay choices for CIFAR-100 in setting dcw. We observe that the performance of
both Layer-wise Re-initialization and Shrink & Perturb varies less over the
grid compared to no re-initialization. This is especially prominent for Shrink
& Perturb. For example, with learning rate 0.005 and weight decay 0,
performance of standard training degrades to 67.6%, whereas the performance of
Shrink & Perturb remains above 73% throughout. Therefore, re-initialization
methods can be beneficial in regimes where thoroughly hyperparameter tuning is
infeasible and robustness becomes crucial. This complies with the findings of
Alabdulmohsin et al. [1] who operated under the setup of using a common set of
hyperparameters for all datasets and architectures, which may be sub-optimal
in certain cases for standard training, and found re-initialization methods
outperform standard training.

###  5.5 Impact of the Number of Re-initialization Stages

Recall that the number of stages $T$ used for re-initialization is an
additional hyperparameter that we tuned in our experiments. Note that $T=1$
trivially corresponds to standard training (i.e. no re-initialization). We
show how test accuracy varies as a function of $T$ in Figure 5 for the
different methods over each of the CIFAR datasets in settings $\varnothing$
and dcw. In setting $\varnothing$, typically a large number of stages, in the
range 5-20, is most beneficial for the re-initialization methods. In fact, for
Shrink & Perturb and Layer-wise Re-initialization, any number of stages in the
range shown (2-25) improves upon standard training. However, in setting dcw,
the profile of the curves changes drastically. Increasing the number of stages
monotonically lowers the test accuracy, which is consistent with our finding
that in this setting, re-initialization methods offer little advantage. See
Figure 12 in Appendix A for further experiments with MobileNetV2 on the CIFAR
datasets.

##  6 Re-initialization Under Label Noise

Figure 6: Re-initialization is beneficial for learning under label noise. Both
re-initialization methods improve performance, with distillation leading to a
further improvement. Moreover, the benefit of re-initialization compared to
standard training increases with more label noise.

Having studied how different components of the training protocol interact with
re-initialization, we next focus on learning under noisy data. In particular,
we add label noise to the data which makes learning more challenging. We
consider the impact on performance when a randomly-chosen fraction $q$ of the
training data points have random labels, whereas the test set remains the
same. The addition of noise makes learning more difficult as models would
require more regularization to prevent them from overfitting to the noise.
Therefore, learning under label noise can help distinguish different
regularization techniques. We remain in setting dcw, where data augmentation,
weight decay and learning rate schedules are used. Moreover, we follow the
setup described in Section 4.1; on the CIFAR datasets, learning rate and
weight decay are tuned separately for each method, while on Tiny ImageNet, we
use the hyperparameters obtained from tuning on standard training only.

Figure 6 shows the performance of different methods as we add $q=20\%$ and
$40\%$ label noise on the CIFAR and Tiny ImageNet datasets. Surprisingly, re-
initialization methods consistently improve upon standard training here. For
CIFAR datasets, both Shrink & Perturb and Layer-wise Re-initialization are
beneficial each on their own, and performance further improves with the
addition of distillation. The results are similar for Tiny ImageNet, where the
difference between standard training and re-initialization is even starker:
Shrink & Perturb with distillation improves upon standard training by over 15
percentage points in test accuracy when we have 40% label noise.

We can therefore conclude that the effects of re-initialization do not
completely overlap with those of data augmentation, weight decay and learning
rate schedules. Indeed, even though re-initialization does not improve
generalization in setting dcw without label noise, it shows a substantial
improvement on generalization with label noise on top of standard
regularization techniques. We suspect that one reason re-initialization helps
in this scenario relates to the order in which neural networks learn training
examples. “Difficult examples” (such as mislabelled inputs) tend to be learned
later during training by deeper layers [3]. Re-initialization might be
exploiting this property of learning by naturally selecting the clean data:
since periodically re-initializing “resets” training to an extent, the network
might be prevented from confidently fitting to the noise (which is learned
later) to instead focus on learning the correctly labelled examples (which are
learned earlier). This may also explain why Layer-wise Re-initialization
outperforms Shrink & Perturb here due to its layer-wise structure that re-
initializes deeper layers more. This also raises the question of whether the
performance gap between re-initialization methods and standard training can be
closed by tuning the epoch budget to reduce overfitting. As shown in Figure
13, this is not sufficient to close to gap.

##  7 Conclusion, Limitations & Future Work

We have investigated when re-initialization methods, a set of techniques that
are simple to implement with almost no computational overhead, improve
generalization compared to standard training. We found that such methods are
beneficial for improving performance in the absence of other regularization.
However, in a setting resembling SOTA training protocols (i.e. including data
augmentation, weight decay and learning rate schedules), which is arguably
more important and common, re-initialization methods do not improve over
standard training, although optimal performance becomes more robust to the
choice of hyperparameters. Finally, under label noise, re-initialization does
have a significant and helpful effect on learning that other regularization
techniques are not able to offer.

In future work, we hope to study the implications of our work on online
learning, where Shrink & Perturb was first proposed and shown to be helpful.
Does the generalization gap due to warm-starting still appear if models are
trained with sufficient regularization? Does Shrink & Perturb continue to be
beneficial? One limitation of our work is that, although we observe clear
empirical trends in when re-initialization works, a deeper understanding of
why it does or does not work is missing and would be very desirable to gain in
future work. Another limitation of our study is that we restrict ourselves to
CIFAR-10/100 and Tiny ImageNet datasets and convolution-based network
architectures for classification. Although this restriction allowed us to
thoroughly explore the interaction between re-initialization and other
techniques on these standard benchmarks and carefully tune hyperparameters
given a limited compute budget, it would be interesting to extend the scope of
our study to other architectures, tasks and data modalities as future work.

## References

  * Alabdulmohsin et al. [2021] Alabdulmohsin, I., Maennel, H., and Keysers, D.  The impact of reinitialization on generalization in convolutional neural networks.  _arXiv preprint arXiv:2109.00267_ , 2021.
  * Ash & Adams [2020] Ash, J. and Adams, R. P.  On warm-starting neural network training.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 3884–3894. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/288cd2567953f06e460a33951f55daaf-Paper.pdf.
  * Baldock et al. [2021] Baldock, R. J. N., Maennel, H., and Neyshabur, B.  Deep learning through the lens of example difficulty.  2021\.
  * Bartlett et al. [2017] Bartlett, P. L., Foster, D. J., and Telgarsky, M.  Spectrally-normalized margin bounds for neural networks.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, pp. 6241–6250, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Brock et al. [2021] Brock, A., De, S., Smith, S. L., and Simonyan, K.  High-performance large-scale image recognition without normalization.  _arXiv preprint arXiv:2102.06171_ , 2021.
  * Caccia et al. [2021] Caccia, L., Xu, J., Ott, M., Ranzato, M., and Denoyer, L.  On anytime learning at macroscale.  _arXiv preprint arXiv:2106.09563_ , 2021.
  * Chen et al. [2020] Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Cubuk et al. [2018] Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., and Le, Q. V.  Autoaugment: Learning augmentation policies from data.  _arXiv preprint arXiv:1805.09501_ , 2018.
  * Cubuk et al. [2020] Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pp. 702–703, 2020.
  * Foret et al. [2020] Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B.  Sharpness-aware minimization for efficiently improving generalization.  _arXiv preprint arXiv:2010.01412_ , 2020.
  * Furlanello et al. [2018] Furlanello, T., Lipton, Z., Tschannen, M., Itti, L., and Anandkumar, A.  Born again neural networks.  In Dy, J. and Krause, A. (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1607–1616. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/furlanello18a.html.
  * Han et al. [2016] Han, S., Pool, J., Narang, S., Mao, H., Gong, E., Tang, S., Elsen, E., Vajda, P., Paluri, M., Tran, J., et al.  Dsd: Dense-sparse-dense training for deep neural networks.  _arXiv preprint arXiv:1607.04381_ , 2016.
  * He et al. [2016a] He, K., Zhang, X., Ren, S., and Sun, J.  Identity mappings in deep residual networks.  In Leibe, B., Matas, J., Sebe, N., and Welling, M. (eds.), _Computer Vision – ECCV 2016_ , pp. 630–645, Cham, 2016a. Springer International Publishing.  ISBN 978-3-319-46493-0.
  * He et al. [2016b] He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 770–778, 2016b.  doi: 10.1109/CVPR.2016.90.
  * Hinton et al. [2015] Hinton, G., Vinyals, O., and Dean, J.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Huang et al. [2017] Huang, G., Li, Y., Pleiss, G., Liu, Z., Hopcroft, J. E., and Weinberger, K. Q.  Snapshot ensembles: Train 1, get M for free.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=BJYwwY9ll.
  * Igl et al. [2021] Igl, M., Farquhar, G., Luketina, J., Boehmer, W., and Whiteson, S.  Transient non-stationarity and generalisation in deep reinforcement learning.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=Qun8fv4qSby.
  * Izmailov et al. [2018] Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D. P., and Wilson, A. G.  Averaging weights leads to wider optima and better generalization.  In Globerson, A. and Silva, R. (eds.), _UAI_ , pp. 876–885. AUAI Press, 2018.  URL http://dblp.uni-trier.de/db/conf/uai/uai2018.html#IzmailovPGVW18.
  * Krizhevsky et al. [2009] Krizhevsky, A., Nair, V., and Hinton, G.  Cifar-10 (canadian institute for advanced research), 2009.  URL http://www.cs.toronto.edu/~kriz/cifar.html.
  * Le & Yang [2015] Le, Y. and Yang, X.  Tiny imagenet visual recognition challenge, 2015.
  * Li et al. [2020] Li, X., Xiong, H., An, H., Xu, C.-Z., and Dou, D.  Rifle: Backpropagation in depth for deep transfer learning through re-initializing the fully-connected layer.  In _ICML_ , 2020.
  * Li et al. [2019] Li, Y., Wei, C., and Ma, T.  Towards explaining the regularization effect of initial large learning rate in training neural networks.  _arXiv preprint arXiv:1907.04595_ , 2019.
  * Li & Arora [2019] Li, Z. and Arora, S.  An exponential learning rate schedule for deep learning.  In _International Conference on Learning Representations_ , 2019.
  * Liu et al. [2021] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  _arXiv preprint arXiv:2103.14030_ , 2021.
  * Loshchilov & Hutter [2016] Loshchilov, I. and Hutter, F.  Sgdr: Stochastic gradient descent with warm restarts.  _arXiv preprint arXiv:1608.03983_ , 2016.
  * Neyshabur et al. [2018] Neyshabur, B., Bhojanapalli, S., and Srebro, N.  A PAC-bayesian approach to spectrally-normalized margin bounds for neural networks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=Skz_WfbCZ.
  * Sandler et al. [2018] Sandler, M., Howard, A. G., Zhu, M., Zhmoginov, A., and Chen, L.  Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation.  _CoRR_ , abs/1801.04381, 2018.  URL http://arxiv.org/abs/1801.04381.
  * Smith [2017] Smith, L. N.  Cyclical learning rates for training neural networks.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 464–472, 2017.  doi: 10.1109/WACV.2017.58.
  * Taha et al. [2021] Taha, A., Shrivastava, A., and Davis, L. S.  Knowledge evolution in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12843–12852, 2021.
  * Tan & Le [2019] Tan, M. and Le, Q.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International Conference on Machine Learning_ , pp. 6105–6114. PMLR, 2019.
  * Zhao et al. [2018] Zhao, K., Matsukawa, T., and Suzuki, E.  Retraining: A simple way to improve the ensemble accuracy of deep neural networks for image classification.  In _2018 24th International Conference on Pattern Recognition (ICPR)_ , pp. 860–867, 2018.  doi: 10.1109/ICPR.2018.8545535.

##  Appendix A Additional Results

###  A.1 Results on ImageNet

Our findings on the dependence of the benefits of Shrink & Perturb on other
regularization were shown to hold for the CIFAR datasets and Tiny ImageNet. In
this section, we preliminarily explore whether they also hold for ImageNet. As
Table 4 shows, in setting CW, which lacks data augmentation, Shrink & Perturb
clearly outperforms no re-initialization. However, in setting dcw with full
regularization, Shrink & Perturb even slightly underperforms compared to
standard training. These conclusions also match our findings in Sections 4 and
5.

Table 4: Test accuracy (%) of standard training and Shrink & Perturb on
ImageNet with ResNet-18.

Setting Abbrev. |  Data Aug. |  Cosine Anneal. |  Weight Decay |  No Re-initialization (standard training) |  Shrink & Perturb
---|---|---|---|---|---
CW | ✗ | ✓ | ✓ | $59.1$ | $63.7$
DCW | ✓ | ✓ | ✓ | $70.8$ | $69.7$

###  A.2 Implications for Online Learning

Shrink & Perturb was originally proposed by Ash & Adams [2] in the context of
online learning (cf. Section 3.2), so we explore whether our conclusions also
hold there. In particular, how do different methods compare in the absence of
other regularization (similar to the setting originally studied by Ash & Adams
[2]) and with full regularization? In Figure 7, we assume that training data
accumulates by arriving sequentially in five “chunks” leading to five stages
of training, and we compare three methods: (1) initializing the network from
scratch and re-training it every time a new chunk arrives, (2) warm-starting
it (i.e. continuously fine-tuning) and (3) applying Shrink & Perturb every
time a new chunk arrives. In line with the findings of Ash & Adams [2],
without regularization, the warm-started network significantly underperforms
the network initialized from scratch, whereas applying Shrink & Perturb
outperforms both (left plot in Figure 7). However, under sufficient
regularization, all three methods interestingly perform approximately the same
(right plot in Figure 7). This also closely matches our findings from Sections
4 and 5.

We also find that the warm-starting generalization gap [2] (i.e. warm-started
models underperforming models initialized from scratch in each stage) does not
exist on ImageNet in a setting with full regularization as shown in Figure 8.

Figure 7: The benefits of Shrink & Perturb and the warm-starting
generalization gap in the online learning setup of Ash & Adams [2] also depend
on whether other regularization is used. Results shown are for a ResNet-18
trained on CIFAR-100. See Section A.2 for discussion. Figure 8: There is no
warm-starting generalization gap for ResNet-18 on ImageNet in the online
learning setup of Ash & Adams [2] under full regularization. Learning curves
show the test accuracy on ImageNet where in the first stage we train on half
the training data, followed by the full training data in the second stage. See
Section A.2 for discussion.

###  A.3 Additional Figures

Figures 13, 9, 10, 11 and 12 are discussed in Sections 5, 4 and 6.

Figure 9: Example test accuracy curves on the CIFAR-100 dataset with ResNet-18
in setting $\varnothing$. BAN involves 10 stages. For each method and dataset,
the learning rate is tuned separately. Notice that each re-initialization
causes a large drop in performance in contrast with e.g. Shrink & Perturb as
shown in Figure 2. Figure 10: Example train accuracy curves for standard
training and Shrink & Perturb. In each plot, Shrink & Perturb is shown with
its optimal number of stages. Observe that in all cases the models achieve
close to 100% accuracy. Figure 11: Example train negative log-likelihood
curves for standard training and Shrink & Perturb. In each plot, Shrink &
Perturb is shown with its optimal number of stages. Observe that in all cases
the models achieve close to zero loss. Figure 12: Effect of Shrink & Perturb
on the CIFAR datasets in settings $\varnothing$ and dcw with a MobileNetV2
architecture. Our results in this figure indicate that the overall trend
observed in our experiments with ResNet-18 also appears for MobileNetV2. In
particular, in setting $\varnothing$, Shrink & Perturb improves performance
for any number of stages shown, whereas in setting dcw, it provides no benefit
over standard training (no re-initialization). Figure 13: Under label noise,
tuning the epoch budget for standard training does not close the gap with re-
initialization methods. See the discussion in Section 6. Algorithm 1 Training
with re-initialization function $R$.

Input: Training data $\mathcal{D}$, number of re-initialization stages $T$,
initialization distribution $p_{\text{init}}$, total epochs $N$.

$P$ $\leftarrow\lfloor N/T\rfloor$ # epochs per stage

for $t=1$ to $T$ do

if $t==1$ then

${\boldsymbol{\theta}}^{\text{start}}\sim p_{\text{init}}$ # initialize
training

else

${\boldsymbol{\theta}}^{\text{start}}\leftarrow
R({\boldsymbol{\theta}}_{\text{init}},{\boldsymbol{\theta}}^{\text{end}})$,
where ${\boldsymbol{\theta}}_{\text{init}}\sim p_{\text{init}}$ is an i.i.d.
initialization. # re-initialize network

end if

${\boldsymbol{\theta}}^{\text{end}}\leftarrow$ parameters after training for
$P$ epochs on $\mathcal{D}$, starting from
${\boldsymbol{\theta}}^{\text{start}}$, optionally with distillation from
teacher network $f_{{\boldsymbol{\theta}}^{\text{end}}}$ if $t>1$. #
optimization objective is defined in Equation 1

end for

return ${\boldsymbol{\theta}}^{\text{end}}$

##  Appendix B Hyperparameters & Pseudo-code of Re-initialization Methods

Hyperparameters for our experiments are described in Tables 5 and 6, and the
pseudo-code for training with re-initialization is shown in Algorithm 1.

General
---
Total training epochs | $200$
Stages $\times$ epochs per stage | $\left\\{2\times 100,5\times 40,10\times 20,20\times 10,25\times 8\right\\}$
Batch size | $125$
Momentum | $0.9$
Learning rate grid | $\left\\{0.005,0.01,0.03,0.05,0.1\right\\}$
Weight decay grid | $\left\\{0,0.0001,0.0005,0.001,0.005\right\\}$
Distillation strength ${\beta_{\text{distill}}}$ (if used) | $1$
Data augmentation (if used) | Random horizontal flips and
size-preserving crops after padding 4 pixels.
Data normalization | $\mu=\left(0.485,0.456,0.406\right)$
$\sigma=\left(0.229,0.224,0.225\right)$
Shrink & Perturb
Shrink $\lambda$ (default unless explicitly defined otherwise) | 0.4
Perturb $\gamma$ (default unless explicitly defined otherwise) | 0.1
Layer-wise Re-initialization
$\left(K,M\right)$ | $\left\\{\left(2,1\right),\left(5,1\right),\left(5,2\right)\right\\}$
Table 5: Hyperparameters common to all models trained on CIFAR-10 and
CIFAR-100 ($32\times 32$ images) [19]. General
---
Total training epochs | 500
Stages $\times$ epochs per stage | $\left\\{2\times 100,5\times 40,10\times 20,20\times 10,25\times 8\right\\}$
Batch size | $100$
Momentum | $0.9$
Learning rate grid | $\left\\{0.01,0.05,0.1\right\\}$
Weight decay grid | $\left\\{0,0.0001,0.0005,0.005\right\\}$
Distillation strength ${\beta_{\text{distill}}}$ (if used) | $2$
Data augmentation (if used) | Random horizontal flips and
size-preserving crops after padding 4 pixels.
Data normalization | $\mu=\left(0.485,0.456,0.406\right)$
$\sigma=\left(0.229,0.224,0.225\right)$
Shrink & Perturb
Shrink $\lambda$ (default unless explicitly defined otherwise) | 0.4
Perturb $\gamma$ (default unless explicitly defined otherwise) | 0.1
Table 6: Hyperparameters common to all models trained on Tiny ImageNet
($64\times 64$ images) [20].