# Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free
Ensembles of DNNs

Uri Stern, Daphna Weinshall

###### Abstract

The infrequent occurrence of overfit in deep neural networks is perplexing. On
the one hand, theory predicts that as models get larger they should eventually
become too specialized for a specific training set, with ensuing decrease in
generalization. In contrast, empirical results in image classification
indicate that increasing the training time of deep models or using bigger
models almost never hurts generalization. Is it because the way we measure
overfit is too limited? Here, we introduce a novel score for quantifying
overfit, which monitors the forgetting rate of deep models on validation data.
Presumably, this score indicates that even while generalization improves
overall, there are certain regions of the data space where it deteriorates.
When thus measured, we show that overfit can occur with and without a decrease
in validation accuracy, and may be more common than previously appreciated.
This observation may help to clarify the aforementioned confusing picture. We
use our observations to construct a new ensemble method, based solely on the
training history of a single network, which provides significant improvement
in performance without any additional cost in training time. An extensive
empirical evaluation with modern deep models shows our method’s utility on
multiple datasets, neural networks architectures and training schemes, both
when training from scratch and when using pre-trained networks in transfer
learning. Notably, our method outperforms comparable methods while being
easier to implement and use, and further improves the performance of
competitive networks on Imagenet by 1%.

##  1 Introduction

Overfit of train data constitutes a fundamental problem in machine learning.
Theoretical analysis predicts that as a model acquires additional degrees of
freedom, its ability to fit a certain training data increases. As a result, a
model’s generalization error is expected to increase when it becomes too
specialized for a specific training set. Accordingly, in deep learning we
expect to see _increased generalization error_ as the number of parameters
and/or training epochs increases. Surprisingly, even vast deep neural networks
with many billions of parameters rarely fulfil this expectation. In fact, even
in larger models that do exhibit slightly inferior performance (Liu et al.
2022) with increased size, we do not see overfit as a function of epochs. More
typically, even substantial increase in the number of parameters will still
lead to improved performance, or to more bizarre behaviors like the double
descent in test error (Annavarapu 2021, see Section 3). Clearly there is a gap
between our classical understanding of overfit and the empirical results
obtained when using modern neural networks.

To bridge this gap, we introduce a novel perspective on overfit. Instead of
gauging it solely through a reduction in _test accuracy_ , we propose to track
what we term _the model’s forget fraction_. This metric represents the portion
of test data111We use the common terminology, though for estimation purposes,
the various scores are evaluated on validation data. that the model initially
classifies correctly but misclassifies as training proceeds. In Section 3 we
examine various benchmark datasets, where we measure this phenomenon even if
no overfit is observed using the traditional definition, namely, when test
accuracy increases as learning proceeds. Interestingly, we observe that models
can suffer from a significant level of forgetting, which indicates that our
score measures something new. Importantly, this occurs even after the
deployment of modern methods to reduce overfit, such as data augmentation,
while using competitive networks.

Using this new perspective with its corresponding scores and to better
understand the phenomenon, we analyze in Section 3 the curious phenomenon of
“epoch wise double descent”. Here, models trained over data with label noise
show _two distinct periods of descent in test error_ (i.e., improvement in
generalization), the second of which occurring _after_ the model has memorized
the noisy labels. Our empirical analysis shows that the second period of
descent is caused by learning _new patterns_ in the data, rather than
relearning old ones forgotten at the overfit stage. This phenomenon, we show
empirically, is caused by the _simultaneous learning_ of general patterns
learned from clean data (which enhances performance continuously as learning
progresses), and irrelevant specific patterns learned from noisy data (which
degrade performance).

Based on these empirical observations, we propose in Section 4 a method that
can effectively reduce the forgetting of test data. The challenge is to reduce
overfit when test accuracy decreases with time, and to further improve the
test accuracy even when it does not decrease with time (i.e., when overfit
does not occur in the traditional sense). Accordingly, we construct a new
prediction method, which aims to combine knowledge gained both in mid-training
and after training. More specifically, the method delivers a weighted average
of the class probability output vector between the final model and a set of
checkpoints of the model from mid-training, where the checkpoints and their
weights are chosen in an iterative manner using a validation dataset and our
forget metric. The method is outlined in Alg. 1, see App B.

In Section 5 we describe the empirical validation of our method in a series of
experiments over image classification of datasets with and without label
noise, using various network architectures, including in particular modern
networks over Imagenet, thus showing that our method is universally useful.
When compared with alternative methods that use the network’s training
history, our method shows comparable or better performance, while being more
general and easy to use (both in implementation and hyper-parameter tuning).
Specifically, in contrast with other methods, it does not depend on additional
training choices that require much more time and effort to tune the new hyper-
parameters.

#### Our main contributions

(i) Novel perspective on overfit, entailing new scores to assess local
overfit.  (ii) Empirical evidence that overfit occurs ”locally” even without a
decrease in overall generalization.  (iii) A simple and effective method to
reduce overfit.

##  2 Related Work

#### Studies of overfit and double descent

Overfit in deep neural networks, and specifically the double descent and
epoch-wise double descent phenomena, has garnered increasing attention in
recent years (see recent review by Ganaie et al. 2022). Double descent with
respect to model size has been studied empirically in (Belkin et al. 2019;
Nakkiran et al. 2021), while epoch-wise double descent (which is the
phenomenon analyzed here) was studied in (Stephenson and Lee 2021; Heckel and
Yilmaz 2020). These studies analyzed when and how epoch-wise double descent
occurs, specifically in data with label noise, and explored ways to avoid it
(sometimes at the cost of reduced generalization). In contrast, our research
delves into the concept of double descent by investigating the extent to which
neural networks forget. Our findings reveal that such phenomena are present,
to some degree, even in datasets without label noise. We then use our
observation to improve performance at _inference time_ , rather than change
the training scheme as done in most previous work. Our line of study is
complementary to - and should not be confused with - the study of ”benign
overfitting”, e.g., the fact that models can achieve perfect fit to the train
data while still obtaining good performance over the test data.

#### Study of forgetting in prior work

Most relevant studies focus on the forgetting of training data, namely, the
fact that some training points are memorized early on, but are then forgotten.
This may occur when the network is not able to memorize all the training set.
(Toneva et al. 2018), for example, analyzed the forgetting of points in the
_train data_ , which is then used to score the importance of individual
datapoints for training. In contrast, our work focuses on the forgetting of
test points, which cannot be verified during training since the label of these
points is not known. Another phenomenon, which may be confused with the one we
are discussing, is ”catastrophic forgetting” (McCloskey and Cohen 1989;
Ratcliff 1990). This occurs in a _continual learning_ scenario when the
training data changes with time and the network does not have access to
training data encountered at the beginning of training. In the scenario
considered here this problem does not exist, since data does not expire.

#### Ensemble learning

Ensemble learning has been studied in machine learning for decades (Polikar
2012), including many recent works that employ deep neural networks ensembles,
see (Ganaie et al. 2022; Yang, Lv, and Chen 2023) for recent surveys. As
ensembles are expensive, many works attempt to reduce their cost, specifically
their training cost (Ganaie et al. 2022). This line of works, to which our
work belongs, is called ”implicit ensemble learning”, in which only a single
network is learned in a way that ”mimics” ensemble learning. A notable work in
this field is dropout (Srivastava et al. 2014) and its variants, where some
parts of the network are dropped at random during training, creating multiple
”independent” networks inside the network.

Utilizing checkpoints from the training history as a ’cost-effective’ ensemble
has also been considered. This was achieved by either considering the last
epochs and averaging their probability outputs (Xie, Xu, and Chuang 2013), or
by employing exponential moving average (EMA) on all the weights throughout
training (Polyak and Juditsky 1992). While the latter method has demonstrated
some success in reducing overfit, it has been reported to fail in some cases
(Izmailov et al. 2018).

A number of methods (Izmailov et al. 2018; Garipov et al. 2018; Huang et al.
2017) adopted a somewhat different approach that involves changing the
training protocol of the network, such that it will converge to several local
minima throughout training. These multiple solutions are then combined to form
an ensemble classifier. These methods show great promise, impacting a range of
fields such as medical applications (Nguyen and Pernkopf 2020; Annavarapu
2021), fault diagnosis (Wen, Gao, and Li 2019), attack detection (Rouzbahani,
Bahrami, and Karimipour 2021) and land use classification (Noppitak and
Surinta 2022). However, these methods should be used with care, as they
require the tuning of two inter-connected training schemes (the old and the
new), and sometimes prolong the training time substantially at a potentially
large financial cost in practical settings. Additionally, the new training
scheme is not guaranteed to produce good and diverse local minima, and can
even hurt performance as reported by Guo, Jin, and Liu (2023). We conducted
comprehensive comparisons to these methods (see Table 3), demonstrating that
in all instances, our approach either matches or outperforms them, all the
while maintaining a significantly simpler design.

(a) Cifar100, 20% sym noise

(b) TinyImagenet, 20% sym noise

(c) Cifar100, 20% sym noise

(d) TinyImagenet, 20% sym noise

Figure 1: (a)-(b): Blue denotes test accuracy. Among those correctly
recognized in each epoch $e$, orange denotes the fraction that remains
correctly recognized at the end. The test accuracy (the blue curve) shows a
clear double ascent of accuracy, which is much less pronounced in the orange
curve. During the decrease in test accuracy - the range of epochs between the
first and second dashed red vertical lines - the large gap between the blue
and orange plots indicates the fraction of test data that has been correctly
learned in the first ascent and then forgotten, without ever being re-learned
in the later recovery period of the second ascent. (c)-(d): Comparison between
the amounts of the clean and noisy data with large loss (above a fixed
threshold) at each epoch just before and during the second ascent of test
accuracy (the range of epochs after the second dashed red vertical line). A
positive value indicates that there are more clean data examples with large
loss at this epoch, indicating that the model will now learn more general and
correct patterns than wrong pattern caused by the noisy labels.

##  3 Overfit Revisited

The textbook definition of overfit entails the co-occurrence of elevated train
accuracy and reduced generalization. Let $acc(e,S)$ denote the accuracy over
set $S$ in epoch $e$ \- some epoch in mid-training, $E$ the total number of
epochs, and $T$ the test dataset. Using test accuracy to approximate
generalization, this implies that overfit occurs at epoch $e$ when
$acc(e,T)\geq acc(E,T)$. However, test accuracy is a global measure that may
obscure more subtle expressions of overfit, for example, when overfit only
occurs in sub-regions of a continually evolving feature space. In this
section, we attempt to expand our view of overfit, and develop a more
sensitive metric that can capture the second form of overfit even in the
absence of the first one.

We begin with the observation that portions of the test data $T$ may be
’forgotten’ by the network during training. We argue that this phenomenon
indicates a form of local overfit, which can persist and negatively impact
specific sub-regions of the dataset even as overall test accuracy continues to
improve. Based on this observation, we propose to expand the definition of
overfit. This broader definition gains further validation from our examination
of the ’epoch-wise double descent’ phenomenon, which frequently occurs during
training on datasets that contain significant label noise in the training set.
In such cases, a notable forgetting of the test data coincides with the
memorization of noisy labels, serving as an objective indicator of overfit.
The primary implication of this analysis leads to a surprising revelation:
even when training modern networks on standard datasets (devoid of label
noise), where overfit (as traditionally defined) does not manifest, _the
networks still appear to forget certain sub-regions of the test population_.
This observation, we assert, signifies a significant and more subtle form of
overfit.

#### Looking at overfit in a new way

Let $M_{e}$ denote the subset of test data mislabeled by the network at some
epoch e. We now define two scores - _learning_ $L_{e}$ and _forgetting_
$F_{e}$ \- as follows:

$F_{e}=acc(e,M_{E})\times\frac{|M_{E}|}{|T|},\quad
L_{e}=acc(E,M_{e})\times\frac{|M_{e}|}{|T|}$ (1)

$F_{e}$, referred to henceforth as the _forget fraction_ , represents the
fraction of the test data that is correctly classified at epoch $e$ but
incorrectly classified at the end of training (epoch $E$). This subset of the
test data is known at epoch $e$, but later forgotten. On the other hand,
$L_{e}$ denotes the fraction that will be known post-training but is
misclassified at epoch $e$. Clearly $acc(E,T)=acc(e,T)+L_{e}-F_{e}$.
Therefore, in line with the classical definition of overfit, if $L_{e}<F_{e}$,
overfit indeed occurs sometime after epoch $e$.

#### What happens in the absence of classical overfit?

If $L_{e}\geq F_{e}~{}\forall e$, then by its classical definition _overfit
does not occur_ since the test accuracy doesn’t ever decrease. Nevertheless,
there might still be numerous test examples that the final model
misclassifies, even though they were classified correctly at some intermediate
stage of training. This happens if at some epoch $e$ $L_{e}>F_{e}$ is still
true, but $F_{e}$ is nevertheless large. As we show later on, this phenomenon
is frequent among neural networks, more so than the traditionally defined
overfit, which makes our definition useful in capturing this type of ill
behavior.

#### Reflections on the epoch-wise double descent phenomenon

Epoch-wise double descent (see Fig. 1) is an empirical observation (Belkin et
al. 2019), which shows that neural networks can improve their performance even
after overfitting, thus causing _double descent in test error_ during training
(note that we show the corresponding _double-ascent in test accuracy_). This
phenomenon is characteristic of learning from data with label noise, and is
strongly related to overfit since the dip in test accuracy co-occurs with the
memorization of noisy labels.

We examine the behavior of the novel score $F_{e}$ in this context and make a
novel revelation: when we focus on the fraction of data acquired by the
network during the second rise in test accuracy, we observe that the data
newly memorized during these epochs often differs from the data forgotten
during the overfit phase (the dip in accuracy). In fact, most of this data has
been previously misclassified (refer to Figs.1(a) and 1(b)). To bolster this
observation, Figs.1(c) and 1(d) further illustrate that during the later
stages of training on data with label noise, when the second increase in test
accuracy occurs, the majority of the data being memorized is, in fact, data
with clean labels.

#### What happens when there is no label noise?

When training deep networks on visual benchmark datasets without added label
noise, double descent rarely occurs, if ever.

(a) TinyImagenet

(b) TinyImagenet

(c) ImageNet

(d) Cifar100

Figure 2: (a) The forget fraction $F_{e}$, as defined in (1), of Resnet18
trained on TinyImagenet. (b) Within the set of wrongly classified test points
after training, we show the fraction that was correctly predicted (y-axis) for
at least x epochs (x-axis). (c) Comparison of the $F_{e}$ score of ConvNeXt
trained on Imagenet, evaluated in three network sizes: small $\rightarrow$
blue, base $\rightarrow$ orange and large $\rightarrow$ green. Clearly,
$F_{e}$ increases with the size of the network. (d) The $F_{e}$ score of
Resnet18 trained on $10/30/50/70/90\%$ of the train data in cifar100
(purple/red/green/yellow/blue line, respectively) in the first 50 epochs of
training (after which the score decreases); $F_{e}$ is significantly larger
for the smallest set of only $10\%$.

What about our new score $F_{e}$? To answer this question we trained various
neural networks (ConvNets: Resnet, ConvNeXt; Visual transformers: ViT, MaxViT)
on various datasets (CIFAR100, TinyImagenet, Imagenet) using a variety of
optimizers (SGD, AdamW) and learning rate schedulers (cosine annealing,
steplr). In Fig. 2(a) and App A we report the results, showing that all
networks forget some portion of the data during training as in the label noise
scenario, even if the test accuracy never decreases. Fig. 2(b) shows that this
effect is not arbitrary: many examples have been correctly classified for a
large portion of the network’s training.

In Figs. 2(c) and 2(d) we connect our results to overfit. We show that when
investigating either larger models or relatively small amounts of train data,
which are scenarios that are expected to increase overfit based on theoretical
considerations, both are associated with larger _forget fraction_ $F_{e}$.

To further validate our results, we examined in App A a different definition
of ”forget”, in which we look at the last epoch in which an example was
classified correctly. Interestingly, we see a dense span of epochs in which
many examples are being classified correctly for the last time, which hints
that subregions of the data population were ”forgotten” in those epochs. In
the appendix we also analyze the number of times a mistake of the last epoch
was previously classified correctly, and show that for many examples, the
correct classification was not a random effect but rather a consistent trend
in significant parts of training.

In summary, our observations add up to the following: Neural networks can, and
often will, ”forget” significant portions of the test population as their
training proceeds. In a sense, the networks _are_ overfitting, but this only
occurs at some limited subregions of the world. The reason that overfit is not
seen using the classical defintion, it appears, is that at the same time the
networks still learn general patterns about the data from some correct - but
hard to learn - train data, which allows for the test accuracy to keep
improving. In the next section we discuss how we can harness this observation
to improve the network’s performance.

##  4 Method: Recovering Forgotten Knowledge

In Section 3 we showed that neural networks often show better performance in
mid-training on a subset of the test data, even when the test accuracy is
monotonically increasing with training epochs. Here we aim to integrate the
knowledge obtained in mid and post training epochs, during inference time, in
order to improve performance. To this end we must determine:  (i) which
versions of the model to use;  (ii) how to combine them with the post-training
model; and  (iii) how much weight to assign the post training model, whose
overall performance is usually better.

Method/Dataset | CIFAR-100 | TinyImagenet | Imagenet
---|---|---|---
architecture | Resnet18 | Resnet18 | Resnet50 | ConvNeXt large | ViT16 base | MaxViT tiny
_single network_ | $78.07\pm.28$ | $64.95\pm.24$ | $75.74\pm.14$ | $82.92\pm.11$ | $79.16\pm.1$ | $82.51\pm.15$
_horizontal (i)_ | $78.15\pm.17$ | $64.89\pm.18$ | $\mathbf{76.46\pm.14}$ | $83.13\pm.1$ | $79.11\pm.1$ | $82.77\pm.1$
_fixed jumps (i)_ | $78.04\pm.23$ | $66.54\pm.35$ | $75.5\pm.09$ | $82.37\pm.1$ | $78.67\pm.08$ | $83.38\pm.1$
_KF (ours) (i)_ | $\mathbf{78.33\pm.08}$ | $\mathbf{66.98\pm.37}$ | $75.88\pm.14$ | $83.18\pm.16$ | $\mathbf{79.93\pm.11}$ | $83.34\pm.04$
_horizontal $(\infty)$_ | $78.23\pm.17$ | $65.11\pm.3$ | $\mathbf{76.42\pm.1}$ | $83.02\pm.06$ | $79.53\pm.13$ | $82.93\pm.14$
_fixed jumps $(\infty)$_ | $\mathbf{79.17\pm.08}$ | $68.24\pm.38$ | $75.72\pm.18$ | $\mathbf{83.86\pm.06}$ | $79.11\pm.13$ | $\mathbf{83.78\pm.15}$
_KF (ours) $(\infty)$_ | $\mathbf{79.13\pm.14}$ | $\mathbf{68.5\pm.36}$ | $\mathbf{76.52\pm.16}$ | $\mathbf{83.96\pm.09}$ | $\mathbf{80.34\pm.08}$ | $\mathbf{83.81\pm.14}$
_improvement_ | $\mathbf{1.05\pm.14}$ | $\mathbf{3.54\pm.14}$ | $\mathbf{.78\pm.04}$ | $\mathbf{1.03\pm 13}$ | $\mathbf{1.17\pm.08}$ | $\mathbf{1.29\pm.02}$
Table 1: Mean (over random validation/test splits) test accuracy (in percent) and standard error on image classification datasets, comparing our method and baselines described in the text. The last row shows the improvement of the best performer over the single network. Suffixes: $(i)$ denotes a limited budget scenario, in which we use our method in a non-iterative manner; $(\infty)$ denotes the unlimited budget scenario, where we use our iterative version of the method. In each case, the baselines employ the same number of checkpoints as our method.  Method/Dataset | Animal10N | CIFAR-100 asym | CIFAR-100 sym | TinyImagenet
---|---|---|---|---
% label noise | 8% | 10% | 20% | 40% | 20% | 40% | 20% | 40%
_single network_ | $85.9\pm.3$ | $72.1\pm.1$ | $67.1\pm.5$ | $49.4\pm.3$ | $65.4\pm.3$ | $56.9\pm.1$ | $56.2\pm.2$ | $49.8\pm.3$
_fixed jumps $(\infty)$_ | $87.1\pm.4$ | $76.2\pm.1$ | $73.9\pm.1$ | $59.9\pm.6$ | $72.8\pm.1$ | $66.5\pm.1$ | $60.0\pm.8$ | $54.16\pm.3$
_horizontal $(\infty)$_ | $86.3\pm.3$ | $75.4\pm.3$ | $73.4\pm.1$ | $58.5\pm.1$ | $71.1\pm.38$ | $65.2\pm.1$ | $59.3\pm.3$ | $51.7\pm.2$
_KF (ours) $(\infty)$_ | $\mathbf{87.8\pm.4}$ | $\mathbf{76.6\pm.3}$ | $\mathbf{74.2\pm.1}$ | $\mathbf{62.1\pm.5}$ | $\mathbf{72.8\pm.1}$ | $\mathbf{67.0\pm.1}$ | $\mathbf{62.8\pm.2}$ | $\mathbf{57.0\pm.5}$
_improvement_ | $\mathbf{1.9\pm.4}$ | $\mathbf{4.4\pm.2}$ | $\mathbf{7.1\pm.6}$ | $\mathbf{12.6\pm.2}$ | $\mathbf{7.4\pm.4}$ | $\mathbf{10.1\pm.1}$ | $\mathbf{6.6\pm.1}$ | $\mathbf{7.2\pm.1}$
Table 2: Mean test accuracy (in percent) and standard error of Resnet 18, comparing our method and the baselines on datasets with large label noise and significant overfit. We include a comparison to the Animal10N dataset, which has innate label noise. Note that as is customary, only the train data has label noise while the test data remains clean for a fair evaluation. Method/Dataset | CIFAR-100 | TinyImagenet | Animal10N | CIFAR-100 asym | CIFAR-100 sym
---|---|---|---|---|---
% label noise | 0% | 0% | 8% | 20% | 40% | 20% | 40%
_FGE $(\infty)$_ | $78.9\pm.4$ | $67.7\pm.1$ | $86.5\pm 0.6$ | $67.1\pm.2$ | $48.1\pm.3$ | $66.5\pm.1$ | $52.1\pm.1$
_SWA $(\infty)$_ | $78.8\pm.1$ | $\mathbf{69.3\pm.6}$ | $\mathbf{88.1\pm.2}$ | $66.6\pm.1$ | $46.9\pm.2$ | $65.6\pm.4$ | $50.0\pm.1$
_snapshot $(\infty)$_ | $78.4\pm.1$ | $\mathbf{69.3\pm.4}$ | $86.8\pm.3$ | $72.1\pm.4$ | $52.8\pm.6$ | $70.8\pm.5$ | $63.8\pm.2$
_KF (ours) $(\infty)$_ | $\mathbf{79.3\pm.2}$ | $\mathbf{69.4\pm.6}$ | $\mathbf{87.8\pm.4}$ | $\mathbf{74.2\pm.1}$ | $\mathbf{62.1\pm.5}$ | $\mathbf{72.8\pm.1}$ | $\mathbf{67.0\pm.1}$
Table 3: Mean test accuracy (in percent) and standard error of Resnet18,
comparing our method and baseline methods that alter the training.

#### Choosing an early epoch of the network

Given a set of epochs $\\{1,\ldots,E\\}$ and corresponding forget rates
$\\{F_{e}\\}_{e}$, we first single out the model $n_{A}$ obtained at epoch
$A=argmax_{e\in\\{1,...,E\\}}F_{e}$. This epoch is most likely to correct
mistakes of the model on ”forgotten” test data.

#### Combining the predictors

How to combine the two models, $n_{A}$ where the forget fraction is maximal,
and $n_{E}$ where train accuracy is maximal? A common practice is to average
models’ output. However, since the performance of $n_{E}$ is typically better
than that of $n_{A}$, we use a weighted average instead, giving $n_{E}$ a
larger weight. This guarantees that our method will not harm the general
performance, as it can always give a zero weight to the early checkpoint
$n_{A}$.

#### Improving robustness

To improve our method’s robustness to the choice of epoch $A$, we use a span
of epochs around $A$, denoted by $\\{n_{A-w},...,n_{A},...,n_{A+w}\\}$. The
vectors of probabilities computed by each checkpoint are averaged before
forming an ensemble with $n_{E}$. In our experiments, we use a fixed width
$w=1$.

#### Working in an iterative manner

As we now have a new predictor, we can now find another alternative predictor
from the training history that maximizes accuracy on the data misclassified by
the new predictor, and combine their knowledge as described. This can be done
iteratively, until no further improvement is achieved.

#### Choosing hyper-parameters

In order to compute $F_{e}$ (for the early epoch choice) and to find the best
weights and epoch span, we use a validation set, which is a part of the
labeled data not shown to the model during initial training. This is done post
training as it has no influence over the training process, and thus _doesn’t
incur additional costs over the training time_. We follow common practice, and
show in Section 6.1 that after finding the best hyper-parameters it is
possible to retrain the model on the complete training set and validation set
and use our method _without hurting performance_ , and while maintaining our
superiority over alternative methods also trained on the full data.

We call our method KnowledgeFusion (KF), and evaluate it in Section 5. Its
pseudocode can be found in App B.

##  5 Empirical evaluation

We now demonstrate the superior performance of our method as compared to the
original predictor, i.e. the network after training, as well as other
baselines. We evaluate our method using various image classification datasets,
neural networks architectures, and training schemes. The main results of our
empirical evaluation are presented in Tables 1-3, followed by an extensive
ablation study (and additional comparisons) in Section 6.

#### Review of empirical results

In Table 1 we report the results of our method using multiple architectures
trained on cifar100, TinyImagenet and Imagenet, with different learning rate
schedulers and optimizers. For comparison, we report the results of both the
original predictor and some simple baselines. We continue with additional
experiments on settings connected to overfit in Table 2 and App D, where we
test our methods on these datasets with injected symmetric and asymmetric
label noise (see App C), as well as on real label noise dataset (Animal10N).
Note that as customary, the label noise exists only in the train data while
the test data remains clean for model evaluation.

In Table 3 and App D we compare our method to additional methods that adjust
the training protocol itself, using both clean and noisy datasets. We employ
these methods using the same network architecture as our own, after suitable
hyper-parameter search. Finally, we compare in App E (Fig. 6) our method with
an ensemble of independent networks, to evaluate how much of the ensemble’s
performance boost can be gained using our method (without the extra cost of
ensemble training), see details in App C.

In each experiment we use half of the _test data_ for validation, to compute
our method’s hyper-parameters (the list of alternative epochs and
$\\{\epsilon_{i}\\}$), and then test the result on the remaining test data.
The accuracy reported here is only on the remaining test data, averaged over
three random splits of validation and test data, using different random seeds.
In Section 6.1 we show that when data is limited, we can train a network on a
subset of the training data while using the left out data for hyper-parameter
tuning. As customary, these same parameters are later used with models trained
on the full data, demonstratively without deteriorating the results.

#### Baselines

Our method incurs the training cost of a single model, and thus, following the
methodology of (Huang et al. 2017), we compare ourselves to methods that
require the same amount of training time. Our baselines are from two groups of
methods. The first group includes methods that do not alter the training
process:

  * •

Single network: the original network, after training.

  * •

Horizontal ensemble (Xie, Xu, and Chuang 2013): this method uses a set of
epochs at the end of the training, and delivers their average probability
outputs (with the same number of checkpoints as we do).

  * •

Fixed jumps: this baseline was used in (Huang et al. 2017), where several
checkpoints of the network, equally spaced through time, are taken as an
ensemble.

The second group includes methods that _alter_ the training protocol. While
this is not a directly comparable set of methods, as they focus on a
complementary way to improve performance, we report their results in order to
further validate the usefulness of our method. This group includes the
following methods:

  * •

Snapshot ensemble (Huang et al. 2017): in this method the network is trained
in several ”cycles”, each ending with a large increase of the learning rate
that pushes the network away from the local minimum. The network is meant to
converge to several different local minima during training, which are used as
an ensemble.

  * •

Stochastic Weight Averaging (SWA) (Izmailov et al. 2018): in this method the
network is regularly trained for a fixed training budget of epochs, and is
then trained using a circular/constant learning rate to converge to several
local minima, whose weights are averaged to get the final predictor. To
achieve fair comparison in training budget, we train the network using our
training method for 75% of the epochs, followed by their unique training for
the remaining 25% epochs.

  * •

Fast Geometric Ensembling (FGE) (Garipov et al. 2018): similar to SWA, except
that the final predictor is constructed by averaging the probability outputs
of each model, instead of their weights. In this comparison we match budgets
as explained above.

Comparisons to additional baselines that are relevant to resisting overfit,
including early stopping and test time augmentation, are discussed in App E.1.

##  6 Ablation Study

In this section we investigate some limitations and practical aspects of our
method. In Section 6.1 we show that a separate validation set is not really
necessary for the method to work well. In Section 6.2 we investigate how many
checkpoints are needed for the method to be effective, showing that only
$5-10\%$ of the past checkpoints are sufficient. In Section 6.3 we investigate
the added value of our method when using only partial hyper-parameter search,
as is common in real applications, which leads to sub-optimal training.
Interestingly, our method is shown to be even more beneficial in the sub-
optimal scenario, and reduces the gap between the optimal and sub-optimal
networks. Finally, in Section 6.4 we show our method is effective in transfer
learning scenerio, when the network’s initial weights are pretrained.

Additional evaluations are described in App E, where we show that: (i) our
method is superior compared to exponantial-moving-average (EMA), early
stopping and test time augmentation (App E.1); (ii) our method’s improvement
can grow as the number of parameters grow (App E.2); (iii) a large portion the
improvement of a regular ensemble of independent networks can often be
obtained using our method at a much lower cost (App E.3); and (iv) our method
does not have negative effects on the model’s fairness (App E.4).

###  6.1 Removing the requirement for validation set

In this experiment, we follow a common practice with respect to the validation
data: we train our model on cifar100 and TinyImagenet using only 90% of the
train data, use the remaining 10% for validation, and finally retrain the
model on the full train data while keeping the same hyper-parameters for
inference. The results are almost identical to those reported in Table 1. This
validates the robustness of our method to the (lack of) a validation set.

###  6.2 number of checkpoints used

Here we evaluate the cost entailed by the use of an ensemble at inference
time. In Fig. 3 we report the improvement in test accuracy as compared to a
single network, when varying the ensemble size. The results indicate that
almost all of the improvement can be obtained using only $5-10\%$ of the
checkpoints, making our method practical in real life.

(a) cifar100

(b) TinyImagenet

(c) cifar100, 20% asym noise

(d) TinyImagenet, 20% sym noise

Figure 3: Improvement achieved by our method when using a different number of
checkpoints (shown on the x-axis).

###  6.3 Optimal vs sub-optimal training

In real life, full search for the optimal training scheme and hyper-parameters
is not always possible, leading to sub-optimal performance. Interestingly, our
method can be used to reduce the gap between optimal and sub-optimal training,
as seen in Table 4, where the gap between optimally and suboptimally trained
MaxViT over Imagenet reduces by almost half when applying our method (on both
models).

training/method | original network | KF
---|---|---
_regular training_ | $82.5\pm.1$ | $83.8\pm.1$
_sub-optimal training_ | $77.3\pm.1$ | $81.0\pm.1$
_improvement_ | $\mathbf{5.2}$ | $\mathbf{2.8}$
Table 4: Mean test accuracy (in percent) and ste, over random validation/test
split. MaxViT is trained to classify Imagenet, comparing optimal and sub-
optimal training with and without KF.

###  6.4 Transfer learning

Another popular method to improve performance and reduce overfit employs
transfer learning, in which the model weights are initialized using pre-
trained weights over a different task, for example Imagenet pretrained
weights. This is followed by _fine tuning_ either the entire model or only
some of its layers (its head for example). In Table 5 we show that our method
is complementary to the use of transfer learning in both cases, as it can
still improve performance in this scenario. Note that when finetuning only the
last layers, our method is almost free of overhead costs, as one needs to save
and use at inference most of the model only once. Thus, the only overhead
involves the memory and inference costs of the different head checkpoints in
finetuning, whose size is insignificant compared to the rest of the model.

Method/Dataset | CIFAR-100 | TinyImagenet
---|---|---
_fully finetuned Resnet18_ | $80.72\pm.53$ | $75.01\pm.12$
_fully finetuned Resnet18 + KF_ | $\mathbf{81.64\pm.27}$ | $\mathbf{75.6\pm.18}$
_partially finetuned Resnet18_ | $61.7\pm.60$ | $54.78\pm.13$
_partially finetuned Resnet18 + KF_ | $\mathbf{65.24\pm.67}$ | $\mathbf{59.5\pm.03}$
Table 5: Mean test accuracy over random validation/test split. Our method
applied to Resnet18 pre-trained on Imagenet, while finetuning the entire model
(top) or only the head (bottom).

##  7 Method: Discussion and Limitations

Our method provides a significant improvement of around $1\%$ on modern neural
networks over Imagenet, which typically implies more than $5\%$ reduction in
test error. It is often complementary to other methods (such as EMA) that aim
to reduce overfit, and often succeeds to reduce overfit when such methods fail
(see Section 6.4 and App E.1). Our method is especially useful in transfer
learning settings when only a few layers are finetuned, as it:  (i)
significantly improves performance; and  (ii) adds very little overhead in
inference and memory costs, since most of the model is saved used at inference
time only once.

In more difficult settings, such as a small network for complex data (e.g.,
Resnet18 over TinyImagenet) or datasets with label noise, our method improves
performance even further, leading to a nice error reduction of
${\scriptstyle\sim}15\%$ in the reasonable setting of $10\%$ asymmetric noise.
This may be the case when dealing with complex and confusing natural datasets
such as Animal10N (Song, Kim, and Lee 2019).

When compared to baselines, we achieve comparable or better results (see, for
example, our improvement in ViT16 over Imagenet and Resnet18 over
TinyImagenet). Another large advantage of our method is that it is independent
of training choices. In contrast, the horizontal method seems to show little
improvement when the cosine-annealing learning rate scheduler is used (all
experiments but Resnet50 over Imagenet), while fixed-jumps with no cycles
(Huang et al. 2017) is known to show little improvement (or none at all) when
step-size learning rate scheduler is used (Resnet50 over Imagenet in our
experiments). Finally, while the more complicated baselines proved more useful
than the other baselines (but not more than our method), this was only made
possible by an extensive hyper-parameter search.

While being simple and useful, our method has a few limitations as it
requires: (i) validation data to tune the hyper-parameters; (ii) using
multiple checkpoints, which incurs overhead in inference time and memory
usage; (iii) the occurrence of forgetting to have any impact. These
limitations can be mitigated, however, as shown in Section 6:  (i) A subset of
the train set can be effectively used for validation.  (ii) A few checkpoints
can already achieve most of the method’s benefit. We note that these
checkpoints can run in parallel on multiple GPUs, and the memory overhead is
not likely to be excessive for big datasets. For example, the Imagenet dataset
weighs 167GB222https://www.kaggle.com/competitions/imagenet-object-
localization-challenge/data., while a regular resnet50 weighs around 230MB.
(iii) The method can revert to the original single network if no improvement
on a validation set is seen, a benefit lacking in the methods listed in Table
3, as it doesn’t require any changes to the learning process. Lastly, our
empirical evaluation shows that our method has no negative effects on the
model’s fairness, making it safe to use.

##  8 Conclusions and Future Work

We revisited the problem of _overfit_ in deep learning, proposing to track the
forgetting of validation data in order to detect local overfit. We connected
our new perspective with the _epoch wise double descent_ phenomenon,
empirically extending its scope while demonstrating that a similar effect
occurs in benchmark datasets with clean labels. Inspired by these new
empirical observations, we constructed a simple yet general method to improve
classification at inference time. We then empirically demonstrated its
effectiveness on many datasets and modern network architectures. The method
improves modern networks by around 1% accuracy over Imagenet, and is
especially useful in some transfer learning settings where its benefit is
large and its overhead is very small. In future work we will investigate and
characterize the ”forgotten” examples, and seek ways to achieve better and
more effective combinations of checkpoints.

## References

  * Annavarapu [2021] Annavarapu, C. S. R. 2021.  Deep learning-based improved snapshot ensemble technique for COVID-19 chest X-ray classification.  _Applied Intelligence_ , 51: 3104–3120.
  * Belkin et al. [2019] Belkin, M.; Hsu, D.; Ma, S.; and Mandal, S. 2019.  Reconciling modern machine-learning practice and the classical bias–variance trade-off.  _Proceedings of the National Academy of Sciences_ , 116(32): 15849–15854.
  * Deng et al. [2009] Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Dosovitskiy et al. [2020] Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_.
  * Ganaie et al. [2022] Ganaie, M. A.; Hu, M.; Malik, A.; Tanveer, M.; and Suganthan, P. 2022.  Ensemble deep learning: A review.  _Engineering Applications of Artificial Intelligence_ , 115: 105151\.
  * Garipov et al. [2018] Garipov, T.; Izmailov, P.; Podoprikhin, D.; Vetrov, D. P.; and Wilson, A. G. 2018\.  Loss surfaces, mode connectivity, and fast ensembling of dnns.  _Advances in neural information processing systems_ , 31.
  * Guo, Jin, and Liu [2023] Guo, H.; Jin, J.; and Liu, B. 2023.  Stochastic weight averaging revisited.  _Applied Sciences_ , 13(5): 2935.
  * He et al. [2016] He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heckel and Yilmaz [2020] Heckel, R.; and Yilmaz, F. F. 2020.  Early stopping in deep networks: Double descent and how to eliminate it.  _arXiv preprint arXiv:2007.10099_.
  * Huang et al. [2017] Huang, G.; Li, Y.; Pleiss, G.; Liu, Z.; Hopcroft, J. E.; and Weinberger, K. Q. 2017\.  Snapshot ensembles: Train 1, get m for free.  _arXiv preprint arXiv:1704.00109_.
  * Izmailov et al. [2018] Izmailov, P.; Podoprikhin, D.; Garipov, T.; Vetrov, D.; and Wilson, A. G. 2018.  Averaging weights leads to wider optima and better generalization.  _arXiv preprint arXiv:1803.05407_.
  * Krizhevsky, Hinton et al. [2009] Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Le and Yang [2015] Le, Y.; and Yang, X. 2015.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7): 3.
  * Liu et al. [2022] Liu, Z.; Mao, H.; Wu, C.-Y.; Feichtenhofer, C.; Darrell, T.; and Xie, S. 2022.  A ConvNet for the 2020s.  _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * McCloskey and Cohen [1989] McCloskey, M.; and Cohen, N. J. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, 109–165. Elsevier.
  * Nakkiran et al. [2021] Nakkiran, P.; Kaplun, G.; Bansal, Y.; Yang, T.; Barak, B.; and Sutskever, I. 2021\.  Deep double descent: Where bigger models and more data hurt.  _Journal of Statistical Mechanics: Theory and Experiment_ , 2021(12): 124003.
  * Nguyen and Pernkopf [2020] Nguyen, T.; and Pernkopf, F. 2020.  Lung sound classification using snapshot ensemble of convolutional neural networks.  In _2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)_, 760–763. IEEE.
  * Noppitak and Surinta [2022] Noppitak, S.; and Surinta, O. 2022.  dropCyclic: snapshot ensemble convolutional neural network based on a new learning rate schedule for land use classification.  _IEEE Access_ , 10: 60725–60737.
  * Patrini et al. [2017] Patrini, G.; Rozza, A.; Krishna Menon, A.; Nock, R.; and Qu, L. 2017.  Making deep neural networks robust to label noise: A loss correction approach.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1944–1952.
  * Polikar [2012] Polikar, R. 2012.  Ensemble learning.  _Ensemble machine learning: Methods and applications_ , 1–34.
  * Polyak and Juditsky [1992] Polyak, B. T.; and Juditsky, A. B. 1992.  Acceleration of stochastic approximation by averaging.  _SIAM journal on control and optimization_ , 30(4): 838–855.
  * Ratcliff [1990] Ratcliff, R. 1990.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  _Psychological review_ , 97(2): 285.
  * Rouzbahani, Bahrami, and Karimipour [2021] Rouzbahani, H. M.; Bahrami, A. H.; and Karimipour, H. 2021.  A snapshot ensemble deep neural network model for attack detection in industrial internet of things.  _AI-Enabled Threat Detection and Security Analysis for Industrial IoT_ , 181–194.
  * Song, Kim, and Lee [2019] Song, H.; Kim, M.; and Lee, J.-G. 2019.  Selfie: Refurbishing unclean samples for robust deep learning.  In _International Conference on Machine Learning_ , 5907–5915. PMLR.
  * Srivastava et al. [2014] Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1): 1929–1958.
  * Stephenson and Lee [2021] Stephenson, C.; and Lee, T. 2021.  When and how epochwise double descent happens.  _arXiv preprint arXiv:2108.12006_.
  * Toneva et al. [2018] Toneva, M.; Sordoni, A.; Combes, R. T. d.; Trischler, A.; Bengio, Y.; and Gordon, G. J. 2018.  An empirical study of example forgetting during deep neural network learning.  _arXiv preprint arXiv:1812.05159_.
  * TorchVision [2016] TorchVision. 2016.  TorchVision: PyTorch’s Computer Vision library.  https://github.com/pytorch/vision.
  * Tu et al. [2022] Tu, Z.; Talebi, H.; Zhang, H.; Yang, F.; Milanfar, P.; Bovik, A.; and Li, Y. 2022\.  Maxvit: Multi-axis vision transformer.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV_ , 459–479. Springer.
  * Wang et al. [2020] Wang, Z.; Qinami, K.; Karakozis, I. C.; Genova, K.; Nair, P.; Hata, K.; and Russakovsky, O. 2020.  Towards fairness in visual recognition: Effective strategies for bias mitigation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8919–8928.
  * Wen, Gao, and Li [2019] Wen, L.; Gao, L.; and Li, X. 2019.  A new snapshot ensemble convolutional neural network for fault diagnosis.  _Ieee Access_ , 7: 32037–32047.
  * Xie, Xu, and Chuang [2013] Xie, J.; Xu, B.; and Chuang, Z. 2013.  Horizontal and vertical ensemble with deep representation for classification.  _arXiv preprint arXiv:1306.2759_.
  * Yang, Lv, and Chen [2023] Yang, Y.; Lv, H.; and Chen, N. 2023.  A survey on ensemble learning under the era of deep learning.  _Artificial Intelligence Review_ , 56(6): 5545–5589.
  * Zhao et al. [2017] Zhao, J.; Wang, T.; Yatskar, M.; Ordonez, V.; and Chang, K.-W. 2017.  Men also like shopping: Reducing gender bias amplification using corpus-level constraints.  _arXiv preprint arXiv:1707.09457_.

## Appendix

##  Appendix A Additional forget fraction examples

In this appendix, we show more examples of various neural network trained on
different datasets which show significant forgetting during training (Fig 4),
which demonstrates the generality of this phenomenon. We also show in Fig 5
that (i) many ”forgotten” examples were correctly classified in a significant
amount of epochs, indicating that their correct classification somewhere mid-
training was not random, and (ii) that when observing the last epoch in which
a correct classification was made for an example, one can see a dense span of
epochs in which many examples where correctly predicted for the last time,
which further supports that some subset of the data population was ”forgotten”
by the network in this span of epochs.

(a) Resnet50 on Imagenet, steplr scheduler

(b) Densenet121 on cifar100

(c) MaxViT tiny on Imagenet

(d) Densenet121 on TinyImagenet

Figure 4:  The forget fraction $F_{e}$, as defined in (1), of common neural
networks trained on image classification datasets.

(a) cifar100

(b) cifar100

(c) TinyImagenet

(d) TinyImagenet

Figure 5: On the left: histogram of the new forget definition suggested by the
reviewers - the last epoch in which a test data example mis-classified post
training is correctly classified. One can see that within a span of 50 epochs
a large fraction of those data examples are being ”forgotten”, as opposed to a
random ”forgetting” time which might have been expected. On the right:
histogram of the amount of epochs a correct prediction was made for test data
mis-classified post training; One can see that a large fraction of the
mistakes were correctly predicted for over 25% of the training, showing this
correct prediction did not happen by random. Notably, both datasets are
without label noise, and show no signs of overfit in the validation accuracy.

##  Appendix B Pseudo-code for our method

In Alg 1 and Alg 2 we show how to implement our method and how to calculate
its hyper-parameters, respectively. In the pseudo-codes we call functions that
(i) calculate the probability for each class for a given example and a list of
predictors (get_class_probabilities) (ii) calculate the forget value per epoch
on some validation data, given the predictions at each epoch
(calc_forget_per_epoch) (iii) calculate the validation accuracy, to determine
the best weights for each epoch (validation_acc).

Algorithm 1 Knowledge Fusion (KF)

Input:A set of checkpoints during training of the neural network
{$n_{0}$,…,$n_{E}$}, w, test-point $x$

Output: prediction for $x$

{$A_{1}$,…,$A_{k}$}, {$\epsilon_{1},...,\epsilon_{k}$} $\leftarrow$
calc_early_epochs_and_epsilons({$n_{0}$,…,$n_{E}$}) #use Alg.2

$class\\_prob\\_per\\_checkpoint\leftarrow$

get_class_probabilities({$n_{0}$,…,$n_{E}$}, $x$)

$prob\leftarrow class\\_prob\\_per\\_checkpoint[E]$

for $i\leftarrow 1$ to $k$ do

$prob_{A}\leftarrow\mathbf{mean}(class\\_prob\\_per\\_checkpoint[A_{i}-w:A_{i}+w])$

$prob\leftarrow\epsilon_{i}*prob_{A}+(1-\epsilon_{i})*prob$

end for

$prediction\leftarrow\mathbf{argmax}(prob)$

Return $prediction$

Algorithm 2 KF - hyper-parameter calculation

Input:all past checkpoints during training of the neural network
{$n_{0}$,…,$n_{E}$}, w and validation data $V$, w and validation data $V$

Output: list of alternative epochs and their weights

$class\\_prob\\_per\\_checkpoint\leftarrow$
get_class_probabilities({$n_{0}$,…,$n_{E}$}, $V$)

$prob\leftarrow class\\_prob\\_per\\_checkpoint[E]$

explore = {$n_{0}$,…,$n_{E}$}

Alternative_epochs = {}

epsilons = {}

while explore is not empty do

$F$ = calc_forget_per_epoch($prob$, $class\\_prob\\_per\\_checkpoint$)

$alt\\_epoch$ = argmax($F[explore]$)

Alternative_epochs.append($alt\\_epoch$)

explore.remove($alt\\_epoch-1$, $alt\\_epoch$, $alt\\_epoch+1$)

for $epsilon\in\\{0,0.01,...,1\\}$ do

$prob_{A}\leftarrow\mathbf{mean}(class\\_prob\\_per\\_checkpoint[A_{i}-w:A_{i}+w])$

$combined\\_prob\leftarrow\epsilon*prob_{A}+(1-\epsilon)*prob$

if validation_acc($combined\\_prob$) ¿= validation_acc($prob$) then

$best\\_prob=combined\\_prob$

$best\\_epsilon=combined\\_prob$

end if

end for

$prob=best\\_prob$

epsilons.append(argmax($best\\_epsilon$))

end while

Return Alternative_epochs, epsilons;

##  Appendix C Full implementation details

In our experiments we use three image classification datasets - Imagenet [3],
cifar100 [12] and TinyImagenet [13]. In the Imagenet experiments, we train all
networks (Resnet [8], ConvNeXt [14], ViT [4] and MaxViT [29]) using the
torchvision
code333https://github.com/pytorch/vision/tree/main/references/classification
for training [28] with the recommended hyper-parameters, except ConvNeXt,
which was trained using the official
code444https://github.com/facebookresearch/ConvNeXt with the recommended
hyper-parameters, without using exponential moving average (EMA) (see App E.1
for comparison to using EMA). With cifar100 and TinyImagenet, we train all
networks for 200 epochs. For the clean versions of cifar100 and TinyImagenet
we use batch-size of 32, learning rate of 0.01, SGD optimizer with momentum of
0.9 and weight decay of 5e-4, cosine annealing scheduler, and standard
augmentations (horizontal flip, random crops).

We use similar settings for our transfer learning experiments, in which the
images are resized to $224\times 224$, the learning rate is set to 0.001 and
the network is initialized using Imagenet weights. In training, either the
whole network is finetuned or only a new head (instead of the original fully
connected layer), which consists of two dense layers, the first with output
size of 100 times the embedding size.

For noisy labels experiments, we train using cosine annealing with warm
restarts (restarting the learning rate every 40 epochs), using a larger
learning rate of 0.1 and updating it after every batch. We also use a larger
batch size of 64 in cifar100 and 128 in TinyImagenet. In the suboptimal
training described in Section 6.3, each image was cut before training into its
central 224 over 224 pixels (images smaller than this size were first resized
such that the smallest dimension was of size 224, then cut into 224 over 224).

To obtain a fair comparison, we train the competitive methods in Table 3 from
scratch using our network architecture and data. For [10] we train as
instructed by the paper, while for [11, 6] we use our training scheme (as
these methods are meant to be added to an existing training scheme) and
performed hyper-parameter search to optimize the methods’ performance in the
new setting. Experiments were conducted on a cluster of GPU type A5000.

#### Injecting label noise

For the label noise experiments we inject noisy labels using two standard
methods [19]:

  1. 1.

Symmetric noise: a fraction $p\in\\{0.2,0.4,0.6\\}$ of the labels is selected
randomly. Each selected label is switched to any other label with equal
probability.

  2. 2.

Asymmetric noise: a fraction $p$ of the labels is selected randomly. Each
selected label is switched to another label using a deterministic permutation
function.

##  Appendix D Additional evaluations

We show here additional evaluation settings of our method and baselines on
dataset with injected label noise, see Table 6.

Method/Dataset | CIFAR100 sym | TinyImagenet
---|---|---
% label noise | 60% | 20% | 40%
_FGE_ | $38.3\pm.7$ | $53.8\pm.1$ | $40.4\pm.3$
_SWA_ | $30.5\pm.7$ | $52.5\pm.2$ | $39.4\pm.3$
_snapshot_ | $55.6\pm.2$ | $62.6\pm.1$ | $56.5\pm.3$
_KF (ours)_ | $\mathbf{57.6\pm.2}$ | $\mathbf{62.8\pm.2}$ | $\mathbf{57.0\pm.5}$
Table 6: Mean (over random validation/test split) test accuracy (in percent)
and standard error on image classification datasets with injected label noise,
comparing our method and baselines.

##  Appendix E Additional ablation results

###  E.1 Comparisons to additional baselines

an alternative method to combine different checkpoints is to perform
exponential moving average (EMA) during training, which is known to have some
advantages [21] and is used sometimes to reduce overfit [14, 4], see [29] for
example). In table 7 we explore this option for two datasets and a regular
Resnet18, showing that our method can be of use when EMA doesn’t work, or
improves the performance much less than our method.

Method/Dataset | CIFAR-100 | TinyImagenet
---|---|---
_EMA_ (decay = 0.999) | $-0.34\pm.14$ | $0.73\pm.11$
_EMA_ (decay = 0.9999) | $-0.06\pm.33$ | $2.51\pm.01$
_KF_ | $\mathbf{1.05}\pm.14$ | $\mathbf{3.54}\pm.14$
Table 7: Mean (over random validation/test split) improvement in test accuracy (in percent) and standard error on image classification datasets, comparing our method and EMA with different decay values. We use the best epoch for EMA, calculated using the validation set. Method/Dataset | CIFAR-100 | CIFAR-100 asym | CIFAR-100 sym | TinyImagenet
---|---|---|---|---
% label noise | 0% | 10% | 20% | 40% | 20% | 40% | 0%
_ES_ | $78.13\pm.4$ | $71.91\pm.2$ | $68.54\pm.3$ | $51.53\pm.3$ | $68.16\pm.4$ | $61.17\pm.2$ | $65.44\pm.3$
_TTA_ | $79.21\pm.3$ | $72.97\pm.1$ | $71.00\pm.1$ | $54.53\pm.1$ | $70.14\pm.2$ | $63.59\pm.1$ | $65.67\pm.3$
_ES + TTA_ | $79.11\pm.2$ | $73.14\pm.2$ | $70.46\pm.2$ | $53.93\pm.5$ | $70.21\pm.1$ | $63.57\pm.1$ | $65.74\pm.2$
_KF (ours)_ | $78.71\pm.2$ | $73.61\pm.1$ | $71.24\pm.5$ | $56.19\pm.8$ | $72.21\pm.3$ | $65.75\pm.1$ | $69.00\pm.1$
_KF (ours) + TTA_ | $\mathbf{79.55\pm.3}$ | $\mathbf{74.83\pm.1}$ | $\mathbf{72.65\pm.2}$ | $\mathbf{57.71\pm.3}$ | $\mathbf{72.33\pm.2}$ | $\mathbf{65.71\pm.1}$ | $\mathbf{69.05\pm.3}$
Table 8: Mean test accuracy (in percent) and standard error of resnet 18, comparing our method with Early Stopping (ES) and Test Time Augmentation (TTA) on datasets with and without label noise. Dataset/Method | original model | KF
---|---|---
evaluation metric | natural accuracy | transformed accuracy | bias | natural accuracy | transformed accuracy | bias
_cifar10 w/o color_ | $89.07\pm.48$ | $87.98\pm.38$ | $0.07\pm.001$ | $89.90\pm.40$ | $87.85\pm.48$ | $0.07\pm.002$
_cifar10 center cropped to 28x28_ | $88.45\pm.31$ | $70.44\pm.44$ | $0.13\pm.003$ | $88.92\pm.32$ | $70.21\pm.74$ | $0.13\pm.004$
_cifar10 downsampled to 16x16_ | $85.43\pm.32$ | $76.70\pm.13$ | $0.08\pm.001$ | $86.55\pm.27$ | $77.47\pm 14$ | $0.07\pm.001$
_cifar10 downsampled to 8x8_ | $80.061\pm.33$ | $52.03\pm.49$ | $0.22\pm.002$ | $81.48\pm.44$ | $52.99\pm.49$ | $0.21\pm.003$
_cifar10 with Imagenet replacements_ | $88.45\pm.31$ | $70.44\pm.44$ | $0.13\pm.003$ | $88.92\pm.32$ | $70.44\pm.44$ | $0.13\pm.004$
Table 9: Mean (over random validation/test split) test accuracy and
amplification bias (in percent) and standard error on natural and transformed
test sets, comparing our method and the original model.

Our analysis focused, for the most part, on overfit that exists even in the
scenario in which test accuracy does not decrease as training proceeds - which
means that ”early stopping” - culminating the training when performance over
validation data decreases - has a minimal effect. Still, we wanted to compare
our performance to early stopping (ES) on datasets with and without label
noise. We also included in this comparison the test-time augmentation (TTA)
method, in which a test example is being classified several times, each time
with a different augmentation, and given a final classification based on the
average class probabilities of the different classifications. The results in
Table 8 indicate that our method is comparable or better than both methods
even when label noise exists in the training dataset (which leads to
deteriorating performance as training proceeds), and that it is complementary
to test-time augmentation.

###  E.2 model size

Method/model size | small | base | large
---|---|---|---
_single network_ | $83.21\pm.01$ | $83.31\pm.15$ | $82.92\pm.09$
_KF (ours)_ | $83.17\pm.04$ | $\mathbf{83.57\pm.15}$ | $\mathbf{83.96\pm.09}$
Table 10: Mean (over random validation/test split) test accuracy (in percent)
and standard error on image classification datasets, comparing our method and
the original predictor (ConvNeXt, trained on Imagenet) with varying number of
parameters

A common practice nowadays is to use very large neural networks, with hundred
of millions parameters, or even more. However, enlarging models does not
always improve performance, as large number of parameters can lead to overfit.
In Fig. 2(c) we show that indeed larger versions of a model can cause
increasing forget fraction, which also improves the benefit of our model (see
Table 10), making it especially useful when one uses a large model.

###  E.3 Comparison to a regular ensemble

A regular ensemble, unlike ours, requires multiple training of independent
networks, which could be unfeasible. Thus, it serves as an ”upper bound” for
our method’s performance. In Fig. 6, we compare our method and a regular
ensemble of the same size, showing our method can achieve much of the
performance gain provided by the regular ensemble. Notably, when label noise
occurs our method can add most, if not all, of the regular ensemble
performance gain.

(a) TinyImagenet

(b) TinyImagenet, 20% sym noise

(c) CIFAR100

(d) CIFAR100, 20% asym noise

Figure 6: Comparing our method with limited number of checkpoints and an
ensemble of the same size of independent networks

###  E.4 Fairness

In this section we study our method’s effect on the model’s _fairness_ , i.e.
the effect non-relevant features have on the classification of test data
examples. We follow [30] and train and test our models on datasets in which
they might learn spurious correlations. To create those datasets, we divide
the classes into two groups: in each class of the first group 95% of the
training images goes through a transformation (and the rest remain unchanged),
and vice versa for the classes of the second group. The transformation we use
are: removing color, lowering the images resolution (by down sampling and up
sampling), and replacing images with downsampled images for the same class in
Imagenet. We use cifar10 in our evaluation as done in [30], and use also
cifar100 with the remove color transformation (the rest of the transformations
were less appropriate for this datasets, as it contains similar classes that
could actually become harder to seperate at a lower resolution). We use the
same method as before for our validation data, and thus the validation is of
the same distribution as the test data.

Our evaluation use the following metrics: (i) the test accuracy on two test
sets (with/out the transformation), which should be lower if the model learns
more spurious correletions, and (ii) the amplification bias defined in [34],
which is defined is follows:

$\frac{1}{|C|}\sum_{c\in C}\frac{max(c_{T},c_{N})}{c_{T}+c_{N}}-0.5$ (2)

When $C$ is the group of classes, $c_{T}$ is the number of images from the
transformed test set predicted to be of class c, and $c_{N}$ is the number of
images from the natural test set predicted to be of class c - we would like
those to be as close as possible, since the transformation shouldn’t change
the prediction, and thus the lower the score the better.

The results of our evaluation are presented in Table 9. To summarize, our
method improves the average performance on both datasets without deteriorating
the amplification bias, which indicates that our method has no negative
effects on the model’s fairness.