# Which Pretrain Samples to Rehearse when Finetuning Pretrained Models?

Andrew Bai

UCLA
<EMAIL_ADDRESS>
Chih-Kuan Yeh

Google Inc.
<EMAIL_ADDRESS>
Cho-Jui Hsieh

Google Inc., UCLA
<EMAIL_ADDRESS>
Ankur Taly

Google Inc.
<EMAIL_ADDRESS>

###### Abstract

Fine-tuning pretrained foundational models on specific tasks is now the de
facto approach for text and vision tasks. A known pitfall of this approach is
the forgetting of pretraining knowledge that happens during finetuning.
Rehearsing samples randomly from the pretrain dataset is a common approach to
alleviate such forgetting. However, we find that random mixing unintentionally
includes samples which are not (yet) forgotten or unlearnable by the model. We
propose a novel sampling scheme, mix-cd, that identifies and prioritizes
samples that actually face forgetting, which we call _collateral damage_.
Since directly identifying collateral damage samples is computationally
expensive, we propose a procedure to estimate the distribution of such samples
by tracking the statistics of finetuned samples. Our approach is lightweight,
easy to implement, and can be seamlessly integrated into existing models,
offering an effective means to retain pretrain performance without additional
computational costs.

##  1 Introduction

The pretrain-finetune framework has become a prevalent and widely adopted
practice in the training of machine learning models (Fu et al., 2023; Zhang et
al., 2023a). This framework is characterized by a two-step process:
pretraining on a large and diverse dataset to learn general features and
patterns, followed by fine-tuning on a more specific task with a smaller
dataset. During fine-tuning, the model leverages the patterns learned during
pretraining making the overall approach much more computationally efficient
and better performing than training model scratch. Knowledge transfer between
pretraining and finetuning tasks can occur even when the two tasks are
considerably different. For instance, BERT models (Devlin et al., 2018) that
are pre-trained on a generic masked language modeling objective can be
efficiently adapted to a variety of NLP tasks.

Unfortunately, finetuning on a specific task, which involves adjusting the
models’ parameters based on a smaller task-specific dataset, can lead to
unintentional deterioration in pretraining task performance. For the lens of
continual learning, this is essentially the problem of _catastrophic
forgetting_ (Goodfellow et al., 2013) of the pretraining task when learning
the finetuning tasks. Furthermore, finetuning with insufficient or biased data
may result in overfitting to the peculiarities of the finetuning dataset,
further degrading the model’s ability to generalize.

A common strategy to preserve pretraining performance is to randomly mix
pretrain samples during finetuning (He et al., 2021). This strategy is rooted
in _rehearsal-based_ continual learning approaches (Bonicelli et al., 2022;
Zhang et al., 2022; Yoon et al., 2022; Mai et al., 2022). In the continual
learning setting, rehearsal-based approaches are criticized for the memory
overhead and privacy concerns around storing samples from earlier tasks. This
does not apply to our setting as the pretraining dataset is always available
for rehearsal. However, we do have to account for the additional training cost
from adding pretraining samples. Consequently, the strategy is applied under a
fixed budget for number of pretraining samples to mix111We use the terms “mix”
and “rehearse” interchangeably in this paper. They both refer to adding
(pretrainig) samples to the finetuning set..

Through empirical investigations, we find that selecting pretraining samples
uniformly at random is not optimal, especially when operating under a fixed
training budget. In hindsight, this is not surprising as the pretraining
dataset often contains diverse examples, and not all of them are impacted
equally by the finetuning process. Consequently, in this work we study the
problem of: _efficiently identifying the pretraining samples to rehearse
during finetuning to balance pretrain and finetune task performances._

We propose a finetuning procedure, mix-cd, that mixes in _high-confidence
collateral damage_ during finetuning. Collateral damage is defined as being
predicted correctly by the pretrained model, but incorrectly by the finetuned
one. High-confidence samples are ones where the pretrained model was
confidently correct. Since collateral damage regions can shift during the
course of finetuning, our strategy adapts the set of pretraining samples being
mixed in during each finetuning iteration.

A key technical challenge in this approach is that obtaining collateral damage
information requires make a forward pass with the finetuned model on the
pretrain dataset. This incurs a significant computation cost. To overcome
this, we propose a procedure called mix-cd-sample that efficiently estimates
the collateral damage density within the pretrain distribution. The estimated
density is updated throughout the finetuning process to keep track of the
dynamic changes in where collateral damage occurs.

Overall, our scheme retains the desirable quality of being general,
lightweight, and easy to implement, and can serve as a drop-in replacement for
the vanilla random-sampling approach. Through experiments on different image
and text tasks, we demonstrate that our scheme outperforms random-sampling and
several other baselines on balancing pretraining and finetuning task
peformances.

##  2 Background and Related Work

###  2.1 Applications of pretrain-finetune framework

The pretrain-finetune framework finds applications across various domains and
tasks within the field of machine learning. In natural language processing,
pretrained language models such as BERT (Devlin et al., 2018), T5 (Raffel et
al., 2020), and others are extensively fine-tuned for specific tasks like
sentiment analysis (Sun et al., 2019), text summarization (Liu and Lapata,
2019), and question answering (Roberts et al., 2020). Large generative
language models such as GPT (Brown et al., 2020) and Llama (Touvron et al.,
2023) are instruction-tuned (Wei et al., 2021) on human-provided feedback to
align their generation with human response. In computer vision, pretrained
vision transformers are commonly fine-tuned for image classification, object
detection (Li et al., 2022), and segmentation tasks (Thisanke et al., 2023).
Transfer learning through pretrain-finetune is also prevalent in medical
imaging (He et al., 2023) for tasks like disease diagnosis and organ
segmentation. Additionally, it is widely employed in recommendation systems
(Zhang et al., 2023b), speech recognition (Guillaume et al., 2022), and
reinforcement learning (Luo et al., 2023).

###  2.2 Retain Pretrain Performance

One major challenge for the pretrain-finetune framework is to retain the
pretrain performance while improving the finetune task. In some cases where
the finetuned model is only expected to perform well on a limited set of
finetune examples, disregarding the degradation in pretrain task is
acceptable. However, studies have shown that keeping the pretrain performance
is beneficial to not overfit on the finetune data and other desiderata (Lin et
al., 2023; He et al., 2021). While weight regularization and rehearsal are the
two main types of technique to prevent pretrain task forgetting, we focus on
the latter for this work as the two techniques are orthogonal and can be
combined to amplify their effects.

##### Forgetting prevention by weight regularization

Weight regularization (Lin et al., 2023) prevents pretrain forgetting by
directly operating on restricting the weights, either by constraining model
weight during the finetuning process directly or manipulating the finetuned
weights post-hoc. The model weights can be constraint during the finetuning by
anchoring it to the pretrain weight (Panigrahi et al., 2023; Xuhong et al.,
2018; Kirkpatrick et al., 2017). The constraint can also be in the form of
low-rank weight adaptation with LoRA (Hu et al., 2021). On the other hand,
Wortsman et al. (2022) proposed WiSE-FT to ensemble the pretrained and
finetuned weights post-hoc and achieve superior performance on both tasks. We
explore combining rehearsal methods with WiSE-FT in Appendix C.1.

##### Forgetting prevention by rehearsal

Rehearsal-based methods prevents pretrain forgetting by mixing a portion of
pretrain data into the finetuning phase. The typical approach for multi-task
learning is to random uniformly sample from the pretrain data and mix them
into the finetuning (He et al., 2021; Kazemi et al., 2023). In continual/life-
long learning settings that assume only a small subset of pretrain data is
accessible during finetune time, maintaining a replay buffer to store
important pretrain samples is necessary. This is typically the case for
continual learning tasks (Zhuang et al., 2022). For continual learning, online
selection of rehearsal samples have been a well studied topic. Aljundi et al.
(2019) proposed Maximally Interfered Sampling, where they sample high loss
difference samples from a small replay buffer. The idea is similar to our
method but different. We samples more efficiently by reusing information
collected from samples in previous iterations and we prioritize not only high
loss difference but actual collateral damage samples.

##  3 Evaluation Protocol and Key Ideas

Our objective is to: _Improve performance on the finetune task while avoiding
performance deterioration on the pretrain task._ In this section, we define
our evaluation protocol, and motivate the design of our method via some key
empirical observations.

###  3.1 Evaluation protocol

Our objective requires balancing finetune task performance with pretraing task
performance. Thus, we compare different mixing strategies by examining the
Pareto frontier of the pretrain and finetune tasks performance; see Fig 1. We
track the performance of each strategy after finetuning on every fix number of
finetune samples, which we call “iterations”. At each iteration, we plot the
finetune task performance (y-axis) and pretrain task performance (x-axis),
which gives us a performance curve for each strategy.

In some cases, the curve from one strategy may completely dominate the curve
from another. For example, the green curve strictly dominates the orange curve
on the left hand side of Fig 1, and thus, the green strategy is strictly
better. In cases where strategies are not mutually dominating, we can compare
the average performance (weighted by the evaluation set size) across the
pretrain and finetune task instead. This can be done visually by examining the
curves relative to the light-blue bands in the background. Each band represent
the same average accuracy across both tasks, i.e., an equivariant-performance
zone. For example, although the red curve does not dominate the orange curve
on the left hand side of Fig 1, the red curve is situated in a more top-right
blue band, indicating better average performance.

Figure 1: Preliminary observations suggest that while randomly mixing in
pretrain data helps mitigate collateral damage, prioritizing certain parts of
the pretrain data distribution benefits the joint performance on the pretrain
and finetune tasks even more.

###  3.2 Key Ideas

To establish intuition for which samples to rehearse to prevent pretrain task
forgetting, we prioritized different pretrain regions and show the performance
in Fig 1. The complete experiment details can be found in Section 5.1. As
expected, the strategy finetune-only which only considers pretraining samples,
does poorly on pretrain task performance when compared with random, which
mixes randomly drawn pretraining samples for rehearsal. However, we argue that
random is still sub-optimal as it does not take into account the “utility” of
the pretrain samples. Pretrain samples that were originally predicted
incorrectly by the pretrained model are unlikely to help pretrain task
performance. To confirm this, we construct the strategy reject pretrain
incorrect, which modifies random to reject samples that were predicted
incorrectly by the pretrained model. We find that this strategy dominates
random.

Following the same intuition, we expect pretrain samples that were predicted
correctly initially and also _after_ finetuning, to not help pretrain task
performance either. This is because such samples are not impacted by
finetuning, and therefore cannot undo the deterioration caused by it. To also
confirm this, we construct the strategy reject finetune correct, which rejects
samples that were predicted correctly by the finetuned model. We find that
this strategy increases the performance on the pretrain task significantly.

This leaves us with samples that we were predicted correctly initially but
were predicted incorrectly after finetuning. Such samples form the _collateral
damage_ of finetuning, i.e., these are samples for which the correct
predictions were essentially “forgotten” during finetuning. The forgotten
aspect suggests that since the model predicted correctly on these samples
before, it could predicted correctly again after reviewing them. This brings
us to our first technical idea.

Idea 1: Mix collateral damage pretrain samples that are forgotten during
finetuning.

Indeed, the strategy select collateral damage which mixes in randomly chosen
collateral damage samples is better than other strategies in Fig 1. Our
ultimate goal is to remedy the deterioration in pretrain task performance on
the holdout testing set. But, does fixing every collateral damage training
sample generalize equally? Not necessarily. For example, fixing collateral
damage samples that are incorrectly labeled will not generalize. We posit that
samples that are predicted correct initially but have low prediction
confidence may not generalize well. This is because such samples may have
noisy labels or may be hard samples for the model. In classification settings,
these are samples that fall close to the decision boundary. Instead, we focus
on samples that are predicted confidently, and lie further away from the
decision boundary. This brings us to our second idea.

Idea 2: Mix collateral damage samples that are predicted with high confidence.

The confident variant does not seem to be significantly better than the
original select collateral damage, but we will later see the importance of
confidence filtering when collateral damage samples are estimated instead. In
the next section, we use these two ideas to propose our finetuning procedure.

##  4 Methodology

We introduce our procedure in two steps. We first discuss a basic procedure,
called mix-cd-exact (Algorithm 1), that is a straightforward
operationalization of the ideas from 3.2. It performs well but is
computationally expensive (and impractical), as it involves inferencing the
finetuning model over a large number of pretrain samples at each iteration.
Our main procedure, called mix-cd-sample, avoid this inference cost by
estimating the collateral damage distribution at each iteration using only the
samples mixed in during the previous iterations. Since these mixed in samples
are already part of the finetuning set, we get inference (forward pass) on
them for free as part of the standard training loop. This makes the procedure
as efficient as mixing in randomly chosen pretraining samples during
finetuning.

Algorithm 1 mix-cd-exact

1: Input: number of iterations $N$, pretrain dataset $Z_{p}$, finetune dataset
$Z_{f}$, pretrain model $f$, mix ratio $\beta$, pretrain threshold $c_{p}$,
finetune threshold $c_{f}$, number of finetune samples per iteration $n_{f}$.

2: Initialize finetune model $f^{\prime}\leftarrow f$

3: for $n=1$ to $N$ do

4: Initialize dataset $D_{f}\leftarrow$ $n_{f}$ samples from $Z_{f}$

5: Initialize dataset $D_{p}\leftarrow\\{\\}$

6: repeat

7: $z_{p}\leftarrow$ uniform sample from $Z_{p}$

8: if $\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})=1$ then

9: $D_{p}\leftarrow D_{p}\cup\\{z_{p}\\}$

10: end if

11: until $|D_{p}|\geq\beta\cdot|D_{f}|$

12: Train $f^{\prime}$ for one iteration on the set $D_{f}~{}\cup~{}D_{p}$

13: end for

###  4.1 Formal Definition of Collateral Damage.

Let us denote the pretrained model as $f$, finetuned model as $f^{\prime}$,
pretrain data samples as $z_{p}$, and fine tune data samples as $z_{f}$. Let
$\phi$ denote the indicator function for collateral damage. For classification
tasks, a sample $z_{p}=(x,y)$ suffers from collateral damage, denoted by
$\phi_{f,f^{\prime}}(z_{p})$, if it is predicted correctly by $f$ but
incorrectly by $f^{\prime}$.

$\displaystyle\phi_{f,f^{\prime}}(z_{p}):=$
$\displaystyle\big{(}\operatorname*{arg\,max}f(x)\equiv
y\big{)}\wedge\big{(}\operatorname*{arg\,max}f^{\prime}(x)\neq y\big{)}$

Next, we incorporate the notion of confidence into classification collateral
damage. Given prediction confidence thresholds $c_{p}$ and $c_{f}$, a sample
$z_{p}$ is a confident collateral damage if $z_{p}$ is a collateral damage and
the top predicted class confidence is greater than $c_{p}$ on $f$ and greater
than $c_{f}$ on f’.

$\displaystyle\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p}):=\phi_{f,f^{\prime}}(z_{p})$
$\displaystyle\wedge\big{(}\max f(z_{p})>c_{p}\big{)}$
$\displaystyle\wedge\big{(}\max f^{\prime}(z_{p})>c_{f}\big{)}$

We can extend the definition of collateral damage for generative tasks using
the losses incurred by the pretrained and finetuned models. Given loss
thresholds $\tau_{p}$ and $\tau_{f}$, a sample $z_{p}$ suffers from collateral
damage if its loss on $f$ is less than $\tau_{p}$ and loss on $f^{\prime}$ is
greater than $\tau_{f}$.

$\phi_{f,f^{\prime}}^{\tau_{p},\tau_{f}}(z_{p})=\big{(}\mathsf{loss}(f,z_{p})<\tau_{p}\big{)}\wedge\big{(}\mathsf{loss}(f^{\prime},z_{p})>\tau_{f}\big{)}$

Algorithm 2 mix-cd-sample

1: Input: number of iterations $N$, pretrain dataset $Z_{p}$, finetune dataset
$Z_{f}$, pretrain model $f$, mix ratio $\beta$, number of partitions $K$,
pretrain threshold $c_{p}$, finetune threshold $c_{f}$, number of finetune
samples per iteration $n_{f}$.

2: // Initialize estimates $\hat{\alpha}_{k}$

3: for $k=1$ to $K$ do

4: Initialize $\hat{\alpha}_{k}\leftarrow 0.5$

5: Initialize $u_{k}\leftarrow 0,n_{k}\leftarrow 0$

6: end for

7: Initialize finetune model $f^{\prime}\leftarrow f$

8: for $n=1$ to $N$ do

9: Initialize dataset $D_{f}\leftarrow$ $n_{f}$ samples from $Z_{f}$

10: Initialize dataset $D_{p}\leftarrow\\{\\}$

11: repeat

12: $z_{p}\leftarrow$ sample from $Z_{p}$ with probability
$\hat{\alpha}_{b(z_{p})}\cdot P(b(z_{p}))$

13: $D_{p}\leftarrow D_{p}\cup\\{z_{p}\\}$

14: until $|D_{p}|\geq\beta\cdot n_{f}$

15: Train $f^{\prime}$ for one iteration on the set $D_{f}~{}\cup~{}D_{p}$

16: // Update estimates $\hat{\alpha_{k}}$

17: for $k=1$ to $K$ do

18: Update $u_{k},n_{k}$ according to Eq 1 and Eq 2

19: $\hat{\alpha}_{k}\leftarrow u_{k}/n_{k}$

20: end for

21: end for

###  4.2 Basic Procedure: mix-cd-exact

Algorithm 1 provides our basic finetuning procedure mix-cd-exact. We denote
the pretraining and finetuning sets by $Z_{p}$ and $Z_{f}$ respectively. The
finetuning procedure proceeds in iterations, with each iteration involving a
training run through $n_{f}$ samples from $Z_{f}$. Let $\beta$ denote the
pretrain data proportion (with respect to the number of finetuning samples) to
mix into the finetuning process. For example given $\beta=0.1$, 100 pretrain
samples are reviewed for every 1,000 finetune samples trained. At the
beginning of each iteration, $n_{f}$ finetune samples and $\beta\cdot n_{f}$
pretrain samples are mixed together to form the training set for finetuning.

For the first iteration, the pretrain samples are picked uniformly at random
(as there is no collateral damage information), while for later iterations,
the samples are drawn from the subset that suffers from high confidence
collateral-damage (i.e., $\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})=1$). We
assume that the pretrain model’s losses and predictions over pretrain samples
$z_{p}$ are cached from the pretraining run, and are available at no
additional computational cost. The main computation cost then is in assessing
collateral damage ($\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})$).

Let $\gamma$ denote the proportion of high confidence collateral damage
samples in the pretrain set. Obtaining $\beta\cdot n_{f}$ high confidence
collateral damage samples would in expectation require inferencing
$f^{\prime}$ on $(\beta/\gamma)\cdot n_{f}$ samples.222To assess collateral
damage, we also need to perform inference with $f$ but we assume that those
predictions are cached from the pretraining run. For $\gamma=0.1$ and
$\beta=0.1$, this amounts to running inference on a dataset the same exact
size as the finetune samples for training. This is prohibitively expensive.
Besides, this computational cost may be better spent on finetuning with a
higher proportion ($\beta$) of randomly chosen pretrain samples. Assuming the
computation cost for a forward pass is similar to a backward pass, the same
inferencing computation budget could be spent on additional finetuning on half
of the samples instead. In the next section, we discuss a mechanism for
avoiding this extra inference cost.

###  4.3 Main procedure: mix-cd-sample

Our main procedure mix-cd-sample is defined in Algorithm 2. The key difference
from mix-cd-exact is that we estimate the probability that a pretraining
sample $z_{p}$ suffers from collateral damage without inferencing on
$f^{\prime}$. We first partition the pretrain data distribution into $K$ bins.
At each fine-tuning iteration, we estimate the conditional probability
(denoted by $\alpha_{k}$) that a sample from bin $k$ suffers from collateral
damage. Formally,

$\alpha_{k}:=P(\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})=1~{}|~{}b(z_{p})=k)$

where $b(z_{p})\in[K]$ is the bin that sample $z_{p}$ falls in. We discuss
different partitioning schemes in Section 5.4. Once we have estimates
$\hat{\alpha}_{k}$, we select a randomly drawn pretraining sample $z_{p}$ with
probability $\hat{\alpha}_{b(z_{p})}\cdot P(b(z_{p}))$.

We note that even with perfect estimation of $\alpha_{k}$, our sampling scheme
via binning is not guaranteed to draw only collateral damage samples. Yet, in
our experiments (see Section 5.2.2), we find that our finetuning procedure is
comparable in performance to the basic procedure mix-cd-exact that always
samples collateral damage points. Our conjecture is that pretrain task
forgetting only requires few collateral damage points to fix (quality over
quantity). This is aligned with previous works that claim the efficacy of
random uniform sampling with high $\beta$ values, where mixing more pretrain
samples increases the chances of including collateral damage samples (He et
al., 2021). We discuss this further in Section 5.3.

##### Estimating $\alpha_{k}$.

A straightforward scheme for estimating $\alpha_{k}$ is to sample uniformly
from the pretrain distribution, perform inference on the samples using the
finetuning model, then compute the fraction of samples falling in bin $k$ that
suffer from collateral damage. This provides an unbiased estimate of
$\alpha_{k}$. However, it still requires additional inference on the sampled
pretrain points. To completely avoid any additional inference, we propose
estimating $\alpha_{k}$ at each iteration using the pretrain samples mixed
into the finetuning step during the previous iteration. For the first
iteration, the pretrain data samples are drawn uniformly at random with
$\alpha_{k}$ set to $0.5$ for all $k$. For subsequent iterations, we maintain
running counts of the number of samples ($n_{k}$) mixed in from bin $k$, and
the number of collateral damage samples ($u_{k}$) among them. Specifically, at
the end of each iteration, we update these counts as follows. Let $D_{p}$ be
the pretraining samples mixed in during the iteration.

$n_{k}\leftarrow n_{k}+|\\{z_{p}\in D_{p}~{}|~{}b(z_{p})=k\\}|$ (1)
$u_{k}\leftarrow u_{k}+|\\{z_{p}\in
D_{p}~{}|~{}b(z_{p})=k,\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})=1\\}|$ (2)

for all $k\in[K]$. We then set our estimate $\hat{\alpha}_{k}:=u_{k}/n_{k}$.
These updates do not incur any additional cost since $D_{p}$ is already part
of the finetuning set. Consequently, we would have the forward pass
(inference) output of the finetune model already available. This enables us to
compute $\phi_{f,f^{\prime}}^{c_{p},c_{f}}(z_{p})$ at no additional inference
cost.

We note that the our estimation procedure is not unbiased as we use samples
seen during finetuning to estimate collateral damage distribution for unseen
samples. However, in our experiments (see Section 5.2.2) we find that the bias
isn’t large, and the resulting estimates $\hat{\alpha}_{k}$ are close to the
true collateral damage ratios $\alpha_{k}$ at each iteration (see Section
5.3). We conjecture that even though a collateral damage sample might be fixed
after finetuning on it, the finetune task continues to cause more damage in
the corresponding bin. Thus, sampling more data from such bins is effective at
countering forgetting.

####  4.3.1 Partitioning Pretrain Data

The intuition behind our procedure mix-cd-sample is that by partitioning the
pretrain distribution into bins, we can identify regions that suffer more from
collateral damage. It is then possible to prioritize such regions over others
when identifying pretraining samples to mix during finetuning. Our procedure
benefits from any type of partitioning as long as the collateral damage is not
conditionally independent on the partitions. We note that the multiple
partitions can be combined to form even more finer-grained partitions. In our
experiments, we find that partitions based on combining pretraining loss and
auxiliary labels perform the best (see Section 5.4). We discuss some partition
strategies that work well with mix-cd-sample, are common to obtain in
datasets.

##### Partition with pretrain loss

The pretrain data can be partitioned according to their losses on the
pretrained model, by setting partition thresholds with fixed-sized loss
quantiles. These losses are typically available for free from the last
iteration pretraining. The losses contain valuable information for predicting
whether collateral damage will occur. For instance, in classification
settings, correctly predicted samples with lower loss are typically further
away from the decision boundary. Consequently, these samples are less likely
to be forgotten during finetuning.

##### Partition with auxiliary information

The pretrain data can also be partitioned with auxiliary information such as
class labels and/or other meta labels for partioning pretrain samples. Usually
these meta labels convey semantic meaning that helps distinguish whether
certain regions would suffer more from collateral damage. For example, the
ground truth class labels naturally partition multi-class classification
datasets. For multilingual translation datasets, the language serves as a
natural partition. For instruction-tuning datasets, the source instruction-
tuning task also naturally partitions the pretrain data.

##  5 Experiments and Discussion

Figure 2: Pareto frontiers of pretrain and finetune performance. Each data
point indicate one iteration of finetuning with 5 iterations in total. Mix-cd-
sample outperforms other baselines.

We will first demonstrate the efficacy of mix-cd in multiple finetuning
settings. Then we will investigate the impact of the prediction of collateral
damage with bins and the estimation of collateral damage ratios for the bins.
The highest and lowest collateral damage bins are visualized to reveal which
part of the pretrain distribution is prioritized.

###  5.1 Experiment Setup

We experiment on three different tasks that commonly utilize a pretrain-
finetune pipeline: image classification, text classification, and multiligual
translation. More technical details can be found in Appendix A.

##### Image classification: ImageNet-CUB

We pretrain a ResNet-18 model on ImageNet-1k (Russakovsky et al., 2015), and
finetune it on CUB-200-2021 (Wah et al., 2011), a multi-class bird image
classification dataset. The ground truth class labels of ImageNet-1k are used
for partitioning. The pretrain and finetune task performances are defined as
the top-1 classification accuracy on the holdout sets.

##### Text classification: MNLI-Scitail

We pretrain a DistilBert model on MNLI (Kim et al., 2019) for natural language
inference (NLI), and finetune it on Scitail (Khot et al., 2018), a NLI dataset
for scientific statements. The ground truth class labels and genre labels are
used for partitioning. The pretrain and finetune task performances are defined
as the classification accuracy on the holdout sets.

##### Multilingual translation: translating Danish to English

The experimental setting for multilingual translation is slightly different
from classification tasks. Instead of finetuning on a new dataset, we take
mBart50 (Tang et al., 2020), a multilingual translation model that translates
from 50 different languages to English, and finetune it to perform translation
on one additional language. The training data pairs (both pretrain and
finetune) are taken from Opus100, a multilingual, English-centric dataset that
consists of sentence pairs translating from 100 other languages to English. We
finetune the model on Danish, which is previously not supported by the
pretrained mBart50 model. The pretrain language labels are used for
partitioning. The pretrain task performance is defined as the average loss of
all pretrain language samples and the finetune task performance is defined as
the average loss of Danish samples.

##### Training configuration

All pretrained models are finetuned for 5 iterations. For each experiment, we
report the joint performance of the pretrain and finetune task on holdout test
sets, evaluated at the end of each finetuning iteration. The results are
averaged over 10 repetitions for classification tasks and 5 for generative
tasks. The pretrain data proportion $\beta$ is set to 0.1 for all methods and
all experiments. In some previous works $\beta$ were set to higher values
(e.g. 4) to recover more pretrain performance in less iterations, at the cost
of more expensive computation (He et al., 2021). We opted for a lower $\beta$
to highlight the importance of prioritization when the budget for pretrain
samples is lower.

###  5.2 Mix-cd Outperforms Baselines

To demonstrate the general effectiveness of mix-cd in diverse pretrain-
finetune settings, we compared it with other finetuning strategies of equal
computation cost. We assume the computation cost of a forward pass is
approximately the same as a backward pass. Recall an iteration of finetuning
refers to training the model on every $n_{f}$ finetune samples.

####  5.2.1 Baseline Descriptions

The most basic baselines include finetune-only which only includes finetune
samples and random which samples random pretrain data uniformly to mix with
the finetune samples. Other baselines can be categorized into two types:
stream-based and finetune-task-agnostic.

##### Stream-based baselines

These methods sample a batch of pretrain candidates at each step, calculate
the priority for each sample, then select the top priority samples from the
batch to train. Since the priority calculation requires additional forward
passes, there is a tradeoff between how big a batch to sample initially and
how many high priority samples to keep and train. We searched for the
proportion to balance computation budget and report the performance of the
best hyperparameter. mix-uncertainty prioritizes pretrain samples with high
uncertainty, a common objective for active learning. The uncertainty is
estimated with prediction entropy for classification tasks and sequence log
likelihood for generative tasks. mix-rho prioritizes pretrain samples with
high reducible holdout loss (Mindermann et al., 2022), a state-of-the-art
technique for active learning tasks. The reducible holdout loss is defined as
the difference between loss on the current finetune model and loss on the
pretrain model.

##### Finetune-task-agnostic baselines

These baselines select important pretrain data before the finetuning begins,
typical for continual learning settings where the previous learned tasks are
assumed to be inaccessible after being trained on. Let $N$ denote the number
of iterations of finetuning. Each method pre-select $N\cdot n_{f}\cdot\beta$
highest priority samples from the pretrain dataset. mix-high-pretrain-loss
adopts pretrain loss while mix-coreset adopts the online coreset selection
objective (Yoon et al., 2022) as the priority.

Figure 3: Visualizing classes from ImageNet-1k suffering the most and least
collateral damage when finetuned on CUB (bird classification). Bird classes
suffer the most collateral damage due to competition with the finetune task.

####  5.2.2 Result analysis

Mix-cd-sample consistently outperforms the random baseline over all experiment
settings. This supports mix-cd-sample as the drop-in replacement for random
since the performance gain comes at no additional computation cost is
incurred. Stream-based baselines perform similar to or worse than random since
for the given computation budget it is better off to include everything from a
pretrain data batch for finetuning (which is basically random), instead of
spending budget filtering for high priority samples. The performance of
finetune-task-agnostic baselines vary significantly for different tasks. mix-
coreset generally performs the worse since the selection objective does not
take the finetune task information into consideration. mix-high-pretrain-loss
on the other hand is the worst for classification tasks but the best for
translation. Whether the heuristic of selecting high loss pretrain samples is
effective depends on inter-dataset competition and the noisiness of ground
truth labels. High loss would likely indicate mislabel if the dataset labels
were noisy. High loss samples would also cause more collateral damage within
the pretrain dataset if the inter-dataset competition is high (e.g. 1000
classes in ImageNet). In this case, the translation dataset has both clean
labels and low inter-dataset competition, which explains the exceptional
efficacy of mix-high-pretrain-loss.

###  5.3 Are the collateral damage estimations accurate?

Mix-cd-sample relies on accurately estimating the useful ratios for each cell
to sample accordingly. The estimation relies on tracking the forward process
output of mixed pretrain samples and may be inaccurate due to insufficient
number of pretrain samples. The forward output process might also yield
outdated information since the model had performed back-propagation to update
according to the mixed pretrain samples. We compare the estimated useful
ratios with actual useful ratios to demonstrate the accuracy of the
estimation. The actual useful ratios require full-inference on the pretrain
training dataset, infeasible in normal settings. Fig 4 shows the estimated
collateral ratios closely matches the exact values. The error decreases as the
finetuning progresses, indicating better estimation with more samples
collected.

Figure 4: Mean absolute error between estimated and exact collateral damage
ratios. Low estimation error explains the small performance gap between mix-
cd-sample and mix-cd-sample-oracle.

Fig 5 compares the performance between variants of mix-cd. mix-cd-sample-
oracle uses the exact collateral damage ratios calculated with the entire
pretrain dataset to sample from the pretrain distribution. It is an
theoretical upper bound for the performance of mix-cd-sample and empirically
performs slightly better since the ratios is not estimated. mix-cd-exact
performs slightly better than other variants on text classification since
collateral damage conditioned on partition is not unbiased. However, the small
performance difference suggests the impact of the bias on maintaining pretrain
performance is insignificant. Interestingly, mix-cd-exact is worse on image
classification due to the inter-dataset competition in ImageNet. Mixing exact
collateral damage samples introduces strong signal that could exacerbate the
pretrain forgetting. Other design choices of mix-cd are extensively explored
in Appendix B.

One important step of mix-cd is to update the collateral damage ratios at the
end of every iteration and select new pretrain samples to mix accordingly.
Does the collateral damage ratios actually change during finetuning? To
illustrate the importance of updating the ratios, we compare mix-cd-sample
with a variant that freezes the ratios after the first iteration. Fig 5
highlights the importance of updating the collateral damage ratios to reflect
collateral damage caused by the most up-to-date finetune model.

Figure 5: Ablation study on variants of mix-cd. oracle uses exact collateral
damage statistics for sampling while freeze stops updating the ratios after
the first iteration.

###  5.4 What partitions are prioritized?

Fig 3 shows the top and bottom 3 collateral damage classes for ImageNet-CUB.
The bird classes (38 in total) suffer the most collateral damage due to
competition with the finetune task, a fine-grained bird classification
dataset. If we rank the cells according useful ratios from high to low, the
median rank of bird class cells is 106.5 out of 1,000 cells. It might seem
obvious to prioritize bird cells when the finetuning class also happens to be
classifying birds. However, explicit prioritization of cells is not as simple
when the competition between the two tasks is not as intuitive. Estimating
collateral damage ratios adapts to general tasks better.

##  6 Conclusion

In this paper we proposed a rehearsal-based sampling strategy to prioritize
collateral damage samples during finetuning. The simplicity and effectiveness
makes it an appealing drop-in replacement for the typical uniform sampling
strategy. Future works can investigate better hybrid methods to combine both
rehearsal-based and weight-regularization forgetting prevention techniques.

##### Limitations

We assume the last-epoch prediction of the pretrain data is cached during the
pretraining phase. The assumption is rather realistic but nevertheless may not
hold in all applications. More investigation is also needed to examine whether
the original pretrain performance can be fully recovered with mix-cd.

## References

  * Aljundi et al. (2019) Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars.  Online continual learning with maximally interfered retrieval.  In _Proceedings of the 33rd International Conference on Neural Information Processing Systems_ , pages 11872–11883, 2019.
  * Bonicelli et al. (2022) Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara.  On the effectiveness of lipschitz-driven rehearsal in continual learning.  _Advances in Neural Information Processing Systems_ , 35:31886–31901, 2022.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fu et al. (2023) Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier.  On the effectiveness of parameter-efficient fine-tuning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 12799–12807, 2023.
  * Goodfellow et al. (2013) Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio.  An empirical investigation of catastrophic forgetting in gradient-based neural networks.  _arXiv preprint arXiv:1312.6211_ , 2013.
  * Guillaume et al. (2022) Séverine Guillaume, Guillaume Wisniewski, Cécile Macaire, Guillaume Jacques, Alexis Michaud, Benjamin Galliot, Maximin Coavoux, Solange Rossato, Minh-Châu Nguyên, and Maxime Fily.  Fine-tuning pre-trained models for automatic speech recognition, experiments on a fieldwork corpus of japhug (trans-himalayan family).  In Sarah Moeller, Antonios Anastasopoulos, Antti Arppe, Aditi Chaudhary, Atticus Harrigan, Josh Holden, Jordan Lachler, Alexis Palmer, Shruti Rijhwani, and Lane Schwartz, editors, _Proceedings of the Fifth Workshop on the Use of Computational Methods in the Study of Endangered Languages_ , pages 170–178, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.computel-1.21.  URL https://aclanthology.org/2022.computel-1.21.
  * He et al. (2023) Kelei He, Chen Gan, Zhuoyuan Li, Islem Rekik, Zihao Yin, Wen Ji, Yang Gao, Qian Wang, Junfeng Zhang, and Dinggang Shen.  Transformers in medical image analysis.  _Intelligent Medicine_ , 3(1):59–78, 2023.
  * He et al. (2021) Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James Glass, and Fuchun Peng.  Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1121–1133, 2021.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Kazemi et al. (2023) Mehran Kazemi, Sid Mittal, and Deepak Ramachandran.  Understanding finetuning for factual knowledge extraction from language models.  _arXiv preprint arXiv:2301.11293_ , 2023.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark.  Scitail: A textual entailment dataset from science question answering.  In _AAAI Conference on Artificial Intelligence_ , 2018.  URL https://api.semanticscholar.org/CorpusID:24462950.
  * Kim et al. (2019) Seonhoon Kim, Inho Kang, and Nojun Kwak.  Semantic sentence matching with densely-connected recurrent and co-attentive information.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 6586–6593, 2019.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Li et al. (2022) Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In _European Conference on Computer Vision_ , pages 280–296. Springer, 2022.
  * Lin et al. (2023) Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al.  Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models.  _arXiv preprint arXiv:2309.06256_ , 2023.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_ , 2019.
  * Luo et al. (2023) Yicheng Luo, Jackie Kay, Edward Grefenstette, and Marc Peter Deisenroth.  Finetuning from offline reinforcement learning: Challenges, trade-offs and practical solutions.  _arXiv preprint arXiv:2303.17396_ , 2023.
  * Mai et al. (2022) Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ , 469:28–51, 2022.
  * Mindermann et al. (2022) Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al.  Prioritized training on points that are learnable, worth learning, and not yet learnt.  In _International Conference on Machine Learning_ , pages 15630–15649. PMLR, 2022.
  * Panigrahi et al. (2023) Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora.  Task-specific skill localization in fine-tuned language models.  _arXiv preprint arXiv:2302.06600_ , 2023.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer.  How much knowledge can you pack into the parameters of a language model?  _arXiv preprint arXiv:2002.08910_ , 2020.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115:211–252, 2015.
  * Sun et al. (2019) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang.  How to fine-tune bert for text classification?  In _Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18_ , pages 194–206. Springer, 2019.
  * Tang et al. (2020) Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan.  Multilingual translation with extensible multilingual pretraining and finetuning.  _arXiv preprint arXiv:2008.00401_ , 2020.
  * Thisanke et al. (2023) Hans Thisanke, Chamli Deshan, Kavindu Chamith, Sachith Seneviratne, Rajith Vidanaarachchi, and Damayanthi Herath.  Semantic segmentation using vision transformers: A survey.  _Engineering Applications of Artificial Intelligence_ , 126:106669, 2023.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wah et al. (2011) C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  Caltech ucsd birds dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * Wortsman et al. (2022) Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al.  Robust fine-tuning of zero-shot models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7959–7971, 2022.
  * Xuhong et al. (2018) LI Xuhong, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  In _International Conference on Machine Learning_ , pages 2825–2834. PMLR, 2018.
  * Yoon et al. (2022) Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang.  Online coreset selection for rehearsal-based continual learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=f9D-5WNG4Nv.
  * Zhang et al. (2023a) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_ , 2023a.
  * Zhang et al. (2023b) Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He.  Collm: Integrating collaborative embeddings into large language models for recommendation.  _arXiv preprint arXiv:2310.19488_ , 2023b.
  * Zhang et al. (2022) Yaqian Zhang, Bernhard Pfahringer, Eibe Frank, Albert Bifet, Nick Jin Sean Lim, and Yunzhe Jia.  A simple but strong baseline for online continual learning: Repeated augmented rehearsal.  _Advances in Neural Information Processing Systems_ , 35:14771–14783, 2022.
  * Zhuang et al. (2022) Chen Zhuang, Shaoli Huang, Gong Cheng, and Jifeng Ning.  Multi-criteria selection of rehearsal samples for continual learning.  _Pattern Recognition_ , 132:108907, 2022.

##  Appendix A Experiment Technical Details

###  A.1 Image classification: ImageNet-CUB

We pretrain a ResNet-18 model on ImageNet-1k (Russakovsky et al., 2015), and
finetune it on CUB-200-2021 (Wah et al., 2011), a multi-class bird image
classification dataset. An additional linear classification head is attached
to the final average pooling layer to perform classification on the CUB task.
The model is finetuned with AdamW with learning rate of $10^{-4}$ and weight
decay of $10^{-5}$. To implement mix-cd-sample, we use the 1,000 ground truth
class labels of ImageNet-1k for partitioning. During each iteration, we
finetune on 1,200 samples from the CUB-200-2021 training set (iterating over
the entire training set of 5,994 samples in 5 iterations). The pretrain
dataset consists of 100,000 random uniform samples (10%) drawn from the
ImageNet-1k training set. The pretrain and finetune task performances are
defined as the top-1 classification accuracy on the holdout sets.

###  A.2 Text classification: MNLI-Scitail

We pretrain a DistilBert model on MNLI (Kim et al., 2019), which is a natural
language inference (NLI) dataset, and finetune on Scitail (Khot et al., 2018),
a natural language entailment dataset for scientific statements. NLI tasks aim
to determine the relationship (entailment, contradiction, or neutral) between
a pair of input sentences. The model is finetuned with AdamW with learning
rate of $2\cdot 10^{-6}$ and weight decay of $10^{-5}$. There are 393,000
samples in the MNLI pretrain training dataset. In additional to relation
label, additional genre labels (e.g. fiction, government, travel) for the
sentence pairs are also provided. To implement mix-cd-sample, we use the
ground truth class labels and genre labels for partitioning. For each
iteration, we finetune on 5,000 samples from the Scitail training set
(iterating over the entire training set of 23,600 samples after 5 iterations).
The pretrain and finetune task performances are defined as the classification
accuracies on MNLI and Scitail, respectively.

###  A.3 Multilingual translation: translating Danish to English

The experimental setting for multilingual translation is slightly different
from classification tasks. Instead of finetuning on a new dataset, we take a
multilingual translation model that translates from 50 different languages to
English, and finetune it to perform translation on one additional language. To
implement mix-cd-sample, we use the language type for partitioning. We would
like to prevent any deterioration in the performance of the existing 50
languages due to finetunung. It is expected for the translation for some
languages in the pretrain language to deteriorate after finetuning. We
leverage the pretrain language as auxiliary information for partitioning to
identify and fix the languages with more collateral damage.

The pretrain model of choice is mBart50 (Tang et al., 2020), a generative
language model pretrained on translation sentence pairs of 50 different
languages to English. The model is finetuned with AdamW with learning rate of
$10^{-5}$ and weight decay of $10^{-5}$. The training data pairs (both
pretrain and finetune) are taken from Opus100, a multilingual, English-centric
dataset that consists of sentence pairs translating from 100 other languages
to English. We finetune the model on Danish, which is previously not supported
by the pretrained mBart50 model. For each iteration, we subsample 10,000 new
Danish-English sentence pairs to finetune. The pretrain dataset consists of
10,000 random uniform samples from the languages that mBart50 was originally
capable of translating. The pretrain task performance is defined as the
average loss of all pretrain language samples and the finetune task
performance is defined as the average loss of Danish samples.

##  Appendix B Ablation Study on Design Choices

###  B.1 Ablation on partitioning

Fig 6 shows that applying more partitioning methods benefits mix-cd. In
general, a more fine-grained partition allows more accurate estimation of
collateral damage ratios within each partition cell. Combining orthogonal
partition schemes that utilizes different information increases the fineness
of the partitioned cells.

Figure 6: Ablation study on how the pretrain distribution is partitioned. Mix-
cd performs better when both pretrain loss and auxiliary information is
utilized to better localize the collateral damage regions.

Figure 7: Comparison of estimated and exact (oracle) collateral damage ratios
of auxiliary label partitions.

Figure 8: Comparison of estimated and exact (oracle) collateral damage ratios
of pretrain loss partitions.

##### Which pretrain loss partitions suffer more collateral damage?

Fig 8 shows the collateral damage ratios of different cells partitioned with
pretrain loss. The middle cell has the highest collateral damage ratio and
thus is prioritized with mix-cd for ImageNet-CUB while the last cell is
prioritized for MNLI-Scitail. This shows that the best pretrain loss partition
to prioritize may differ between different datasets, justifying our approach
to adaptively estimate the collateral damage ratios.

##### Which auxiliary information partitions suffer more collateral damage?

Fig 7 shows that the collateral damage ratios of different cells partitioned
with auxiliary labels vary significantly between different cells. This
explains why treating the pretrain data distribution uniformly is an inferior
sampling strategy. We further examine whether prioritized cells make sense
intuitive sense in the ImageNet-CUB task.

###  B.2 How to select the confidence threshold?

Figure 9: Ablation study on the choice of generalization thresholds, $c_{p}$
and $c_{f}$. Filtering with both the pretrain prediction $c_{p}$ as well as
finetune prediction $c_{f}$ is the best indicator for generalizable collateral
damage and performs the best. Figure 10: Ablation study on the confidence
threshold $c_{f}$ and $c_{p}$ values. We fix the two values to be the same in
the study. The best value is somewhere between so we adopted 0.5 as a
heuristic to avoid hyperparameter tuning. Also observe that all the choices of
the threshold are no worse than the random baseline.

Recall the purpose of the confidence threshold is to filter for collateral
damage pretrain samples with generalizable signal (see Section 3.2). Fig 9
ablates where the confidence threshold is applied and concludes that setting
confidence thresholds for both the pretrain prediction (guarantee the sample
was initially, confidently correct) and finetune prediction (guarantee the
sample is now confidently incorrect) is the best strategy.

Fig 10 compares the performance for different confidence threshold values.
Lower threshold enforces a stricter restriction definition on generalizable
collateral damage samples. This leads to better quality but less quantity. On
the contrary, higher threshold enforces a more relaxed restriction on the
generalizability. More samples will be considered but with worse quality.
Results show that the optimal value lies somewhere in between the extremes and
varies slightly for different tasks. Regardless, most threshold choices
outperform the random baseline suggesting mix-cd is not sensitive to the
hyperparameter and 0.5 is a good-enough heuristic.

###  B.3 How does prioritizing other types of samples compare to mix-cd?

We explore alternative definitions of important samples to prioritize to
combat forgetting. The definitions are inspired by common finetuning
techniques that also don’t require additional computation cost. We apply the
same ratio tracking and sampling scheme, only with the definition of samples
prioritized changed. We compare these variants with mix-cd-sample in Fig 11.

Figure 11: Ablation study on the definition of usefulness. The common practice
of selecting high loss (strong signal) samples for finetuning does not
generalize to all settings.

mix-cd-sample dominates the performance of other variants. Variants that only
leverage the pretrain loss does not take the relationship between the pretrain
and finetune tasks into account. Additionally, choosing pretrain samples with
the strongest signal (highest pretrain loss, highest loss difference) is also
sub-optimal since strong-signal pretrain samples may directly compete with the
finetune task performance.

##  Appendix C Interesting Explorations

###  C.1 Is combining rehearsal methods with weight-regularization techniques
better?

In previous studies, rehearsal and weight-regularization methods are rarely
compared with each other. Needless to say, the combination of these two
methods are underexplored. Technically, the two types of methods operate on
orthogonal directions and should benefit each other when applied
simultaneously. Rehearsal methods operate on data selection on weight-
regularization on the model weight directly. Thus, we explore combining
random, mix-cd-sample, and finetune-only with WiSE-FT (Wortsman et al., 2022).
WiSE-FT is a simple post-hoc weight regularization that directly takes the
linear combination of the finetuned model weights and pretrained model weights
with a hyperparameter $\alpha$ controlling the combination proportions. (Lin
et al., 2023) have demonstrated that WiSE-FT is consistently the best-
performing method out of all weight-regularization methods benchmarked on
various foundation model finetuning tasks. We experiment with two different
combination ratios 0.25 and 0.5.

Fig 12 shows that all combined variants outperform the rehearsal-only variants
significantly in the pretrain performance. This shows promising result for
future works to explore integrating them into hybrid methods. The relative
performance between combined variants are similar to the rehearsal-only ones,
except for finetune-only on ImageNet-CUB. The combined version outperforms
other baselines significantly, which is unexpected given the poor performance
of finetune-only on the pretrain task. More investigation is needed to examine
the difference in mechanisms for rehearsal and weight regularization methods
to prevent forgetting.

Figure 12: Combination of rehearsal baselines with the weight averaging
technique (WiSE-FT) for weight regularization.

###  C.2 How many examples less are needed for mix-cd-sample to match the
performance of random?

Another perspective to compare the data efficiency of the finetuning
strategies is to evaluate how many less samples are needed for one method to
achieve the same performance of the other method. Our goal is to find out how
many less samples does the model need to train on for mix-cd-sample to match
the performance of random. Fig 13 shows that mix-cd-sample only needs 90% of
pretrain samples to match random on ImageNet-CUB and only 50% on MNLI-Scitail.
This, again, highlights the importance of sample selection especially under
low pretrain data budget.

Figure 13: Many pretrain samples less does mix-cd-sample need to perform as
well as uniform? 90% for ImageNet-CUB and 50% for MNLI-Scitail.