# Continual Learning of Diffusion Models with Generative Distillation

Sergi Masip
Master in Computer Vision (Barcelona)
<EMAIL_ADDRESS>
&Pau Rodríguez
Apple
<EMAIL_ADDRESS>
&Tinne Tuytelaars
KU Leuven
<EMAIL_ADDRESS>&Gido M. van de Ven
KU Leuven
<EMAIL_ADDRESS>
Work done under the Erasmus+ Traineeship program at KU Leuven.Work done
externally as part of the Master in Computer Vision (Barcelona).

###### Abstract

Diffusion models are powerful generative models that achieve state-of-the-art
performance in tasks such as image synthesis. However, training them demands
substantial amounts of data and computational resources. Continual learning
would allow for incrementally learning new tasks and accumulating knowledge,
thus reusing already trained models would be possible. One potentially
suitable approach is generative replay, where a copy of a generative model
trained on previous tasks produces synthetic data that are interleaved with
data from the current task. However, standard generative replay applied to
diffusion models results in a catastrophic loss in denoising capabilities. In
this paper, we propose generative distillation, an approach that distils the
entire reverse process of a diffusion model. We demonstrate that our approach
significantly improves the continual learning performance of generative replay
with only a moderate increase in the computational costs.

##  1 Introduction

Diffusion models for likelihood-based data generation [1, 2, 3] have achieved
state-of-the-art performance in many generative modelling tasks such as image
synthesis. They are defined by a forward process that adds noise to an image
and a reverse process that denoises it. To achieve impressive results, these
models must be trained on vast datasets and require a lot of computational
resources, making their training very costly. Furthermore, incorporating new
knowledge incrementally is challenging due to a phenomenon known as
catastrophic forgetting [4, 5], as a result of which the performance of neural
networks on previously learned tasks severely degrades when trained on new
tasks. For this reason, despite the computational cost, most practitioners
retrain their models using all the data seen so far each time they incorporate
new data. Therefore, it would be valuable to devise a way to leverage already
trained diffusion models and continuously expand their knowledge.

The continual learning field studies how to mitigate catastrophic forgetting
to enable neural networks to incrementally learn new tasks, domains or classes
while retaining previous knowledge [6]. Among the existing approaches for
continual learning, rehearsal methods, which store a fixed replay buffer of
samples from previous tasks, have shown promising results for diffusion models
[7]. Nevertheless, accessing previous data is sometimes infeasible due to
limited resources or privacy issues. An alternative is generative replay [8],
where the replay buffer is replaced with a generative model that produces
synthetic samples. Previous works [7, 9] that used generative replay for
continually training diffusion models employed the original sampler from
“Denoising Diffusion Probabilistic Models” (DDPM) [2] to generate the
synthetic samples.

However, one limitation of DDPM is that its generation times are prohibitively
slow. This restriction forced the authors of [7, 9] to generate a fixed buffer
of synthetic samples, thereby limiting the diversity advantage that diffusion
models could offer. An alternative solution is to employ the sampler
introduced in “Denoising Implicit Probabilistic Models” (DDIM) [10], which
allows for faster generation and, therefore, drawing a considerably larger
amount of samples. Nonetheless, we show that generative replay with DDIM
results in a fast accumulation of errors. We note that, when training on a new
task, standard generative replay transfers knowledge from the previous task
model only at the endpoint of the reverse process of the diffusion model
(i.e., through the generated replay samples), which we believe is inefficient.

In this paper, we propose to transfer knowledge more effectively by combining
generative replay with knowledge distillation. This approach, generative
distillation, distils the entire reverse process of a teacher diffusion model
trained on previous tasks to the student model being trained on the current
task. The teacher generates a sample with DDIM, then Gaussian noise is added
according to the variance schedule [2] and, finally, the objective matches the
noise predictions of both the teacher and the student. We conduct an
experiment on Fashion-MNIST [11] where we continually train a diffusion model
with generative distillation and demonstrate its effectiveness compared to
generative replay at mitigating the catastrophic loss in denoising
capabilities, while increasing the computational cost by just one forward
pass.

##  2 Background and related work

##### Diffusion models

Diffusion models are likelihood-based models [1, 2, 3] defined by two
processes: (1) a forward process that adds Gaussian noise to an image
according to a variance schedule and (2) a learnable reverse process that
follows the reverse trajectory of the forward process. The forward and reverse
processes are discretized into $T$ timesteps. A diffusion model, denoted as
$\epsilon_{\theta}$ and parameterized by $\theta$, can be trained to match the
noise introduced during the forward process at any timestep $t$ following a
predefined variance schedule. To train this model, one can minimize the
following simplified objective [2]:

$L_{\text{simple}}(\theta)=\mathbb{E}_{t\sim\text{Unif}(T),x_{0}\sim
D,\epsilon\sim\mathcal{N}(0,I)}\bigr{[}\|\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{t}}x_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,t)\|^{2}\bigr{]},$
(1)

where $x_{0}$ is a clean sample from the data distribution $D$,
$\epsilon\sim\mathcal{N}(0,I)$ is the ground truth Gaussian noise, and
$\bar{\alpha}$ is a noise schedule function. The standard way of sampling from
a diffusion model (e.g., with DDPM) is computationally expensive as it
requires many denoising steps (e.g., 1000 in [2]). This greatly limits the
number of samples that can be generated for replay with a finite computational
budget. One widely used alternative is DDIM, which permits sampling using a
smaller number of denoising steps, thus allowing the generation of more images
given the same budget. However, the fewer denoising steps are used, the
blurrier and less detailed the sampled images tend to be.

##### Knowledge distillation in diffusion models

Existing studies use distillation in diffusion models to train a student such
that it can generate the same quality images as its teacher, but in fewer
generation steps [12, 13, 14, 15, 16]. In this paper, we use this technique
for a different purpose and we demonstrate its effectiveness for continual
learning of diffusion models.

##### Continual learning

An important problem in continual learning is catastrophic forgetting [4, 5],
a phenomenon where a sequentially trained neural network rapidly and
drastically forgets its knowledge of previous tasks when it learns a new task.
One approach to mitigate catastrophic forgetting is generative replay [8].
Given a sequence of $B$ tasks $\mathcal{T}=\\{T_{1},T_{2},...,T_{B}\\}$, a
generative model $G_{i}$ is trained on data from task $T_{i}$ interleaved with
replay samples produced by $G_{i-1}$, a copy of the generative model after it
finished training on the previous task.

##### Continual learning and diffusion models

The topic of continual learning of diffusion models remains relatively
unexplored. We are aware of three works that use generative replay in
diffusion models. Zając et al. [7] test different continual learning
approaches on diffusion models, and Gao and Liu [9] use a continually trained
classifier to guide the generation process, as in [17]. These works, however,
do not study the use of DDIM in generative replay. In other recent work, Smith
et al. [18] observe a catastrophic accumulation of errors when using standard
generative replay.

##  3 Generative distillation for diffusion models

(a) Generative Replay

(b) Generative Distillation

Figure 1: Conceptual flow of generative replay and generative distillation. In
generative replay, the current task model $\epsilon_{\theta_{i}}$ is trained
to match the noise added to the replay sample, whereas, in generative
distillation, $\epsilon_{\theta_{i}}$ is trained to match the noise prediction
of the previous task model $\epsilon_{\hat{\theta}_{i-1}}$.

With standard generative replay, the previous task model
$\epsilon_{\hat{\theta}_{i-1}}$ is used in the training process of the current
task model $\epsilon_{\theta_{i}}$ only for sampling images to replay (fig.
1(a), see section A.3 for details). This means that knowledge from
$\epsilon_{\hat{\theta}_{i-1}}$ is transferred to $\epsilon_{\theta_{i}}$ only
at the endpoint of the reverse process of the diffusion model, i.e. after
going through all the denoising steps. Therefore, if the previous task model
generates imperfect images (e.g., using a small number of DDIM steps), the
current task model learns trajectories that lead to these images, causing an
accumulation of errors. To tackle this issue, we introduce knowledge
distillation into generative replay for diffusion models. This approach,
_generative distillation_ , transfers the knowledge from
$\epsilon_{\hat{\theta}_{i-1}}$ to $\epsilon_{\theta_{i}}$ at each point of
the reverse trajectory, while using synthetic images sampled from
$\epsilon_{\hat{\theta}_{i-1}}$ as inputs. First,
$\epsilon_{\hat{\theta}_{i-1}}$ generates a clean sample using DDIM and, then,
Gaussian noise is added to obtain a noisy sample $z_{t}$ at timestep $t$ such
that

$z_{t}=\sqrt{\bar{\alpha}_{t}}\cdot\text{DDIM}_{N}(\epsilon_{\hat{\theta}_{i-1}})+\sqrt{1-\bar{\alpha}_{t}}\cdot\epsilon,$
(2)

where $\text{DDIM}_{N}(\epsilon_{\hat{\theta}_{i-1}})$ denotes a function that
uses DDIM to generate a clean sample using $N$ generation steps employing the
diffusion model trained on previous tasks $\epsilon_{\hat{\theta}_{i-1}}$.
Given the noisy sample $z_{t}$, the distillation objective for previous tasks
during generative distillation is the following:

$L_{\text{distil}}(\theta_{i})=\mathop{\mathbb{E}}{}_{t\sim\text{Unif}(T),z_{t}\sim\epsilon_{\hat{\theta}_{i-1}}}\bigl{[}||\epsilon_{\hat{\theta}_{i-1}}(z_{t},t)-\epsilon_{\theta_{i}}(z_{t},t)||^{2}_{2}\bigr{]},$
(3)

where $\epsilon_{\theta_{i}}$ is the model being trained. In fig. 1(b), a
conceptual flow of generative distillation is shown. The final loss that we
aim to minimize throughout the continual learning process is a weighted
combination of both the standard loss (eq. 1) for data from the current task
and the distillation loss for replayed data:

$L_{\text{GD}}=rL_{\text{simple}}+(1-r)\lambda L_{\text{distil}},$ (4)

where $r$ is a ratio that gives the loss on the current task a decreasing
proportional importance relative to the number of tasks seen so far (i.e.,
$r=\frac{1}{i}$ with $i$ the number of tasks learned so far [19]), and
$\lambda$ is a weighting parameter to balance the magnitudes of the standard
and distillation losses.

##  4 Experiment

We divide the Fashion-MNIST dataset into five tasks, each comprising two
randomly selected classes (fig. 2(a)), that must be learned one after the
other. To generate replay samples, we use the previous task model
$\epsilon_{\hat{\theta}_{i-1}}$ with 2 DDIM steps for both generative replay
and generative distillation. To quantitatively compare the trained diffusion
models, we report the commonly used Fréchet Inception

Table 1: Quantitative results. The FID and KLD are computed after training on all tasks. Reported for each metric is mean ± standard error over 3 random seeds. Strategy |  $\downarrow\text{FID}$ | $\downarrow\text{KLD}$
---|---|---
Joint (upper target) |  15.5 $\pm$ 1.30 |  0.08 $\pm$ 0.02
Finetune (lower target) |  68.8 $\pm$ 7.00 |  4.82 $\pm$ 1.42
Generative replay |  211.8 $\pm$ 6.9 |  3.39 $\pm$ 0.30
Generative distillation | 23.5 $\pm$ 1.1 | 0.25 $\pm$ 0.10

Distance (FID) [20], as well as the Kullback-Leibler divergence between the
ground truth class frequency distribution and a distribution of predicted
class membership for images generated by the diffusion model. This last
metric, denoted KLD, evaluates to what extent the diffusion model generates
samples from all classes so far (rather than, for example, only from classes
of the last task). For further details on the experiment, models and metrics,
please refer to appendix A.

(a) Task order

(b) Joint

(c) GR

(d) GD

Figure 2: Qualitative results. (a) The order of the tasks the models were
trained on. (b-d) Images generated at the end of training by a diffusion model
that was jointly trained on all tasks, continually trained using generative
replay (GR) or continually trained using generative distillation (GD). When
introducing distillation into generative replay, the model keeps its denoising
capabilities and diversity.

Figure 3: Quality of the diffusion model throughout the continual learning
process. For each approach, the FID and KLD of the diffusion model are
computed after every task. Displayed are the means ± standard errors over 3
random seeds. Joint: training using data from all tasks so far (upper target),
Naive: continual fine-tuning (lower target), GR: generative replay, GD:
generative distillation.

The results in table 1 indicate a severe loss in denoising capabilities when
using standard generative replay. As shown in fig. 2(c), after training on the
last tasks, the generative replay trained model generates images that are too
smooth, indicating that the model quickly forgets how to add detail to the
images. On the other hand, generative distillation preserves previous
knowledge much more effectively and the model still generates high-quality and
detailed images (fig. 2(d)), while using the same computational budget plus
one extra forward pass. The performance of generative replay can be improved
by increasing the number of DDIM steps when generating the replay samples, but
generative distillation clearly outperforms generative replay in all settings
we consider (section B.2 and figs. 10, 11, 12 and 13). Interestingly, the FID
of the generative distillation trained model slightly improved from tasks 2 to
4 (fig. 3), which may indicate positive transfer between tasks.

As an additional way of evaluating the performance of the continually trained
diffusion model, we include a classifier trained with generative replay using
the continually trained diffusion model as

Table 2: Evaluation with a continually trained classifier. Reported is mean ± standard error over 3 seeds. Strategy | $\uparrow\text{ACC}$
---|---
Joint (upper target) |  0.92 $\pm$ 0.00
Finetune (lower target) |  0.17 $\pm$ 0.03
Generative replay |  0.50 $\pm$ 0.04
Generative distillation | 0.72 $\pm$ 0.03

generator (see section A.2 for details). In Table 2, we report the accuracy of
the classifier at the end of training. This evaluation confirms the benefit of
incorporating distillation into generative replay.

To further illustrate the potential of distillation, in appendix B we provide
additional experiments in which we apply distillation to data from the current
task, to Gaussian noise and to replay samples generated using 10 and 100 DDIM
steps. We note that generative distillation could also be used in combination
with other samplers [e.g., 21, 22, 15].

##  5 Conclusions

Continually training a diffusion model with standard generative replay results
in a catastrophic loss in its denoising capabilities, leading the model to
produce blurry images. We have conducted an experiment on Fashion-MNIST where
we have demonstrated that including knowledge distillation into the generative
replay process (generative distillation) mitigates this catastrophic
forgetting and markedly enhances the performance of the continually trained
diffusion model.

## Acknowledgements

This project has been supported by funding from the European Union under the
Erasmus+ Traineeship program, under the Horizon 2020 research and innovation
program (ERC project KeepOnLearning, grant agreement No. 101021347) and under
Horizon Europe (Marie Skłodowska-Curie fellowship, grant agreement No.
101067759).

## References

  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33:6840–6851, 2020.
  * Song et al. [2020a] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In _International Conference on Learning Representations_ , 2020a.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of Learning and Motivation_ , volume 24, pages 109–165. Elsevier, 1989.
  * Ratcliff [1990] Roger Ratcliff.  Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.  _Psychological Review_ , 97:285–308, 1990.
  * van de Ven et al. [2022] Gido M van de Ven, Tinne Tuytelaars, and Andreas S Tolias.  Three types of incremental learning.  _Nature Machine Intelligence_ , 4(12):1185–1197, 2022.
  * Zając et al. [2023] Michał Zając, Kamil Deja, Anna Kuzina, Jakub M Tomczak, Tomasz Trzciński, Florian Shkurti, and Piotr Miłoś.  Exploring continual learning of diffusion models.  _arXiv preprint arXiv:2303.15342_ , 2023.
  * Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  In _Advances in Neural Information Processing Systems_ , volume 30, 2017.
  * Gao and Liu [2023] Rui Gao and Weiwei Liu.  DDGR: Continual learning with deep diffusion-based generative replay.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.
  * Song et al. [2020b] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2020b.
  * Xiao et al. [2017] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Luhman and Luhman [2021] Eric Luhman and Troy Luhman.  Knowledge distillation in iterative generative models for improved sampling speed.  _arXiv preprint arXiv:2101.02388_ , 2021.
  * Salimans and Ho [2021] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2021.
  * Berthelot et al. [2023] David Berthelot, Arnaud Autef, Jierui Lin, Dian Ang Yap, Shuangfei Zhai, Siyuan Hu, Daniel Zheng, Walter Talbot, and Eric Gu.  TRACT: Denoising diffusion models with transitive closure time-distillation.  _arXiv preprint arXiv:2303.04248_ , 2023.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  In _Proceedings of the 40th International Conference on Machine Learning_ , 2023.
  * Gu et al. [2023] Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Lingjie Liu, and Joshua M Susskind.  BOOT: Data-free distillation of denoising diffusion models with bootstrapping.  In _ICML 2023 Workshop on Structured Probabilistic Inference & Generative Modeling_, 2023.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat GANs on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Smith et al. [2023] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin.  Continual diffusion: Continual customization of text-to-image diffusion with C-LoRA.  _arXiv preprint arXiv:2304.06027_ , 2023.
  * van de Ven et al. [2020] Gido M. van de Ven, Hava T. Siegelmann, and Andreas S. Tolias.  Brain-inspired replay for continual learning with artificial neural networks.  _Nature Communications_ , 11:4069, 2020.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  GANs trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Lomonaco et al. [2021] Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni.  Avalanche: an end-to-end library for continual learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2nd Continual Learning in Computer Vision Workshop, 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 770–778, 2016.
  * Li and Hoiem [2017] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2935–2947, 2017.

##  Appendix A Extra details on the experiments

Code is publicly available at
https://github.com/Atenrev/difussion_continual_learning.

###  A.1 Model architectures

####  A.1.1 Diffusion model

We employ a custom 2D U-Net based on the one used in [2]. We set the input
size to $32\times 32$ pixels, with a single input channel and a single output
channel. We use four decreasing feature map resolutions ($32\times 32$ to
$4\times 4$) with progressively increasing output channel dimensions: 64, 128,
256, and 512, respectively, each block containing one layer. The normalization
is applied with 32 groups, and the model has two convolutional residual blocks
per resolution level and self-attention blocks between the convolutional
blocks at resolutions $8\times 8$ and $4\times 4$. The total amount of
parameters is 42.9 million.

####  A.1.2 Continually trained classifier

Table 3: Architecture used for the continually trained classifier. Layer | Type | Filters | Kernel Size | Activation
---|---|---|---|---
1 | Conv2D | 32 | 3x3 | ReLU
2 | Conv2D | 32 | 3x3 | ReLU
3 | MaxPool2D | - | 2x2 | -
4 | Dropout (p=0.25) | - | - | -
5 | Conv2D | 64 | 3x3 | ReLU
6 | Conv2D | 128 | 3x3 | ReLU
7 | MaxPool2D | - | 2x2 | -
8 | Dropout (p=0.25) | - | - | -
9 | Conv2D | 128 | 1x1 | ReLU
10 | AdaptiveMaxPool2D | 1 | - | -
11 | Dropout (p=0.25) | - | - | -
12 | Linear | 10 | - | -

####  A.1.3 Classifier used for computing the KLD metric

In the implementation of the KLD metric we use a ResNet18, which we had
trained to classify the ten classes of Fashion-MNIST. For the training of this
model, we split the training set of Fashion-MNIST into a training set
consisting of 50,000 samples and a validation set consisting of 10,000. We
evaluated the model’s accuracy on the validation set at the end of each epoch.
Upon completing the training process, we chose the model from the epoch where
it achieved the best validation accuracy. We applied the following
augmentations, hyperparameters, and architectural changes:

Augmentations:

  * •

Randomly resized the images to 32x32 pixels.

  * •

Applied random horizontal flips to the images.

  * •

Applied random rotations up to 10 degrees.

  * •

Applied random affine transformations with a translation of up to 0.1 in both
directions.

  * •

Added Gaussian noise with a mean of 0 and a standard deviation of 0.1 to the
images.

Hyperparameters:

  * •

Batch size: 64

  * •

Number of training epochs: 100

  * •

Learning rate: 0.0005

Architectural change to ResNet18:

  * •

Replaced the final fully connected layer with a new linear layer with the
number of output classes set to 10.

###  A.2 Generative replay in the classifier

We continually train a classifier with the architecture displayed in table 3
along with the diffusion model. This provides an additional way to evaluate
the performance of the diffusion model when used as a generator in generative
replay of a classifier [8]. If the generator was ideal and approximated the
real data distribution accurately, generative replay would be close to
performing experience replay with an unlimited replay buffer.

For tasks $T_{i},i>1$, we train the classifier $S_{i}$ on samples from the
current task $T_{i}$, interleaved with synthetic samples generated by
$\epsilon_{\hat{\theta}_{i-1}}$, a copy of the knowledge-accumulating
generator after training on the previous task. To generate the replay samples
to train the classifier, we always use 10 DDIM steps. Since we use an
unconditional diffusion model, we employ soft labelling, following van de Ven
et al. [19], which has shown greater effectiveness than hard labelling. These
soft labels result from running the synthetic samples through the previous
classifier $S_{i-1}$. Our objective is to minimize two distinct losses: one
for the current task and one for previous tasks. The first loss
$L_{\text{curr}}^{\text{clf}}$ is the usual cross-entropy classification
defined as:

$L_{\text{curr}}^{\text{clf}}=\mathbb{E}_{(x_{\text{curr}},y)\sim
D_{i}}\biggr{[}-\sum_{c=1}^{N_{\text{classes}}}y_{c}\log
p_{\theta}(Y=c|x_{\text{curr}})\biggl{]}.$ (5)

The second loss is the distillation loss that makes use of soft labels defined
as:

$L_{\text{\text{replay}}}^{\text{clf}}=\mathbb{E}_{x_{\text{prev}}\sim\epsilon_{\theta_{i-1}},\tilde{y}\sim
S_{i-1}}\biggr{[}-R^{2}\sum_{c=1}^{N_{\text{classes}}}{\tilde{y}_{c}}\log
p_{\theta}^{R}(Y=c|x_{\text{prev}})\biggl{]}$ (6)

where $x_{\text{prev}}$ is the input data for the current task labelled with a
hard target $y$, $x_{\text{prev}}$ is the input data for previous tasks
generated by the previous generator, $\tilde{y}$ are the soft targets for
$x_{\text{prev}}$, and $R$ is a temperature parameter that we set to $2$.

The final objective is the joint loss that combines both with a ratio of
relative importance $r=\frac{1}{i}$, with $i$ the number of tasks so far, that
gives $L_{\text{replay}}$ an importance proportional to the number of tasks
seen so far:

$L_{\text{classifier}}=rL_{\text{curr}}^{\text{clf}}+(1-r)L_{\text{replay}}^{\text{clf}}$
(7)

The experimental setup is discussed in section A.5.

###  A.3 Standard generative replay

With standard generative replay, as with generative distillation, in each
replay step the previous task model $\epsilon_{\hat{\theta}_{i-1}}$ generates
a clean sample using DDIM to which Gaussian noise
$\epsilon\sim\mathcal{N}(0,I)$ is added to obtain a noisy sample $z_{t}$ at
timestep $t$:

$z_{t}=\sqrt{\bar{\alpha}_{t}}\cdot\text{DDIM}_{N}(\epsilon_{\hat{\theta}_{i-1}})+\sqrt{1-\bar{\alpha}_{t}}\cdot\epsilon,$
(8)

where $\text{DDIM}_{N}(\epsilon_{\hat{\theta}_{i-1}})$ is a clean sample
generated by $\epsilon_{\hat{\theta}_{i-1}}$ using DDIM with $N$ generation
steps and $\bar{\alpha}$ is the noise schedule function. Note that eq. 8 is
the same as eq. 2 in the main text. The replay loss for previous tasks is
then:

$L_{\text{replay}}(\theta_{i})=\mathop{\mathbb{E}}{}_{t\sim\text{Unif}(T),z_{t}\sim\epsilon_{\hat{\theta}_{i-1}}}\bigl{[}||\epsilon-\epsilon_{\theta_{i}}(z_{t},t)||^{2}_{2}\bigr{]}.$
(9)

where $\epsilon_{\theta_{i}}$ is the model being trained. As in generative
distillation, the final loss is a weighted combination of the standard loss
(eq. 1 in the main text) for data from the current task and the replay loss
for replayed data:

$L_{\text{GR}}=rL_{\text{simple}}+(1-r)L_{\text{replay}},$ (10)

where $r$ is the task importance ratio, which gives $L_{\text{replay}}$ an
increasing proportional importance relative to the number of tasks seen so
far.

###  A.4 Additional distillation approaches

The implementation of the distillation with current data is based on eq. 4,
except that in $L_{\text{distil}}$ in eq. 3, $z_{t}$ is equal to the noisy
data from the current batch used in the standard training loss
$\bar{x_{t}}:\sqrt{\bar{\alpha}_{t}}\cdot
x_{t}+\sqrt{1-\bar{\alpha}_{t}}\cdot\epsilon$ such that $z_{t}:=\bar{x_{t}}$.
Distillation with Gaussian noise is the same, except that now Gaussian noise
is used as the “noisy data input”, such that $z_{t}\sim\mathcal{N}(0,I)$ for
all timesteps.

###  A.5 Experimental setup

In our experiment, we continually train a diffusion model along a classifier.
When training on a new task, a copy of the diffusion model from previous tasks
$\epsilon_{\hat{\theta}_{i-1}}$ generates replay samples for the training of
both the diffusion model (only on generative methods) and the classifier. To
generate the replay samples to train the diffusion model, for all experiments
reported in the main text, we use 2 DDIM steps. In the appendix B, we
additionally report results for experiments in which we use 10 or 100 DDIM
steps to generate the replay samples to train the diffusion model (The replay
samples to train the classifier are always generated using 10 DDIM steps, see
section A.2). No augmentations are applied to any of the samples. We
standardize the ground truth data to fall within the interval of $[-0.5,0.5]$
and resize it to $32x32$.

For each task, the diffusion model is trained for 100 epochs (9400 gradient
updates) and the classifier for 20 epochs (1800 gradient updates). The batch
size is set to 128 for the first task and 256 for the subsequent ones to
accommodate the synthetic samples (each batch then contains 128 samples from
the current task and 128 replayed samples). For distillation with current data
the batch size is always 128. We train both models using Adam [23] as the
optimizer. For the diffusion model, we use a learning rate of $2\times
10^{-4}$, while for the classifier we use $1\times 10^{-3}$. We implement all
the strategies using the Avalanche library [24].

In order to find the best $\lambda$ for each distillation approach, we sweep
over different sets of lambda values based on the observed loss magnitudes. We
sweep across 3 random seeds over the values of
$\\{0.1,0.25,0.5,0.75,1.0,1.25,1.5,1.75,2.0,2.25,2.5,2.75,3.0\\}$ for
generative distillation, $\\{1,4,8,12,16,20,24,28\\}$ for distillation with
Gaussian noise, and $\\{0.1,0.25,0.5,0.75,1.0,1.25,1.5,1.75\\}$ for
distillation with current data. We select the lambda that yields the best FID
value for the comparisons.

###  A.6 Evaluation

The models are evaluated after finishing the training on each task. For these
evaluations, we only use the part of the test set corresponding to all the
classes seen so far. To compute the Fréchet Inception distance (FID), we
generate 10,000 images with the diffusion model using 10 DDIM steps. We use
these same 10,000 images to compute the Kullback-Leibler divergence (KLD)
between the ground truth class frequency distribution and the generated class
frequency distribution. Since we are using an unconditional diffusion model
architecture, we use a ResNet18 [25] trained on Fashion-MNIST to label the
generated samples; see section A.1.3 for further details on the training of
this classifier. As an additional metric, we report the accuracy of the
continually trained classifier with generative replay (see section A.2 for
details).

For each metric, we include baselines that can be interpreted as upper and
lower targets. The upper target is attained through joint training of either
the classifier or the diffusion model, wherein we sequentially train the
models using all the data from tasks encountered so far. The lower target is
obtained by naïvely fine-tuning the models on the current task. The upper
target yields desirable metrics, while the lower target serves as a
performance baseline.

To visualize example images sampled from the trained diffusion models (Figures
2,6-15), we always use 10 DDIM steps.

##  Appendix B Extra experiments

###  B.1 Samples from a model trained jointly on all data

(a) DDIM (2 steps)

(b) DDIM (10 steps)

(c) DDIM (100 steps)

(d) DDPM

Figure 4: Samples from a jointly trained model with different sampling
methods. Relatively smooth samples are generated with 2 DDIM steps, minor
artefacts are present with 100 DDIM steps.

To motivate the choice of the number of teacher DDIM steps, we trained a
diffusion model in Fashion-MNIST with an independently and identically
distributed stream of data for 100 epochs (equivalent to 46,900 gradient
updates). We set the batch size to 128 and the learning rate to $2\times
10^{-4}$. With this model, we used DDIM to generate images using 2, 10 and 100
steps. For comparison, we also sampled images using DDPM. As illustrated in
fig. 4, employing 2 DDIM steps resulted in smooth samples that lack fine
details. Using 10 DDIM steps provided decent looking samples; increasing the
DDIM step count to 100 did not seem to enhance the visual quality of the
sampled images much.

###  B.2 Results

Table 4: Full comparison results among different approaches. Reported for each metric is the mean ± standard error over 3 random seeds. The number of generation steps used by the generator from the previous task to train the current task model (if any) is indicated in the Teacher steps column. All models used 10 DDIM steps to generate images for the classifier during generative replay. Strategy | Teacher steps | $\lambda$ | $\downarrow\text{FID}$ | $\downarrow\text{KLD}$ | $\uparrow\text{ACC}$
---|---|---|---|---|---
Joint (classifier) | - | - | - | - |  0.92 $\pm$ 0.00
Finetune (classifier) | - | - | - | - |  0.17 $\pm$ 0.03
Joint (generator) | - | - |  15.49 $\pm$ 1.30 |  0.08 $\pm$ 0.02 | -
Finetune (generator) | - | - |  68.76 $\pm$ 7.00 |  4.82 $\pm$ 1.42 | -
Distil. w/ Gaussian noise | - | 8 |  38.87 $\pm$ 2.23 |  0.85 $\pm$ 0.05 |  0.63 $\pm$ 0.04
Distil. w/ current data | - | 1.25 |  65.01 $\pm$ 3.31 |  0.37 $\pm$ 0.09 |  0.35 $\pm$ 0.01
Generative replay | 2 | - |  211.79 $\pm$ 6.92 |  3.39 $\pm$ 0.30 |  0.50 $\pm$ 0.04
Generative distillation | 2 | 0.75 | 23.53 $\pm$ 1.06 | 0.25 $\pm$ 0.10 | 0.72 $\pm$ 0.03
Generative replay | 10 | - |  121.42 $\pm$ 13.04 |  0.98 $\pm$ 0.19 |  0.58 $\pm$ 0.04
Generative distillation | 10 | 0.75 | 20.29 $\pm$ 3.07 | 0.13 $\pm$ 0.04 | 0.79 $\pm$ 0.01
Generative replay | 100 | - |  57.54 $\pm$ 8.61 |  0.61 $\pm$ 0.21 |  0.66 $\pm$ 0.02
Generative distillation | 100 | 0.75 | 23.69 $\pm$ 3.42 | 0.12 $\pm$ 0.04 | 0.80 $\pm$ 0.01

Figure 5: Quality of the diffusion model throughout the continual learning
process (with additional approaches). Each point represents a metric achieved
after training in a specific task. Displayed are the means ± standard errors
over 3 random seeds. Joint: training using data from all tasks seen so far
(upper target), Naive: fine-tuning on the current task (lower target),
GR($n$): generative replay with a teacher using $n$ DDIM steps, LwF:
distillation with current data, GGD: distillation with Gaussian noise, GD:
generative distillation.

We provide a table (table 4) and graph of metrics (fig. 5) wrapping all the
results obtained after conducting the main and extra experiments. Note that
increasing the number of DDIM steps used by the teacher in generative replay
mitigates the catastrophic loss in denoising capabilities to some extent.
However, even when using 100 teacher DDIM steps with generative replay, it is
still outperformed by generative distillation using just 2 teacher DDIM steps.
Using 100 teacher DDIM steps in generative distillation does not seem to
provide significant benefits over using 10 teacher DDIM steps, despite the
greater computational costs.

Another approach we implemented for the diffusion model distils the knowledge
of the model from the previous task using the current task’s data. This can be
seen as a Learning without Forgetting (LwF) [26] like approach applied to
diffusion models. This approach exhibited better performance than naïve
generative replay both in quality and diversity when the teacher used up to 10
DDIM steps. However, the overall quality of the samples degrades steadily and
more artefacts show up as more tasks are learned.

We also considered an extreme case equivalent to the model from the previous
task using 0 DDIM steps, that is, distilling with Gaussian noise as input for
all timesteps. Surprisingly, this approach surpasses distillation with current
data in FID. In contrast, it shows a worse KLD, indicating that distilling
with noise sacrifices diversity over image quality. The reason for poorer
diversity when using Gaussian noise might be that the noise doesn’t align well
with the actual distribution, whereas the lower quality obtained when
distilling with current data could be due to conflicting gradients calculated
in the loss function. Distilling with Gaussian noise achieves the second best
ACC only behind generative distillation, which shows the potential of
distillation for transferring knowledge even when employing only noise.

###  B.3 Samples throughout the continual learning process

In this section, we display samples generated by the diffusion model
throughout the continual learning process following the different approaches
we have discussed. This allows the reader to qualitatively evaluate the
diffusion model’s performance under each approach and throughout their
continual training. The samples were obtained using a consistent
representative random seed across all tasks and approaches.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 6: Samples for joint training. This can be interpreted as an upper
target.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 7: Samples for finetune. This can be interpreted as a lower target.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 8: Samples for generative replay (2 teacher DDIM steps). Standard
generative replay causes a catastrophic degradation of the denoising
capabilities of the diffusion model for samples from previous tasks.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 9: Samples for generative distillation (2 teacher DDIM steps).
Generative distillation markedly improves the quality with only a moderate
increase in the computational cost.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 10: Samples for generative replay (10 teacher DDIM steps). Increasing
the number of teacher DDIM steps reduces but does solve the catastrophic loss
in denoising capabilities.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 11: Samples for generative distillation (10 teacher DDIM steps).
Generative distillation markedly improves the quality with only a moderate
increase in the computational cost.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 12: Samples for generative replay (100 teacher DDIM steps). Increasing
the number of teacher DDIM steps reduces but does not solve the catastrophic
loss in denoising capabilities.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 13: Samples for generative distillation (100 teacher DDIM steps).
Generative distillation markedly improves the quality with only a moderate
increase in the computational cost.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 14: Samples for LwF-like distillation. The LwF-like approach improves
over naïve generative replay but still has an image quality degradation over
the continual learning process.

(a) Task 1

(b) Task 2

(c) Task 3

(d) Task 4

(e) Task 5

Figure 15: Samples for distillation with Gaussian noise. Similar to distilling
with current data, distilling with noise preserves knowledge of classes
similar to the ones in the last task learned.