# Data-Free Federated Class Incremental Learning with Diffusion-Based
Generative Memory

Naibo Wang
Institute of Data Science
National University of Singapore
Singapore
<EMAIL_ADDRESS>
&Yuchen Deng
Institute of Data Science
National University of Singapore
Singapore
<EMAIL_ADDRESS>
&Wenjie Feng
Institute of Data Science
National University of Singapore
Singapore
<EMAIL_ADDRESS>
&Jianwei Yin
College of Computer Science and Technology
Zhejiang University
Hangzhou, China
<EMAIL_ADDRESS>
&See-Kiong Ng
Institute of Data Science
National University of Singapore
Singapore
<EMAIL_ADDRESS>

###### Abstract

Federated Class Incremental Learning (FCIL) is a critical yet largely
underexplored issue that deals with the dynamic incorporation of new classes
within federated learning (FL). Existing methods often employ generative
adversarial networks (GANs) to produce synthetic images to address privacy
concerns in FL. However, GANs exhibit inherent instability and high
sensitivity, compromising the effectiveness of these methods. In this paper,
we introduce a novel data-free federated class incremental learning framework
with diffusion-based generative memory (DFedDGM) to mitigate catastrophic
forgetting by generating stable, high-quality images through diffusion models.
We design a new balanced sampler to help train the diffusion models to
alleviate the common non-IID problem in FL, and introduce an entropy-based
sample filtering technique from an information theory perspective to enhance
the quality of generative samples. Finally, we integrate knowledge
distillation with a feature-based regularization term for better knowledge
transfer. Our framework does not incur additional communication costs compared
to the baseline FedAvg method. Extensive experiments across multiple datasets
demonstrate that our method significantly outperforms existing baselines,
e.g., over a 4% improvement in average accuracy on the Tiny-ImageNet dataset.

##  1 Introduction

Federated learning (FL) [31, 11] is a promising paradigm that facilitates
collaborative machine learning across multiple clients, allowing them to
construct a unified global model without sharing their local datasets [58, 3].
FL notably enhances data privacy [16] and security [39], while enabling the
derivation of a model with superior inferential capacities compared to those
models developed through individual client-based training [62]. FL has
attracted significant attention in both research and industry communities such
as healthcare [6], autonomous driving [12], and finance [36].

Despite its wide application, most FL frameworks [30, 15, 33] operate under
assumptions that are too constraining for realistic scenarios. A prevalent
assumption is that the local data distributions of clients are static and
unchanging over time, which is rarely the case in real-world settings where
data is often dynamic and changes with the environment [70, 73, 46]. For
instance, in healthcare, models initially trained on historical disease data
must generalize to newly emerging diseases. A relevant example is the
necessity for models to develop in response to the new variants of COVID-19
[7, 71], which continue to evolve because of the virus’s high mutation rate.
Therefore, it is crucial for models to quickly adapt to new data while
maintaining performance on previous data distributions.

An intuitive solution to address the challenge of continuously emerging data
classes is to train new models from scratch. However, this approach is
impractical due to the significant additional computational costs involved. An
alternative way is to apply transfer learning techniques to a previously
trained model, but this approach is hindered by catastrophic forgetting [23,
43], which leads to degraded performance on earlier classes. In centralized
settings, such challenges have been extensively studied within the framework
of continual learning (CL) [67, 51, 64], where various algorithms have been
developed to mitigate catastrophic forgetting from multiple perspectives.

Despite these advancements, most continual learning methods cannot be directly
adapted to the federated learning environment due to the essential disparities
between the two frameworks. For instance, experience replay [54]—a widely-used
strategy that involves storing a subset of past data to preserve some
knowledge of previous distributions during training—poses privacy concerns and
is not suitable for FL. Therefore, recent studies [2, 52, 40] have introduced
Federated Continual Learning (FCL), a paradigm addressing catastrophic
forgetting in FL environments experiencing evolving data classes. A common
scenario in FCL involves the dynamic integration of data with new classes into
local clients, a process known as Federated Class Incremental Learning (FCIL)
[10, 34]. FCIL enables local clients to continuously gather new data with new
classes at any time.

Existing FCIL methods primarily depend on either an unlabeled surrogate
dataset to facilitate FL training [40] or require a memory buffer for storing
historical data [10], which are not suitable in privacy-sensitive FL
environments, such as hospitals or banks where long-term data retention is
discouraged or prohibited. An alternative approach involves utilizing data-
free techniques that do not require real-data storage and employing the
generative adversarial network (GAN) [18] to simulate historical data. These
approaches [2, 69] have attracted considerable attention and have been proven
effective in mitigating the issue of catastrophic forgetting in FCIL. However,
the effectiveness of these strategies is frequently compromised by the
inherent instability and high sensitivity of GANs [60]. Such instability
impairs their capability to develop a robust global model in FCIL. Compared to
GANs, diffusion models [9] offer distinct advantages including precise control
over the generation process, an interpretable latent space, robustness against
overfitting, and enhanced stability [66]. Consequently, diffusion models are
capable of producing images of superior quality.

In this paper, we propose a Data-Free Federated Class Incremental Learning
framework with Diffusion-based Generative Memory (DFedDGM) that employs the
diffusion model to generate stable, high-quality images to mitigate the
catastrophic forgetting issue in FCIL. We design a new balanced sampler to
help train the diffusion models to alleviate the common non-IID [29] problem
in FL. Additionally, since employing the diffusion model would not always
produce higher-quality images with accurate labels which results in
performance degradation, we introduce a novel entropy-based sample filtering
approach from an information theory perspective to filter out low-confidence
samples. Specifically, entropy in information theory quantifies the average
level of uncertainty, making it an appropriate criterion for filtering
uncertain samples. To this end, we achieve this goal by removing the
generative replay samples with low entropy values while preserving those with
high entropy values.

Finally, we propose integrating traditional knowledge distillation [21] with a
feature-based regularization term to enhance knowledge transfer from previous
tasks to the new task while minimizing the feature drift from earlier tasks.
The proposed feature distance loss addresses a common issue in existing
continual learning methods, which typically focus on minimizing the KL
divergence in the prediction space between the teacher and student model [32]
while ignoring the features that contain rich semantic information. To the
best of our knowledge, this is the first work to explore the diffusion model
in FCIL, providing a novel way to address catastrophic forgetting in this
field. With the help of the diffusion model, our approach does not require
historical samples of previous tasks or external datasets. This data-free
approach is particularly useful in scenarios where data sensitivity is a
concern. Our framework does not require any additional communication costs
compared with the baseline method FedAvg [44]. The efficacy and efficiency of
our framework are substantiated through the extensive experimental results of
our paper.

We summarize our contributions as follows:

  * •

We propose a novel data-free federated class incremental learning framework
DFedDGM to alleviate the catastrophic forgetting issue in FCIL by
incorporating diffusion-based generative memory. Our framework incurs no
additional communication costs compared to the baseline method FedAvg.

  * •

We design a novel balanced sampler to assist in the training of diffusion
models to address the prevalent non-IID problem in federated learning.

  * •

We introduce a novel entropy-based sample filtering approach to remove lower-
quality generative samples from an information theory perspective.

  * •

We conduct extensive experiments on three datasets with various non-IID and
task settings to show the efficacy of our method. Our approach consistently
outperforms existing FCIL methods, e.g., over a 4% improvement in average
accuracy on the Tiny-ImageNet dataset.

##  2 Related Work

Continual Learning. Continual learning (CL) [67, 28, 50, 37] has been
extensively studied in recent years to address the problem of catastrophic
forgetting [43, 23, 27] in machine learning. In the CL framework, the training
data is presented to the model as a sequence of datasets, commonly referred to
as tasks. At each time step, the model has access to only one dataset (task)
and seeks to perform well on both current and previous tasks. There are
primarily three incremental learning (IL) settings in CL: Task-IL, Domain-IL,
and Class-IL. In Task-IL [41, 49], tasks are distinct with separate output
spaces identified by task IDs during training and inference phases. In
contrast, Domain-IL [45, 24] maintains a consistent output space across tasks
without the provision of task IDs. Class-IL [46, 42, 4] represents a more
complex scenario in which each new task introduces additional classes to the
output space, progressively increasing the total number of classes. In this
study, we focus on class incremental learning (Class-IL), due to its greater
relevance to real-world applications.

Federated Learning. Federated learning (FL) is a distributed learning
framework that constructs a global model on a central server by aggregating
parameters that are independently learned from private data on multiple client
devices. FedAvg [44] is a popular FL method whose performance is limited due
to the dispersed nature of the data (non-IID). Many FL methods aim to mitigate
the issue of data heterogeneity by refining the training process to enhance
the global model, such as FedProx [30], FedDisco [63], FedDC [15], FedNP [61],
and CCVR [38]. Several methods involve training generative models using
distributed resources to enhance model performance [68, 72]. In this study, we
tackle a more complex situation involving statistical heterogeneity in
federated learning where users’ local data changes over time.

Federated Continual Learning. A few studies have explored continual learning
within a federated learning framework. One pioneering work, FedWeIT [65],
focuses on the Task-IL setting, which requires task IDs during FL training and
inference. For federated class incremental learning (FCIL), FLwF2T [57] and
GLFC [10] have introduced distillation-based methods to address catastrophic
forgetting from both local and global perspectives, respectively. Similarly,
CFeD [40] applies knowledge distillation on both the server and client sides
using a surrogate dataset to alleviate forgetting. More recently, FedET [35]
utilizes an enhanced transformer to facilitate the absorption and transfer of
new knowledge for NLP tasks. Moreover, TARGET [69], MFCL [2], and FedCIL [52]
all employ generative models to create synthetic data from previous tasks to
reduce forgetting, with their efficacy independently confirmed in their
respective papers. In this paper, we aim to improve the global model by
generating more stable and high-quality synthetic images using diffusion
models.

##  3 Problem Definition

Federated Learning (FL). Assume there are $N$ different clients, each client
$i$ has its own private dataset
$D_{i}=\\{(\mathbf{x}_{k},y_{k})\\}_{k=1}^{n_{i}}$ with size $n_{i}$. The goal
of federated learning is for clients to collaboratively develop a global model
$\theta_{g}$ without sharing their private datasets:

$\theta_{g}=\mathop{\operatorname*{arg\,min}}\limits_{\theta}\sum_{i=1}^{N}\mathbb{E}_{(\mathbf{x},y)\sim{\mathbb{P}}_{D_{i}}}[L(\theta;\mathbf{x},y)],$
(1)

where $L(\theta;\mathbf{x},y)$ is the loss function evaluated on a dataset
$D=\\{(\mathbf{x},y)\\}$ with model $\theta$, and ${\mathbb{P}}_{D_{i}}$ is
the data distribution of private dataset ${D_{i}}$.

Class Incremental Learning (Class-IL). Consider a set of $T$ tasks
$\mathcal{T}=\\{\mathcal{T}^{t}\\}_{t=1}^{T}$, each task $\mathcal{T}^{t}$
contains a dataset $D^{t}=\\{(\mathbf{x}_{k}^{t},y_{k}^{t})\\}_{k=1}^{n^{t}}$
with size $n^{t}$. Every new task $\mathcal{T}^{t}$ introduces
$n^{t}_{\mathcal{C}}$ new classes
$\mathcal{C}^{t}=\\{c_{j}^{t}\\}_{j=1}^{n^{t}_{\mathcal{C}}}$ that are not
present in the previous tasks. During the training of task $\mathcal{T}^{t}$,
datasets from all previous tasks $\\{D^{i}|i<t\\}$ are inaccessible. The
objective of class incremental learning is to develop a model $\theta^{T}$
that not only performs effectively on the last task $\mathcal{T}^{T}$, but
also preserves its performance on all previous tasks
$\\{\mathcal{T}^{t}\\}_{t=1}^{T-1}$ to avoid catastrophic forgetting:

$\theta^{T}=\operatorname*{arg\,min}_{\theta}\sum_{t=1}^{T}\mathbb{E}_{(\mathbf{x},y)\sim{\mathbb{P}}_{D^{t}}}[L(\theta;\mathbf{x},y)],$
(2)

Federated Class Incremental Learning (FCIL). FCIL incorporates both federated
learning and class incremental learning principles to develop a global model
that can incrementally learn new classes across multiple distributed clients
while maintaining data privacy. Within this framework, we have in total $T$
tasks $\mathcal{T}=\\{\mathcal{T}^{t}\\}_{t=1}^{T}$, each client $i$ will
independently learn from $T$ local tasks
$\mathcal{T}_{i}=\\{\mathcal{T}^{t}_{i}\\}_{t=1}^{T}$ in a class-incremental
way, and $\mathcal{T}^{t}=\cup_{i=1}^{N}\mathcal{T}^{t}_{i}$. At a given step
$t$, client $i$ will only have access to dataset $D_{i}^{t}$ from task
$\mathcal{T}^{t}_{i}$. During training, clients will communicate with the
central server to update and obtain the global model. The objective of FCIL is
to develop a comprehensive global model, $\theta_{g}^{T}$, that effectively
predicts all classes from all $T$ tasks encountered by all $N$ clients:

$\theta_{g}^{T}=\operatorname*{arg\,min}_{\theta}\sum_{i=1}^{N}\sum_{t=1}^{T}\mathbb{E}_{(\mathbf{x},y)\sim{\mathbb{P}}_{D_{i}^{t}}}[L(\theta;\mathbf{x},y)],$
(3)

##  4 Federated Class Incremental Learning with DFedDGM

In FCIL, storing previous data is considered a violation of the FL setup and
is thus prohibited. In this paper, we propose a data-free framework DFedDGM
which employs diffusion models to generate synthetic datasets to tackle the
catastrophic forgetting problem in FCIL while adhering to the data privacy
standards of FL. The overview of our framework is shown in Figure 1.

Figure 1: Overview of our framework. A diffusion model is trained with the
help of the Balanced Sampler (II). We utilize the diffusion model to generate
synthetic images, and assign labels to these images by the global model
$\theta_{g}^{t-1}$ from the previous task. An Entropy Filter subsequently
screens these samples. We combine three losses
($\mathcal{L}_{CE}^{t},\mathcal{L}_{KD}^{t}$ and $\mathcal{L}_{FD}^{t}$) to
train the local model $\theta_{i}^{t}$ (III).

Previous studies focused on training a GAN on the server side to generate
synthetic data. However, the quality of the generated images tends to be
unstable and highly sensitive to the training hyperparameters due to the
inherent limitations of GAN [1]. Moreover, the model-inversion technique
necessitates pre-assigned labels for image generation, rather than utilization
of labels predicted by the global model. This method often fails to accurately
capture the true distribution or nuances of the original data [20].
Consequently, it is advisable to train a generator directly on the client side
to produce higher-quality images. The global model will then label these
images, thereby creating synthetic samples for subsequent tasks.

Diffusion models offer several advantages [66] over GANs, including improved
control during the generation phase, increased robustness to overfitting, and
enhanced stability. In this study, we propose training a diffusion model
subsequent to local training on every client $i$ using dataset $D_{i}^{t}$
from the current task $\mathcal{T}^{t}_{i}$ to assist future task training. We
first introduce how to train diffusion models in Section 4.1, then elaborate
on the local training process using the diffusion model in Section 4.2.

###  4.1 Train Diffusion Models with Balanced Sampler

Due to the pervasive non-IID issue in federated learning (FL) [30], employing
the traditional random sampling method from dataset $D_{i}^{t}$ to train the
diffusion model is inadvisable. This method could lead to the generation of
images with a highly unbalanced distribution. In extreme cases, it is
imaginable that images from certain classes might not be generated at all.
This imbalance may lead the training of the local model to diverge from the
global optimum. Therefore, it is essential to develop a training strategy for
the diffusion model to address the non-IID issues during image generation.

0: Local dataset $D_{i}^{t}$ of task $\mathcal{T}^{t}_{i}$ on client $i$,
diffusion model $\mathcal{D}_{i}^{t-1}$ from task $\mathcal{T}^{t-1}_{i}$ of
client $i$, local training epochs $E_{\mathcal{D}}$, batch size $B$, number of
new classes $n^{t}_{i}$ for task $\mathcal{T}^{t}_{i}$ on client $i$.

0: Diffusion model $\mathcal{D}_{i}^{t}$.

1: $\mathcal{D}_{i}^{t}\leftarrow\mathcal{D}_{i}^{t-1}$ // For the first task
$t=1$, random initialize $\mathcal{D}_{i}^{1}$

2: $B_{\mathcal{C}}\leftarrow\lceil\frac{B}{n^{t}_{i}}\rceil$ // Number of
samples selected from every class for every batch

3:
$N_{\mathcal{C}}\leftarrow\max\\{|D_{i}^{t}\textbf{[}c_{j}^{t}\textbf{]}|\\}_{j=1}^{n^{t}_{i}}$
// The sample numbers of the class with the most samples

4: $E_{B}\leftarrow\lceil\frac{N_{\mathcal{C}}}{B_{\mathcal{C}}}\rceil$ //
Total number of batches for every epoch

5: for local epoch $e=1,2,\ldots,E_{\mathcal{D}}$ do

6: for class $c_{j}^{t}=1,2,\ldots,n^{t}_{i}$ do

7:
$\mathcal{S}_{j}^{t}\leftarrow\mathcal{X}(D_{i}^{t}\textbf{[}c_{j}^{t}\textbf{]})$
// $\mathcal{X}(D)$ means select the images part of $D$

8: for
$l=1,2,\cdots,\lceil\frac{N_{\mathcal{C}}}{|\mathcal{S}_{j}^{t}|}\rceil-1$ do

9:
$\mathcal{S}_{j}^{t}\leftarrow\mathcal{S}_{j}^{t}\cup\textit{Reshuffle}(\mathcal{X}(D_{i}^{t}\textbf{[}c_{j}^{t}\textbf{]}))$

10: end for

11: end for

12: for batch $b=1,2,\ldots,E_{B}$ do

13: $\mathcal{S}\leftarrow\emptyset$ // Sample set for current batch

14: for class $c_{j}^{t}=1,2,\ldots,n^{t}_{i}$ do

15:
$\mathcal{S}\leftarrow\mathcal{S}\cup\mathcal{S}_{j}^{t}\textbf{[}(b-1)\times
B_{\mathcal{C}}:b\times B_{\mathcal{C}}\textbf{]}$

16: end for

17: $\mathcal{D}_{i}^{t}\leftarrow$ Update$(\mathcal{D}_{i}^{t},\mathcal{S})$
// Train diffusion model with sample set $\mathcal{S}$ by method like DDPM

18: end for

19: end for

Algorithm 1 Training procedure for the diffusion model with Balanced Sampler

As shown in Algorithm 1, we have designed a mechanism named Balanced Sampler
to help train the diffusion model, enabling it to generate more balanced
images across different classes. Due to the non-IID nature of FL, the number
of samples for each class on a client can vary significantly. Therefore,
instead of randomly selecting $B$ samples from all samples for each batch, our
approach involves selecting an equal number of $B_{\mathcal{C}}$ samples from
each class to form a training data batch. This strategy allows the diffusion
model to consistently learn from an equitable distribution of samples across
all classes in every iteration, effectively addressing the challenges posed by
non-IID data.

In detail, we generate $E_{B}$ batches of samples per training epoch. Every
batch consists of $B_{\mathcal{C}}$ randomly selected samples from each class,
leading to a total of $B$ samples per batch (Lines 2-4). However, since the
number of samples is imbalanced, there will be a batch $b$ in which all
samples of a specific class $c_{j}^{t}$ are selected in earlier batches,
leaving some samples from other classes unselected. In such cases, we
reshuffle the samples from class $c_{j}^{t}$ and start a cycle of new random
selection for class $c_{j}^{t}$ into the batch. This process of reshuffling is
repeated until all samples from the class with the largest sample size have
been completely incorporated for one training epoch (Lines 6-18). This
sampling strategy ensures that the diffusion model consistently interacts with
samples from every class in each training batch, thereby producing more class-
balanced images.

All the available training methods for diffusion models, such as DDPM [22] and
DDIM [56], are applicable for training our generative model. Furthermore, the
diffusion model $\mathcal{D}_{i}^{t}$ is initialized with the model
$\mathcal{D}_{i}^{t-1}$, which was trained on the previous task
$\mathcal{T}^{t-1}_{i}$ (Line 1), enabling it to retain the data distributions
from previous tasks. This allows the model to remember and generate samples
for both the current and previous tasks, thus effectively addressing the issue
of catastrophic forgetting.

###  4.2 Entropy Filter for Selecting High-Quality Generative Replay Samples

In our framework, client $i$ will utilize its local dataset $D_{i}^{1}$ to
train the local model $\theta^{1}_{i}$ at the first learning task
$\mathcal{T}^{1}_{i}$. For subsequent tasks $\\{\mathcal{T}^{t}_{i}|t>1\\}$,
the diffusion model $\mathcal{D}_{i}^{t-1}$ trained from the previous task
$\mathcal{T}^{t-1}_{i}$ will generate synthetic data to help the client in
relieving catastrophic forgetting.

Given random noises $\textbf{z}\sim\textbf{{N}}({\bf 0},{\bf I})$, the number
of samples to be generated $n_{s}$, and the retention ratio $\lambda$ of the
Entropy Filter (to be discussed later), we can utilize existing sampling
methods, such as DDPM [22] or DDIM [56], to generate $\frac{n_{s}}{\lambda}$
synthetic images $\\{\mathbf{x}_{i,s}^{t}\\}$ with the diffusion model
$\mathcal{D}_{i}^{t-1}:\\{\mathbf{x}_{i,s}^{t}\\}\leftarrow\mathcal{D}_{i}^{t-1}(\textbf{z},\frac{n_{s}}{\lambda})$.
Since diffusion models produce images without labels, it is necessary to
assign labels to these generated images for subsequent training.

Given that the global model generally outperforms local models on clients in
FL, we utilize the global model $\theta_{g}^{t-1}$ to annotate the generated
images on the clients:
$\\{y_{i,s}^{t}\\}=\\{\theta_{g}^{t-1}(\mathbf{x}_{i,s}^{t})\\}$. However, the
global model does not always exhibit high confidence in the labels assigned to
certain images. Since labels are determined by selecting the class with the
highest probability after applying the softmax function to the logits from the
final layer of the model, there are cases where a class is chosen as the label
simply because its probability is slightly higher than that of other classes.
These low-confidence labels often do not accurately match the generated
images, leading to fake samples that impair training performance. Hence, it is
essential to design a method to filter out such data.

To address the above issue, we propose an Entropy Filter, inspired by
information theory. According to information theory, entropy measures the
amount of information conveyed by an event, whereby higher entropy corresponds
to greater information value. For each image, we calculate the entropy of the
probability distribution across all classes to evaluate the global model’s
confidence in the assigned label. A higher entropy value indicates greater
confidence in the image and we should maintain these samples. The entropy of
the predictions achieved by the global model is defined as:

$H(\theta_{g}^{t-1}(\mathbf{x}))=-\sum\nolimits_{k=1}^{t-1}\sum\nolimits_{j=1}^{n_{\mathcal{C}}^{k}}p(c_{j}^{k})\log
p(c_{j}^{k}),$ (4)

where $p(c_{j}^{k})$ is the probability of class $c_{j}^{k}$ calculated by
global model $\theta_{g}^{t-1}$. We calculate the entropy of all generated
images and sort them in descending order. We then retain only the top
$\lambda$ portion of the samples, thereby filtering out samples for which the
global model exhibits low confidence:

$D^{t}_{i,s}=\\{(\mathbf{x}_{i,s}^{t},y_{i,s}^{t})\mid
i\in\mathcal{I}_{\lambda}\\},\mathcal{I}_{\lambda}=\\{i\mid
i\leq\lambda\cdot|\\{H(\theta_{g}^{t-1}(\mathbf{x}_{i,s}^{t}))\\}|\\},$ (5)

where $|\\{H(\theta_{g}^{t-1}(\mathbf{x}_{i,s}^{t}))\\}|$ is the original
number of generated images ($\frac{n_{s}}{\lambda}$). The refined generated
dataset $D^{t}_{i,s}$ is then combined with the real dataset
$D^{t}_{i}=\\{(\mathbf{x}_{i}^{t},y_{i}^{t})\\}$ for local model training.

###  4.3 The Final Objective Function

Following the standard FL paradigm, we first apply the Cross-Entropy (CE) loss
function to minimize the divergence between the predicted and true
distributions of new classes in task $\mathcal{T}^{t}_{i}$ from dataset
$D^{t}_{i}$, as well as the true distribution of old classes across task
$\mathcal{T}^{1}_{i}$ to $\mathcal{T}^{t-1}_{i}$ with dataset $D^{t}_{i,s}$:

$\mathcal{L}_{CE}=CE(\theta^{t}_{i}(\mathbf{x}),y;D_{i}^{t},D^{t}_{i,s}).$ (6)

However, solely applying cross-entropy is insufficient to effectively address
the problem of forgetting. To address this issue, we propose to use Knowledge
Distillation [21] to transfer knowledge from the previous global model
$\theta^{t-1}_{g}$ to the current local model $\theta^{t}_{i}$, thereby
aligning the output distributions of the two models. It is essential when
dealing with continuously changing data distributions, as it enhances the
model’s robustness and generalization capabilities across various tasks:

$\mathcal{L}_{KD}=KL(\theta^{t}_{i}(\mathbf{x}),\theta^{t-1}_{g}(\mathbf{x});D_{i}^{t},D^{t}_{i,s}),$
(7)

where $KL$ is the Kullback-Leibler divergence between the outputs of the
student model $\theta^{t}_{i}(\mathbf{x})$ and the teacher model
$\theta^{t-1}_{g}(\mathbf{x})$ over the data distributions from $D_{i}^{t}\cup
D^{t}_{i,s}$.

Finally, to ensure that the most significant features with richer semantic
information from old models are transferred, while allowing the model to learn
new features from new tasks better [55], we introduce an additional Feature
Distance loss to control the drift of feature distribution:

$\mathcal{L}_{FD}=||h_{i}^{t}(\mathbf{x};D_{i}^{t},D^{t}_{i,s})-h_{g}^{t-1}(\mathbf{x};D_{i}^{t},D^{t}_{i,s})||^{2}_{2},$
(8)

where $||\cdot||^{2}_{2}$ denotes the L2 distance.
$h_{i}^{t}(\mathbf{x};D_{i}^{t},D^{t}_{i,s})$ and
$h_{g}^{t-1}(\mathbf{x};D_{i}^{t},D^{t}_{i,s})$ are feature representations
from the feature extractors of the current local model and the previous global
model. Minimizing this loss ensures the features remain stable over time in
the model’s feature space, which is crucial for tasks requiring consistent
feature interpretations amidst continuously evolving data distributions.

To summarize, the total loss function for model $\theta_{i}^{t}$ is formed as
follows:

$\mathcal{L}(\theta_{i}^{t})=\mathcal{L}_{CE}+\alpha\mathcal{L}_{KD}+\gamma\mathcal{L}_{FD},$
(9)

where $\alpha$ and $\gamma$ are two hyperparameters controlling the effect of
both losses on model training. The values of $\alpha$ and $\gamma$ are
determined empirically in the experiments.

###  4.4 Model Aggregation and Analysis

In our framework, diffusion models on local clients remain private and are not
shared to avoid potential privacy concerns and reduce communication costs. The
server simply collects and aggregates the local models from clients to produce
a comprehensive global model. Our approach greatly diminishes communication
overhead compared to studies that create a synthetic dataset on the server and
distribute it to clients. The pseudocode of our framework is provided in
Algorithm 2 in the appendix.

Communication Cost. In our method, each client sends only their local model to
the central server. Thus, the overall communication cost for $T$ tasks on $N$
clients is $O(TNRM)$, where $R$ is the communication rounds per task and $M$
is the size of the local model, consistent with FedAvg [44].

##  5 Experiments

###  5.1 Experimental Setup

Datasets and Settings. We evaluate the Average Accuracy ($Acc$) and Average
Forgetting of different methods with three challenging datasets: EMNIST-
Letters [8], CIFAR-100 [25], and Tiny-ImageNet [26], consisting of 26, 100,
and 200 classes, respectively. By default, each dataset is partitioned into
$T=5$ non-overlapping tasks and distributed among $N=10$ clients using a
Dirichlet distribution [47] with $\beta=0.5$. The clients train local models
with the ResNet-18 [19] structure. For the diffusion model, we employ the same
architecture as DDPM [22] on the clients.

Baselines. We compare our method with the following baselines: FedAvg [44],
FedProx [30], FedEWC [27], FLwF-2T [57], FedCIL [52], TARGET [69], and MFCL
[2]. FedAvg and FedProx are two representative methods in federated learning,
which we have adapted to FCIL setting; FedEWC is the federated adaptation of
the incremental moment matching approach commonly used in continual learning;
FLwF-2T utilized knowledge distillation to address catastrophic forgetting in
FCIL; FedCIL is a generative replay method for clients to train the ACGAN [48]
to produce synthetic samples from previous tasks. Lastly, TARGET and MFCL are
two recent generator-based methods to invert images from the global model on
the server with different types of losses.

For more implementation details about the experimental setup, such as the
details of metrics, optimizer, learning rate and batch size we employed,
please refer to Section B of the appendix.

###  5.2 Performance of DFedDGM

Table 1: Performance (%, mean$\pm$std) comparison of our method to other
baselines on $T$=5 tasks.

Dataset | EMNIST-Letters | CIFAR-100 | Tiny-ImageNet
---|---|---|---
Metric | Acc ($\uparrow$) |  $\mathcal{F}$ ($\downarrow$) | Acc ($\uparrow$) |  $\mathcal{F}$ ($\downarrow$) | Acc ($\uparrow$) |  $\mathcal{F}$ ($\downarrow$)
FedAvg | 45.41$\pm$0.49 | 77.48$\pm$0.68 | 38.42$\pm$0.44 | 61.65$\pm$0.81 | 24.53$\pm$1.44 | 42.75$\pm$0.98
FedProx | 46.83$\pm$0.92 | 79.42$\pm$1.33 | 38.48$\pm$0.49 | 61.03$\pm$0.44 | 24.64$\pm$1.02 | 43.18$\pm$1.70
FedEWC | 62.92$\pm$0.87 | 70.46$\pm$0.66 | 40.74$\pm$0.54 | 56.83$\pm$0.68 | 25.38$\pm$1.28 | 36.27$\pm$1.16
FLwF-2T | 76.44$\pm$0.95 | 32.26$\pm$0.84 | 50.43$\pm$0.18 | 33.28$\pm$0.66 | 26.02$\pm$0.62 | 31.64$\pm$0.53
FedCIL | 79.59$\pm$0.43 | 31.07$\pm$0.29 | 53.18$\pm$0.74 | 32.66$\pm$0.29 | 27.45$\pm$1.74 | 30.02$\pm$0.58
TARGET | 77.36$\pm$0.59 | 28.83$\pm$0.37 | 54.01$\pm$0.56 | 29.41$\pm$0.73 | 27.88$\pm$0.89 | 29.83$\pm$0.76
MFCL | 80.21$\pm$0.44 | 29.25$\pm$0.58 | 48.87$\pm$0.26 | 31.47$\pm$0.67 | 29.08$\pm$0.73 | 27.61$\pm$0.63
DFedDGM (Ours) | 85.33$\pm$0.36 | 26.34$\pm$0.28 | 57.08$\pm$0.58 | 27.64$\pm$0.74 | 33.55$\pm$0.69 | 23.54$\pm$0.72

Main results. Table 1 presents the performance of different methods. Results
show that our DFedDGM method consistently outperforms all other methods across
all datasets. Specifically, DFedDGM exceeds existing methods by over 5% in
average accuracy ($Acc$) on the EMNIST-Letters dataset, and reduces average
forgetting ($\mathcal{F}$) by more than 4% on the Tiny-ImageNet dataset.
Notably, FedAvg and FedProx exhibit the highest levels of forgetting as they
are not designed for FCIL. FedEWC, which employs regularization constraints
during training, failed to prevent forgetting due to insufficient data
availability. In contrast, distillation-based approaches, including FLwF-2T,
TARGET, MFCL, and our method, effectively improve average accuracy and
mitigate catastrophic forgetting by transferring knowledge from old to new
tasks. This underscores the efficacy of knowledge distillation.

On the other hand, generative methods like FedCIL, TARGET, MFCL, and our
proposed method significantly enhance accuracy and mitigate forgetting by
generating synthetic datasets for future tasks. This underscores the capacity
of synthetic data to accurately reflect the distribution of historical data.
Among these approaches, our method exhibits the least forgetting and the
highest accuracy, substantiating its effectiveness in preserving knowledge
from previous tasks.

Figure 2: Test average accuracy vs. the number of observed tasks for (a)
CIFAR-100 on $T=5$ tasks, (b) Tiny-ImageNet on $T=5$ tasks, (c) Tiny-ImageNet
on $T=10$ tasks.

Effect of the number of tasks. Figure 2 depicts the model’s performance on
both current and prior tasks after the completion of each task. The curves
indicate that our proposed model consistently surpasses other baseline methods
in all incremental tasks, irrespective of the number of tasks (whether $T=5$
or $T=10$). This result highlights the efficacy of our approach in enabling
multiple local clients to learn new classes sequentially while alleviating the
issue of forgetting.

Impact of data distribution. We performed experiments across various non-IID
settings by adjusting the Dirichlet parameters to $\\{$0.05, 0.1, 0.3,
0.5$\\}$, as well as the IID setting. Figure 3 (a) shows that our approach
consistently outperforms the Highest Baseline, which is the highest accuracy
achieved by any of the baseline methods, across all non-IID settings. This
highlights the notable efficacy of our approach in alleviating catastrophic
forgetting, even when faced with extreme data distributions.

(a) Non-IIDness

(b) Ablation method

Figure 3: (a) Average accuracy under different data distribution settings on
the CIFAR-100 dataset, (b) Ablation studies on the CIFAR-100 dataset.

###  5.3 Ablation Studies

In Figure 3 (b), we illustrate the significance of each component in our
framework. We can see that the Balanced Sampler is crucial for the
performance, underscoring the necessity of generating balanced samples to
mitigate the effects of non-IID data. Removing the Entropy Filter leads to a
performance decline due to the inclusion of excessive synthetic samples with
low-confidence labels, highlighting the importance of filtering data based on
prediction entropy. Additionally, the knowledge distillation (KD) loss is also
vital for performance enhancement, validating its importance in mitigating
forgetting; the FD loss also contributes to an accuracy improvement,
emphasizing the importance of maintaining temporal consistency within the
feature space for continual learning.

Lastly, we evaluate the performance of our method by varying the number of
generated samples $n_{s}$ for the diffusion model. As illustrated in Figure 3
(b), the optimal value for $n_{s}$ is 2000, and performance declines when
generating either too few samples (500) or too many samples (3000). Generating
too few samples does not provide sufficient synthetic data for the model to
learn from previous tasks, while generating too many samples impedes the
model’s ability to learn new information from the current task. Therefore,
striking a balance of $n_{s}$ is crucial for achieving optimal model
performance.

###  5.4 Further Analysis

Figure 4: Real vs synthetic data generated by the diffusion model for the
Tiny-ImageNet dataset.

Privacy of DFedDGM. In contrast to existing FCIL methods [69, 2], which
require clients to share locally trained generative models or synthetic
images, our diffusion model is never shared with others. Consequently, our
method does not introduce additional privacy concerns and retains the same
privacy protections as FedAvg [44]. Specifically, our method can effectively
defend attacks such as model poisoning [13], backdoor attacks [5], and
gradient inversion [17] through techniques like secure aggregation [14] and
differential privacy [59].

Figure 4 presents several synthetic images that exhibit notable differences
from real data. Nevertheless, they are clear to identify and contain essential
class-specific knowledge to accurately represent the entire class, which
facilitates knowledge transfer and substantially mitigates catastrophic
forgetting.

Limitations. Due to the inherent limitations of the diffusion model [22], our
method introduces additional computational overhead and extends the training
time compared to other approaches. However, unlike other generative methods,
our framework does not incur additional communication costs, which is a more
critical issue in FL. Compared to other GAN-based methods, our approach
produces more stable images and offers significant improvements in model
performance. Furthermore, the training time can also be reduced by employing
advanced techniques such as the quick sampling method DDIM [56], which we plan
to incorporate in future work to better adapt to the FCIL scenarios.

##  6 Conclusion

This paper presents a novel framework, DFedDGM to mitigate the issue of
catastrophic forgetting in federated class incremental learning (FCIL). We
employ diffusion models to generate high-quality synthetic images on the
client side in a data-free manner, and utilize knowledge distillation to
efficiently transfer information from previous tasks. Extensive experiments
demonstrate the effectiveness of our method compared to existing FCIL
frameworks.

## References

  * Alqahtani et al. [2021] Hamed Alqahtani, Manolya Kavakli-Thorne, and Gulshan Kumar.  Applications of generative adversarial networks (gans): An updated review.  _Archives of Computational Methods in Engineering_ , 28:525–552, 2021.
  * Babakniya et al. [2023] Sara Babakniya, Zalan Fabian, Chaoyang He, Mahdi Soltanolkotabi, and Salman Avestimehr.  A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Balkus et al. [2022] Salvador V Balkus, Honggang Wang, Brian D Cornet, Chinmay Mahabal, Hieu Ngo, and Hua Fang.  A survey of collaborative machine learning using 5g vehicular communications.  _IEEE Communications Surveys & Tutorials_, 24(2):1280–1303, 2022.
  * Belouadah and Popescu [2019] Eden Belouadah and Adrian Popescu.  Il2m: Class incremental learning with dual memory.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 583–592, 2019.
  * Chen et al. [2022] Chien-Lun Chen, Sara Babakniya, Marco Paolieri, and Leana Golubchik.  Defending against poisoning backdoor attacks on federated meta-learning.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 13(5):1–25, 2022.
  * Chen et al. [2023] Yiqiang Chen, Wang Lu, Xin Qin, Jindong Wang, and Xing Xie.  Metafed: Federated learning among federations with cyclic knowledge distillation for personalized healthcare.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * Ciotti et al. [2020] Marco Ciotti, Massimo Ciccozzi, Alessandro Terrinoni, Wen-Can Jiang, Cheng-Bin Wang, and Sergio Bernardini.  The covid-19 pandemic.  _Critical reviews in clinical laboratory sciences_ , 57(6):365–388, 2020.
  * Cohen et al. [2017] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik.  Emnist: Extending mnist to handwritten letters.  In _2017 international joint conference on neural networks (IJCNN)_ , pages 2921–2926. IEEE, 2017.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Dong et al. [2022] Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu.  Federated class-incremental learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10164–10173, 2022.
  * Duan [2023] Moming Duan.  Towards open federated learning platforms: Survey and vision from technical and legal perspectives.  _arXiv preprint arXiv:2307.02140_ , 2023.
  * Elbir et al. [2022] Ahmet M Elbir, Burak Soner, Sinem Çöleri, Deniz Gündüz, and Mehdi Bennis.  Federated learning in vehicular networks.  In _2022 IEEE International Mediterranean Conference on Communications and Networking (MeditCom)_ , pages 72–77. IEEE, 2022.
  * Fang et al. [2020] Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong.  Local model poisoning attacks to $\\{$Byzantine-Robust$\\}$ federated learning.  In _29th USENIX security symposium (USENIX Security 20)_ , pages 1605–1622, 2020.
  * Fereidooni et al. [2021] Hossein Fereidooni, Samuel Marchal, Markus Miettinen, Azalia Mirhoseini, Helen Möllering, Thien Duc Nguyen, Phillip Rieger, Ahmad-Reza Sadeghi, Thomas Schneider, Hossein Yalame, et al.  Safelearn: Secure aggregation for private federated learning.  In _2021 IEEE Security and Privacy Workshops (SPW)_ , pages 56–62. IEEE, 2021.
  * Gao et al. [2022] Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu.  Feddc: Federated learning with non-iid data via local drift decoupling and correction.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10112–10121, 2022.
  * Gao et al. [2021] Wei Gao, Shangwei Guo, Tianwei Zhang, Han Qiu, Yonggang Wen, and Yang Liu.  Privacy-preserving collaborative learning with automatic transformation search.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 114–123, 2021.
  * Geiping et al. [2020] Jonas Geiping, Hartmut Bauermeister, Hannah Droge, and Michael Moeller.  Inverting gradients-how easy is it to break privacy in federated learning?  _NeurIPS_ , 33:16937–16947, 2020.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2020] Yingzhe He, Guozhu Meng, Kai Chen, Xingbo Hu, and Jinwen He.  Towards security threats of deep learning systems: A survey.  _IEEE Transactions on Software Engineering_ , 48(5):1743–1770, 2020.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526, 2017.
  * Kiyasseh et al. [2021] Dani Kiyasseh, Tingting Zhu, and David Clifton.  A clinical deep learning framework for continually learning from cardiac signals across diseases, time, modalities, and institutions.  _Nature Communications_ , 12(1):4221, 2021.
  * Krizhevsky et al. [2009] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Le and Yang [2015] Ya Le and Xuan Yang.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7):3, 2015.
  * Lee et al. [2017] Sang-Woo Lee, Jin-Hwa Kim, Jaehyun Jun, Jung-Woo Ha, and Byoung-Tak Zhang.  Overcoming catastrophic forgetting by incremental moment matching.  _Advances in neural information processing systems_ , 30, 2017.
  * Lesort et al. [2020] Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Díaz-Rodríguez.  Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges.  _Information fusion_ , 58:52–68, 2020.
  * Li et al. [2022] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He.  Federated learning on non-iid data silos: An experimental study.  In _ICDE_ , pages 965–978. IEEE, 2022.
  * Li et al. [2020a] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks.  _Proceedings of Machine learning and systems_ , 2:429–450, 2020a.
  * Li et al. [2020b] Tian Li, Virginia Smith, et al.  Federated learning: Challenges, methods, and future directions.  _IEEE Signal Processing Magazine_ , 37(3):50–60, 2020b.
  * Li et al. [2023] Xiaorong Li, Shipeng Wang, Jian Sun, and Zongben Xu.  Variational data-free knowledge distillation for continual learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Li et al. [2021] Xiaoxiao Li, Meirui Jiang, Xiaofei Zhang, Michael Kamp, and Qi Dou.  Fedbn: Federated learning on non-iid features via local batch normalization.  _arXiv preprint arXiv:2102.07623_ , 2021.
  * Liu et al. [2023a] Chenghao Liu, Xiaoyang Qu, Jianzong Wang, and Jing Xiao.  Fedet: A communication-efficient federated class-incremental learning framework based on enhanced transformer.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23_ , pages 3984–3992, 2023a.
  * Liu et al. [2023b] Chenghao Liu, Xiaoyang Qu, Jianzong Wang, and Jing Xiao.  Fedet: a communication-efficient federated class-incremental learning framework based on enhanced transformer.  _arXiv preprint arXiv:2306.15347_ , 2023b.
  * Long et al. [2020] Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang.  Federated learning for open banking.  In _Federated Learning: Privacy and Incentive_ , pages 240–254. Springer, 2020.
  * Lopez-Paz and Ranzato [2017] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Luo et al. [2021] Mi Luo, Fei Chen, Dapeng Hu, Yifan Zhang, Jian Liang, and Jiashi Feng.  No fear of heterogeneity: Classifier calibration for federated learning with non-iid data.  _Advances in Neural Information Processing Systems_ , 34:5972–5984, 2021.
  * Ma et al. [2020] Chuan Ma, Jun Li, Ming Ding, Howard H Yang, Feng Shu, Tony QS Quek, and H Vincent Poor.  On safeguarding privacy and security in the framework of federated learning.  _IEEE network_ , 34(4):242–248, 2020.
  * Ma et al. [2022] Yuhang Ma, Zhongle Xie, Jue Wang, Ke Chen, and Lidan Shou.  Continual federated learning based on knowledge distillation.  In _IJCAI_ , pages 2182–2188, 2022.
  * Masana et al. [2021] Marc Masana, Tinne Tuytelaars, and Joost Van de Weijer.  Ternary feature masks: zero-forgetting for task-incremental learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3570–3579, 2021.
  * Masana et al. [2022] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer.  Class-incremental learning: survey and performance evaluation on image classification.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5513–5533, 2022.
  * McCloskey and Cohen [1989] Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier, 1989.
  * McMahan et al. [2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pages 1273–1282. PMLR, 2017.
  * Mirza et al. [2022] M Jehanzeb Mirza, Marc Masana, Horst Possegger, and Horst Bischof.  An efficient domain-incremental learning approach to drive in all weather conditions.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3001–3011, 2022.
  * Mittal et al. [2021] Sudhanshu Mittal, Silvio Galesso, and Thomas Brox.  Essentials for class incremental learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3513–3522, 2021.
  * Ng et al. [2011] Kai Wang Ng, Guo-Liang Tian, and Man-Lai Tang.  Dirichlet and related distributions: Theory, methods and applications.  2011\.
  * Odena et al. [2017] Augustus Odena, Christopher Olah, and Jonathon Shlens.  Conditional image synthesis with auxiliary classifier gans.  In _International conference on machine learning_ , pages 2642–2651. PMLR, 2017.
  * Oren and Wolf [2021] Guy Oren and Lior Wolf.  In defense of the learning without forgetting for task incremental learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2209–2218, 2021.
  * Pan et al. [2020] Pingbo Pan, Siddharth Swaroop, Alexander Immer, Runa Eschenhagen, Richard Turner, and Mohammad Emtiyaz E Khan.  Continual deep learning by functional regularisation of memorable past.  _Advances in neural information processing systems_ , 33:4453–4464, 2020.
  * Parisi et al. [2019] German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural networks_ , 113:54–71, 2019.
  * Qi et al. [2023] Daiqing Qi, Handong Zhao, and Sheng Li.  Better generative replay for continual federated learning.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=cRxYWKiTan.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_ , pages 234–241. Springer, 2015.
  * Shin et al. [2017] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  _Advances in neural information processing systems_ , 30, 2017.
  * Smith et al. [2021] James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira.  Always be dreaming: A new approach for data-free class-incremental learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9374–9384, 2021.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=St1giarCHLP.
  * Usmanova et al. [2021] Anastasiia Usmanova, François Portet, Philippe Lalanda, and German Vega.  A distillation-based approach integrating continual learning and federated learning for pervasive services.  _arXiv preprint arXiv:2109.04197_ , 2021.
  * Wang et al. [2022] Junbo Wang, Amitangshu Pal, Qinglin Yang, Krishna Kant, Kaiming Zhu, and Song Guo.  Collaborative machine learning: Schemes, robustness, and privacy.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * Wei et al. [2020] Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor.  Federated learning with differential privacy: Algorithms and performance analysis.  _IEEE transactions on information forensics and security_ , 15:3454–3469, 2020.
  * Wiatrak et al. [2019] Maciej Wiatrak, Stefano V Albrecht, and Andrew Nystrom.  Stabilizing generative adversarial networks: A survey.  _arXiv preprint arXiv:1910.00927_ , 2019.
  * Wu et al. [2023] Xueyang Wu, Hengguan Huang, Youlong Ding, Hao Wang, Ye Wang, and Qian Xu.  Fednp: Towards non-iid federated learning via federated neural propagation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 10399–10407, 2023.
  * Yang et al. [2019] Qiang Yang, Yang Liu, Tianjian Chen, and Yongxin Tong.  Federated machine learning: Concept and applications.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ , 10(2):1–19, 2019.
  * Ye et al. [2023] Rui Ye, Mingkai Xu, Jianyu Wang, Chenxin Xu, Siheng Chen, and Yanfeng Wang.  Feddisco: Federated learning with discrepancy-aware collaboration.  In _International Conference on Machine Learning_ , pages 39879–39902. PMLR, 2023.
  * Yoon et al. [2018] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang.  Lifelong learning with dynamically expandable networks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=Sk7KsfW0-.
  * Yoon et al. [2021] Jaehong Yoon, Wonyong Jeong, Giwoong Lee, Eunho Yang, and Sung Ju Hwang.  Federated continual learning with weighted inter-client transfer.  In _International Conference on Machine Learning_ , pages 12073–12086. PMLR, 2021.
  * Yu et al. [2023] Jiwen Yu, Xuanyu Zhang, Youmin Xu, and Jian Zhang.  Cross: Diffusion model makes controllable, robust and secure image steganography.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Zenke et al. [2017] Friedemann Zenke, Ben Poole, and Surya Ganguli.  Continual learning through synaptic intelligence.  In _International conference on machine learning_ , pages 3987–3995. PMLR, 2017.
  * Zhang et al. [2022] Jie Zhang, Chen Chen, Bo Li, Lingjuan Lyu, Shuang Wu, Shouhong Ding, Chunhua Shen, and Chao Wu.  Dense: Data-free one-shot federated learning.  _Advances in Neural Information Processing Systems_ , 35:21414–21428, 2022.
  * Zhang et al. [2023] Jie Zhang, Chen Chen, Weiming Zhuang, and Lingjuan Lyu.  Target: Federated class-continual learning via exemplar-free distillation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4782–4793, 2023.
  * Zhang et al. [2020] Junting Zhang, Jie Zhang, Shalini Ghosh, Dawei Li, Serafettin Tasci, Larry Heck, Heming Zhang, and C-C Jay Kuo.  Class-incremental learning via deep model consolidation.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 1131–1140, 2020.
  * Zhang et al. [2021a] Weishan Zhang, Tao Zhou, Qinghua Lu, Xiao Wang, Chunsheng Zhu, Haoyun Sun, Zhipeng Wang, Sin Kit Lo, and Fei-Yue Wang.  Dynamic-fusion-based federated learning for covid-19 detection.  _IEEE Internet of Things Journal_ , 8(21):15884–15891, 2021a.
  * Zhang et al. [2021b] Yikai Zhang, Hui Qu, Qi Chang, Huidong Liu, Dimitris Metaxas, and Chao Chen.  Training federated gans with theoretical guarantees: A universal aggregation approach.  _arXiv preprint arXiv:2102.04655_ , 2021b.
  * Zhao et al. [2020] Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia.  Maintaining discrimination and fairness in class incremental learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 13208–13217, 2020.

##  Appendix A DFedDGM Algorithm

Algorithm 2 summarizes our framework. All clients conduct local training
independently and then train their diffusion models locally. The server
receives only the local models from clients to perform model aggregation, and
redistributes the global model to clients.

Algorithm 2 DFedDGM

0: Local datasets $D=\\{D_{i}\\}_{i=1}^{N}$,
$D_{i}=\\{D_{i}^{t}\\}_{t=1}^{T}$, number of clients selected per round $m$,
learning rate $\eta$, local training epochs $E$, FL communication rounds per
task $R$, number of new classes $n^{t}_{\mathcal{C}}$ in task
$\mathcal{T}^{t}$, number of generated images $n_{s}$, retention ratio of the
entropy filter $\lambda$, training hyperparameters $\alpha$, $\gamma$.

0: Final global model $\theta_{g}^{T}$.

1: Initialize global model $\theta_{g}^{1}$ of task $\mathcal{T}^{1}$

2: $n_{\mathcal{C}}\leftarrow 0$ // Number of observed classes

3: for task step $t=1,2,\ldots,T$ do

4: $n_{\mathcal{C}}\leftarrow n_{\mathcal{C}}+n^{t}_{\mathcal{C}}$

5: $\theta_{g}^{t}\leftarrow$
UpdateStructure$(\theta_{g}^{t},n_{\mathcal{C}})$ // Incorporate new classes
into the classification layer

6: for round $r=1,2,\ldots,R$ do

7: $C_{r}\leftarrow$ RandomSelect($N,m$) // Random select $m$ clients from $N$
clients

8: for client $i\in C_{r}$ in parallel do

9: $\theta_{i}^{t}\leftarrow$
ClientUpdate$(\theta_{g}^{t},D_{i}^{t},\theta_{g}^{t-1},\mathcal{D}_{i}^{t-1})$
// No need of $\mathcal{D}_{i}^{0}$ and $\theta_{g}^{0}$ for $t=1$

10: $\mathcal{D}_{i}^{t}\leftarrow$
TrainDiffusionModel($D_{i}^{t},\mathcal{D}_{i}^{t-1}$) // Refer to Algorithm
1.

11: end for

12: $\theta_{g}^{t}\leftarrow$
GlobalAggregation$(\\{\theta_{i}^{t}\\}_{i\in{C_{r}}})$ // Average the local
models from selected clients

13: end for

14: end for

15:
ClientUpdate$(\theta_{g}^{t},D_{i}^{t},\theta_{g}^{t-1},\mathcal{D}_{i}^{t-1}):$

16: $\theta_{i}^{t}\leftarrow\theta_{g}^{t}$ // Initialize local model as
global model

17: for local epoch $e=1,2,\ldots,E$ do

18: if $t=1$ then

19:
$\theta_{i}^{t}\leftarrow\theta_{i}^{t}-\eta\nabla_{\theta}\mathcal{L}_{CE}(\theta_{i}^{t};D_{i}^{t})$

20: else

21: $\textbf{z}\sim\textbf{{N}}({\bf 0},{\bf I})$ // Sample noises

22:
$\\{\mathbf{x}_{i,s}^{t}\\}\leftarrow\mathcal{D}_{i}^{t-1}(\textbf{z},\frac{n_{s}}{\lambda})$
// Generate $\frac{n_{s}}{\lambda}$ synthetic images by diffusion model

23:
$D^{t}_{i,s}=\\{(\mathbf{x}_{i,s}^{t},y_{i,s}^{t})\\}\leftarrow\textbf{EntropyFilter}(\\{\theta_{g}^{t-1}(\mathbf{x}_{i,s}^{t})\\},\lambda)$
// Filter out part of generated samples and assign labels by model
$\theta_{g}^{t-1}$ for the rest to form the synthetic dataset $D^{t}_{i,s}$

24:
$\mathcal{L}(\theta_{i}^{t})\leftarrow\mathcal{L}_{CE}(\theta_{i}^{t};D_{i}^{t},D^{t}_{i,s})+\alpha\mathcal{L}_{KD}(\theta_{i}^{t},\theta_{g}^{t-1};D_{i}^{t},D^{t}_{i,s})+\gamma\mathcal{L}_{FD}(h_{i}^{t},h_{g}^{t-1};D_{i}^{t},D^{t}_{i,s})$

25:
$\theta_{i}^{t}\leftarrow\theta_{i}^{t}-\eta\nabla_{\theta}\mathcal{L}(\theta_{i}^{t})$

26: end if

27: end for

##  Appendix B Experimental Setup

Evaluation Metrics. We employ the Average Accuracy ($Acc$) and Average
Forgetting ($\mathcal{F}$) to conduct our evaluations. Let Accuracy
($Acc^{t}$) represent the model’s accuracy at the end of task
$\mathcal{T}^{t}$ across all observed classes, then the Average Accuracy is
the average of all $Acc^{t}$ for all $T$ tasks:

$Acc=\frac{1}{T}\sum_{t=1}^{T}Acc^{t},$ (10)

Let Forgetting ($\mathcal{F}^{t}$) be the difference between the model’s
highest accuracy on task $\mathcal{T}^{t}$ and its accuracy on task
$\mathcal{T}^{t}$ at the end of all task training. Then, the Average
Forgetting is the average of $\mathcal{F}^{t}$ from task $\mathcal{T}^{1}$ to
task $\mathcal{T}^{T-1}$:

$\mathcal{F}^{t}=\max_{l\leq T}Acc^{t}_{l}-Acc^{t}_{T},$ (11)

$\mathcal{F}=\frac{1}{T-1}\sum_{t=1}^{T-1}\mathcal{F}^{t},$ (12)

where $Acc^{t}_{l}$ represents the accuracy on task $\mathcal{T}^{t}$ at
training step $l$ and $Acc^{t}_{T}$ represents the accuracy on task
$\mathcal{T}^{t}$ after the training of all $T$ tasks.

Training details. To ensure a fair comparison, we adapted the federated
learning algorithms FedAvg, FedProx, and the continual learning method EWC to
the federated class incremental learning setting. Each client was trained with
a batch size of 128 for a total of 100 communication rounds, with each round
consisting of 5 local training epochs. We employed the SGD optimizer with a
learning rate of 0.01, a momentum coefficient of 0.9, and a weight decay
parameter of $5\times 10^{-4}$. For other hyperparameters of the baseline
methods, we adhered to the values recommended in their respective papers. For
our method, we applied optimal hyperparameters derived through a grid-search
strategy, setting $\alpha$ to 3 and $\gamma$ to 2. The diffusion models were
trained for 200 epochs for each task using a batch size of 16, with the Adam
optimizer and a learning rate of $5\times 10^{-5}$. We generated images by
denoising random noise over 1000 epochs by the sampling method DDPM. All
results are averaged over three different random seeds.

Environment All our experiments were conducted on a single machine with 1TB
RAM and 256-core AMD EPYC 7742 64-Core Processor @ 3.4GHz CPU. We use the
NVIDIA H100 GPU with 80GB memory. The software environment settings are:
Python 3.10.11, PyTorch 2.1.2 with CUDA 12.2 on Ubuntu 22.04.4 LTS.

##  Appendix C Code for Reproduction

Our code is attached to the supplementary material.

##  Appendix D Analysis on Communication and Computation Cost

Table 2: Cost comparison on the CIFAR-100 dataset on $N$=10 clients when
$T$=5.

Metric | Communication Cost | Client Training Time | Server Runtime | Total Time
---|---|---|---|---
FedAvg | 419.92 GB | 3.13 h | 0.25 h | 3.38 h
FedProx | 419.92 GB | 4.19 h | 0.25 h | 4.44 h
FedEWC | 419.92 GB | 4.46 h | 0.25 h | 4.71 h
FLwF-2T | 419.92 GB | 3.28 h | 0.25 h | 3.53 h
TARGET | 463.87 GB | 4.25 h | 2.95 h | 7.20 h
MFCL | 581.05 GB | 5.57 h | 1.24 h | 6.81 h
DFedDGM (Ours) | 419.92 GB | 29.52 h | 0.25 h | 29.77 h

Table 2 presents the communication and computation costs for various methods
based on ResNet-18 on the CIFAR-100 dataset. As we can see, our method
exhibits significant advantages in terms of communication cost compared to
other generative-based methods, which is a crucial concern in federated
learning scenarios where minimizing communication overhead is essential.
Specifically, the communication cost of our method is comparable to the
baseline method FedAvg, maintaining a communication cost of 419.92 GB. This
cost is considerably lower than that of TARGET and MFCL, thus enhancing the
scalability of the federated learning system and improving data privacy by
reducing the amount of information shared.

Additionally, as our method does not necessitate the server from training the
generative models, the server runtime is notably short at 0.25 hours,
comparable to that of FedAvg. This is particularly important when scaling a
federated learning system to include a large number of clients (e.g., over 100
clients), as it alleviates server-side computational pressure. Although our
method involves additional computational overhead and increases training
duration on the client side due to the inherent limitations associated with
the diffusion model, it offers more stable image generation and significant
enhancements in model performance compared to GAN-based methods. Therefore, it
is a trade-off between performance and computation cost. Moreover, the
training and sampling times can be reduced by utilizing advanced techniques,
such as DDIM, making the method more suitable for the FCIL scenarios.

##  Appendix E Details of the Diffusion Model

###  E.1 Model Structure

In our method, we use the UNet [53] as the base structure to train the
diffusion models. UNet consists of an encoder-decoder structure with skip
connections that facilitate the retention of high-resolution details during
the image reconstruction process. The encoder systematically downsamples the
input image while capturing spatial features, and the decoder subsequently
upsamples the encoded representation, combining fine and coarse features via
skip connections to reconstruct the image.

###  E.2 Training Procedure

We use the DDPM [22] method to train our diffusion models. The training of
DDPM involves learning to denoise progressively noise-added images. This is
formalized in the following steps:

  1. 1.

Forward Diffusion Process: We define a forward diffusion process to
iteratively add Gaussian noise to a data sample $\mathbf{x}_{0}$ over $T$ time
steps. At each time $t$, the noisy image $\mathbf{x}_{t}$ can be described as:

$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\textbf{{N}}(\mathbf{x}_{t};\sqrt{\alpha_{t}}\mathbf{x}_{t-1},(1-\alpha_{t})\mathbf{I}),$
(13)

where $\alpha_{t}$ are predefined constants.

  2. 2.

Reverse Diffusion Process: The objective of the DDPM is to learn the reverse
process $p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$, which entails denoising
the image $\mathbf{x}_{t}$ to recover $\mathbf{x}_{t-1}$. The reverse process
is parameterized by a neural network (in our case, UNet), denoted by $\theta$.
The likelihood of the training data is maximized by minimizing the following
variational bound:

$\mathcal{L}=\mathbb{E}_{q(\mathbf{x}_{1:T}|\mathbf{x}_{0})}\left[\sum_{t=1}^{T}\frac{1}{2\sigma_{t}^{2}}\left\|\mathbf{x}_{t}-\mathbf{\mu}_{\theta}(\mathbf{x}_{t},t)\right\|^{2}\right].$
(14)

###  E.3 Sampling Procedure

Once the model is trained, images can be generated via the learned reverse
diffusion process. The sampling process reverts the sequence of noise addition
steps from a pure Gaussian noise $\mathbf{x}_{T}$ back to the data space:

  1. 1.

Initial Gaussian Sample: Begin with $\mathbf{x}_{T}\sim\textbf{{N}}({\bf
0},{\bf I})$.

  2. 2.

Iterative Denoising: For $t$ from $T$ to 1, use the trained model
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})$ to iteratively denoise
$\mathbf{x}_{t}$ to recover $\mathbf{x}_{t-1}$. The denoising step can be
expressed as:

$\mathbf{x}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\mathbf{x}_{t}-\frac{1-\alpha_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)\right)+\sigma_{t}\mathbf{z},$
(15)

where $\mathbf{z}\sim\textbf{{N}}({\bf 0},{\bf I})$ and
$\mathbf{\epsilon}_{\theta}(\mathbf{x}_{t},t)$ denotes the model prediction of
the added noise.

##  Appendix F Hyperparameter tuning for DFedDGM

Hyperparameters are crucial in determining the performance of algorithms.
Here, we demonstrate the sensitivity of algorithm performance to each
hyperparameter. Our analysis involves modifying one parameter at a time while
keeping the others constant. The hyperparameters we have tuned are:

  * •

$\lambda$: retention ratio for generated samples filtered by entropy.

  * •

$\alpha$: weight of knowledge distillation loss ($\mathcal{L}_{KD}$).

  * •

$\gamma$: weight of feature distance loss ($\mathcal{L}_{FD}$).

  * •

$n_{d}$: number of epochs to train a diffusion model.

Table 3 presents the average accuracy Acc and average forgetting $\mathcal{F}$
for each hyperparameter on the CIFAR-100 dataset with $T=5$ tasks. A minor
discrepancy may exist between this value and the results presented in the main
manuscript since the ablation was conducted using a single seed, while the
results in the primary manuscript represent the average of three different
seeds.

Table 3: Effect of different hyperparameters for the CIFAR-100 dataset.

$\lambda$ | $\textit{Acc}(\uparrow)$ | $\mathcal{F}(\downarrow)$ | $\alpha$ | $\textit{Acc}(\uparrow)$ | $\mathcal{F}(\downarrow)$ | $\gamma$ | $\textit{Acc}(\uparrow)$ | $\mathcal{F}(\downarrow)$ | $n_{d}$ | $\textit{Acc}(\uparrow)$ | $\mathcal{F}(\downarrow)$
---|---|---|---|---|---|---|---|---|---|---|---
0.5 | 56.17 | 28.49 | 1 | 56.31 | 28.95 | 1 | 56.66 | 28.08 | 20 | 49.28 | 31.42
0.7 | 56.53 | 28.27 | 3 | 57.11 | 27.84 | 2 | 57.11 | 27.84 | 50 | 55.15 | 29.76
0.8 | 56.89 | 28.01 | 5 | 56.29 | 29.02 | 5 | 56.15 | 28.52 | 100 | 56.01 | 29.17
0.9 | 57.11 | 27.84 | 7 | 55.87 | 29.47 | 8 | 54.03 | 30.11 | 200 | 57.11 | 27.84
1.0 | 55.69 | 29.33 | 9 | 55.24 | 29.88 | 10 | 53.24 | 30.98 | 400 | 57.08 | 27.89

From Table 3, it is evident that the performance of our method remains robust
to variations in hyperparameters. Meanwhile, we have the following findings:

  * •

Setting the parameter $\lambda$ to a small value, such as 0.5, does not
necessarily improve model performance. This is attributed to the fact that
only samples with very low-confidence labels might adversely impact model
training, whereas the majority of samples are generally suitable for effective
training.

  * •

The values of $\alpha$ and $\gamma$ should not be excessively large, as this
would harm the model performance. It is crucial to balance the cross-entropy
loss with the knowledge distillation (KD) and feature distance (FD) losses.
This balance enables the model to effectively learn from previous tasks
without compromising its performance on the current task.

  * •

Once the diffusion model has been sufficiently trained (i.e., $n_{d}\geq
200$), further training does not substantially improve the quality of the
generated images or the model performance. This observation highlights the
stability of the diffusion model as a generative tool.

The robustness of the model performance concerning each hyperparameter is
crucial in both federated learning and continual learning, which remains a
topic requiring further investigation.


# Feature Denoising for Low-Light Instance Segmentation using Weighted Non-
Local Blocks

###### Abstract

Instance segmentation for low-light imagery remains largely unexplored due to
the challenges imposed by such conditions, for example shot noise due to low
photon count, color distortions and reduced contrast. In this paper, we
propose an end-to-end solution to address this challenging task. Based on Mask
R-CNN, our proposed method implements weighted non-local (NL) blocks in the
feature extractor. This integration enables an inherent denoising process at
the feature level. As a result, our method eliminates the need for aligned
ground truth images during training, thus supporting training on real-world
low-light datasets. We introduce additional learnable weights at each layer in
order to enhance the network’s adaptability to real-world noise
characteristics, which affect different feature scales in different ways.
Experimental results show that the proposed method outperforms the pretrained
Mask R-CNN with an Average Precision (AP) improvement of +10.0, with the
introduction of weighted NL Blocks further enhancing AP by +1.0.

Index Terms—  Instance segmentation, low-light images, feature denoising, non-
local means, end-to-end

##  1 Introduction

Instance segmentation is a crucial task in computer vision because it allows
systems to interpret scenes with higher levels of detail, by understanding the
position and shape of the objects within them. This can be highly beneficial
in scenarios such as autonomous driving, video surveillance, augmented
reality, multimedia editing, medical imaging and agricultural monitoring. Its
importance is underscored by numerous recent research contributions in this
field and various benchmarks and datasets, including Microsoft COCO [1],
ADE20K [2], Pascal VOC [3] and Cityscapes [4].

(a)

(b)

(c)

(d)

Fig. 1: Results of applying instance segmentation to (a) a low-light image
using (b) a pretrained Mask R-CNN model [5], (c) a RetinexFormer [6] for LLIE,
and (d) our method.

While its applications span many sectors, there has been limited research on
instance segmentation in low-light conditions. The is due to the fact that
low-light images are heavily affected by noise from a low photon count in
darker conditions, making the task much more challenging. A common approach is
to first apply a low-light image enhancement (LLIE) method to these images.
This enables existing instance segmentation methods, typically designed for
normal-light conditions, to run inference upon the enhanced images. LLIE
methods include EnlightenGAN [7], Zero-DCE [8], and more recently
RetinexFormer [6]. Specifically, Zero-DCE [8], and RetinexFormer [6] also use
low-light object detection benchmarks (ExDark [9], DARK FACE [10]) to show
their low-light enhancing performances.

To the best of our knowledge to date, the only work published on low light
instance segmentation is by by Chen et al. [11]; however their method was only
applicable for RAW images to take advantage of the higher bit-depth. In this
paper, we propose an end-to-end approach that takes low-light sRGB images as
inputs; these are much more widely used than RAW images and can be applied to
video sequences.

Our goal is to develop a robust and effective end-to-end method for low-light
instance segmentation. This implies eliminating any explicit pre-processing
steps in order to simplify the mask generating procedure. However most models
are not designed to handle images with noise and low-contrast. In our
experiments, we have found that both low-contrast and noise impact the
performance of these models, as also confirmed by the study in [12].

Our focus is to build upon existing feature extractors to learn to ignore the
presence of noise in the feature maps. This is achieved by integrating
denoising blocks with non-local (NL) means as its denoising operation into the
backbone. Specifically, our framework is based on Mask R-CNN [5] and leverages
an NL block [13] for denoising in the feature extractor. We however extend
this with a weighted NL block, where the additional learnable weights ensure
that the network can learn the noise characteristics that affect each scale of
feature space differently. We trained our framework with a synthetic dataset;
this is due to the infancy of performing high-level computer vision tasks
under low-light conditions. We used the popular Microsoft COCO [1] dataset and
synthesised low-light conditions following the pipeline proposed by [14].

##  2 Related Work

###  2.1 Instance Segmentation

Instance segmentation aims to produce a segmentation mask, bounding box and
class label for every object detected in an image. One of the most popular
techniques for achieving this is Mask R-CNN [5], introduced by He et al. as an
extension to the popular object detection model, Faster R-CNN [15]. The
architecture is the same except Mask R-CNN [5] introduced a new mask head to
also predict the instance masks from the Regions of Interest (RoIs) proposed
by the Region Proposal Network (RPN). With the notable work from [16], there
has been a surge in transformer-based methods in recent years. For example,
DETR-based [17] instance segmentation methods use CNN-based feature extraction
backbones and concatenate positional embeddings before passing into a
transformer.

###  2.2 Low-light image enhancement

Traditional LLIE methods treat the problem as two separate tasks: contrast
enhancement and denoising. Contrast enhancement methods include gamma
correction and adaptive histogram equalizatiom (AHE) [18]; however these lack
consideration of illumination factors, leading to cognition approaches
inspired by Retinex Theory [19]. Contrast-enhancement of low-light images
however, amplifies noise which necessitates the use of denoising algorithms
for further refinement.

Recent advancements in LLIE have explored the use of deep learning. Lore et
al. first introduced LLNet [20], an autoencoder trained with synthetic low-
light images. Supervised methods, including RetinexNet [21], KiND [22], and
AGLLNet [14], require large sets of paired normal/low-light images for
training, often generated synthetically. Unsupervised methods gained
popularity due to the scarcity of paired normal-light images in low-light
datasets. Examples include EnlightenGAN [7] and Zero-DCE (and its extended
version Zero-DCE++) [8]. RetinexFormer [6], a transformer-based method, is
currently the state-of-the-art approach, leveraging the Retinex theory [19] to
separate color vision into reflectance and luminance components.

###  2.3 Low-light scene understanding

Several datasets have been released for higher computer vision tasks in low-
light conditions. Semantic segmentation is a more common task, with datasets
like NightCity [23] and BDD100K-night [24] focusing on analyzing road scenes
to enhance autonomous driving at night. Additionally datasets such as ExDark
[9] and DARK FACE [10] serve as benchmarks for low-light object detection.

Chen et al. [11] investigated instance segmentation in extreme low-light
conditions, leveraging the higher bit-depth of RAW images to enhance scene
parsing. Their approach introduces three key contributions: an adaptive
weighted downsampling (AWD) layer incorporating spatial and channel
information for downsampling, a smooth-oriented convolutional block replacing
standard 3$\times$3 convolutions during training and re-parameterizing back
into a 3$\times$3 convolution block, and a disturbance suppression loss (DSL)
measuring Euclidean distance between feature maps from clean and noisy images.
Training involves the synthesis of both a clean RAW low-light dataset and a
noisy RAW low-light dataset for the disturbance suppression loss. They
provided a low-light dataset with paired short-exposure/long-exposure images
for instance segmentation training and validation.

Fig. 2: Proposed architecture inspired by Mask R-CNN [5]. A weighted non-local
(NL) block [13] used for feature denoising in the feature extractor. The
extracted feature maps are passed into a Region Proposal Network (RPN) and
Region of Interest (RoI) pooling layer to get the features for the proposed
regions. These features are passed into the box and mask heads for
predictions. The image used in this diagram was taken from the MS COCO dataset
[1].

##  3 Synthetic Low-Light Pipeline

For training and evaluating our method, we require annotated low-light images
for instance segmentation. However, to the best of our knowledge, there
currently exists only one small dataset, released by Chen et al. [11]. This
real-world low-light dataset however contains only 2230 paired short-
exposure/long-exposure images with instance-level segmentation annotations.
Due to the limited number of training paired samples with imagery only in
extreme low-light conditions, we decided not to use this dataset. Instead we
chose to synthetically create an annotated low-light dataset.

We used the popular Microsoft COCO [1] dataset and passed it through the
synthetic low-light pipeline proposed by Lv et al. [14]. This method was
chosen over a simplified model (e.g. [25]) because of the simulated in-camera
processing which also factors in spatial correlations of noise caused by
demosaicing thereby generating more realistic noise characteristics.

##  4 Proposed Method

###  4.1 Instance Segmentation

We propose an end-to-end method for instance segmentation in low-light without
the need for paired normal-light images. Following the popular Mask R-CNN [5],
the network is constructed of a backbone, a Region Proposal Network (RPN),
Region of Interest (RoI) pooling, and box and mask heads.

The image is first passed into the modified backbone to extract features. The
feature maps are then passed into the RPN first introduced by [15] to generate
object proposals with a objectiveness score. These proposals and the feature
maps are passed into the RoIPool layer which extracts the feature maps within
the proposals. These RoIs are then passed into the box head and mask head to
make predictions on the class, bounding box and segmentation mask for each
RoI.

For our backbone, we chose to use ResNet-50 [26] with a feature pyramid
network (FPN) [27]; this is a popular backbone capable of learning features at
different scales. We modify this backbone by adding our weighted NL blocks at
the end of res2, res3, res4 and res5 to reduce the feature noise that has been
propagated into the feature extractor. The remainder of the network uses the
originally proposed parameters from Mask R-CNN [5]. For training, we froze the
weights of everything except the backbone to allow the model to specifically
learn to denoise the feature maps.

###  4.2 Weighted non-local blocks for feature denoising

Traditional non-local blocks were first introduced by Wang et al. [13] when
they explored implementing non-local means into neural networks. Non-local
means [28] is a class of traditional computer vision denoising algorithm that
process images on a global scale, unlike local smoothing filters (e.g.
bilateral filter [29], Gaussian filter etc.) which only consider neighbouring
pixels. Non-local means filtering involves calculating the mean of all pixels
in the image, with the weighting determined by the similarity of each pixel to
the target one. Wang et al. [13] applied this idea in their neural networks to
learn long-range dependencies between pixels spatially and temporally. They
also found that the self-attention mechanism [16] is a special form of NL
operations in the embedded gaussian form [13].

Xie et al. [30] explored using both non-local means algorithms and local
smoothing filtering algorithms for feature denoising to defend against
adversarial attacks; attacks that intentionally insert noise into images to
confuse systems. They found that although the local denoising operations did
improve the robustness of these systems, NL operations were more effective.

In this paper we consider three implementations of NL means for denoising: dot
product, Gaussian and embedded Gaussian. The NL blocks consist of an
implementation of non-local mean followed by a 1$\times$1 convolution, added
to the block’s input via an identity skip connection. They achieve feature
denoising as they pass the feature maps through a denoising operation (i.e.
non-local means). We modified these blocks to include learnable weight $w$ as
shown in Fig. 3. This allows the model to determine the amount of denoising
for each ResNet layer. We refer to it as a weighted NL block, which computes
the following:

$\mathbf{z}=wW_{z}\mathbf{y}+(1-w)\mathbf{x}$ (1)

where $\mathbf{x}$ is the input feature map, $\mathbf{y}$ is the output from
the NL means operation, $W_{z}$ is the weight matrix from the 1$\times$1
convolutional layer after the NL means operation and $\mathbf{z}$ is the
output.

Fig. 3: A weighted non-local block for feature denoising with learnable weight $w$. The blue box illustrates the non-local operation used for denoising an input feature map $\mathbf{x}$. This figure shows the embedded Gaussian form. Easy |  |  |  |
---|---|---|---|---
Medium |  |  |  |
Difficult |  |  |  |
| Ground truth | Pretrained | Finetuned | Ours
Fig. 4: Visual comparison of our proposed method against pretrained and finetuned Mask R-CNN [5] models, along with the ground truth, with varying levels of difficulty for the models. The easy example case shows when these methods were all successful. The medium example case shows when our proposed method performed successfully while the other two methods failed. The difficult example case shows when all of the methods struggle. The images used in this diagram were taken from the MS COCO dataset [1]. Table 1: Comparison of methods on the synthetic low-light COCO minival dataset Method | LLIE | AP | AP50 | AP75 | APS | APM | APL
---|---|---|---|---|---|---|---
Pretrained | - | 6.9 | 12.4 | 6.9 | 2.3 | 7.4 | 12.4
Pretrained | EnlightenGAN [7] | 5.5 | 10.0 | 5.5 | 1.7 | 6.1 | 10.0
Pretrained | Zero-DCE++ [8] | 5.6 | 10.0 | 5.6 | 1.8 | 6.1 | 9.6
Pretrained | AGLLNet [14] | 6.1 | 11.0 | 6.1 | 1.8 | 7.2 | 10.5
Pretrained | RetinexFormer [6] | 5.7 | 10.3 | 5.7 | 1.8 | 6.5 | 9.9
Finetuned | - | 15.9 | 28.6 | 15.6 | 4.8 | 15.8 | 27.8
Ours (weighted non-local block) | - | 16.9 | 30.7 | 16.6 | 5.6 | 17.2 | 28.9
|  |  |
---|---|---|---
Original | Ground Truth | Low-Light | Proposed
|  |  |
EnlightenGAN [7] | Zero-DCE++ [8]
|  |  |
AGLLNet [14] | RetinexFormer [6]
Fig. 5: Visual comparison of our proposed method against two-stage methods
(enhanced first then passed through a pretrained Mask R-CNN [5]). The first
row shows the original image with the ground truth annotations, and the low-
light image with predictions from our proposed method. The second and third
row shows the enhanced images and the predictions using a pretrained Mask
R-CNN [5]. The image used in this diagram was taken from the MS COCO dataset
[1].

##  5 Experimental Results

For all of our experiments, we trained the models for 10 epochs with an
initial learning rate of 0.0005 that decreased by a factor of 10 every 3
epochs, with a batch size of 4. Due to limited computational resources, the
number of channels inside the weighted NL block after res2 were divided into 4
instead of 2 as proposed originally by Wang et al. [13]. We used the Microsoft
COCO [1] dataset and ran it through the low-light synthetic pipeline. We used
118,287 images for training and 5,000 for evaluation. We did not use their
test set as it does not have public annotations available. We used the metrics
provided by MS COCO: AP, AP50, AP75, APS, APM, and APL. AP calculates the mean
of the average precisions across multiple IoU thresholds, 0.5 to 0.95, in 0.05
increments. AP50 and AP75 calculate the average precision with an IoU
threshold of 0.5 and 0.75 respectively. APS, APM, and APL calculate the
average precision for small, medium and large objects, which are predefined by
[1].

Table 1 shows our results compared to several different methods based around
Mask R-CNN [5]. We experimented with one-stage methods and two-stage methods
(using LLIE methods [8, 7, 14, 6] for the pre-processing step). We did not run
our experiments with a weighted non-local block on res2 due to memory
limitations.

###  5.1 One-stage experiments

Our baseline experiments compare our proposed method with a pretrained Mask
R-CNN model [5] and also with finetuning the pretrained model on synthetic
low-light training data generated from [1]. Table 1 shows a significant
improvement between the fine-tuned model and the pre-trained model.
Additionally, there is further enhancement when the weighted NL blocks are
integrated, resulting in a +1.0 AP improvement.

Both Table 1 and Fig. 4 illustrate the efficacy of our method in enhancing
performance, while also indicating areas for further refinement. Generally,
the models exhibit proficiency in segmenting larger objects but encounter
challenges with smaller ones, likely attributed to injected noise. The easy
example case in Fig. 4 had one large object of a cat and all models were able
to accurately generate an instance mask for it. In the medium case, the
pretrained Mask R-CNN method failed significantly and falsely labelled many
pixels due to the low-light conditions. The finetuned Mask R-CNN performed
better although it still suffered with accuracy when segmenting the tennis
racket, while our method did not have this issue. In the difficult case, there
are small cows in the distance but none of the methods were able to pick them
up. These findings are backed by the results from Table 1, particularly the
metrics APS, APM and APL. These metrics clearly show that our method improves
on the finetuned method for different sized objects, indicating that the
weighted NL block successfully removes the noise in the feature extractor.

###  5.2 Two-stage experiments

We also compared our method with the two-stage approaches, where the pre-
processing employed popular LLIE methods, including EnlightenGAN [7], Zero-
DCE++ [8], AGLLNet [14], and RetinexFormer [6]. All of these methods used the
weights released by them (for RetinexFormer, we used the weights trained from
the LOL [21] dataset). It can be seen from Table 1 that these approaches
underperformed compared to using a pretrained model, primarily because the
enhanced images were significantly impacted by noise (see Fig. 5, even with
the transformer-based method RetinexFormer which reformulated the equation
from the Retinex theory to also model noise and reduce the noise
amplification. In the example shown, the two-stage methods unsuccessfully
generated masks for the giraffe We found that, of all of the LLIE methods
used, AGLLNet performed the best. This may be because there model was trained
with the synthetic data generated using the same synthetic low-light pipeline
as ours.

###  5.3 Ablation study

We compared our approach with using different NL operations in order to
determine which was the most effective method. As seen in Table 2, they all
show similar performances, but the embedded Gaussian form shows some
improvement compared to the other two. The embedded Gaussian form likely had a
higher performance due to its self-attention properties, as described by Wang
et al. [13].

Table 2: Ablation study of different NL operations for NL block with weights $w_{1}=0$, $w_{2}=0$, $w_{3}=0.5$, $w_{4}=0$ Method | AP | AP50 | AP75
---|---|---|---
Dot Product | 16.1 | 29.3 | 15.7
Gaussian | 16.0 | 29.1 | 15.7
Embedded Gaussian | 16.6 | 30.2 | 16.4

##  6 Conclusion

This paper introduces a framework for low-light instance segmentation without
the need for pre-processing. We enhance popular instance segmentation models
by proposing a weighted non-local block for feature denoising, improving
performance for smaller objects that would previously most likely be obscured
by noise. Experimental results show that our method improves the average
precision by +10.0 over a vanilla pretrained Mask R-CNN. Our approach is
applicable to various low-light computer vision tasks, including object
detection and semantic segmentation, by integrating weighted NL blocks into
the backbones of other architectures. Future work should address performance
on even smaller objects, enhance other aspects such as the FPN, and assess our
method’s efficacy on real low-light datasets.

## References

  * [1] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco: Common objects in context,”  in European Conference on Computer Vision, 2014, pp. 740–755.
  * [2] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba,  “Scene parsing through ade20k dataset,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [3] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman,  “The pascal visual object classes challenge: A retrospective,”  International Journal of Computer Vision, vol. 111, no. 1, pp. 98–136, Jan. 2015.
  * [4] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, et al.,  “The cityscapes dataset for semantic urban scene understanding,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [5] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick,  “Mask r-cnn,”  in 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 2980–2988.
  * [6] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang,  “Retinexformer: One-stage retinex-based transformer for low-light image enhancement,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023, pp. 12504–12513.
  * [7] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang,  “Enlightengan: Deep light enhancement without paired supervision,”  IEEE Transactions on Image Processing, vol. 30, pp. 2340–2349, 2021.
  * [8] Chongyi Li, Chunle Guo Guo, and Chen Change Loy,  “Learning to enhance low-light image via zero-reference deep curve estimation,”  in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [9] Yuen Peng Loh and Chee Seng Chan,  “Getting to know low-light images with the exclusively dark dataset,”  Computer Vision and Image Understanding, vol. 178, pp. 30–42, 2019.
  * [10] Wenhan Yang, Ye Yuan, Wenqi Ren, Jiaying Liu, Walter J. Scheirer, et al.,  “Advancing image understanding in poor visibility environments: A collective benchmark study,”  IEEE Transactions on Image Processing, vol. 29, pp. 5737–5752, 2020.
  * [11] Linwei Chen, Ying Fu, Kaixuan Wei, Dezhi Zheng, and Felix Heide,  “Instance segmentation in the dark,”  International Journal of Computer Vision, vol. 131, no. 8, pp. 2198–2218, 2023.
  * [12] Anqi Yi and Nantheera Anantrasirichai,  “A comprehensive study of object tracking in low-light environments,”  arXiv:2312.16250, 2024.
  * [13] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He,  “Non-local neural networks,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [14] Feifan Lv, Yu Li, and Feng Lu,  “Attention guided low-light image enhancement with a large scale low-light simulation dataset,”  International Journal of Computer Vision, vol. 129, no. 7, pp. 2175–2193, 2021.
  * [15] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun,  “Faster r-cnn: Towards real-time object detection with region proposal networks,”  in Advances in Neural Information Processing Systems, 2015, vol. 28.
  * [16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, et al.,  “Attention is all you need,”  in Advances in Neural Information Processing Systems, 2017, vol. 30.
  * [17] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko,  “End-to-end object detection with transformers,”  in European conference on computer vision. Springer, 2020, pp. 213–229.
  * [18] Stephen M. Pizer, E. Philip Amburn, John D. Austin, Robert Cromartie, Ari Geselowitz, et al.,  “Adaptive histogram equalization and its variations,”  Computer Vision, Graphics, and Image Processing, vol. 39, no. 3, pp. 355–368, 1987.
  * [19] Edwin H Land,  “The retinex theory of color vision,”  Scientific american, vol. 237, no. 6, pp. 108–129, 1977.
  * [20] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar,  “Llnet: A deep autoencoder approach to natural low-light image enhancement,”  Pattern Recognition, vol. 61, pp. 650–662, 2017.
  * [21] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu,  “Deep retinex decomposition for low-light enhancement,”  in British Machine Vision Conference, 2018.
  * [22] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo,  “Kindling the darkness: A practical low-light image enhancer,”  in Proceedings of the 27th ACM International Conference on Multimedia, 2019, pp. 1632–1640.
  * [23] Xin Tan, Ke Xu, Ying Cao, Yiheng Zhang, Lizhuang Ma, and Rynson W. H. Lau,  “Night-time scene parsing with a large real dataset,”  IEEE Transactions on Image Processing, vol. 30, pp. 9085–9098, 2021.
  * [24] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell,  “Bdd100k: A diverse driving dataset for heterogeneous multitask learning,”  in IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [25] N. Anantrasirichai, J. Burn, and David R. Bull,  “Robust texture features based on undecimated dual-tree complex wavelets and local magnitude binary patterns,”  in 2015 IEEE International Conference on Image Processing (ICIP), 2015, pp. 3957–3961.
  * [26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
  * [27] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie,  “Feature pyramid networks for object detection,”  in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 936–944.
  * [28] A. Buades, B. Coll, and J.-M. Morel,  “A non-local algorithm for image denoising,”  in IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, vol. 2, pp. 60–65.
  * [29] C. Tomasi and R. Manduchi,  “Bilateral filtering for gray and color images,”  in Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271), 1998, pp. 839–846.
  * [30] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L. Yuille, and Kaiming He,  “Feature denoising for improving adversarial robustness,”  in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.