# Decentralized Distributed Learning with Privacy-Preserving Data Synthesis

M. Pennisi  F. Proietto Salanitri  G. Bellitto  B. Casella  M. Aldinucci  S.
Palazzo  C. Spampinato  M. Pennisi, F. Proietto Salanitri, G. Bellitto, S.
Palazzo, C. Spampinato are with the Department of Electrical, Electronics and
Computer Engineering, University of Catania, Italy (email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
B. Casella and M. Aldinucci are with the Department of Computer Science,
University of Turin, Italy (email<EMAIL_ADDRESS>
Equal contribution by M. Pennisi and F. Proietto Salanitri. Equal supervision
by S. Palazzo and C. Spampinato.

###### Abstract

In the medical field, multi-center collaborations are often sought to yield
more generalizable findings by leveraging the heterogeneity of patient and
clinical data. However, recent privacy regulations hinder the possibility to
share data, and consequently, to come up with machine learning-based solutions
that support diagnosis and prognosis. Federated learning (FL) aims at
sidestepping this limitation by bringing AI-based solutions to data owners and
only sharing local AI models, or parts thereof, that need then to be
aggregated. However, most of the existing federated learning solutions are
still at their infancy and show several shortcomings, from the lack of a
reliable and effective aggregation scheme able to retain the knowledge learned
locally to weak privacy preservation as real data may be reconstructed from
model updates. Furthermore, the majority of these approaches, especially those
dealing with medical data, relies on a centralized distributed learning
strategy that poses robustness, scalability and trust issues. In this paper we
present a decentralized distributed method that, exploiting concepts from
experience replay and generative adversarial research, effectively integrates
features from local nodes, providing models able to generalize across multiple
datasets while maintaining privacy. The proposed approach is tested on two
tasks — tuberculosis and melanoma classification — using multiple datasets in
order to simulate realistic _non-i.i.d._ data scenarios. Results show that our
approach achieves performance comparable to both standard (non-federated)
learning and federated methods in their centralized (thus, more favourable)
formulation.

Decentralized Learning, Federated Learning, Privacy in Machine Learning,
Pattern recognition and classification

##  1 Introduction

Recent advances of deep learning in the medical imaging domain have shown
that, while data-driven approaches represent a powerful and promising tool for
supporting physicians’ decisions, the availability of large-scale datasets
plays a key role in the effectiveness and reliability of the resulting models
[1, 2, 3]. However, the curation of large medical imaging datasets is a
complex task: data collection at single institutions is relatively slow and
the integration of historical data may require significant efforts to deal
with different data formats, storage modalities and acquisition devices;
moreover, medical institutions are often reluctant to share their own data,
due to privacy concerns. As a consequence, this affects the quality,
reliability and generalizability of models trained on local datasets, which
unavoidably suffer from bias and overfitting issues [4]. In order to overcome
the lack of large-scale datasets, methodological solutions can be adopted: in
particular, federated learning [5] encompasses a family of strategies for
distributed training over multiple nodes, each with its own private dataset,
which typically communicate with a central node by sending local model
updates, used to train the main model. In this scenario, no data is explicitly
shared between nodes, thus addressing the required privacy issues. However,
this family of techniques generally performs well when dataset distributions
are approximately _i.i.d._ and local gradients/models contribute to learning
shared features: unfortunately, in practice this hypothesis rarely holds, due
to differences in the acquisition and in the clinical nature of data collected
by multiple institutions. Moreover, the presence of a central node, besides
representing a single point of failure, requires that all nodes trust it to
correctly and fairly treat updates from all sources: indeed, privacy issues
arise when transferring local updates to the “semi-honest” central node [6],
which might attempt to reconstruct original inputs from gradients or parameter
variations [7, 8, 9].

In this work, we propose a decentralized distributed learning approach that
combines principles of experience replay [10, 11, 12, 13] and generative
models [14, 15, 16] to train local independent models that approximately
converge to the same solution, while enforcing privacy preservation through
the transmission of synthetic data built in a way to obfuscate real data
patterns. Specifically, the distributed training procedure envisages multiple
nodes that initially train their local models on their own datasets, in order
to generate a privacy-preserving synthesized version thereof. Once local
training is completed in a node, its model and a “buffer” of generated
synthetic data are randomly sent to another node of the network. The receiving
node then adapts the incoming model using its own data and the buffer data, in
order to limit model’s forgetting. Data privacy is ensured through a privacy-
preserving generative adversarial network (GAN) that employs a specific loss
designed to maximize the distance from real data, while keeping a high level
of realism and — as importantly — clinically-consistent features, in order to
allow models to be still trained effectively.

Our approach is tested on two tasks, each simulating both _i.i.d._ and
_non-i.i.d._ scenarios: 1) classification of tuberculosis with X-ray data,
using Montgomery County X-ray and Shenzhen Hospital X-ray sets [17, 18, 19],
and 2) melanoma classification using skin images of the ISIC 2019 dataset [20,
21, 22]. Results show how our approach is able to reach performance similar to
using all real data together in a single node. Privacy-preserving capabilities
are measured quantitatively by evaluating LPIPS distance [23] between real
images and samples generated, respectively, through latent space optimization
on a standard GAN and by the proposed approach. Qualitatively, we also show
several examples of generated images with corresponding closest match in the
real dataset, demonstrating significant differences that prevent tracing back
to the original real distribution.

In summary, the overall contributions of the proposed work are the following:

  * •

We propose, for the first time, a decentralized distributed learning strategy
designed for medical imaging data, which performs at least on par with
centralized learning approaches and yields performance similar to standard
(non-federated) training settings in low-data availability scenarios;

  * •

The distributed learning strategy is model and dataset-agnostic;

  * •

We also propose a privacy-preserving mechanism that supports synthetic data
sharing through a GAN-based technique designed to minimize patient information
leak;

  * •

We demonstrate that distributing the learning at the data level is an
effective, secure and scalable solution to create models able to generalize
across multiple datasets while keeping local data private.

##  2 Related Work

Figure 1: Overview of the proposed architecture. Each node initially trains a
_privacy-preserving GAN_ , that is used to sample synthetic data from the
local distribution, without retaining features that may be used to identify
patients. Then, each node iteratively receives the local model and a buffer of
synthetic samples from a random node, and fine-tunes the received model on its
own private data, using the buffer to prevent forgetting of previously-learned
features.

Data scarcity in the medical domain often calls for multi-institutional
collaborations, in order to collect enough evidence to draw statistically-
significant conclusions, as well as to create automated analysis methods
(based on machine learning) to support decision making. However, due to the
privacy-sensitive nature of medical information, data cannot be shared,
hindering the possibility to create robust learning methods able to
effectively support physicians in their clinical routines. Federated Learning
(FL) [24] has recently emerged as a family of distributed learning strategies
that allow nodes to keep training data private, while supporting the creation
of a shared model. In a typical FL setting, a central server sends a model to
a set of client nodes; each node fine-tunes the model on its own data, then
sends local model updates back to the server; the server aggregates the
updates by all nodes into the global model, which is sent back to nodes
iteratively until convergence. Given the constraints existing in the medical
domain, especially in terms of data sharing, it represents an appropriate test
bench for federated learning methods [25, 26, 27, 28, 29]. The most
straightforward way to aggregate information from multiple nodes is through
averaging local gradients of each client, as proposed in FedAvg [24]. FedProx
[30] demonstrates theoretical convergence guarantee of FedAvg when training
over statistically-heterogeneous (_non-i.i.d._) data. Statistical data
heterogeneity is particularly challenging as it leads to catastrophic
forgetting in FL settings [31, 32]. FedCurv [33] addresses this issue by
adding a penalty term to the loss function in order to drive the local models
to a shared optimum. FedMA [34] builds a shared global model in a layer-wise
manner by matching and averaging hidden elements with similar feature
extraction signatures. Our method differs from existing feature integration
approaches in that, instead of averaging model updates or gradients, which can
be subject to input reconstruction attacks [8, 35, 7], each node attempts to
learn features that perform well on both its own dataset and other nodes’, in
a more principled way than crude parameter averaging.

Another novelty of the current work w.r.t. the existing state of the art in
the medical domain lies in the network topology. Unlike most solutions in
medical federated learning, our strategy is decentralized, i.e., nodes
directly exchange information among themselves. Indeed, the presence of a
central node that aggregates local updates simplifies the communication
protocols when the number of clients is very large (thousands or millions),
but introduces several downsides: it represents a single point of failure; it
can become a bottleneck when the number of clients increases [36]; in general,
it may not always be available or desirable in collaborative learning
scenarios [31, 37]. In decentralized learning, the central node is replaced by
peer-to-peer communication between clients: there is no longer a global shared
model as in standard FL, but the communication protocol is designed so that
all local models approximately converge to the same solution. Decentralized
distributed learning is particularly suitable to application in the medical
domain, where the number of nodes (i.e., institutions) is relatively low;
however, research is still ongoing, and no effective solutions have emerged.
In [38], a Bayesian approach is proposed to learn a shared model over a graph
of nodes, by aggregating information from local observational data with the
model of each node’s one-hop neighbors. A secure weight averaging algorithm is
proposed in [39], where model parameters are not shared between nodes, but all
converge to the same numerical values (with the disadvantages associated to
parameter averaging with _non-i.i.d._ data distributions). Other approaches
implement different communication strategies based on parameter sharing (e.g.,
gossip algorithms [37] or decentralized variants on FedAvg [40, 24]). In
general, most of the existing solutions do not target, nor are they tested on,
the medical domains — most employ toy datasets, such as MNIST and CIFAR10. A
work which is similar in spirit to ours is BrainTorrent [26], where a use case
of decentralized learning for MRI brain segmentation is presented. However,
like other approaches, simple parameter averaging is used to integrate
features from multiple nodes.

Finally, a key component in our approach lies in the use of generative
adversarial networks (GAN) to synthesize samples that can be shared between
nodes in a privacy-preserving way. Among similar approaches proposed in the
literaure, GS-WGAN [41] proposes a gradient-sanitized Wasserstein GAN to
improve differential privacy, by carefully distorting gradient information in
a way that reduces loss of information and generates more informative samples.
Federated CycleGAN [42] is designed to perform unsupervised federated image
translation. FedDPGAN [43] designs a distributed DPGAN [35] trained in a FL
framework, to generate models for COVID-19 diagnosis from chest X-ray images,
without data sharing. While CycleGAN, GS-WGAN and FedGPGAN share either
gradient or model updates of the generator with the server, introducing
privacy concerns related to the risk of input reconstruction, in our approach
nodes share a subset of synthetic images, generated in a way that discourages
privacy-sensitive information leaking, to be used with an experience replay
mechanism and enhance feature reuse among nodes.

##  3 Method

###  3.1 Overview

An overview of the proposed decentralized distributed learning approach is
shown in Fig. 1. In this scenario, a _federation_ consists of a set of $N$
peer nodes, each owning a private dataset.

Before the distributed training algorithm is started, each node internally
trains a _Privacy-Preserving Generative Adversarial Network_ , which is used
to generate synthetic samples from the private data distribution. The training
objective of the GAN is designed to enforce the constraint that sampled data
do not include privacy-sensitive, while maintaining the clinical features
required for successful training.

At each round of distributed training, each node receives the current model
and a set of synthetic samples — “buffer” — from a randomly-chosen predecessor
node, and sends its own model and buffer to a randomly-chosen successor node.
The input model is then fine-tuned on both the private dataset and the buffer,
in a way that is reminiscent of experience replay techniques in continual
learning (e.g., [13]), in order to learn features that transfer between nodes
and that can handle _non-i.i.d._ distributions. At the end of the current
round (i.e., after performing several training iterations), the locally-
trained model is sent to a randomly-chosen successor node, and the whole
procedure is repeated.

In this work we specifically address the problem of federated learning for
medical image classification; thus, the method is presented by considering
this task, but the whole strategy can be applied to any other task without
losing generalization.

###  3.2 Privacy-preserving GAN

In the proposed method, nodes exchange both models and data, implementing a
knowledge transfer procedure based on experience replay (see Sect. 3.3 below).
Of course, sharing real samples would go against federated learning policies;
hence, exchanged samples are generated so that they are representative of the
local data, while taking precautions against privacy violations — which may
happen, for instance, if the generative model overfits the source dataset.

Formally, we assume that each node $n_{i}$ owns a private dataset
$\mathcal{D}_{i}=\left\\{\left(\mathbf{x}_{1},\mathbf{y}_{1}\right),\left(\mathbf{x}_{2},\mathbf{y}_{2}\right),\dots,\left(\mathbf{x}_{M},\mathbf{y}_{M}\right)\right\\}$,
where each $\mathbf{x}_{j}\in\mathcal{X}$ represents a sample in the dataset,
and each $\mathbf{y}_{j}\in\mathcal{Y}$ represents the corresponding
target111The proposed approach is task-agnostic, as long as it is possible to
sample from the $\mathcal{Y}$ distribution. For simplicity, within the scope
of this work, we will focus on classification tasks, and we will assume that
targets are class labels.. The local dataset can then be used to train a
conditional GAN [15], consisting of a generator $G$, that synthesizes samples
for a given label by modeling
$P\left(\mathbf{x}|\mathbf{y},\mathbf{z}\right)$), where
$\mathbf{z}\in\mathcal{Z}$ is a random vector sampled from the generation
latent space, and a discriminator $D$, which outputs the probability of an
input sample being real, modeling
$P\left(\text{real}|\mathbf{x},\mathbf{y}\right)$. The standard GAN
formulation introduces a discrimination loss, which trains $D$ to distinguish
between real and synthetic samples:

$\mathcal{L}_{D}=-\mathbb{E}_{\mathbf{x},\mathbf{y}}\left[\log\left(D\left(\mathbf{x},\mathbf{y}\right)\right)\right]-\mathbb{E}_{\mathbf{z},\mathbf{y}}\left[\log\left(1-D\left(G\left(\mathbf{z},\mathbf{y}\right),\mathbf{y}\right)\right)\right],$
(1)

and a generation loss, which trains $G$ to synthesize samples that appear
realistic to the discriminator:

$\mathcal{L}_{G}=-\mathbb{E}_{\mathbf{z},\mathbf{y}}\left[D\left(G\left(\mathbf{z},\mathbf{y}\right),\mathbf{y}\right)\right].$
(2)

While it has been theoretically proven that, at convergence, the distribution
learned by the generator matches and generalizes from the original data
distribution [32], unfortunately GAN architectures may be subject to training
anomalies, including mode collapse and overfitting: as a consequence, the
basic GAN formulation may lead to the generation of samples that are near
duplicates of the original samples, which would be unacceptable in a federated
learning scenario.

In order to mitigate this risk, we introduce a _privacy-preserving loss_ ,
enforcing the generation of samples that do not retain potentially identifying
information, but still include features that are clinically relevant to the
target $\mathbf{y}$ of the synthetic sample. In other words, if $\mathbf{y}$
encodes the diagnosis of a certain disease, we want the generator to learn how
to synthesize samples conditioned by $\mathbf{y}$, that exhibit evidence of
that disease but cannot be traced back to any of the dataset’s samples of the
same disease.

To do so, our privacy-preserving loss aims at penalizing the model
proportionally to the similarity between pairs of real and synthetic samples.
We measure “similarity” by means of the LPIPS metric [23], which has been
shown to capture perceptual similarity better than traditional metrics (e.g.,
SSIM) by calibrating the distance between feature vectors extracted from a
pre-trained VGG model [44].

In practice, given a batch of real samples
$\left\\{\mathbf{x}_{1}^{(r)},\mathbf{x}_{2}^{(r)},\dots,\mathbf{x}_{b}^{(r)}\right\\}$
and a batch of synthetic samples
$\left\\{\mathbf{x}_{1}^{(s)},\mathbf{x}_{2}^{(s)},\dots,\mathbf{x}_{b}^{(s)}\right\\}$,
the privacy-preserving loss term is computed as:

$\mathcal{L}_{\text{PP}}=\frac{1}{b}\sum_{\mathbf{x}^{(r)}}\sum_{\mathbf{x}^{(s)}}d_{L}\left(\mathbf{x}^{(r)},\mathbf{x}^{(s)}\right),$
(3)

where $d_{L}$ is the LPIPS distance. Note that, in this formulation, we ignore
the $\mathbf{y}$ targets associated to each $\mathbf{x}$: we want to prevent
the model from generating near-duplicates of real samples in general,
regardless of class correspondence. Also, we intentionally employ a pairwise
metric on samples, rather than an aggregated metric such as Frêchet Inception
Distance [45], since we want to prevent similarity between samples, not
between distributions, which would conflict with the GAN objective. The
combined effect of the three loss terms — $\mathcal{L}_{D}$,
$\mathcal{L}_{G}$, $\mathcal{L}_{\text{PP}}$ — pushes the generator to explore
the sample space to match the dataset distribution, while “avoiding” latent
space mappings that would project to actual real samples.

###  3.3 Decentralized learning with experience replay

Current approaches for federated learning are mostly based on parameter
averaging (e.g., FedAvg), which is however a very crude way to combine
knowledge from multiple sources: feature locations are not aligned over
different models and may be disrupted by updates, before slowly converging to
consensus: hypothetically, two models could learn the same set of features at
different locations of the same layer, to only have them cancel each other
when averaging. In a decentralized scenario, this issue is even exacerbated,
due to the lack of an entity that enforces global agreement on node features.
In our approach, we address the problem by taking inspiration from continual
learning principles [46], where it is necessary to learn how to perform a task
without forgetting a previously-learned one: as a consequence, models are
encouraged to reuse and adapt features so that they can equally serve the
current and previous tasks.

Given these premises, we define a decentralized learning scenario where a node
receives another node’s model and surrogate data (generated through the
privacy-preserving GAN, since original data cannot be shared) — the _“previous
task”_ — and fine-tunes that model on its own private date — the _“current
task”_ — while using received data as a reference to what is necessary to
retain/adapt from the knowledge learned by the previous node.

In the following, we describe our method from the point of view of a single
node, $n_{i}$, which receives information from a predecessor node $n_{i-1}$.
Note that, for a given $n_{i}$, the predecessor node $n_{i-1}$ is not fixed:
in a practical asynchronous implementation, a node may receive a model and
buffer from any random node in the federation at any time, using queues to
handle incoming data.

As above, the local dataset at node $n_{i}$ is
$\mathcal{D}_{i}=\left\\{\left(\mathbf{x}^{(d)}_{1},\mathbf{y}^{(d)}_{1}\right),\left(\mathbf{x}^{(d)}_{2},\mathbf{y}^{(d)}_{2}\right),\dots,\left(\mathbf{x}^{(d)}_{M},\mathbf{y}^{(d)}_{M}\right)\right\\}$,
while $f_{\bm{\theta}}$ and
$\mathcal{B}_{i}=\left\\{\left(\mathbf{x}^{(b)}_{1},\mathbf{y}^{(b)}_{1}\right),\left(\mathbf{x}^{(b)}_{2},\mathbf{y}^{(b)}_{2}\right),\dots,\left(\mathbf{x}^{(b)}_{B},\mathbf{y}^{(b)}_{B}\right)\right\\}$
are, respectively, the model (with parameters $\bm{\theta}$) and buffer
received from node $n_{i-1}$. During a round of the learning algorithm, node
$n_{i}$ performs several training iterations on both $\mathcal{D}_{i}$ and
$\mathcal{B}_{i}$, using the latter to prevent forgetting of previously-
learned features, according to the experience replay strategy [47, 13].
Formally, the training loss is defined as:

$\mathcal{L}_{n_{i}}=\mathop{\mathbb{E}}_{\begin{subarray}{c}\mathbf{x}^{(d)},\mathbf{y}^{(d)},\\\
\mathbf{x}^{(b)},\mathbf{y}^{(d)}\end{subarray}}\left[L\left(f_{\bm{\theta}}\left(\mathbf{x}^{(d)}\right),\mathbf{y}^{(d)}\right)+L\left(f_{\bm{\theta}}\left(\mathbf{x}^{(b)}\right),\mathbf{y}^{(b)}\right)\right],$
(4)

where $L$ is the task-specific loss function (e.g., cross-entropy for
classification).

After optimizing the $\mathcal{L}_{n_{i}}$ objective through mini-batch
gradient descent for a certain number of training iterations, the resulting
model $f_{\bm{\theta}^{\prime}}$, with updated parameters
$\bm{\theta^{\prime}}$, is sent to a random node of the federation, along with
a buffer of locally-generated synthetic samples. The number of training
iterations and the size of the buffer is discussed in the next section.

Unlike other distributed learning approaches, each node does not have its own
local model: as the distributed learning algorithm proceeds, a node
iteratively receives a model from another node and updates it with local
information, while preserving previously-learned knowledge, before sending it
to the next node. As experimentally shown in the next section, at the end of
this procedure all models approximately converge to a solution that performs
similarly on all private datasets in the federation.

##  4 Experimental Results

We test our federated learning approach on two applications: 1) tuberculosis
classification from X-ray images, and 2) skin lesion classification. In this
section we present the employed benchmarks, the training procedure and report
the obtained results to demonstrate the advantages of the proposed approach
w.r.t. the state of the art.

###  4.1 Datasets

X-ray image datasets for tuberculosis classification. We employ the Montgomery
County X-ray Set and the Shenzhen Hospital X-ray Set [17, 18, 19]. The
Montgomery Set contains 138 frontal chest X-ray images (80 negatives and 58
positives), captured with a Eureka stationary machine (CR) at 4020$\times$4892
or 4892$\times$4020 pixel resolution. The Shenzhen dataset was collected using
a Philips DR Digital Diagnostic system. It includes 662 frontal chest X-ray
images (326 negatives and 336 positives), with a variable resolution of
approximately 3000$\times$3000 pixels.

Skin lesion classification. We employ the “ISIC 2019” dataset, which contains
25,331 skin images belonging to nine different diagnostic categories. The data
provided belongs to three different sources: 1) the BCN20000 [20] dataset,
consisting of 19,424 images of skin lesions captured from 2010 to 2016 in the
Hospital Clínic in Barcelona; 2) the HAM10000 dataset [21], which contains
10,015 skin images collected over a period of 20 years from two different
sites, the Department of Dermatology at the Medical University of Vienna,
Austria, and the skin cancer practice of Cliff Rosendahl in Queensland,
Australia; 3) the MSK4 [22] dataset, which is anonymous and includes 819
samples. Among all skin lesion classes, we only consider the melanoma class,
posing the problem as a binary classification task.

In our federated learning framework, we consider two settings:

  * •

_non-i.i.d. setting_ , the most realistic one, where each dataset is
associated to a single node: i.e., two nodes for tuberculosis classification
(one with the Montgomery County X-ray Set and the other with the Shenzhen
Hospital X-ray set); three nodes for melanoma classification (one with the
BCN20000 dataset, one with the HAM10000 dataset and the last one with the MSK4
dataset). This setting is intended to test performance under clear data
distribution shifts among nodes.

  * •

_i.i.d. setting_ , where a single dataset is randomly split and equally-
distributed over multiple nodes. This setting evaluates how the approach
scales with the number of nodes.

In all tasks and datasets, we employ 80% of the data to train both the
privacy-preserving GAN and the classification model, while the remaining 20%
of each dataset is used as test set. Test sets are balanced w.r.t. the label
to avoid performance biases due to class imbalance.

###  4.2 Training procedure and metrics

Table 1: Comparison between the proposed approach and centralized baselines. In the distributed learning setting, results are obtained with a buffer size of 512, 100 rounds and 100 epochs per round. Methods | Tuberculosis | Melanoma |
---|---|---|---
Shenzhen | Montgomery | Avg. | BCN | HAM | MSK4 | Avg.
Non-distributed training on dataset union | 0.823 | 0.721 | 0.772 | 0.784 | 0.815 | 0.798 | 0.799
Centralized distributed learning with synthetic data | 0.624 | 0.495 | 0.559 | 0.568 | 0.443 | 0.583 | 0.531
Decentralized distributed learning (ours) | 0.712 | 0.772 | 0.742 | 0.776 | 0.765 | 0.656 | 0.732
Table 2: Accuracy convergence among federated node models. Results obtained with a buffer size of 512.  |  | Tuberculosis | Melanoma
---|---|---|---
Rounds | Epochs | Shenzhen | Montgomery | BCN | HAM | MSK4
10 | 1 | 0.544 $\pm$ 0.040 | 0.525 $\pm$ 0.042 | 0.614 $\pm$ 0.064 | 0.586 $\pm$ 0.082 | 0.603 $\pm$ 0.076
10 | 0.665 $\pm$ 0.217 | 0.620 $\pm$ 0.145 | 0.673 $\pm$ 0.122 | 0.653 $\pm$ 0.166 | 0.636 $\pm$ 0.036
100 | 0.670 $\pm$ 0.220 | 0.750 $\pm$ 0.173 | 0.724 $\pm$ 0.077 | 0.672 $\pm$ 0.143 | 0.657 $\pm$ 0.046
100 | 1 | 0.642 $\pm$ 0.155 | 0.631 $\pm$ 0.129 | 0.716 $\pm$ 0.059 | 0.659 $\pm$ 0.142 | 0.646 $\pm$ 0.046
10 | 0.708 $\pm$ 0.170 | 0.772 $\pm$ 0.101 | 0.727 $\pm$ 0.083 | 0.710 $\pm$ 0.115 | 0.655 $\pm$ 0.043
100 | 0.712 $\pm$ 0.127 | 0.772 $\pm$ 0.045 | 0.776 $\pm$ 0.049 | 0.765 $\pm$ 0.079 | 0.656 $\pm$ 0.033

####  4.2.1 Decentralized distributed learning

In all settings, we employ ResNet-18 [48] as classification model, trained by
minimizing the cross-entropy loss with mini-batch gradient descent using the
Adam optimizer. Mini-batch size is set to 32 and 8 for the Shenzhen and
Montgomery datasets, respectively, and to 64 for skin lesion datasets. The
learning rate is set to $10^{-4}$. Data augmentation is carried out with
random horizontal flip; for skin images we additionally apply random 90-degree
rotations. All images are resized to 256$\times$256.

The node federation is trained for $T$ rounds. At each round of training, each
node receives the model and a buffer of synthetic data from a randomly-chosen
predecessor, fine-tunes the received model to its private data for $E$ epochs,
and sends the updated model and its own buffer of synthetic samples to a
randomly-chosen successor. In our implementation, at each round nodes are
randomly ordered to establish each node’s predecessor and successor: given our
focus on medical applications, we can assume that the number of nodes is low
enough that synchronization is not an issue. However, asynchronicity can be
achieved by assuming that nodes can store incoming data in queue: if the
distribution of successor nodes is uniform and computation times are similar
for all nodes, this is on average equivalent to the synchronous case.

####  4.2.2 GAN training

The proposed privacy-preserving GAN uses StyleGAN2-ADA [49] as a backbone,
because of its suitability in low-data regimes and its generation
capabilities. Training is carried out in two steps: 1) the GAN is initially
trained for 800,000 iterations without any privacy-preserving loss, to support
learning of high-quality visual features; 2) we enable the privacy-preserving
loss and fine-tune the mode for additional 150,000 iterations, in order to
limit the embedding of patient-specific patterns in the GAN latent space. In
our classification setting, GANs are trained in a label-conditioned fashion
with a mini-batch size of 32 and learning rate 0.0025 for both the generator
and the discriminator. We evaluate the generator’s capability to synthesize
realistic images by means of the Frêchet Inception Distance (FID) [45]. In
order to quantitatively evaluate privacy preservation, we generate, for each
real image, its closest synthetic sample by means of latent space projection
(described in Sect. 4.3.2), and compute the average LPIPS distance [23]
between such pairs: a sufficiently high value implies the impossibility to
reconstruct real images from the generator.

###  4.3 Results

####  4.3.1 Distributed training

We initially evaluate the performance (in terms of classification accuracy) of
our approach, trained for 100 rounds with 100 epochs per round, in the
_non-i.i.d._ setting, and compare its results to those achieved in case of a)
non-distributed training of a single model on the union of the datasets, and
b) centralized distributed learning where a central node receives a set of 256
privacy-preserving synthetic samples from each node and trains a global model.
Classification accuracy is reported for each dataset: in distributed learning
scenarios, performance is computed using the final model at the node where
that dataset is located. From the results reported in Table 1, two insights
can be drawn: 1) on large datasets, as in the case of Shenzhen and all the
melanoma datasets, our approach performs worse non-distributed training (as
can be expected); 2) surprisingly, on smaller datasets such as Montgomery, our
approach is able to outperform non-distributed training: a possible
explanation lies in the data augmentation effect induced by the use of
synthetic examples, which has been shown to reduce overfitting and help
improve performance in low-data regimes [50].

We proceed to evaluate the agreement between models trained on each node, by
measuring the standard deviation of the classification accuracy obtained by
all nodes’ models for each specific dataset, for different combinations of
numbers of rounds and epochs per round. Table 2 shows that, in our default
configuration with 100 rounds and 100 epochs per round, models tend to
converge to similar predictions with limited variability, lower than 0.06 in
four cases out of six. The performance on the Shenzhen (tubercolosis) dataset
exhibits a larger variability, which can be explained by the lower
generalization capabilities of the model trained on the very small Montgomery
dataset.

A key component of the proposed architecture is the sharing of synthetic
samples among all federated nodes. We evaluate the impact of the buffer size
on classification performance, and compare with the case when the buffer is
removed (i.e., only the model is shared). Results, evaluated on the
tuberculosis task (which is the only one we use also for the subsequent
evaluations on the architectural components) with 100 training rounds and
different epochs per round, are reported in Table 3 and indicate that the
experience replay strategy is necessary to allow the whole system to
effectively learn with _non-i.i.d._ data. In general, the presence of larger
buffer size ensures that common and robust features are learned, leading to
significant performance improvements, especially on small datasets such as
Montgomery with a performance gain of about 9 percent points between no buffer
and a buffer size of 2048 samples. However, in all the experiments, we use
buffer size of 512 as it represents the best compromise between accuracy and
memory consumption on all the employed datasets.

Table 3: Effect of buffer size on classification accuracy. |  | Tuberculosis
---|---|---
Buffer Size | Epochs | Shenzhen | Montgomery
0 | 1 | 0.548 $\pm$ 0.081 | 0.526 $\pm$ 0.042
10 | 0.667 $\pm$ 0.208 | 0.671 $\pm$ 0.160
100 | 0.672 $\pm$ 0.197 | 0.705 $\pm$ 0.237
256 | 1 | 0.652 $\pm$ 0.176 | 0.666 $\pm$ 0.113
10 | 0.674 $\pm$ 0.205 | 0.746 $\pm$ 0.095
100 | 0.686 $\pm$ 0.157 | 0.757 $\pm$ 0.075
512 | 1 | 0.642 $\pm$ 0.155 | 0.631 $\pm$ 0.129
10 | 0.708 $\pm$ 0.170 | 0.772 $\pm$ 0.101
100 | 0.712 $\pm$ 0.127 | 0.772 $\pm$ 0.045
1024 | 1 | 0.663 $\pm$ 0.162 | 0.659 $\pm$ 0.111
10 | 0.686 $\pm$ 0.196 | 0.752 $\pm$ 0.101
100 | 0.704 $\pm$ 0.143 | 0.788 $\pm$ 0.049
2048 | 1 | 0.661 $\pm$ 0.174 | 0.685 $\pm$ 0.083
10 | 0.694 $\pm$ 0.206 | 0.758 $\pm$ 0.104
100 | 0.712 $\pm$ 0.123 | 0.794 $\pm$ 0.074
Table 4: Comparison between our approach and FedAvg [24]. | _Non-i.i.d._ | _i.i.d._
---|---|---
| Tuberculosis | Melanoma | Tuberculosis | Melanoma
| 2 Nodes | 3 Nodes | 2 Nodes | 4 Nodes | 8 Nodes | 2 Nodes | 4 Nodes | 8 Nodes
FedAvg [24] | 0.744 | 0.782 | 0.777 | 0.808 | 0.838 | 0.754 | 0.712 | 0.686
Ours | 0.745 | 0.716 | 0.793 | 0.820 | 0.828 | 0.818 | 0.803 | 0.792

We then evaluate the capability of the proposed method to scale with the size
of the federated network, without losing effectiveness. We quantify this
property in the _i.i.d._ setting on both tuberculosis (Shenzhen dataset) and
skin lesion classification (BCN dataset) tasks, by equally splitting the
available data on multiple nodes, and training for 100 rounds, with 100 epochs
per round. Fig 2 shows how the proposed approach is able to keep
classification accuracy high (on the two datasets there is a performance drop
less than 3 percent points from 2 to 8 nodes). A slight decrease in
performance can be observed as the number of nodes increase, which can be
explained by the lower size of the training set available for each node.

Figure 2: Effect of the number of nodes on the proposed approach, in the
_i.i.d._ setting (buffer size 256, 100 epochs, 100 rounds).

Finally, we compare our method with the FedAvg approach [24], which is a
suitable baseline as it has been shown to perform generally better than
decentralized methods [40, 38] and results are in Table 4 in both _i.i.d._ and
_non-i.i.d._ settings for the tubercolosis and melanoma classification
tasks222Results for FedAvg were computed using the Intel OpenFL framework
[51]. Since centralized methods benefit from the global view of all models by
the central node, FedAvg’s performance is slightly better on the _non-i.i.d._
scenario (in this case, with varying number of nodes), while, remarkably, our
approach outperforms on average it on the _i.i.d._ settings, demonstrating
better scalability capabilities. Considering all the obtained performance on
the various scenarios, we can claim that our approach is at least on par with
FedAvg, demonstrating that a principled feature integration approach may lead
to better convergence properties over naïve parameter averaging even in a more
complex case as the decentralized one.

####  4.3.2 Privacy-preserving Data Synthesis

After demonstrating the impact that the experience replay mechanism has on the
proposed decentralized learning strategy, it is important to ensure that no
sensitive information is leaving each node. Therefore, we quantify how much
information of real samples is stored by our privacy-preserving method, and in
particular in the mapping between latent space and synthetic images. To do so,
we employ the projection method proposed in [16]: given a real image
$\mathbf{x}$, we find an intermediate latent point $\mathbf{w}$ such that the
generated image $G(\mathbf{w})$ is most similar to $\mathbf{x}$, by optimizing
$\mathbf{w}$ to minimize the LPIPS distance [23] between $\mathbf{x}$ and
$G(\mathbf{w})$.

In practice, for each image of the dataset used for GAN training, we perform
backprojection to find its most similar synthetic sample, and measure the
LPIPS distance between the original and projected images. Fig. 3 shows the
histograms of the resulting distances on the Shenzhen dataset, using GAN
models trained with and without the proposed privacy-preserving loss (both
models start from the same $\mathbf{w}$, for fairness). The histograms show
that standard GAN training, with no privacy-preserving loss, tends to yield
distances close to 0, demonstrating that real images are indeed encoded into
the generator latent space; our model significantly mitigates this issue, by
synthesizing samples that are significantly more different than the original
ones.

Figure 3: Quantitative analysis of privacy-preserving generation. In blue,
LPIPS distance histogram between real images and the corresponding images
obtained through latent space projection using a GAN trained without the
proposed privacy-preserving loss. In red, LPIPS distance histogram between
real images and the closest images generated with the proposed approach.

In order to qualitatively substantiate these findings, Fig. 4 compares
original samples from the Shenzhen dataset with the corresponding projections,
generated with and without our privacy-preserving loss333We show only X-Ray
synthesized samples as the effect is of our privacy-preserving strategy is
more appreciable than in skin lesion data.. It is easy to notice that
generated samples with a traditional GAN highly resemble real data, making it
impossible to share such samples, albeit synthetic, in a privacy-safe manner,
as they clearly contain patient information. Instead, comparing real images
with the projections obtained from privacy-preserving GAN confirms the
inability of the generator to find latent representations that recover real
images used during training.

Figure 4: Qualitative samples of our privacy-preserving generation. Top row:
real images from the Shenzhen dataset. Middle row: projection with a standard
GAN. Bottom row: projection with our privacy-preserving GAN.

Given the high realism of generated samples, we run additional tests meant to
push the level of privacy preservation: a) models are not shared among nodes —
only synthetic buffers are sent and received (_“buffer-only sharing”_); b)
models are trained only using synthetic data, even for the local dataset
(_“synthetic-only training”_). Fig. 5 shows the internal architecture of each
node in the two variants. Results obtained with these alternative privacy-
enhanced strategies are provided in Table 5, when training on the Shenzhen and
Montgomery datasets in the _non-i.i.d._ setting for 100 rounds, with 100
epochs per round and a buffer size of 512. It can be noted that the “buffer-
only sharing” variant achieves comparable performance to our standard setting,
giving encouraging perspectives for further analysis. Instead, the “synthetic-
only training” variant performs poorly, showing that using features from real
images to train local models is essential to reach good global performance:
nevertheless, further research on the generation process might lead to
improvements in this aspect.

Figure 5: Privacy-enhanced alternative architectures. (a) _“Buffer-only sharing”_ configuration: a local node model is trained on real data, but only a buffer of synthetic samples is shared with other nodes. (b) _“Synthetic-only training”_ : Even within the dataset owner node, models are trained on synthetic data only. Table 5: Classification accuracy of the proposed privacy-enhanced strategies in the _non-i.i.d._ setting.  Configuration | Shenzhen | Montgomery
---|---|---
Standard | 0.712 | 0.772
Buffer-only sharing | 0.696 | 0.679
Synthetic-only training | 0.585 | 0.517

##  5 Conclusion

In this paper, we presented a novel decentralized distributed learning
framework, that replaces traditional parameters averaging with a more
principled feature integration approach based on the combination of experience
replay and privacy-preserving generative models. In our approach, nodes
communicate with each other by sharing local models and buffers of synthetic
samples; local model updates are carried out in a way that encourages the
reuse and adaptation of features learned by other nodes, thus avoiding
potentially disruptive effects due to blind feature averaging. Experimental
results show that our method performs at least on par with state-of-the-art
centralized approaches, and outperforms them in low-data availability
scenarios, which is a typical setting in the medical domain. Additionally,
quantitative and qualitative analysis shows that our privacy-preserving
generation approach is able to synthesize samples that are significantly
different from real data, while correctly supporting the learning of
discriminative features. In the future, we aim at investigating some
unexplored properties of our method: for instance, unlike all other existing
methods based on parameter averaging is required, our approach does not
strictly require that all nodes share the same model architecture. Model
heterogeneity could therefore be employed to create a shared ensemble and
combine different feature learning capabilities.

## References

  * [1] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-Ilcus, C. Chute, H. Marklund, B. Haghgoo, R. Ball, K. Shpanskaya _et al._ , “Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 33, no. 01, 2019, pp. 590–597.
  * [2] X. Wang, Y. Peng, L. Lu, Z. Lu, M. Bagheri, and R. M. Summers, “Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 2097–2106.
  * [3] J. P. Cohen, P. Morrison, L. Dao, K. Roth, T. Q. Duong, and M. Ghassemi, “Covid-19 image data collection: Prospective predictions are the future,” _arXiv 2006.11988_ , 2020. [Online]. Available: https://github.com/ieee8023/covid-chestxray-dataset
  * [4] J. R. Zech, M. A. Badgeley, M. Liu, A. B. Costa, J. J. Titano, and E. K. Oermann, “Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study,” _PLoS medicine_ , vol. 15, no. 11, p. e1002683, 2018.
  * [5] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” _ACM Transactions on Intelligent Systems and Technology (TIST)_ , vol. 10, no. 2, pp. 1–19, 2019.
  * [6] D. Evans, V. Kolesnikov, M. Rosulek _et al._ , “A pragmatic introduction to secure multi-party computation,” _Foundations and Trends in Privacy and Security_ , vol. 2, no. 2-3, pp. 70–246, 2018.
  * [7] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [8] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, “Inverting gradients-how easy is it to break privacy in federated learning?” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 16 937–16 947, 2020.
  * [9] B. Zhao, K. R. Mopuri, and H. Bilen, “idlg: Improved deep leakage from gradients,” _arXiv preprint arXiv:2001.02610_ , 2020.
  * [10] R. Ratcliff, “Connectionist models of recognition memory: constraints imposed by learning and forgetting functions.” _Psychological review_ , vol. 97, no. 2, p. 285, 1990.
  * [11] A. Robins, “Catastrophic forgetting, rehearsal and pseudorehearsal,” _Connection Science_ , vol. 7, no. 2, pp. 123–146, 1995.
  * [12] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne, “Experience replay for continual learning,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [13] P. Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,” _Advances in neural information processing systems_ , vol. 33, pp. 15 920–15 930, 2020.
  * [14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [15] M. Mirza and S. Osindero, “Conditional generative adversarial nets,” _arXiv preprint arXiv:1411.1784_ , 2014.
  * [16] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila, “Analyzing and improving the image quality of stylegan,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8110–8119.
  * [17] S. Candemir, S. Jaeger, K. Palaniappan, J. P. Musco, R. K. Singh, Z. Xue, A. Karargyris, S. Antani, G. Thoma, and C. J. McDonald, “Lung segmentation in chest radiographs using anatomical atlases with nonrigid registration,” _IEEE transactions on medical imaging_ , vol. 33, no. 2, pp. 577–590, 2013\.
  * [18] S. Jaeger, S. Candemir, S. Antani, Y.-X. J. Wáng, P.-X. Lu, and G. Thoma, “Two public chest x-ray datasets for computer-aided screening of pulmonary diseases,” _Quantitative imaging in medicine and surgery_ , vol. 4, no. 6, p. 475, 2014.
  * [19] S. Jaeger, A. Karargyris, S. Candemir, L. Folio, J. Siegelman, F. Callaghan, Z. Xue, K. Palaniappan, R. K. Singh, S. Antani _et al._ , “Automatic tuberculosis screening using chest radiographs,” _IEEE transactions on medical imaging_ , vol. 33, no. 2, pp. 233–245, 2013.
  * [20] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpern, S. Puig _et al._ , “Bcn20000: Dermoscopic lesions in the wild,” _arXiv preprint arXiv:1908.02288_ , 2019\.
  * [21] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Scientific data_ , vol. 5, no. 1, pp. 1–9, 2018.
  * [22] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler _et al._ , “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi),” in _IEEE international symposium on biomedical imaging (ISBI 2018)_. IEEE, 2018, pp. 168–172.
  * [23] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 586–595.
  * [24] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial intelligence and statistics_. PMLR, 2017, pp. 1273–1282.
  * [25] W. Li, F. Milletarì, D. Xu, N. Rieke, J. Hancox, W. Zhu, M. Baust, Y. Cheng, S. Ourselin, M. J. Cardoso _et al._ , “Privacy-preserving federated brain tumour segmentation,” in _International workshop on machine learning in medical imaging_. Springer, 2019, pp. 133–141.
  * [26] A. G. Roy, S. Siddiqui, S. Pölsterl, N. Navab, and C. Wachinger, “Braintorrent: A peer-to-peer environment for decentralized federated learning,” _arXiv preprint arXiv:1905.06731_ , 2019.
  * [27] I. Dayan, H. R. Roth, A. Zhong, A. Harouni, A. Gentili, A. Z. Abidin, A. Liu, A. B. Costa, B. J. Wood, C.-S. Tsai _et al._ , “Federated learning for predicting clinical outcomes in patients with covid-19,” _Nature medicine_ , vol. 27, no. 10, pp. 1735–1743, 2021.
  * [28] I. Feki, S. Ammar, Y. Kessentini, and K. Muhammad, “Federated learning for covid-19 screening from chest x-ray images,” _Applied Soft Computing_ , vol. 106, p. 107330, 2021.
  * [29] J. Pang, Y. Huang, Z. Xie, J. Li, and Z. Cai, “Collaborative city digital twin for the covid-19 pandemic: A federated learning solution,” _Tsinghua science and technology_ , vol. 26, no. 5, pp. 759–771, 2021.
  * [30] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” _Proceedings of Machine Learning and Systems_ , vol. 2, pp. 429–450, 2020.
  * [31] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _Foundations and Trends in Machine Learning_ , vol. 14, no. 1–2, pp. 1–210, 2021.
  * [32] I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio, “An empirical investigation of catastrophic forgetting in gradient-based neural networks.” _arXiv preprint arXiv:1312.6211_ , 2013.
  * [33] N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor-Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,” _arXiv preprint arXiv:1910.07796_ , 2019.
  * [34] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni, “Federated learning with matched averaging,” _arXiv preprint arXiv:2002.06440_ , 2020\.
  * [35] L. Xie, K. Lin, S. Wang, F. Wang, and J. Zhou, “Differentially private generative adversarial network,” _arXiv preprint arXiv:1802.06739_ , 2018\.
  * [36] X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? a case study for decentralized parallel stochastic gradient descent,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [37] P. Vanhaesebrouck, A. Bellet, and M. Tommasi, “Decentralized collaborative learning of personalized models over networks,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 509–517.
  * [38] A. Lalitha, O. C. Kilinc, T. Javidi, and F. Koushanfar, “Peer-to-peer federated learning on graphs,” _arXiv preprint arXiv:1901.11173_ , 2019.
  * [39] T. Wink and Z. Nochta, “An approach for peer-to-peer federated learning,” in _2021 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W)_. IEEE, 2021, pp. 150–157.
  * [40] T. Sun, D. Li, and B. Wang, “Decentralized federated averaging,” _arXiv preprint arXiv:2104.11375_ , 2021.
  * [41] D. Chen, T. Orekondy, and M. Fritz, “Gs-wgan: A gradient-sanitized approach for learning differentially private generators,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 673–12 684, 2020.
  * [42] J. Song and J. C. Ye, “Federated cyclegan for privacy-preserving image-to-image translation,” _arXiv preprint arXiv:2106.09246_ , 2021.
  * [43] L. Zhang, B. Shen, A. Barnawi, S. Xi, N. Kumar, and Y. Wu, “Feddpgan: federated differentially private generative adversarial networks framework for the detection of covid-19 pneumonia,” _Information Systems Frontiers_ , vol. 23, no. 6, pp. 1403–1415, 2021.
  * [44] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * [45] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [46] M. Delange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [47] M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,” _arXiv preprint arXiv:1810.11910_ , 2018.
  * [48] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [49] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila, “Training generative adversarial networks with limited data,” in _NeurIPS_ , vol. 33, 2020, pp. 12 104–12 114.
  * [50] M. Pennisi, S. Palazzo, and C. Spampinato, “Self-improving classification performance through GAN distillation,” in _ICCVW_ , 2021, pp. 1640–1648.
  * [51] G. A. Reina, A. Gruzdev, P. Foley, O. Perepelkina, M. Sharma, I. Davidyuk, I. Trushkin, M. Radionov, A. Mokrov, D. Agapov _et al._ , “Openfl: An open-source framework for federated learning,” _arXiv preprint arXiv:2105.06413_ , 2021.