# Federated Learning and catastrophic forgetting in pervasive computing:
demonstration in
HAR domain

Anastasiia Usmanova Univ. Grenoble Alpes, France  François Portet Univ.
Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000 Grenoble, France
Philippe Lalanda Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LIG, 38000
Grenoble, France  German Vega Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP,
LIG, 38000 Grenoble, France

###### Abstract

Federated Learning has been introduced as a new machine learning paradigm
enhancing the use of local devices. At a server level, FL regularly aggregates
models learned locally on distributed clients to obtain a more general model.
In this way, no private data is sent over the network, and the communication
cost is reduced. However, current solutions rely on the availability of large
amounts of stored data at the client side in order to fine-tune the models
sent by the server. Such setting is not realistic in mobile pervasive
computing where data storage must be kept low and data characteristic
(distribution) can change dramatically. To account for this variability, a
solution is to use the data regularly collected by the client to progressively
adapt the received model. But such naive approach exposes clients to the well-
known problem of catastrophic forgetting. The purpose of this paper is to
demonstrate this problem in the mobile human activity recognition context on
smartphones.

##  1 Introduction

Pervasive computing promotes the integration of smart devices in our living
spaces to develop services providing assistance to people [3]. For a decade,
we have seen the emergence of smarter services based on Machine Learning (ML)
and Deep Learning (DL) techniques. ML enables the production of highly
performing decision systems by identifying patterns that may be hidden within
massive data sets whose exact nature is unknown and therefore cannot be
programmed explicitly. However, bringing such services into production raises
a number of difficult issues. Current solutions rely on complex distributed
architectures, integrating devices, edge, and cloud infrastructure. Simply
put, learning models are built in the cloud using historical data, then
deployed and executed on devices, or on the nearest edge resources. If
possible, additional data is regularly collected by the devices and sent up to
the cloud in order to build more up-to-date models. Such an approach, however,
undergoes major limitations in terms of security (data sent over unprotected
networks), performance (high volume of data), and cost related to
communication and also models retraining (especially when using deep
learning).

Federated Learning (FL) was recently proposed [2] [12] as a new learning
paradigm promoting the execution and specialization of models on devices,
called clients. At one point, clients models are sent to a server where they
are aggregated into a more generic one. This new model is redistributed to the
clients for a new local learning iteration, and so on. Federated Learning
reduces communication costs and improves security because only models, and no
private data, are exchanged [25]. It has immediately attracted attention in
the pervasive field but it rapidly appeared that adaptations were needed to
meet specific challenges of this domain [11]. Originally, Federated Learning
was set out as an alternative form of distributed learning to obtain a single
well-performing model (on the server side) without ever communicating user
data. However, to deal with pervasive computing, this learning approach should
shift from a single server-centric model to a multi client-centric model
objective [22]. Precisely, client models should perform well on local data
(_strong personalization_) as well as on data only seen by others (_good
generalization_).

One particular thorny issue is when a client receives a new model from the
server. This new model must be readjusted to the client in order to recover a
good personalization, while preserving its recently obtained generalization.
The solution commonly used today is to fine-tune (train) the received model
with client data kept for this purpose. In practice, this requires a carefully
selected set of representative training data to obtain satisfactory
performances [11]. However, such setting is not realistic in mobile pervasive
computing where data storage must be kept low and data characteristic
(distribution) can change dramatically. To account for this variability, a
solution is to use the data regularly collected by the client to progressively
adapt the received model. But such naive approach exposes clients to the
problem of _catastrophic forgetting_ [19], which appears when a neural network
is optimized on new data that are too different from the data previously used
for training.

This is related to a more general neural network problem known as the
stability-plasticity dilemma [16], where plasticity and stability refer to
integrating new knowledge and retaining previous knowledge respectively. Such
dilemma is at the heart of the research area of Continual Learning (CL) also
known as lifelong learning [9].

The purpose of this paper is to show that the problem of catastrophic
forgetting is critical in a pervasive computing application using Federated
Learning. It is demonstrated in the mobile Human Activity Recognition (HAR)
domain. The paper is organized as it follows. In Section 2, we provide
background on FL and CL and describe current solutions and issues. In Section
3, we present our experimental settings while section 4 demonstrates the
addressed problem, that is catastrophic forgetting in Federated Learning. The
paper is concluded by a synthesis and an outlook on future work.

##  2 Background

###  2.1 Federated Learning

Federated Learning promotes the computation of local models on devices and the
aggregation of these models on a server to produce a new model that is sent
back to clients for use and further training. When the client model has been
significantly updated or when a certain time has elapsed, the client model is
sent to the server for a new aggregation (figure 1). A core incentive of FL in
pervasive computing is to utilize mutual learning goals between clients to
indirectly share knowledge with one another. However, effective collaboration
between unmoderated clients, is still a challenge. This is due to statistical
heterogeneity (differences in individuals’ usage) and system heterogeneity
(difference between client data caused by different system traits).

A number of studies have proposed intricate methods to mitigate the
detrimental effects of heterogeneity. Specifically, we have identified methods
such as FedProx [23] and FSVRG [20] which apply local model/gradient
controlled learning that limits the divergence of clients. Another known
method, as seen in FeSEM [34] and MLMG [29] is to group clients or create more
than one server model based on clients similarity.

Figure 1: Federated Learning Architecture.

A key point in FL is the way specialized models are aggregated. In the case of
deep learning, two families of algorithms can be considered. The first
strategy is to emphasize generalization. The aggregation algorithm considers
local models in their entirety and builds a new model that potentially calls
into question all layers and all weights associated with neurons. This
approach is shown in FedAvg [27] and FedMA [32] algorithms. The second
strategy focuses more on client specialization. Here, the algorithm does not
question certain parts of the local models. Precisely, only the base layers
are sent to the server for generalization, while the last layers are kept
unchanged. This approach is used in FedPer algorithm [15].

FL has been only recently used to implement pervasive services and,
ultimately, does not address all the specifics of this domain well. The goal
in pervasive computing, by contrast with the original FL, is to increase all
individual clients performances, not only the server model performance. To do
so, FL must account for the fact that data, even though in the same domain,
might evolve very differently according to the clients. Also, mediation
techniques [21] [14] could be integrated into the FL approach in order to
align the data in a more homogeneous way.

###  2.2 Catastrophic Forgetting and Continual Learning

Continual Learning (CL), also called lifelong learning or online machine
learning, is the ability of a model to learn continually from an infinite
stream of data, gradually integrating new acquired knowledge into old
knowledge [8]. Continual Learning highlights the sequential nature of the
learning process. Formally, newly acquired data is passed to the model in the
form of tasks. Tasks are made of labeled classes, uniquely identified and
disjoint from each other, and a corresponding dataset. A model then learns
sequences of tasks. At each training session, a model has only access to the
current task (and possibly to some data kept from previous tasks), which is
not accessible anymore after training.

In the past few years, numerous works have proposed to deal with Continual
Learning and avoid catastrophic forgetting. They can be divided into three
families [26]: regularisation-based methods, exemplar-based methods, and bias-
correction methods. Regularisation-based approaches minimize the impact of
learning on the weights that are important for previous tasks. Among them, EWC
[19] calculates the weights of model parameters with a Fisher Information
Matrix, while PathInt [13] accumulates changes in each parameter during the
learning process. RWalk [7] uses both approaches complemented with exemplars
to improve results. Other approaches use knowledge distillation [5] such as
LwF [24] which trains a current model (student) with a past model as a teacher
to prevent drifting while learning new tasks. The student model can also be
guided using the attention of the teacher model as in LwM [10].

Exemplar-based methods, also called _rehearsal methods_ , rely on the storage
of exemplars from previous tasks. For instance, iCarl [30], selects exemplars
by their feature space representation and performs classification based on a
nearest-class-mean rule in that feature space. EEIL [6] uses an equal number
of exemplars of all classes and a loss composed of a distillation measure to
retain the knowledge acquired from the old classes, and a cross-entropy loss
in order to learn the new classes.

Regarding the bias-correction approaches, we can mention

BiC [33], which uses an additional layer to correct task bias in the model and
train the model in two stages. LUCIT [17] uses a cosine normalization layer
instead of the standard softmax layer and proposes to use margin ranking loss
to prevent inter-task confusion. Finally, IL2M [4] uses statistics of
predictions of classes from previous tasks.

Overall, exemplar-based CL is the most efficient method but requires that a
fair amount of data is kept on device, which is not possible when resources
are limited. However, some works show that it is interesting to explore how a
small amount of exemplars can complement other techniques [6].

##  3 Human Activity Recognition

###  3.1 Datasets and base model

Experiments were performed in the domain of Human Activity Recognition (HAR)
on smartphones. We believe that HAR is an excellent domain to test and better
understand Federated Learning and Catastrophic Forgetting because activities
tend to have generic patterns while being highly idiosyncratic. Collected data
is very heterogeneous because it depends on people, devices, the way devices
are carried, the environment, etc. Also, there are a number of realistic
labeled datasets presenting heterogeneous data collected with different
devices that can be used for our experiments.

For the experiments, we have retained the UCI [1] dataset, which is heavily
used by the HAR community as a benchmark. This dataset was collected with 30
volunteers using a Samsung Galaxy S II placed on the waist. One example of
data contains 128 recordings of accelerometer and gyroscope (both 3
dimensions). There are 6 classes in the dataset (number of examples per class
are in brackets): 0 – Walking (1722), 1 – Walking Upstairs (1544), 2 – Walking
Downstairs (1406), 3 – Sitting (1777), 4 – Standing (1906), 5 – Lying (1944).

HAR based on wearable sensors has prompted numerous research works, be they
academic or industrial [28]. The most common approach is to process windows of
data streams in order to extract a vector of features which, in turn, is used
to feed a classifier. Today, the most popular and effective model is
undoubtedly end-to-end deep neural networks.

For that study, and all the experiments presented in this paper, we used the
Convolution Neural Networks of [11] to take into account the small processing
power on mobile devices. The studies [11] showed that a model 196-16C_4M_1024D
gives the best result over a set of state-of-the-art-model. It includes 196
filters of a 16x1 convolution layer followed by a 1x4 max pooling layer, then
by 1024 units of a dense layer and finally a softmax layer.

We trained the model on the UCI dataset partitioned as follow: 70% of data is
in the train set, 15% in the validation set, and 15% in the test set. The
model was trained using a mini-batch SGD of size 32 and a dropout rate of 0.5
and reached a 94.64% accuracy on the testset. All experiments were written on
Python 3 using the TensorFlow 2 library and run on a CPU Intel(R) Xeon(R)
2.30GHz (2 CPU cores, 12GB available RAM).

###  3.2 PSA and T-SNE

To show how difficult it is for a model to distinguish the different classes
in HAR, we made a Principal Component Analysis (PCA) [18] and a T-SNE analysis
of the UCI dataset projected on a last layer (before activation) of the neural
network.

Figure 2 shows the first three principal components of 200 randomly chosen
examples of each class in a 3-dimensional space. We can see that the class
_Laying_ is very distinguishable from the rest. _Sitting_ and _Standing_ are
close to each other. Walking movements are located in the same space, which
shows they are hard to distinguish.

Figure 2: PCA of 200 examples of the UCI dataset on a last layer of the neural
network (color display needed to read this figure).

Figure 3 presents the t-SNE analyzes of the dataset UCI HAR on a last dense
layer before the final activation of a pretrained CNN. T-Distributed
Stochastic Neighbor Embedding (t-SNE) is a non-linear technique for dimension
reduction by minimizing the divergence between two distributions: a
distribution that measures pairwise similarities in high-dimensional space and
a distribution that measures pairwise similarities of the corresponding low-
dimensional space. It allows to find patterns in the data by identifying
observed clusters based on similarity of data points with multiple features.
As t-SNE follows non-linearity, it can capture the structure of complex
manifolds of higher-dimensional data and outperforms PCA, which is a linear
feature extraction technique.

Figure 3: T-SNE of 200 examples of the UCI dataset on a last layer of the
neural network (color display needed to read this figure).

Again, we can see in this figure that it is very easy to distinguish the
_Laying_ class from the other classes. The _Sitting_ and _Standing_ classes
are located very close to each other. Movement-related classes are located in
the same area, but they are individually hard to distinguish.

We will show in the following sections that such data distribution has a
strong and direct impact on catastrophic forgetting.

##  4 Demonstration of catastrophic forgetting

###  4.1 Experimental settings

In order to demonstrate the problem of catastrophic forgetting, we have
defined a Federated Learning architecture including a remote server and a
number of clients simulated on laptops for reproducibility. Each client hosts
a model to classify the UCI classes numbered from 0 to 5, i.e. _Walking_ is 0,
_Walking Upstairs_ is 1, _Walking Downstairs_ is 2, _Sitting_ is 3, _Standing_
is 4 and _lying_ is 5.

For the first learning round, all clients use the same neural network (the one
defined to compute the PCA). For model aggregation at the server level, the
FedAvg algorithm has been selected for its simplicity and good efficiency in
the HAR domain [11]. Also, for the sake of clarity, all the clients behave
synchronously, so all of them take part in each round.

At each FL communication round, each client trains its local model on data
from a sequence of disjoint tasks. A task is characterized by a set of classes
and a dataset. Formally, each client $k\in\\{1,2,...,K\\}$ has its privately
accessible sequence of $n_{k}$ tasks $\mathcal{T}_{k}$:

$\mathcal{T}_{k}=\left[\mathcal{T}_{k}^{1},\mathcal{T}_{k}^{2},...,\mathcal{T}_{k}^{t},...,\mathcal{T}_{k}^{n_{k}}\right],\mathcal{T}_{k}^{t}=(C^{t}_{k},D^{t}_{k}),$

where $t\in\\{1,...,n_{k}\\}$, $C^{t}_{k}$ is a set of classes representing
the task $t$ of a client $k$ (such that $C^{i}_{k}\cap C^{j}_{k}=\varnothing$
if $i\neq j$) and $D^{t}_{k}=\\{X^{t}_{k},Y^{t}_{k}\\}$ is training data
corresponding to $C^{t}_{k}$.

A task can take place during several FL rounds. Formally, each task
$\mathcal{T}_{k}^{t}$ for client $k$ is trained during $r^{t}_{k}$
communication rounds and $\sum_{t=1}^{n_{k}}r^{t}_{k}=R$, where $R$ is the
total number of rounds between server and clients.

At each communication round, each client uses new training data, in order to
simulate pervasive conditions of limited storage and incremental data
collection. Data can also be new in the sense that it corresponds to a new
task that has not been seen before by a client. Formally, at communication
round $r$ client $k$ uses training data $D_{kr}=\\{X_{kr},Y_{kr}\\}$:

$D_{kr}=D^{t}_{kr}\subset D^{t}_{k},\hskip
5.69054pt\sum_{d=1}^{t-1}r^{d}_{k}<r\leqslant\sum_{d=1}^{t-1}r^{d}_{k}+r^{t}_{k},$

where $D_{k^{\prime}r^{\prime}}\cap
D_{k^{\prime\prime}r^{\prime\prime}}=\emptyset$, if $k^{\prime}\neq
k^{\prime\prime}$ and $r^{\prime}\neq r^{\prime\prime}$

($1\leqslant k^{\prime},k^{\prime\prime}\leqslant K,1\leqslant
r^{\prime},r^{\prime\prime}\leqslant R$);

Figure 4: Demonstration of catastrophic forgetting in standard CL (a) and FCL
(b).

The notion of task is fundamental because it is the basis of the calculation
of forgetting for clients.

###  4.2 Baseline scenario

In order to have a baseline, we set up a first experiment with a single
client, called Client 1. The purpose of this experience is to show how
Continual Learning based on fine tuning in pervasive computing leads to
catastrophic forgetting.

The scenario implemented by this client is the following. Client 1 learns two
successive tasks containing a single activity: _Walking upstairs_ and then
_Walking downstairs_. The client is trained alone iteratively for eight steps,
four on the first activity and four on the second, with randomly chosen UCI
data. Here, the steps are created to mimic the notion of rounds and ease the
comparison with the FL experiment _per se_.

###  4.3 Federated Learning scenario

For the second experiment, we integrated this client in a Federated Learning
process, where it can indirectly benefit from knowledge acquired by other
clients, and we defined a scenario very favourable to catastrophic forgetting.

Precisely, we have defined a scenario based on eight FL communication rounds.
In the first four rounds, client 1 learns a first task (_Walking upstairs_).
Then, in the next four rounds, it learns another task (_Walking downstairs_).
Formally, we can say that Client 1 learns $n_{1}=2$ tasks in total:
$\mathcal{T}_{1}^{1}=(C_{1}^{1},D_{1}^{1})$ and
$\mathcal{T}_{1}^{2}=(C_{1}^{2},D_{1}^{2})$, where $C^{1}_{1}=\\{1\\}$ and
$C^{2}_{1}=\\{2\\}$;
$\mathcal{T}_{1}=\left[\mathcal{T}_{1}^{1},\mathcal{T}_{1}^{2}\right]$,
$r_{1}^{1}=r_{1}^{2}=R/2$. At each round, the model of this client is
initialized with the model aggregated in the server and then trained with data
collected between the communication rounds $D^{t}_{1r}\subset D_{1}^{t}$

We assume $K-1$ other clients which only learn one task. These generic clients
perform online-learning on the same well-balanced task containing all classes
at each round. So, they learn $n_{g}=1$ task in total:
$\mathcal{T}_{g}^{1}=(C_{g}^{1},D_{g}^{1})$, where
$C^{1}_{g}=\\{0,1,2,3,4,5\\}$, and have their privately accessible sequence of
tasks $\mathcal{T}_{g}=\left[\mathcal{T}_{g}^{1}\right]$, so $r_{g}^{1}=R$. In
this task, all clients get the same number of examples (data) at each round,
so the size of their dataset does not influence forgetting.

Hence, for the sake of simplicity, we assume that all other $K-1$ clients
behave similarly, and that we can represent their influence in the FL process
with a single generalized client. This is why, for aggregation at the server
side, the weights of generalized and observed clients are $1/K*(K-1)$ and
$1/K$ respectively.

To build the train and test datasets, we randomly chose examples from the UCI
dataset in accordance with the scenario. For each client $k$ and for each
communication round $r$, we built a train set $D_{kr}$ of the same size.

We computed the models performance on a common test set for all clients and
the server. The test dataset includes 100 examples of each class (600 in
total). We run 8 communication rounds and 10 epochs for the clients local
training. We assumed that we have $K=5$ clients (the influence of $K-1$ of
them represented by a single generalized client). The size of a dataset for
each client $k$ and each round $r$ is $|D_{kr}|=120$. We used a learning rate
$\eta=0.01$, dropout rate equal to 0.5, batch size $B=32$ and SGD optimizer.

###  4.4 Results

Figure 4 shows the percentage of correctly classified examples from the test
set after each of the eight steps/rounds.

The left part of the figure is about the first experiment. It appears that
classes $C_{0}$, $C_{2}$, $C_{3}$, and $C_{4}$ are almost totally forgotten
when _Walking upstairs_ ($C_{1}$) starts. Similarly, when the client moves on
to _Walking downstairs_ ($C_{2}$), $C_{1}$ is immediately forgotten. Only
_Sitting_ ($C_{5}$), a static activity, is preserved even if there is no
example of this class during training. As highlighted by the PCA, $C_{5}$ is
very distinguishable from the others. The initially trained model successfully
learnt it and the parameters used to classify it were not changed with further
training on $C_{1}$ and $C_{2}$.

The right part of the figure is about Federated Learning. It shows slightly
better performances. It is apparent that FL successfully transferred to client
$1$ the knowledge about static actions (classes $C_{3}$, $C_{4}$ and $C_{5}$)
that has been learnt by other clients. Client $1$, which has never seen these
actions, can recognize them with good accuracy. However, $C_{2}$ is forgotten
by Client 1 when $C_{1}$ starts and, similarly, $C_{1}$ is totally forgotten
by Client 1 when it moves on to $C_{2}$. Catastrophic forgetting is undeniable
for those classes.

$C_{0}$ (Walking) $C_{1}$ (Walking Upstairs) and $C_{2}$ (Walking Downstairs)
are three very competitive classes as exhibited in Figure 2 and 3. It seems
thus that the absence of any of these classes will induce forgetting. This
scenario is realistic since, in real-life, a person might spend much time
walking without using stairs. During that time all information acquired in the
model about stairs will be forgotten.

These experiments show that Federated Learning is clearly subject to the
catastrophic forgetting issue when fine-tuning client models, although this
problem can be attenuated by knowledge transfer between clients. However, with
standard FL, this knowledge transfer also goes from Client 1 to the server
model. It can be seen on Figure 4 that the classes $C_{0}$, $C_{1}$ and
$C_{2}$ of the server model have almost half the performances of the
generalized client. Catastrophic forgetting has thus not only an effect on
Client 1 but also on all the other clients. We are convinced that this
forgetting problem is a real problem that current FL techniques cannot avoid
in pervasive computing when data is heterogeneous, devices have limited
resources.

##  5 Conclusion

In this paper, we have highlighted in the Human Activity Recognition domain
that the use of a Federated Learning does not avoid the catastrophic
forgetting problem. Even if some knowledge, coming for various clients, is
retained at the server level, some tasks can be very quickly forgotten. This
is due to the fact that some tasks are basically difficult to distinguish by a
Neural Network (and they affect the same parts of a learning model). This is
for example the case for movement activities in the HAR field.

Dealing with catastrophic forgetting is then a necessary step for providing
lifelong-learning in smart collection of objects. In our current work, we are
investigating the use of distillation technique to transfer generic knowledge
to specialized models. First results have shown this approach can greatly
reduces the catastrophic forgetting effect [31].

##  6 Acknowledgements

This work was partially supported by MIAI@Grenoble-Alpes (ANR-19-P3IA-0003).

## References

  * [1] Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, and Jorge L. Reyes-Ortiz.  A public domain dataset for human activity recognition using smartphones.  In 21th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013, pages 24–26, 2013\.
  * [2] McMahan H. B., Moore E., Ramage D., and Hampson S.  Communication-efficient learning of deep networks from decentralized data.  International Conference on Artificial Intelligence and Statistics (AISTATS), 2017.
  * [3] C. Becker, C. Julien, P. Lalanda, and F. Zambonelli.  Pervasive computing middleware: current trends and emerging challenges.  CCF Transactions on Pervasive Computing and Interaction, vol. 1, 2019.
  * [4] E. Belouadah and A. Popescu.  Il2m: Class incremental learning with dual memory.  International Conference on Computer Vision,, 2019.
  * [5] C. Bucilua, R. Caruana, and A. Niculescu-Mizil.  Model compression.  International Conference on Knowledge Discovery and Data Mining, 2006.
  * [6] Francisco M. Castro, Manuel J. Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari.  End-to-end incremental learning.  European Conference on Computer Vision, 2018.
  * [7] A. Chaudhry, P. K. Dokania, T. Ajanthan, and P. H. Torr.  Riemannian walk for incremental learning: Understanding forgetting and intransigence.  European Conference on Computer Vision, 2018.
  * [8] Z. Chen and B. Liu.  Lifelong machine learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, vol.12, number 3, 2018.
  * [9] Zhiyuan Chen and Bing Liu.  Lifelong machine learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 12(3):1–207, 2018.
  * [10] P. Dhar, R. V. Singh, K.-C. Peng, Z. Wu, and R. Chellappa.  Learning without memorizing.  Conference on Computer Vision and Pattern Recognition, 2019.
  * [11] Sannara Ek, Francois Portet, Philppe Lalanda, and German Vega.  A federated learning aggregation algorithm for pervasive computing: Evaluation and comparison.  In Proceedings of the 19th International Conference on Pervasive Computing and Communications (PerCom 2021), 2021.
  * [12] Keith Bonawitz et al.  Towards federated learning at scale: system design.  In Proceedings of the 2nd SysML Conference, Palo Alto, CA, USA, 2019\.
  * [13] F.Zenke, B.Poole, and S.Ganguli.  Continual learning through synaptic intelligence.  International Conference on Machine Learning, 2017.
  * [14] I. Garcia, G. Pedraza, B. Debbabi, P. Lalanda, and C. Hamon.  Towards a service mediation framework for dynamic applications.  In Proceedings of the IEEE Asia-Pacific Services Computing Conference (APSCC 2010), 2010.
  * [15] Manoj Ghuhan Arivazhagan, Vinay Aggarwal, Aaditya Kumar Singh, and Sunav Choudhary.  Federated Learning with Personalization Layers.  arXiv e-prints, December 2019.
  * [16] S. Grossberg.  Studies of mind and brain : neural principles of learning, perception, development, cognition and motor control.  Boston studies in the philosophy of science 70. Dordrecht: Reidel, 1982\.
  * [17] S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin.  Learning a unified classifier incrementally via rebalancing.  International Conference on Computer Vision,, 2019.
  * [18] Ian T. Jolliffe and Jorge Cadima.  Principal component analysis: a review and recent developments.  Phil. Trans. R. Soc., 2016.
  * [19] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, and A. Grabska-Barwinska et al.  Overcoming catastrophic forgetting in neural networks.  National Academy of Sciences, vol. 114, no. 13, 2017.
  * [20] Jakub Konečný, H. Brendan McMahan, Daniel Ramage, and Peter Richtárik.  Federated optimization: Distributed machine learning for on-device intelligence, 2016.
  * [21] Philippe Lalanda, Luc Bellissard, and Roland Balter.  Asynchronous mediation for integrating business and operational processes.  IEEE Internet Computing, vol. 10, issue 1, 2006.
  * [22] Sangsu Lee, Xi Zheng, Jie Hua, Haris Vikalo, and Christine Julien.  Opportunistic federated learning: An exploration of egocentric collaboration for pervasive computing applications.  In 2021 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 1–8. IEEE, 2021.
  * [23] Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith.  Federated optimization in heterogeneous networks, 2020.
  * [24] Z. Li and D. Hoiem.  Learning without forgetting.  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, 2017.
  * [25] W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y. C. Liang, Q. Yang, D. Niyato, and C. Miao.  Federated learning in mobile edge networks: A comprehensive survey.  IEEE Communications Surveys Tutorials, vol. 22, no. 3, pp. 2031–2063, 2020.
  * [26] Marc Masana, Xialei Liu, Bartłomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and Joost van de Weijer C.  Class-incremental learning: survey and performance evaluation.  arXiv:2010.15277v1, 2020.
  * [27] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54, pages 1273–1282, Fort Lauderdale, USA, 2017.
  * [28] O.D.Lara and M.A.Labrador.  A mobile platform for real-time human activity recognition.  IEEE Consumer Communications and Networking Conference (CCNC), pages 667–671, 2012.
  * [29] Yang Qin and Masaaki Kondo.  Mlmg: Multi-local and multi-global model aggregation for federated learning.  In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops), pages 565–571, 2021.
  * [30] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert.  icarl: Incremental classifier and representation learning.  Conference on Computer Vision and Pattern Recognition, 2017.
  * [31] Anastasiia Usmanova, Francois Portet, Philppe Lalanda, and German Vega.  A distillation-based approach integrating continual learning and federated learning for pervasive services.  arXiv:2109.04197, 2021.
  * [32] Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris S. Papailiopoulos, and Yasaman Khazaeni.  Federated learning with matched averaging.  CoRR, abs/2002.06440, 2020.
  * [33] Y. Wu, Y. Chen, L. Wang, Y. Ye, Z. Liu, Y. Guo, and Y. Fu.  Large scale incremental learning.  International Conference on Computer Vision, 2019.
  * [34] Ming Xie, Guodong Long, Tao Shen, Tianyi Zhou, Xianzhi Wang, and Jing Jiang.  Multi-center federated learning, 2020.