# Federated Continual Learning for Socially Aware Robotics

Luke Guerdan1∗ and Hatice Gunes2 1 School of Computer Science, Carnegie Mellon
University, USA. *Corresponding author<EMAIL_ADDRESS>This work
resulted from his ACS MPhil project at the University of Cambridge. 2
Department of Computer Science and Technology, University of Cambridge, UK.

###### Abstract

From learning assistance to companionship, socially aware and socially
assistive robotics is promising for enhancing many aspects of daily life.
However, socially aware robots face many challenges preventing their
widespread public adoption. Two such major challenges are (1) lack in behavior
adaptation to new environments, contexts and users, and (2) insufficient
capability for privacy protection. The commonly employed _centralized
learning_ paradigm, whereby training data is gathered and centralized in a
single location (i.e., machine / server) and the centralized entity trains and
hosts the model, contributes to these limitations by preventing online
learning of new experiences and requiring storage of privacy-sensitive data.
In this work, we propose a _decentralized learning_ paradigm that aims to
improve the personalization capability of social robots while also paving the
way towards privacy preservation. First, we present a new framework by
capitalising on two machine learning approaches, Federated Learning and
Continual Learning, to capture interaction dynamics distributed physically
across robots and temporally across repeated robot encounters. Second, we
introduce four criteria (adaptation quality, adaptation time, knowledge
sharing, and model overhead) that should be balanced within our decentralized
robot learning framework. Third, we develop a new algorithm – Elastic Transfer
– that leverages importance-based regularization to preserve relevant
parameters across robots and interactions with multiple humans (users). We
show that decentralized learning is a viable alternative to centralized
learning in a proof-of-concept Socially-Aware Navigation domain, and
demonstrate the efficacy of Elastic Transfer across our proposed evaluation
criteria.

##  I Introduction

As robot systems improve, they are moving from laboratory environments into
our daily lives. The homes of tomorrow may enlist Jibo, “the world’s first
family robot”, to lend support with cooking, family photos, and entertainment
[1]. In the classroom, humanoid robots like Pepper may provide emotionally
intelligent tutoring to students who need learning assistance [2]. Yet,
several key barriers remain before social robots such as these can become
fully integrated into daily life.

One barrier to widespread social robot adoption is personalization. Whereas
robots of today are designed for single encounters based on pre-configured
abilities, future systems require adaptation over repeated long-term
interactions [3]. Pioneering studies in longitudinal HRI research have
reported that robots who tailor their behavior towards users’ personality,
preferences, and interaction style foster more engagement and long-term
acceptance [4]. A further barrier to social robot adoption is the lack of
privacy protections. Because robots will support humans with sensitive
activities such as learning and eldercare, data protection must be considered
from the ground up. However, existing systems do not emphasize privacy, thus
reducing user trust in the system and limiting widespread adoption [5].

An underlying cause of these barriers is the centralized machine learning (ML)
process robots use to develop intelligent behaviors. Centralized ML works by
(1) gathering user data in a single location such as a server and (2) training
a model to perform a new function (i.e. classifying user emotions) using the
full dataset. However, this process limits robots’ ability to personalize to
new users, because doing so would require uploading raw user data, retraining
the model, and redistributing the model to all devices. Moreover, in a social
robotics scenario, storing user data on a server imposes privacy
vulnerabilities. Instead, by leveraging decentralized learning, whereby robots
leave raw data on robot devices and learn by collaboratively exchanging model
information, it may be possible to support personalization while also
protecting privacy.

In this work, we introduce a decentralized learning approach for socially
aware robotics (_FCL4SR_ henceforth) by capitalising on two machine learning
approaches, Federated Learning (FL) and Continual Learning (CL) [6], to model
interactions that are distributed _physically_ across robots and _temporally_
across repeated interactions. We outline four criteria that should be
satisfied for _FCL4SR_ settings, including adaptation quality, adaptation
time, effective knowledge sharing, and minimal overhead. As a proof-of-
concept, we compare a range of existing Federated Continual Learning (FCL)
methods on a socially-assistive navigation benchmark with respect to these
four criteria. We also develop a new regularization-based algorithm, _Elastic
Transfer_ , that improves performance with respect to three of the proposed
criteria. This new method improves performance by penalizing changes to
parameters that were important in earlier local tasks as well as tasks
encountered by other clients by unifying existing FL and CL approaches [7, 8].
In summary, this work provides three key contributions by:

  1. 1.

Formulating socially aware robotics as a decentralized learning problem. To
our knowledge, this is the first work to formulate socially aware robotics as
a decentralized learning problem that combines both FL and CL. We provide a
flexible extension to previous CL for socially aware robotics settings
(_FCL4SR_) that supports personalization to new individuals and groups.

  2. 2.

Developing a decentralized learning evaluation framework. We outline which
tradeoffs should be balanced in _FCL4SR_ settings, and apply this framework to
a comprehensive evaluation of 10 existing FCL approaches.

  3. 3.

Proposing a regularization strategy for improving FCL performance. We draw
connections between existing FL and CL methods to propose a new Elastic
Transfer approach for improving knowledge sharing among clients. This provides
the first regularization-based strategy tailored to FCL and improves
adaptation quality, adaptation time, and knowledge sharing.

##  II Related work

In Continual Learning (CL), models learn online from sequentially encountered
data [9]. Given a series of incremental tasks (e.g., interactions with new
users, or repeated interaction sessions with the same user), CL approaches aim
to incrementally learn each task without forgetting knowledge from prior
tasks. Catastrophic forgetting occurs if performance on prior tasks
deteriorates while learning the current task. Several approaches have been
proposed to address catastrophic forgetting in CL. Regularization-based
strategies add a penalty to the loss term that prevents models from diverging
from the one learned on prior tasks. L2-transfer (L2T) [10] applies an L2
penalty, while Elastic Weight Consolidation (EWC) [8] applies a quadratic
penalty based on Fisher information to consider parameter importance.
Architectural CL strategies, such as Additive Parameter Decomposition (APD)
[11], address catastrophic forgetting by modifying the model structure to
accommodate task specific information.

In Federated Learning (FL), a network of distributed phones, robots, or other
clients downloads a common initial model from a server, trains on locally
available data, then transmits updated parameters to the server. The server
aggregates updates into a new global model and re-distributes it to clients.
FedAvg is a standard approach for aggregating client parameter updates on the
server via a weighted average [12]. However, prior work shows that FedAvg
performance is poor in non-iid settings with client dataset size imbalances,
concept drift (same label is assigned for dissimilar features), or concept
shift (different label is assigned to similar features). FedProx handles this
by imposing an L2 penalty in the client objective to prevent divergence from
the most recent aggregate model [13]. FedCurv extends FedProx by imposing an
importance-based elastic penalty based on Fisher information [7].

There are recent works on personalized federated learning that learn a global
model while also fine tuning local models on user-specific information [14,
15]. Personalized federated learning approaches have been reported to be
vulnerable to negative transfer, whereby averaging user-specific models
worsens performance for some subjects [14]. Existing personalized FL methods
have also seen limited consideration of temporal adaptability, where concept
shift and concept drift cause non-iid data over time [16]. In this work, we
consider continual learning as the solution well-suited to address these
issues.

Figure 1: A schematic diagram of Federated Continual Learning for Socially
Aware Robots (_FCL4SR_). In this example, a joint global model $\theta_{G}$ is
learned by a network of three robots (i.e., clients) over the course of
interactions with three groups of participants. Each user’s data
$\mathcal{D}_{i}^{j}$ remains locally on the device, and only parameter
updates are transmitted (federated learning). When robots interact with a new
group of users, local data is erased from the device and robots attempt to
retain knowledge from interactions with prior users (continual learning).

In Federated Continual Learning (FCL), FL and CL are combined to accommodate
learning scenarios that are spatially and temporally distributed. A network of
distributed clients each learns a series of local tasks, where each task may
be available to a subset of other clients, or be available on a single client
[6]. This paradigm introduces a new opportunity for inter-client knowledge
transfer, whereby experiences from previous tasks can be shared among clients
to improve learning. However, it also introduces a challenge of inter-client
interference, in which experiences aggregated from other clients accelerate
catastrophic forgetting.

FedWeIT is an architectural approach that was initially proposed to solve the
FCL problem [6]. This technique decomposes a global model into a set of
client-specific base parameters and task-adaptive parameters that learn
knowledge about each local task. This approach encourages knowledge transfer
by enabling clients to exchange task-adaptive parameters and weight them
locally via an attention mechanism. Recent work has proposed new FCL methods
for addressing inter-client interference and facilitating knowledge transfer
[17, 18, 19, 20, 21]. These approaches sometimes explicitly model learning as
an FCL problem [20], and other times examine FL under various forms of non-iid
data [17, 18, 19, 21]. While FCL has been applied to natural language
processing [22] and healthcare [23] applications, no work to our knowledge has
explored applications of FCL to socially aware robotics.

Decentralized Learning in Socially Aware Robotics. Continual learning [24],
federated learning [25], and a combination of both frameworks [26] have been
applied to broader robotics problems. Continual learning has been shown to be
an effective approach for personalization of socially-aware robots over
repeated interactions [3, 27]. Most closely related to this work, a recently-
proposed framework extends an existing CL taxonomy [24] to affective robotics.
Their framework formulates continual learning of personalized socio-emotional
human robot interactions [28], and has been successfully leveraged for
learning the social appropriateness of domestic robot actions [29]. A user
study applying this framework demonstrated that CL-based personalization over
an interaction session improves participants’ impressions of robot
anthropomorphism and likeability [30].

While results demonstrate the feasibility of CL in human robot interaction,
challenges remain. For instance, strategies based on continual learning only
do not provide a mechanism for exchanging information among multiple robots.
Therefore, performance will be limited by training data encountered previously
on the local device. This concern is especially relevant in socially aware
robotics and human robot interaction contexts, where dataset sizes tend to be
comparably small [28]. In this work, we propose leveraging FL to overcome such
dataset and data sharing limitations, while still ensuring that learning
remains distributed (e.g., does not require centralized storage of raw data on
a server). As a result, our approach also provides a promising step towards
achieving better privacy-preservation.

##  III Approach

In this section, we present a new framework for formulating decentralized
socially aware robot learning as an FCL problem, _FCL4SR_ , and identify key
desiderata that should be achieved by the _FCL4SR_ framework. Below we
describe existing approaches before formulating a new regularization-based
method called Elastic Transfer.

###  III-A Learning Scenario

In Federated Continual Learning for Social Robots (FCL4SR), a network of
social robots adaptively learns new intelligent behaviors over a sequence of
distributed encounters with end-users (Fig. 1). For example, social robots may
learn to detect user-specific nuances in affective signals [28] or social
navigation preferences [31, 29]. In FCL4SR scenarios, data used to learn new
behaviors is partitioned into client-task datasets $\mathcal{D}_{i,j}$
containing data acquired from the $i$’th robot during an interaction with the
$j$’th user. During each task, robots perform multiple rounds of local
training and parameter aggregation to learn the desired behavior (FL step).
The robots then begin interacting with a new set of end-users, and loose
access to previously-acquired raw data (CL step). Thus, _FCL4SR_ only requires
raw user data to be temporarily stored on the robot during the interaction
with a current user, and does not require permanent storage on-device or on a
remote server. As such _FCL4SR_ provides improved privacy protections compared
to permanently storing raw user data on a server (i.e., as is standard in
current centralized learning practices), but does not provide a formal privacy
guarantee (i.e., as would be provided by differential privacy).

Our _FCL4SR_ learning scenario builds upon a prior framework designed to
describe learning personalized affective behaviors in CL. In the prior
framework, Type NC updates occur when a robot learns new affective behavior
categories (i.e. learning to distinguish happiness vs. sadness facial
expressions) over each task. Similarly, Type NI updates occur when a robot
learns variations between the same affective behavior (i.e. different
variations of happiness facial expressions) [28]. In our framework, we use
Type NC updates to describe personalization to new users and Type NI updates
to describe repeated interactions with the same user. Further, interactions
with new users can be distributed spatially (i.e., when multiple robots
interact with different users and share information via FL) and temporally
(i.e., when the same robot interacts with different users over time). Fig. 1
shows a schematic illustration of this learning scenario.

As a concrete example of _FCL4SR_ , consider a hospital where a network of
intelligent robots is deployed for emotional support [23]. In this setting,
robots learn appropriate behaviors over repeated interactions with a patient.
Because users may have different response patterns, it is desirable to update
the model based on individualized data (i.e. personalization). Given the
sensitive setting, it is also critical to limit sharing of sensitive raw-data
via a server or persistent local storage on-device. In FCRL, each robot
gathers data during interactions with a patient, trains on this data locally,
and transmits parameter updates for updating a global model. When the robots
interact with a new set of patients, local data is removed, and the
information encoded in learned parameters remains available during
interactions with future residents. Thus, an FCRL algorithm exchanges
information among robots to accelerate learning of intelligent behaviors with
new groups of end-users. While the example above focuses on a medical setting,
our proposed framework also applies to diverse settings such as tutoring [32],
well-being coaching [30], autism therapy [33], where social robots adaptively
learn new intelligent behaviors from end-users.

###  III-B Evaluation Framework

It is crucial to examine several performance dimensions to gain a holistic
understanding of FCL tradeoffs. For example, one naive approach for solving
the scenarios above would be to train a separate model for each client-task
pairing and load the model dynamically during test time. This Single Task
Learning (STL) method may provide strong performance (high adaptation
quality). However, it would also require many feedback rounds to provide
sufficient data.

To improve the adaptation time, we may enhance STL by running a local CL
algorithm on each robot (termed Local-CL). Here, the initial set of
participants still provide many feedback examples, but later participants
benefit from fine-tuning an initialized model. However, if the first
participants return to interact with the robot in later tasks, the robot may
require additional interactions to recover the initial adaptation quality due
to catastrophic forgetting. A further drawback of CL and STL is that they are
unable to leverage experiences from other robots. Therefore, to facilitate
knowledge sharing among robots, we ca share parameter updates through FL.
Thus, an approach using both CL and FL (FL-CL) such as FedWeIT can be used.

Other trade-offs to consider include privacy and model overhead. Because STL,
Local-CL, and FL-CL employ decentralized learning, they all improve user
privacy. However, it is important to consider overhead factors including model
size, client-server communication costs, and energy consumption [24]. We
summarize these considerations in the following desiderata:

  1. 1.

High Adaptation Quality. After a series of interactions, the robot should be
well-tuned to the specific context. This will be demonstrated by satisfying
the learning objective (i.e. low MSE, high F1-score) at the end of training.

  2. 2.

Minimal Adaptation Time. The robot should quickly adjust to the interaction
context. This will be achieved by (1) generalizing to new predictive tasks
with limited fine-tuning and (2) learning effectively on small datasets.

  3. 3.

Effective Knowledge Sharing. The robot should integrate knowledge from other
robots to achieve high adaptation quality with fast adaptation time.

  4. 4.

Minimal Overhead. The robot’s financial, energy, and connectivity requirements
should be minimized.

###  III-C Elastic Transfer

Based on an evaluation of existing FCL approaches (exp. one and two below), we
formulate a new regularization-based method. Our approach extends EWC in the
CL setting and FedCurv in the FL setting. Both EWC and FedCurv are motivated
by a Bayesian framework. In a CL scenario where an optimal model
$\theta^{1}_{*}$ has been found for Task 1, EWC finds a solution for Task 2
near $\theta^{2}_{*}$ while remaining within the region of low Task 1 error.
This is achieved by applying a quadratic penalty to each parameter
proportional to its importance in previous tasks [8].

The process of optimizing the parameters $\theta$ given task datasets
$\mathcal{D}^{1:T}=\\{\mathcal{D}^{1},\mathcal{D}^{2},...,\mathcal{D}^{T}\\}$
can be framed as computing the posterior
$p\left(\theta\mid\mathcal{D}^{1},\ldots,\mathcal{D}^{T}\right)$ over all
possible values of $\theta$. As the model learns a new task online, it
combines the prior learned from earlier tasks with the $t$’th task’s
likelihood to form an updated posterior given by:

$\begin{split}\frac{p\left(\mathcal{D}^{t}\mid\theta\right)p\left(\theta\mid\mathcal{D}^{1},\ldots,\mathcal{D}^{t-1}\right)}{p\left(\mathcal{D}^{t}\mid\mathcal{D}^{1},\ldots,\mathcal{D}^{t-1}\right)}.\end{split}$
(1)

In this case, only $\mathcal{D}^{t}$ is required while learning task $t$
because the posterior absorbs knowledge from previous tasks. EWC makes use of
Laplace’s approximation to estimate this intractable posterior using a
Gaussian [34]. In particular, given a mean $\theta^{i}_{*}$ centered at the
maximum a posteriori (MAP) estimate of task $i<t$ and a precision defined by
the diagonal Fisher information matrix (FIM) $F^{i}$ evaluated at
$\theta^{i}_{*}$, Laplace’s approximation can be given by
$p(\mathcal{D}^{i}\mid\theta)\approx\mathcal{N}(\theta;\theta^{i}_{*},F^{i})$.
Provided that $\theta^{i}_{*}$ is a local minima, the FIM acts as a surrogate
to the Hessian of the negative log-likelihood of the posterior distribution
[35]. Given this approximation, the MAP estimate $\theta^{t}_{*}$ for task $t$
can be found by minimizing:

$-\log
p\left(\mathcal{D}^{t}\mid\theta\right)+\frac{\lambda}{2}\sum_{i=0}^{t-1}F^{i}\left(\theta-\theta_{*}^{i}\right)^{2}$
(2)

where $\lambda$ controls regularization strength. FedCurv applies this same
posterior decomposition over client devices in an FL setting [7]. During each
communication round $r$, each client $c$ exchanges their MAP estimate
$\hat{\theta}_{c,r}$, and precision estimate $F_{c,r}$ refined over $E$ epochs
local training. The MAP estimate for client $c$’s parameters at round $r$ of
training can be given by minimizing:

$-\log p\left(\mathcal{D}_{c}\mid\theta\right)+\frac{\lambda}{2}\sum_{i\in
C\setminus{c}}F_{i,r-1}\left(\theta-\hat{\theta}_{i,r-1}\right)^{2}$ (3)

where the second term provides an importance-based regularization penalty
proportional to $\lambda$. Because each client trains for $E$ epochs-per-round
rather than to task convergence, the MAP estimates $\hat{\theta}_{c,r}$ maybe
not be at a local minima, and approximates the true MAP. This can reduce
FedCurv performance with small $E$ [7]. A key benefit of EWC and FedCurv is
the adaptive penalty term imposed on the training objective. Whereas FedProx
and L2T impose isotropic L2 penalties that constrain parameters equally,
importance-based regularization penalizes changes to parameters in proportion
to their salience (e.g. elastic regularization).

We extend elastic regularization to improve FCL convergence to be used in
socially aware robotics settings. By constraining learning on local client-
task datasets such that important parameters are not overwritten across
clients and earlier tasks, it may be possible to limit inter-client
interference. Similarly, this process may also promote consolidation of
salient parameters learned across clients, promoting effective knowledge
sharing. Whereas FedWeIT proposes transfer of task-adaptive parameters between
clients, we instead propose transfer of task-adaptive importance estimates
encoded by $\theta*$ and $F*$.

Figure 2: Training on the local dataset $\mathcal{D}_{2}^{2}$.

Fig. 2 summarizes how FedCurv and EWC can be extended to FCL problems. This
schematic illustrates the objective of training a local model on
$\mathcal{D}_{2}^{2}$ in a scenario with two clients and two tasks. We
formulate the process of learning $\mathcal{D}_{2}^{2}$ as incorporating the
likelihood into the conditional prior of other client-task partitions to give
a new posterior. We approximate this term by combining three cases:

  * •

Case 1: Previous task, same client. Here, we approximate
$p\left(\theta\mid\mathcal{D}_{2}^{1}\right)$ using $*\theta_{2}^{1}$,
$*F_{2}^{1}$ (EWC; beige in Fig. 2). Because $*\theta_{2}^{1}$ is a previous
task trained to convergence, we expect $*\theta_{2}^{1}$ to be at a local
minima.

  * •

Case 2: Previous task, different client. Here (shown in green),
$p\left(\theta\mid\mathcal{D}_{1}^{1}\right)$ is an earlier task learned on
client 1. To prevent overwriting knowledge from $\mathcal{D}_{1}^{1}$ while
learning $\mathcal{D}_{2}^{2}$, we also need to account for $*\theta_{1}^{1}$,
$*F_{1}^{1}$ in the local objective.

  * •

Case 3: Same task, different client. In this case (FedCurv; shown in blue),
when we approximate $p\left(\theta\mid\mathcal{D}_{1}^{2}\right)$, we use
$\hat{\theta}_{1}^{2}$, $\hat{F}_{1}^{2}$ from other clients as an MAP and
precision estimate. Because task 2 is also being trained on other clients,
$\hat{\theta}_{1}^{2}$ acts as a rough approximation of the MAP where
$\hat{\theta}_{1}^{2}\approx*\theta_{1}^{2}$ for large $E$.

We leverage Online-EWC to scale the scenario shown in Fig. 2 to a set of
clients $C$, each of which learn a series of $T$ tasks. Online-EWC applies
Laplace’s approximation to the full posterior rather than each term
individually by storing a single FIM estimate $F^{*}_{c}$ and re-centered MAP
estimate $*\theta_{c}^{t-1}$ [36]. The online FIM estimate
$F_{c}^{*}:={\sum_{k=1}^{t-1}*F_{c}^{k}}$ is computed as a running sum of
Fisher information matrices estimated for each of the the previous $t-1$ tasks
on client $c$. The MAP estimate $*\theta_{c,r}^{t}$ in the $r$’th round of
training can be found my minimizing:

$\begin{split}-\log
p\left(\mathcal{D}_{c,t}\mid\theta\right)+\frac{\lambda_{1}}{2}\sum_{i\in
C}*F_{i}(\theta-*\theta_{i}^{t-1})^{2}\\\ +\frac{\lambda_{2}}{2}\sum_{i\in
C\setminus{c}}\hat{F}_{i}(\theta-\hat{\theta}_{i,r-1}^{t})^{2}.\end{split}$
(4)

The first term uses $\lambda_{1}$ to weight refined estimates $*F$, $*\theta$
from previous tasks, while the second uses $\lambda_{2}$ to weight rough
estimates of $\hat{\theta}$, $\hat{F}$ currently being optimized across
clients. The first term applies case 1 when $c=i$, and case 3 when $c\neq i$.
The second term handles case 2. We show the training process for optimizing
this local loss in Alg. 1. Under this regime, storage requirements are linear
in $|C|$, while communication requirements are quadratic in $|C|$.

1

Input: Client-task datasets $\\{\mathcal{D}_{c}^{t}\\}_{c\in
C,t\in\\{1,...,T\\}}$, epochs-per-round $E$, rounds-per-task $R$, Clients $C$,
Number of tasks $T$

Output: Optimized global model $\theta_{G}$

2

3for  _$t\leftarrow 1$ to $T$_ do

4 if  _$t >1$_ then

5 for  _$c\in C$_ do

6 Update $*F_{c}$, $*\theta_{c}^{t-1}$ from previous task

7 Send $*F_{c}$, $*\theta_{c}^{t-1}$ to other clients

8

9 for  _$r\leftarrow 1$ to $R$_ do

10 Select available clients $C_{r}\subseteq C$

11 for  _$c\in C_{r}$_ do

$\hat{\theta}_{c}^{t},\;\hat{F}_{c}^{t}\leftarrow TrainLocal(\theta_{G},E)$

$\triangleright$ minimize e.q. 4 on device

12 Send $\hat{\theta}_{c}^{t},\;\hat{F}_{c}^{t}$ to other clients

13 $\theta_{G}\leftarrow\sum_{\begin{subarray}{c}c\in
C_{r}\end{subarray}}\frac{1}{|C_{r}|}\hat{\theta}_{c,t}$

14

return _$\theta_{G}$_

Algorithm 1 Elastic Transfer

Figure 3: Experimental setup for collection of SocNav1 data [31]

.

##  IV Evaluation

To demonstrate the utility of FCL4SR in socially aware robotics scenarios, we
conduct a case study using a Socially-Aware Navigation (SAN) task. In SAN
tasks, robots learn to position themselves appropriately around humans based
on social conventions. SAN tasks provide a strong application setting for
FCL4SR because humans have differing preferences regarding robot navigation
patterns [37]. These preferences can vary across users, as well as within
users depending on the social context [38]. FCL4SR provides a method for
continuously adapting robots’ understanding of user navigation preferences
over multiple distributed interactions without requiring centralized storage
of user data.

###  IV-A Dataset

We use the SocNav1 benchmark for evaluating socially-aware robot navigation
models [31]. This dataset contains 9,280 annotated examples gathered from 12
participants using a desktop-based tool. Participants were presented a static
social scenario including humans, robots and objects, and were instructed to
“assess the robot’s behavior in terms of the disturbance caused to humans” on
a scale from 0 to 100 (Figure 3). See [31] for additional task details.

###  IV-B Centralized Baseline

The SocNav1 authors developed a graph neural network (GNN) architecture for
learning SocNav1 scenes [39]. We replicated their GNN comparison to determine
suitable hyperparameters and establish a centralized learning baseline
assuming standard train/validation/test partition on a server. We vary the
number of network layers, hidden layer size, weight decay ($\gamma$), and
mini-batch size. A Graph Convolutional Network implemented via Deep Graph
Library (GCN+DGL) with 8 layers and 32 hidden units per layer,
$\alpha=5e^{-3}$, $\gamma=1e^{-4}$, and a mini-batch size of 32 performed the
best (test MSE=.0296). We use this configuration in all following experiments.

###  IV-C FCL4SR Experimental Setup

We construct an FCRL scenario with three clients and four tasks (matching Fig.
1) using the SocNav1 dataset. Specifically, we assign annotations from each of
the 12 SocNav1 participants to a distinct client-task dataset such that each
robot personalizes to the spacial navigation preferences of a new participant
during each task. We split each data into a 70/15/15 train, validation, test
split (Table I).

Because SocNav1 participants demonstrated inter-rater variability on a
commonly-rated subset of scenes [31], this scheme mirrors inter-subject
variability present in real-world human-robot interaction scenarios. We
construct a scenario with three clients and four tasks (as opposed to other
possible factorizations of 12 client-task datasets such as 2 clients, 6 tasks
etc.) because it provides a reasonable trade-off between requiring long-term
retention of knowledge over four users (i.e., tasks) and parameter sharing
among three clients. We augment data via the same procedure employed by
SocNav1 authors, which involves mirroring social scenarios over the vertical
axis [31]. All experiments and results reported in this paper are on the held
out test set with data augmentation.

|  | Task 1 | Task 2 | Task 3 | Task 4
---|---|---|---|---|---
| Client 1 | 111, 24, 24 | 781, 168, 168 | 417, 90, 90 | 86, 19, 19
Client 2 | 86, 18, 19 | 365, 78, 79 | 1750, 375, 375 | 431, 92, 93
Client 3 | 1750, 375, 375 | 103, 22, 23 | 46, 10, 10 | 565, 121, 122
\+ Aug. | Client 1 | 222, 24, 24 | 1562, 168, 168 | 834, 90, 90 | 172, 19, 19
Client 2 | 172, 18, 19 | 730, 78, 79 | 3500, 375, 375 | 862, 92, 93
Client 3 | 3500, 375, 375 | 206, 22, 23 | 92, 10, 10 | 1130, 121, 122

TABLE I: Client-task train, validation, test dataset sizes with and without
data augmentation.

####  IV-C1 Evaluation Measures

We compute AMSE, BWT, and FWT as specified in [24]. We compute the training-
test performance matrix $P$, where rows denote training datasets $Tr^{(i)}$
available to the algorithm during each task, and columns denote the test
datasets $Tr^{(j)}$. We compute AMSE over tasks as
$\frac{1}{T}\sum_{i=1}^{T}P_{i,i}$.We compute BWT by measuring the performance
difference between the previous task and the current task

$BWT=\frac{\sum_{i=2}^{N}\sum_{j=1}^{i-1}\left(P_{i,j}-P_{j,j}\right)}{\frac{N(N-1)}{2}}.$
(5)

Because we measure BWT in the context of MSE, negative BWT indicates
performance is improving over tasks. We compute FWT via the mean of the future
unseen tasks such that

$FWT=\frac{\sum_{i=1}^{j-1}\sum_{j=1}^{N}P_{i,j}}{\frac{N(N-1)}{2}}.$ (6)

Figure 4: AMSE as a function of augmented training dataset size.

####  IV-C2 Baseline Approaches

We compare Elastic Transfer with the set of approaches evaluated by [6] in an
image recognition setting. Single Task Learning (STL) methods train a separate
model on each local client-task dataset. Local-CL methods (reported in
experiments as Local-$\langle$CL method$\rangle$) train a separate model on
each client using CL and do not share parameters via FL. FL-CL approaches
(reported in experiments as $\langle$FL method$\rangle$-$\langle$CL
method$\rangle$) enable clients to communicate using FL as they solve their
local CL problem. We consider both architectural and regularization based
strategies for FL (FedProx [13] and FedCurv [7]) and CL (APD [11], EWC [8],
and L2-transfer), in addition to the FedWeIT algorithm proposed in [6].

Figure 5: Breakdown of client-task partition MSE. Dataset size shown in
rectangles.

###  IV-D Experiments and Results

Our first experiment compares Elastic Transfer with FedWeIT and other
baselines evaluated by [6]. We perform $R=25$ rounds of local training and FL
parameter aggregation per task, with $E=5$ local epochs of SGD per FL round.
To simulate intermittent robot connectivity, we randomly drop one of the three
clients from training each round.

Table II shows results of our first experiment. Elastic Transfer out-performs
other methods including FedWeIT, and attains an AMSE that is comparable to our
centralized benchmark MSE (0.0363 vs. 0.0296; 22.6 % increase). We report
Elastic Transfer performance over different hyperparamter settings in Table
III. Further, other FL-CL approaches out-perform Local-CL approaches that do
not perform FL communication. We measure how much task-specific performance
deteriorates after task-specific data becomes unavailable during training on
later tasks using BWT [24]. Elastic Transfer also out-performs baselines in
this approach, indicating it suffers from less inter-client interference
during training. We also measure adaptation time via FWT. Elastic Transfer
demonstrates the best generalization to unseen participants as measured by
FWT, followed by other FL-CL approaches. To assess one aspect of the overhead
criteria, we report how many static parameters (used for regularization) and
trainable parameters (learned via optimization) are required by each method.
Compared to FedWeIT, Elastic Transfer requires fewer trainable parameters, but
more static parameters.

Approaches that require learning task specific parameters perform poorly in
our evaluation (e.g., FedWeIT, APD). We hypothesize that this is because
learning a task-specific parameter decomposition is challenging in low-data
settings. Therefore, we vary the size of the training data available in local
client-task partitions. If a method demonstrates low AMSE when little data is
available, this suggests that the approach provides improved adaptation
quality by learning more efficiently. In this experiment, we set $R=60$,
$E=1$, with no clients dropped each round. Figure 4 shows AMSE as a function
of dataset size for several approaches. As training data increases, the
relative AMSE gain offered by Elastic Transfer increases. This suggests that
Elastic Transfer is leveraging knowledge sharing among client-task datasets
effectively as data increases. However, all approaches perform similarly in
the lowest data condition (20%).

FL-CL methods such as Elastic Transfer may perform well because they more
effectively share knowledge among client-task datasets. Figure 5 demonstrates
this in practice by comparing client-task dataset MSE among a FL-CL method
(Elastic Transfer) and a Local-CL method (Local-EWC). Local-EWC performs
poorly on small client-task datasets, particularly at the beginning of
training (e.g., client 2, task 1). This is because Local-CL methods rely on
data available to the robot, and do not share knowledge with other robots.
Conversely, FedAvg-EWC performs better on these client-task datasets because
it leverages knowledge from other clients (e.g., client 3, task 1) to
accelerate learning. This shows that FL-CL methods better-facilitate knowledge
sharing and thus improve adaptation time in low-data settings.

| Evaluation Metrics | Parameters
---|---|---
AMSE |  BWT |  FWT |  Static |  Trainable
Local |  |  |  |  |
STL |  0.0735 |  0.0332 |  0.5388 |  0 |  30,724
Local-CL |  |  |  |  |
Local-EWC |  0.0482 |  0.0272 |  0.0641 |  61,448 |  7,681
Local-APD |  0.0717 |  0.1041 |  0.1766 |  0 |  39,176
FL-CL |  |  |  |  |
FedAvg-SGD |  0.0391 |  0.0088 |  0.0369 |  0 |  7,681
FedProx-SGD |  0.0408 |  0.0093 |  0.0371 |  7,681 |  7,681
FedAvg-EWC |  0.0386 |  0.0128 |  0.0373 |  61,448 |  7,681
FedProx-EWC |  0.0422 |  0.0124 |  0.0470 |  69,129 |  7,681
FedAvg-APD |  0.0927 |  0.2378 |  0.6182 |  0 |  39,176
FedProx-APD |  0.0665 |  0.1393 |  0.3187 |  7,424 |  39,176
FedWeIT |  0.0886 |  0.1597 |  0.5603 |  89,100 |  39,272
Elastic Transfer |  0.0363 |  0.0083 |  0.0352 |  115,215 |  7,681
TABLE II: Results comparing Elastic Transfer with previously-proposed
approaches in our FCRL benchmark. All results computed on held-out test data.
Table shows Average MSE (AMSE), Backwards Transfer (BWT), Forwards Transfer
(FWT).

Algorithm | Evaluation Metrics | Parameters
---|---|---
AMSE | BWT | FWT | S | T
$\lambda_{1}$ | $\lambda_{2}$ | $\lambda_{3}$ |  |  |  |  |
$5e^{-1}$ |  | $5e^{-1}$ | 0.0408 | 0.0120 | 0.0380 | 115,215 | 7,681
$5e^{-1}$ | $5e^{-1}$ | 0.0389 | 0.0108 | 0.0407
| $5e^{-1}$ | 0.0383 | 0.0107 | 0.0383
| $5e^{-2}$ | 0.0392 | 0.0114 | 0.0391
$5e^{-2}$ | $5e^{-2}$ | 0.0385 | 0.0115 | 0.0390
| $5e^{-2}$ | 0.0369 | 0.0124 | 0.0355
| $0$ | 0.0389 | 0.0103 | 0.0395
$0$ | $0$ | 0.0397 | 0.0113 | 0.0450
| $0$ | 0.0391 | 0.0085 | 0.0362
$5e^{-2}$ |  | $5e^{-1}$ | 0.0396 | 0.0062 | 0.0400
$5e^{-1}$ | $5e^{-2}$ | 0.0402 | 0.0115 | 0.0381
| $0$ | 0.0376 | 0.0107 | 0.0375
| $5e^{-1}$ | 0.0400 | 0.0111 | 0.0395
$5e^{-2}$ | $5e^{-2}$ | 0.0390 | 0.0080 | 0.0392
| $0$ | 0.0390 | 0.0091 | 0.0391
| $5e^{-1}$ | 0.0384 | 0.0115 | 0.0390
$0$ | $5e^{-2}$ | 0.0391 | 0.0086 | 0.0382
| $0$ | 0.0386 | 0.0128 | 0.0373
0 |  | $5e^{-1}$ | 0.0403 | 0.0112 | 0.0412
$5e^{-1}$ | $5e^{-2}$ | 0.0388 | 0.0122 | 0.0396
| $0$ | 0.0363 | 0.0083 | 0.0352
| $5e^{-1}$ | 0.0392 | 0.0118 | 0.0380
$5e^{-2}$ | $5e^{-2}$ | 0.0374 | 0.0108 | 0.0384
| $0$ | 0.0374 | 0.0101 | 0.0358
| $5e^{-1}$ | 0.0378 | 0.0092 | 0.0382
$0$ | $5e^{-2}$ | 0.0404 | 0.0137 | 0.0411
| $0$ | 0.0388 | 0.0122 | 0.0355

TABLE III: Results of full set of Elastic Transfer hyperparameter
combinations.

##  V Conclusion and Future Work

In this paper, we presented a new decentralized learning framework for
socially aware robotics (FCL4SR), by capitalising on two machine learning
approaches, Federated Learning and Continual Learning, to capture interaction
dynamics distributed physically across robots and temporally across repeated
robot encounters. We outlined four criteria (adaptation quality, adaptation
time, knowledge sharing, and model overhead) that should be balanced within
the FCL4SR framework. We also developed a new regularization-based FCL method
– Elastic Transfer, and evaluated it on a proof-of-concept socially aware
navigation dataset, demonstrating that Elastic Transfer improved adaptation
quality, adaptation time, and knowledge sharing.

Our FCL4SR framework (i.e., Elastic Transfer) only requires temporary local
storage of end-user data during model training, but requires no permanent
storage of end-user data on a server. Therefore, our approach provides an
important first step towards end-user privacy protection compared to current
centralized learning practices involving permanent data storage. However, our
approach does not provide a formal privacy guarantee [40]. Future work could
investigate how our framework might be combined with other privacy-protecting
methods, such as differential privacy. We also see opportunities for future
work to further reduce communication overhead by leveraging similar
reformulations introduced by FedCurv and Online-EWC. We hope that this work
will serve as a stepping stone for improving the personalization capabilities
of socially aware robots while also paving the way towards privacy
preservation in human robot interactions.

## ACKNOWLEDGMENTS

Funding: This work is partially supported by the EPSRC/UKRI under grant ref.
EP/R030782/1 (ARoEQ). Open Access: For open access purposes, the authors have
applied a Creative Commons Attribution (CC BY) licence to any Author Accepted
Manuscript version arising.

## References

  * [1] H. Hodson, “The first family robot,” _New Scientist_ , vol. 223, no. 2978, p. 21, 2014.
  * [2] A. K. Pandey and R. Gelin, “A mass-produced sociable humanoid robot: Pepper: The first machine of its kind,” _IEEE Robotics & Automation Magazine_, vol. 25, no. 3, pp. 40–48, 2018.
  * [3] B. Irfan, A. Ramachandran, S. Spaulding, G. I. Parisi, and H. Gunes, “Lifelong learning and personalization in long-term human-robot interaction (LEAP-HRI),” in _ACM/IEEE International Conference on Human-Robot Interaction, HRI 2022, Sapporo, Hokkaido, Japan, March 7 - 10, 2022_ , 2022, pp. 1261–1264.
  * [4] I. Leite, C. Martinho, and A. Paiva, “Social robots for long-term interaction: a survey,” _International Journal of Social Robotics_ , vol. 5, no. 2, pp. 291–308, 2013.
  * [5] D. J. Butler, J. Huang, F. Roesner, and M. Cakmak, “The privacy-utility tradeoff for remotely teleoperated robots,” in _Proc. of HRI_ , 2015, pp. 27–34.
  * [6] J. Yoon, W. Jeong, G. Lee, E. Yang, and S. J. Hwang, “Federated continual learning with weighted inter-client transfer,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 073–12 086.
  * [7] N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor-Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,” _arXiv preprint arXiv:1910.07796_ , 2019.
  * [8] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska _et al._ , “Overcoming catastrophic forgetting in neural networks,” _PNAS_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [9] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual lifelong learning with neural networks: A review,” _Neural Networks_ , vol. 113, pp. 54–71, 2019.
  * [10] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 12, pp. 2935–2947, 2017\.
  * [11] J. Yoon, S. Kim, E. Yang, and S. J. Hwang, “Scalable and order-robust continual learning with additive parameter decomposition,” _arXiv preprint arXiv:1902.09432_ , 2019.
  * [12] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _AISTATS_. PMLR, 2017, pp. 1273–1282.
  * [13] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” _arXiv preprint arXiv:1812.06127_ , 2018.
  * [14] O. Rudovic, N. Tobis, S. Kaltwang, B. Schuller, D. Rueckert, J. F. Cohn, and R. W. Picard, “Personalized federated deep learning for pain estimation from face images,” 2021.
  * [15] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three approaches for personalization with applications to federated learning,” 2020.
  * [16] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” 2021.
  * [17] W. Huang, M. Ye, and B. Du, “Learn from others and be yourself in heterogeneous federated learning,” in _Proc. of CVPR_ , 2022, pp. 10 143–10 153.
  * [18] M. F. Criado, F. E. Casado, R. Iglesias, C. V. Regueiro, and S. Barro, “Non-iid data and continual learning processes in federated learning: A long road ahead,” _Information Fusion_ , vol. 88, pp. 263–280, 2022.
  * [19] G. Wei and X. Li, “Knowledge lock: Overcoming catastrophic forgetting in federated learning,” in _in Proc. of PAKDD 2022_. Springer, 2022, pp. 601–612.
  * [20] Z. Wang, Y. Zhang, X. Xu, Z. Fu, H. Yang, and W. Du, “Federated probability memory recall for federated continual learning,” _Information Sciences_ , 2023.
  * [21] F. Estévez Casado, D. Lema Pais, M. Fernández Criado, R. Iglesias Rodríguez, C. Vázquez Regueiro, and S. Barro Ameneiro, “Concept drift detection and adaptation for federated and continual learning,” 2021.
  * [22] Y. Chaudhary, P. Rai, M. Schubert, H. Schütze, and P. Gupta, “Federated continual learning for text classification via selective inter-client transfer,” _arXiv preprint arXiv:2210.06101_ , 2022.
  * [23] W. Lu, J. Wang, Y. Chen, X. Qin, R. Xu, D. Dimitriadis, and T. Qin, “Personalized federated learning with adaptive batchnorm for healthcare,” _IEEE Transactions on Big Data_ , 2022.
  * [24] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” _Information Fusion_ , vol. 58, pp. 52–68, 2020.
  * [25] Y. Xianjia, J. P. Queralta, J. Heikkonen, and T. Westerlund, “Federated learning in robotic and autonomous systems,” _arXiv preprint arXiv:2104.10141_ , 2021.
  * [26] B. Liu, L. Wang, and M. Liu, “Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 4555–4562, 2019.
  * [27] A. Logacjov, M. Kerzel, and S. Wermter, “Learning then, learning now, and every second in between: lifelong learning with a simulated humanoid robot,” _Frontiers in Neurorobotics_ , vol. 15, p. 669534, 2021.
  * [28] N. Churamani, S. Kalkan, and H. Gunes, “Continual learning for affective robotics: Why, what and how?” _Age_ , vol. 30, p. 40, 2020.
  * [29] J. Tjomsland, S. Kalkan, and H. Gunes, “Mind your manners! A dataset and a continual learning approach for assessing social appropriateness of robot actions,” _Frontiers Robotics AI_ , vol. 9, p. 669420, 2022.
  * [30] N. Churamani, M. Axelsson, A. Caldır, and H. Gunes, “Continual learning for affective robotics: A proof of concept for wellbeing,” in _in Proc. of ACIIW_. IEEE, 2022, pp. 1–8.
  * [31] L. J. Manso, P. Nuñez, L. V. Calderita, D. R. Faria, and P. Bachiller, “Socnav1: A dataset to benchmark and learn social navigation conventions,” _Data_ , vol. 5, no. 1, p. 7, 2020.
  * [32] P. Alves-Oliveira, P. Sequeira, F. S. Melo, G. Castellano, and A. Paiva, “Empathic robot for group learning: A field study,” _ACM Transactions on Human-Robot Interaction (THRI)_ , vol. 8, no. 1, pp. 1–34, 2019.
  * [33] O. Rudovic, J. Lee, M. Dai, B. Schuller, and R. W. Picard, “Personalized machine learning for robot perception of affect and engagement in autism therapy,” _Science Robotics_ , vol. 3, no. 19, p. eaao6760, 2018.
  * [34] D. J. MacKay, “A practical bayesian framework for backpropagation networks,” _Neural computation_ , vol. 4, no. 3, pp. 448–472, 1992.
  * [35] F. Huszár, “On quadratic penalties in elastic weight consolidation,” _arXiv preprint arXiv:1712.03847_ , 2017.
  * [36] J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, “Progress & compress: A scalable framework for continual learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 4528–4537.
  * [37] J. Rios-Martinez, A. Spalanzani, and C. Laugier, “From proxemics theory to socially-aware navigation: A survey,” _International Journal of Social Robotics_ , vol. 7, no. 2, pp. 137–153, 2015.
  * [38] D. S. Syrdal, K. L. Koay, M. L. Walters, and K. Dautenhahn, “A personalized robot companion? the role of individual differences on spatial preferences in hri scenarios,” in _in Proc. of RO-MAN_. IEEE, 2007, pp. 1143–1148.
  * [39] L. J. Manso, R. R. Jorvekar, D. R. Faria, P. Bustos, and P. Bachiller, “Graph neural networks for human-aware social navigation,” in _Workshop of Physical Agents_. Springer, 2020, pp. 167–179.
  * [40] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.