# EdgeSync: Faster Edge-model Updating via Adaptive Continuous Learning for
Video Data Drift

Peng Zhao, Runchu Dong, Guiqin Wang, and Cong Zhao Peng Zhao, Runchu Dong, and
Guiqing Wang are with School of Computer Science and Technology, and with the
National Engineering Laboratory for Big Data Analytics (NEL-BDA), Xi’an
Jiaotong University, China. e-mail<EMAIL_ADDRESS>Zhao is with
School of Mathematics and Statistics,and with the National Engineering
Laboratory for Big Data Analytics (NEL-BDA), Xi’an Jiaotong University, China.

###### Abstract

Real-time video analytics systems typically place models with fewer weights on
edge devices to reduce latency. The distribution of video content features may
change over time for various reasons (i.e. light and weather change) , leading
to accuracy degradation of existing models, to solve this problem, recent work
proposes a framework that uses a remote server to continually train and adapt
the lightweight model at edge with the help of complex model. However,
existing analytics approaches leave two challenges untouched: firstly,
retraining task is compute-intensive, resulting in large model update delays;
secondly, new model may not fit well enough with the data distribution of the
current video stream. To address these challenges, in this paper, we present
EdgeSync, EdgeSync filters the samples by considering both timeliness and
inference results to make training samples more relevant to the current video
content as well as reduce the update delay, to improve the quality of
training, EdgeSync also designs a training management module that can
efficiently adjusts the model training time and training order on the runtime.
By evaluating real datasets with complex scenes, our method improves about
3.4% compared to existing methods and about 10% compared to traditional means.

###### Index Terms:

Edge Intelligence, Continuous Learning, Video Data Drift, Model Updating.

##  I Introduction

Real-time video analytics has significant potential for applications in
various fields, such as augmented reality, video surveillance, and traffic
detection[1]. Owing to recent advancements in deep neural networks (DNNs), the
performance of video analysis has been greatly enhanced, even surpassing human
accuracy in many scenarios[17, 8, 44]. Although advanced DNNs can generate
accurate inferences for various challenging analytical tasks, the complexity
of the network structures and the large number of parameters increase the
computational burden[13]. This makes it challenging for these models to
perform real-time analytics on resource-constrained devices, such as mobile
terminals and edge devices.

To meet the demands of real-time analysis on edge devices, deep neural
networks with fewer weights and shallower architectures are typically
deployed. However, the distribution of video content in real scenarios often
changes over time (e.g., variations in lighting, crowd density, and weather
conditions), making lightweight models vulnerable to data drift[34][36].
Consequently, it is challenging to maintain the desired accuracy with a model
obtained by single offline training. Continuous learning has recently been
validated as a feasible solution to enhance the adaptability of edge models.
For instance, Mullapudi et al. in [28] demonstrated the effectiveness of this
approach by proposing an online model distillation technique to train a low-
cost student model on live video streaming. Ekya[5] refined this framework by
performing the data annotation in the cloud and jointly adjusting real-time
inference and continuous learning based model retraining on edge servers to
maximize the overall accuracy through a scheduler.

(a) Scene Change Example

(b) Accuracy

(c) Class Distribution

Figure 1: Changes of class distributions and accuracy when a car camera enters
downtown of ideal fast retraining and normal retraining.

Though several previous work have recently been conducted to enhance the
accuracy of edge models through online continuous learning, they still suffer
from the challenge of large update delay of updating new edge models, result
in the decrease of overall inference accuracy. First, lightweight models at
the edge continue to use outdated parameters for inference while retraining
new ones in the cloud. Some methods either upload all samples to the cloud for
labeling and retraining or perform additional operations such as
hyperparameter selection in the cloud, which consumes considerable time and
increases the update latency to data drift, as shown in Fig. 1. Second,
current methods mainly set a static sampling rate and a fixed time interval
for model updating, since video streams vary in different degrees over time,
they do not take into account the degree of influence of different samples and
the upper limit of the model’s own accuracy in the current time period, thus
affecting the quality of the model’s retraining and decreasing the
generalization of the retrained model.

Although several previous work has maintained the accuracy of edge devices,
several challenges remain. First, the update delay of new edge models is
large. Lightweight models at the edge continue to use outdated parameters for
inference while retraining new ones. Current methods either upload all samples
to the cloud for labeling and retraining or perform additional operations such
as hyperparameter selection in the cloud, which consumes considerable time and
increases the response latency to data drift, as shown in Figure 1. Second,
current methods primarily use a static sampling rate and a fixed time interval
for model updates. Since video streams vary to different extents over time,
these methods do not consider the varying impact of different samples or the
model’s upper accuracy limit in the current period. This oversight affects the
quality of model retraining and decreases the generalization of the retrained
model.

In order to solve these problems, in this paper, we propose EdgeSync, an
Faster Edge-model Updating approach that automatically and continually adapts
models based on the scene. It consists of two modules: a sample filtering
module and a model retraining management module. Specifically, to reduce the
network uploading bandwidth as well as improve the quality of training
samples, the sample filtering module filters certain samples of the current
video stream in real time, taking into account the model’s ability to extract
features from samples and the importance of timeliness. To further ensure the
quality of model training while accelerating its update speed, we propose the
model retraining management module, which efficiently makes accurate training
handover decisions with low complexity. Benefiting from these two modules,
EdgeSync can achieve fast model updates in real scenes, which helps improve
the accuracy of single camera inference and allows the cloud to carry out more
camera model update tasks.

The major contributions of this paper are summarized as follows:

  * •

First, we propose a fast method for filtering video streaming samples, which
combines the results of timeliness and adaptability to filter out unnecessary
samples. Benefiting from the module, We can flexibly adjust the number of
samples according to network conditions while improving the quality of samples
used for training in the cloud.

  * •

Next, we propose a re-training manager to adjust the training order and
training time by using the labeled and trained computed features, to further
improve the model update speed, we use offline and online profiling to
accelerate hyper-parameter selection procedure.

  * •

Third, we conduct extensive experiments to evaluate the performance of
EdgeSync in real-world scenarios. Evaluation results demonstrate that EdgeSync
can reduce network bandwidth and speed up model update frequencies. It also
outperforms the baseline schemes in terms of overall accuracy.

The structure of the paper is outlined as follows: In Section 2, we review
related work. Section 3 introduces the system architecture and details the
sample filtering module and retraining management module. Section 4 presents
the evaluation results. Finally, concluding remarks are provided in Section 5.

##  II Related Work

###  II-A Video Analytics Systems

Real-time video analytics uses computer vision algorithms to automatically
analyze and understand the content of video streams generated by one or more
cameras, thus accomplishing complex tasks such as target recognition and
anomaly detection while the video streams are being recorded and transmitted.
To balance accuracy and speed, Chameleon[20] designs a controller to
dynamically select parameters in the video analysis system; Wang et al.[39]
addresses the joint configuration tuning problem for video parameters and
serverless computation resources, they propose an algorithm utilizing Markov
approximation to select optimal configurations for video streams by
considering the accuracy-cost trade-off. PacketGame[42] selectively filters
packets through a specially designed neural network before running the video
decoder in order to increase the number of videos to be processed in parallel;
Ekya[5] performs simultaneous training and inference at the edge server, and
designs a thief scheduler to select the appropriate parameters to balance
training and inference. AMS[22] and JIT[28] create specialized lightweight
DNNs to maintain accuracy under a specific scene, the challenge is that as the
video scene changes, the system must dynamically create new DNNs to adapt to
the new video content, our work is similar with them and focuses on an area
that has not been emphasized: the update quality and speed of specialized
lightweight DNNs, we hope to accelerate the model’s ability to adapt to the
video and improve the overall accuracy by improving the efficiency of model
updating.

###  II-B Continuous Learning

Continuous learning (also called incremental learning) utilizes new data
acquired to continuously adapt the current model to new tasks while remember
basic concepts previously learned[11]. Recent works have addressed
catastrophic forgetting with longer task sequences, [38] employ a meta
optimizer that dynamically adapts the learning rate to prevent forgetting
throughout the learning process in SDML. GPM[31] identifies crucial gradient
subspaces related to previous tasks and mitigates catastrophic forgetting by
implementing gradient steps orthogonal to these subspaces when acquiring
knowledge in a new task. Some of them focus on limited training samples,
MAML[14] tunes the model’s parameters through training via gradient descent on
a new task. DeepBDC[40] acquires image representations by assessing the
disparity between joint characteristic functions of embedded features and the
product of their marginals.

Existing efforts toward learning new tasks continually without forgetting the
past tasks mainly classified into three categories. The first group dedicate
different subsets of network parameters to each task[30][41], the second group
attempt to overcome forgetting in fixed capacity model through structural
regularization which penalizes major changes in the parameters that were
important for the previous tasks[23][18]. The third group of method mitigate
forgetting by either storing a subset of (raw) examples from the past tasks in
the memory for rehearsal[27][2] or synthesizing old data from generative
models to perform pseudo rehearsal[33]. Our method shares the idea with
continuous learning, but uses a different approach and framework, we let
lightweight DNNs focus more on recent video frames based on spatial locality
relation and train it on the fly.

###  II-C Unsupervised Adaptation Methods

Figure 2: Overall architecture of EdgeSync

Unsupervised adaptation methods aim to improve accuracy to tackle potential
distribution shifts between training and testing data. Early unsupervised
domain adaptation methods fine-tune the model rely on both source domain and
target domain[26][21]. Some work attempts to solve this problem based only on
target domain. Sahoo et al.[9] propose a method which only requires unlabeled
test data, SHOT[24] combined entropy minimization with pseudo labeling,
TTT[35] turns a single unlabeled test sample into a self-supervised learning
problem, then update the model parameters before making a prediction. To
reduce the training time, Tent[7] optimize the model for confidence as
measured by test entropy minimization, LAME[6] adapts the model’s output
rather than its parameters by discouraging deviations from the prediction to
find the optimal set of latent assignments which mitigating the effect of
hyperparameters on performance.

In addition, a few work has been proposed tailored for video data. Zeng et
al.[43] propose a new test-time learning scheme that leverages motion cues in
videos to enhance the generalization capability of video classification
models. Lin[25] propose a test-time adaptation method for video action
recognition models, it aligns the training statistics with the online
estimates of target statistics, then enforce prediction consistency among
temporally augmented views of a video sample. Our approach leverage a complex
model in the cloud server to do supervised training with its generated labels,
which is which is more suitable in this framework.

##  III Method

In this section, we present the proposed method EdgeSync for real-time video
analytics. We first introduce the overall system architecture of EdgeSync.
Following that, we present the details of our approach, including the sample
filtering module and the adaptive model training module.

###  III-A System Architecture

Fig.2 illustrates the overall architecture of EdgeSync, an edge-cloud
collaborative system for real-time video analytics. Basically, this system
consists of multiple terminal cameras, several edge servers, and a centralized
cloud server. Each edge device is primarily responsible for locally analyzing
video streams from one or more cameras utilizing a lightweight model. In
contrast, the cloud server is responsible for model management across all
edges, including model training and updates. Specifically, the fundamental
modules within both the edge and cloud servers can be summarized as follows.

Edge server: Each edge deploys a lightweight model with a shallow architecture
and few parameters to perform the rapid local inference of real-time video
streaming. To maintain the inference accuracy and address the challenge of
data drift, the inference results and corresponding video frames are then
buffered in a local cache for online continuous learning. Since sending all
frames directly to the cloud will introduce significant network overhead, the
edge side must selectively upload samples. In this study, we proposed a sample
filtering module, which is used to score the quality of the samples in the
current period of time and dynamically select high-quality samples for
training the new model at the cloud. In particular, this module scores the
quality of the current samples from two aspects. First, it uses the output
confidence distribution derived from the current model deployed at the edge,
calculating the entropy of the distribution. Higher entropy produces more
update information for the current model. Then, it takes into account the
temporal distribution. The distance from the current time point represents the
similarity with the current sample period. The edge then sorts these samples
based on the above two aspects and finally selects the top several samples to
transfer their results and frames to the cloud for retraining. On the other
hand, each edge also receives new model from the cloud server for updating.

Cloud server: The cloud utilizes frames sent from the edges to dynamically
train new edge models and then dispatches the updates to the edges. Upon
receiving the data samples from the edge, a training management module is
designed in the cloud to perform two sequential processes: labeling and
retraining. In particular, to label new data samples, the cloud employs a
complex heavyweight model, leveraging its highly accuracy and abundant
resources.This complex model generates highly accurate predicted values, which
are treated as ground truth labels to supervise the training of the
lightweight edge models. The labeled data are then stored in a storage buffer.
Before retraining, the cloud evaluates the historical accuracy of each edge to
prioritize which one to train. During the training phase, the cloud uses pre-
selected hyperparameters and employs an early-stopping mechanism to accelerate
the training process. Finally, the updated model is sent back to the
corresponding edge. Notably, only the updated parameters, rather than the
entire edge model, are replaced. Additionally, the model and its parameters
are stored in GPU memory to further accelerate model updating and reduce
delays in context switching between the GPU and memory.

###  III-B Sample Filtering Module

In practice, it is unnecessary and time-consuming for the edge to process the
inputs simultaneously. This is because video frames can provide abundant
information, yet not all necessarily contribute to improving model retraining
performance. To improve the quality and accelerate the sending procedure, we
propose a sample filtering module to selectively upload samples at edge.

In the following, we present the detailed definitions. There are $k$ edges,
each of them contains a video stream. For each video stream $v\in V$ in an
update window T, we decide: (1) the model $f$ deployed in each edge, whose
parameter is $\theta$;(2) the frame $x_{i}$ in each window($i$ represents
sequence in current window); (2) the inference result $y_{i}$ for frame
$x_{i}$ using current model $f$; and (4) cache $Y$ to store inference result
and related frame, cache $M$ to store frame quality. It’s worth noting that
the size of update window $T$ changes every time.

In the filtering module, we address two key aspects. Firstly, we evaluate the
adaptability of the edge device model to the current sample. If the current
model has accurately predicted the sample, retraining is unnecessary.
Conversely, samples that the current model performs poorly on necessitate
retraining. Secondly, we consider the timeliness of the current sample. Given
our focus on enhancing lightweight model accuracy through adaptation to local
frames, it is vital to select samples that reflect the current video stream
distribution. The filter must prioritize samples closer to the current
timestamp while ensuring an adequate amount of retraining data.

For the adaptability score $E\left(x;\theta\right)$, we do not use the model
output confidence because modern neural networks are poorly calibrated [15],
instead we use the entropy of the model outputs as a basis for adaptability
judgment $E$, as it has achieved good results in other fields:

$E\left(x;\theta\right)=-f_{\theta}(y|x)\cdot\log{f_{\theta}(y|x)}$ (1)

Where x denotes the current sample, y denotes the result predicted by model f
in current parameter $\theta$. The higher the adaptability score, The lower
the model’s certainty for the current sample. For the timeliness score $I(x)$,
we use the following formula to represent the timeliness importance:

$I\left(i\right)=1/(1+\exp(-i/T))$ (2)

where $T$ denotes the size of current window, and $i$ denotes the relative
index of the arrival of sample $x$ to current timestamp. The index of current
sample is initialized as 0, so older samples tend to have bigger index $i$.

Finally we use weighted average to sum up two scores:

$Q\left(x,i,\alpha,\beta\right)=\alpha E\left(x;\theta\right)+\beta
I\left(i\right)$ (3)

In the formula, $\alpha$ and $\beta$ are used to balance adaptability and
timeliness.

Algorithm 1 Sample Filtering Algorithm

1:Input samples
$Y\leftarrow\\{(x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})\\}$ in cache

2:Define update window size $T$, Percentage of uploaded samples $k$, $\alpha$,
$\beta$
list $M\leftarrow,\emptyset$

3:for $(i,x,y)\in Y$ do

4: $adaptability_{i}\leftarrow E(x;\Theta)$, $\triangleright$ Calculate
adaptability score via 1

5: $timeliness_{i}\leftarrow I(i)$ $\triangleright$ Calculate timeliness score
via 2

6: $quality_{i}\leftarrow Q(x,i,\alpha,\beta)$ $\triangleright$ Get overall
score for $x$ via 3

7: append $(quality_{i},x,y,i)$ to list $M$

8:end for

9:$M\leftarrow sort(M)$ sort element in $M$ by $quality_{i}$

10:$M\leftarrow$ get top-k percentage elements from sorted list $M$ and remove
element $quality$

11:send $M$ to cloud

The pseudo code in Algorithm 1 summarizes the process of sample filtering
module. Subsequently, the inference results $y$ and corresponding video frames
$x$ are buffered in a local cache to facilitate online continuous learning.
The module assesses the quality of current samples from two perspectives.
Firstly, it computes the adaptability score using Formula 1, followed by the
calculation of the timeliness score using Formula 2. Based on these criteria,
the edge device ranks the samples and selects the top few for transmitting
their results and frames to the cloud for retraining.

###  III-C Training Manager

The lightweight model at the edge maintains its accuracy for its video stream
through continuous training, we dynamically update the model parameters and
adjust training sequence through the retraining managing module to maximize
the overall accuracy.

####  III-C1 Select a model for retraining

Different edge may have different degrees of change in video content, for
example, When monitoring road conditions, engineers will place cameras in
different locations in the city for monitoring. Due to different geographical
locations, the video streams faced by each camera have different change
patterns, Therefore, we need to make profiling on these video streams and
select a appropriate one for retraining in order to let the edge adapt this
current local distribution faster which scene change more difficultly and
frequently.

The cloud receives some samples and corresponding inference results
transmitted from the edge. We propose to build up profiling problem based on
the model inference result level rather than the sample feature level, as it
provides a robuster signal for measuring degrees of scene changes[22], as well
as avoiding additional computing overhead like some methods, which add a
feature extractor at cloud. We then choose to use history window-based
detection to handle these inference results. Specifically, in cloud, it first
sends the samples to a complex model to generate a pseudo label, then it
compare the inference results with these label to get a variables
$acc\in\\{0,1\\}$ to denote the correctness of each sample, and append them at
the end of list $W=\\{acc_{1},acc_{2},\ldots,acc_{n}\\}$ in cloud storage,
which $n$ represent the capability of the list, it is noting that each edge
will have different $W$ in cloud. Retrain managing module will used $W$ to
compare the urgency degree $d$ for edge:

$\displaystyle d$
$\displaystyle=\sum_{i=0}^{m}{\left({wa}_{0}-{wa}_{i}\right)\cdot(1/(1+e^{-i/tm})\cdot
m)}$ (4) $\displaystyle wa_{i}$ $\displaystyle=\sum_{j=l\cdot i+1}^{(l+1)\cdot
i}{acc_{j}}$ (5)

In the formula, $l$ is the length of each batch $wa$, which sums up $acc$
variables, $m$ is the number of batch $wa$ in list $W$.

There we also consider temporally important differences for window $W_{i}$ via
exponential decay weight, which is similar with the sample filtering module.
Retrain manager continuously remove old samples if the number of $acc$ exceeds
list capability.

Algorithm 2 Training Manager Algorithm

1:Input: samples and inference results received from edge:
$M_{e}=\\{(x_{1e},y_{1e},1),(x_{2e},y_{2e},2),\ldots,(x_{re},y_{re},r)\\}$,
hyperparameters $h$ obtained in offline phase, early stop threshold $k$,
complex model $f$

2:Define: list W’s size $n$, segment’s size $m$

3:for $e\in\text{edges}$ do

4: for $(x,y,i)\in M_{e}$ do

5: $label\leftarrow f(x)$

6: $acc\leftarrow label==y$

7: $e.\text{bank}\leftarrow\text{append}(e.\text{bank},(acc,i))$
$\triangleright$ append acc and index to storage

8: end for

9: if $\text{len}(e.\text{bank})>n$ then

10: remove elements in e.bank exceeds its size n

11: end if

12: divide $e.\text{bank}$ into $m$ batches

13: $d_{e}\leftarrow\text{Score}(e.\text{bank},m)$ $\triangleright$ calculate
degree of e via 4

14:end for

15:$e\leftarrow\arg\max d$ $\triangleright$ find the edge with max $d$

16:$epoch\leftarrow 0$, $max\\_epoch\leftarrow 0$

17:$start\\_time\leftarrow\text{current\\_time}()$,
$max\\_evaluation\leftarrow 0$

18:while $epoch-max\\_epoch>k$ and
$\text{current\\_time}()-start\\_time>max\\_time$ do

19: train the model of edge $e$ with hyperparameters $h$

20: $evaluation\leftarrow e.\text{eval}()$ $\triangleright$ evaluate the
performance of model of edge $e$

21: if $evaluation>max\\_evaluation$ then

22: $max\\_evaluation\leftarrow evaluation$, $max\\_epoch\leftarrow epoch$

23: end if

24: $epoch\leftarrow epoch+1$

25:end while

####  III-C2 Adapt update frequency for retraining

Ideally, if the content of the video stream changes quickly, the system should
adapt to the current scene quickly, then allocate more time on retraining
model deployed on this edge and quickly send back the trained model, in order
to reduces the update window size. Model profiling described above can help
distinguish change degree to optimize time allocation, but it cannot control
the model training time, so it is crucial to let retrain managing module
dynamically adjust the training time.

Ekya shows that different training hyperparameters have a non-negligible
impact on the accuracy, so we first need to determine the appropriate training
hyperparameters. We found that Ekya’s micro-profiler is not suitable for our
system, for two main reasons. Firstly, our system requires more frequent model
updates, leading to a higher proportion of time spent on online profiling
methods within each update cycle. Secondly, Ekya performs training and
inference simultaneously, necessitating consideration of numerous
hyperparameters, whereas we require only a subset of these hyperparameters for
training. We have determined that an offline profiling method is adequate for
achieving desirable outcomes. This approach allows us to focus exclusively on
determining retraining times for the currently selected edge model during
online operations, thereby shortening the total update life cycle.

Specifically, use we use Bayesian Hyperparameter Optimization(BHO) in offline
phase to efficiently explore training hyperparameters that achieve the best
performance. Central to BHO are the objective, prior, and acquisition
functions. The objective function evaluates the predictive performance of a
model for a given hyperparameter set. The prior function encapsulates initial
beliefs about the hyperparameter space, which is updated upon the acquisition
of new data. Specifically, We use Gaussian Process as the prior function:
$f(h_{1:k})\sim\mathcal{GP}(m(h_{1:k}),k(h_{1:k},h_{1:k}))$, where $m$ denotes
the mean function, $k$ characterizes the covariance function modeling the
relationship among input hyperparameters $h_{1}$ to $h_{k}$ and $f$ is
objective function. We adopt Expected Improvement (EI) as our chosen approach
for the acquisition function:
$\text{EI}(h)=\mathbb{E}\left[\max(f(h)-f_{\text{best}},0)\right]$ considering
its high performance in our tasks.

In the offline phase, we first initially collect a set of video data covering
various different scenarios, then use BHO to get a unique set of
hyperparameters for each video. Then we use the mean of these hyperparameters
as the starting point, $h_{0}$. Following this, we randomly sample segments
from each video and combine them to update the hyperparameters, Retrain
Managing module stops BO when the improvement is less than a threshold. It is
noting that in the process of training, we freeze the backbone and feature
extraction layers and merely tune the parameters of the last layer and
classification prediction layer, this approach is based on the understanding
that the model’s representations evolve from generic characteristics (like
patterns and color gradients) to ones that are more tailored to the specific
task (such as identifying objects) with the increase in layer depth, as
documented in [3][16].

In online phase, we use an early-stopping strategy to dynamically adjust the
number of training rounds: record the validation loss in each training, if the
k-th epoch has passed and the loss has not been reduced, then stop the
training phase. Inspire from the Just-In-Time (JIT)[28], we define a maximum
allowable training time threshold and stop training immediately upon
surpassing this threshold. This measure is designed to prevent the issue of
long training rounds encountered in certain scenarios. This not only able to
reduce the time of updating the model but also enhance the model’s
generalizability[19], reducing the misleading of the noise sample to the
model. Algorithm 2 describes overall procedure.

####  III-C3 Send retrained parameters back to edge

After finishing training, Retrain Managing module returns the model to the
relevant edge. It’s important to note that only the updated parameters, not
the entire edge model, are transmitted via the network. Moreover, to speed up
model updates and decrease context switching delays between the GPU and
memory, the model and frozen parameters are permanently stored in the edge’s
GPU memory.

##  IV experiments

###  IV-A Experiment Settings

####  IV-A1 Datasets

We evaluate EdgeSync using 22 videos collected from YouTube, with a frame rate
of 30 frames per second. Considering our objective of incorporating meaningful
data drifts into our video, we adopt videos that have a length of at about 20
minutes, then bind them with a total video length of about 7 hours. These
videos have different light intensities (day or night), clarity conditions
(sunny, rainy, snowy)and varying speeds (walking, driving), thus contain a
variety of data drift scenarios with varying degrees of difficulty. We
randomly select two frames per second to convert these videos into images and
label them with a chronological order; we distribute this data equally to each
camera when multiple cameras are present in each experiment, we do not use a
video twice to ensure these cameras have different scenes from each other.
Note that we do not use common public datasets such as Cityscape[10]. This is
because we want the video data to contain multiple complex scenes under the
condition that the timing relationship is correct, and at the same time, we
ensure that the length of a single video is not too small.

####  IV-A2 Compared Approaches

To verify the effectiveness of EdgeSync, we consider following methods:

  * •

No Adaptation: Without any adaptation, we execute the pre-trained model on the
edge device.

  * •

One-Time Adaptation[29]: We fine-tune those edge models using the first 100
seconds of samples at the beginning of each video like transfer learning,
after which no further adjustments to the parameters are made. This can be
seen as a baseline of not having continuous adaptation.

  * •

AMS[22]: The overall architecture of AMS is the same as EdgeSync, which keeps
repeating the training and updating of the edge model with the help of the
cloud, it uses a fixed epoch to train the edge model and updating a small
fraction of model parameters, in our experiments, we update the parameters of
last feature layer and final classification layer for a more fair comparison.

  * •

Ekya[5]: Ekya place both inference and retraining tasks on edge servers, it
uses a fixed period of time as window, and using a micro-profiler to determine
the resource allocation for each window. We place retraining task and the
micro-profiler in the cloud to determine training hyper-parameters for current
window with the size of 200s, for micro-profiler, we consider number of layers
to retrain, the fraction of data between retraining windows to use for
retraining, momentum in SGD and weight decay, then set fraction of training
data and Early termination epoch to 20% and 5 respectively.

####  IV-A3 Models

On edge devices, we utilize MobileNetV2 [32], which demonstrates real-time
inference speeds (30 frames-per-second). This efficiency holds true even on
edge devices with lower computational power, such as the NVIDIA Jetson Nano
and Jetson TX2. Conversely, on the cloud server, we leverage a high-
performance golden model, ResNeXt101[37], to acquire accurate ground truth
labels. Both of them are pretrained on ImageNet datasets[12].

####  IV-A4 Implementation Details

We adopt the NVIDIA GeForce RTX 2080Ti GPU as our edge device. In order to
enable the edge device to simulate the real scenario, we calculate the time
interval between two model updates, after which we reduce the actual inference
speed of the edge device in this time interval based on the FPS of the Jetson
Nano. We conduct all evaluations using a single NVIDIA Tesla V100 GPU in the
cloud. The models on both the edge and cloud sides are implemented in Python
3.8 and PyTorch 2.0 with CUDA 11.7. At the edge, we apply Algorithm 1 with
parameters $k=0.7$, signifying that 30% of samples are filtered in the current
window. Additionally, we set $\alpha=1.0$ and $\beta=1.0$. In Algorithm 2, we
configure the patience parameter to 5, while the memory bank capacity $N$ and
segment size $m$ are set to 90 and 10, respectively.

TABLE I: Comparing end-to-end performance and bandwidth consumption of different methods. Method | Accuracy | Data Upload | Data Download
---|---|---|---
No Adaptation | 62.40% | 0.0Kbps | 0.0Kbps
One-Time Adaptation | 63.96% | 42.47Kbps | 11.13Kbps
AMS | 68.87% | 254.84Kbps | 352.74Kbps
Ekya | 68.70% | 254.84Kbps | 358.4Kbps
EdgeSync | 72.09% | 178.39Kbps | 1.836Mbps
TABLE II: Time spent of different methods within a cycle. Method | Label time | Retraining time | Model profiling time | Network communication time | Total time
---|---|---|---|---|---
EdgeSync | 9.02s | 8.77s | 1ms | 3.52s | 21.311s
EdgeSync* | 9.67s | 13.72s | 0.1ms | 3.52s | 26.91s
AMS | 43.83s | 66.25s | 0.1ms | 3.52s | 113.6s
Ekya | 33.2s | 58.49s | 7.84s | 3.52s | 103.05s
Figure 3: Impact of accuracy with varying numbers of cameras.

###  IV-B Results and Discussions

####  IV-B1 Comparison to baselines

We first compare the end-to-end accuracy of EdgeSync with the baselines which
7 number of concurrent cameras replaying videos from our dataset, Table I
shows the overall accuracy of thees five methods. To assess various methods’
accuracy, we opt for a classification task, comparing edge device inference
results with labels from the teacher model for video frames. Accuracy is
measured as the ratio of correct predictions to total cases across six
categories: people, bicycles, cars, motorcycles, buses, and trucks. From the
results, it can be seen that continual training the edge model provides
significant accuracy improvement, EdgeSync performs the best among others,
with an 8% increase in accuracy compared to No Adaptation method; No
Adaptation has the lowest accuracy due to the lack of adaptation to a specific
scene; One-Time Adaptation has slightly better accuracy than no-adjustment,
but the overall accuracy is poor due to the difficulty of adapting to the
future video content; the AMS continues to utilize the samples from the most
recent period of time for training, so the edge model is able to maintain a
certain degree of accuracy, which results in 4.9% increase in accuracy
compared to 4.9% improvement over One-Time Adaptation; The accuracy of Ekya is
comparable to that of AMS, although the performance relies on different ideas.
While AMS dedicates the entirety of the cloud’s time to model training and
sample labeling, applying a uniform treatment to each edge model, Ekya adopts
a dynamic approach. It considers the impact of current window configurations
on accuracy during continuous training, strategically selecting more suitable
configurations at a minor cost to the window’s start time. This approach
becomes particularly valuable when the cloud has to manage numerous edge
tasks, as the time spent on configuration selection becomes a larger
proportion of the overall time. Consequently, this finally leads to a limited
enhancement of the overall accuracy of the edge.

(a) Verification accuracy

(b) Hyper-parameters profiling cost

Figure 4: Impact of hyper-parameters profiling in retraining windows.

####  IV-B2 Time Spent

Table II shows how long it takes, on average, to update a model using
different methods for running various components within a single update
window. EdgeSync* represents the EdgeSync method without using the retraining
manager module, the time cost of each retraining process consists of four
parts: labeling, edge model profiling, retraining and edge-cloud
communication. EdgeSync takes lowest average total time compare to other
methods, The time costs for the four components in EdgeSync are 42.32%,
41.15%, 0.01%, and 16.52%, respectively. as discussed before, reducing the
time required for model updating can prevent the retrained model from becoming
outdated, thus adapts to complex environments. If we remove the retraining
manager module in EdgeSync, it will result in a longer single update time by
26.27%, attributed to the rise in sample labeling and extended training
duration. AMS takes 43.83s for labeling and 66.25s for retraining, Since the
sample time horizon of AMS training is longer, the trained model for edge has
better generalization, but when the edge video scene changes rapidly, the
adaptability of AMS will decrease. Ekya employs a time-consuming heuristic
that evaluates each pair of candidates to identify the pair that can enhance
accuracy the most. According to the table, the profiling for every edge
consumes 7.84 seconds, constituting a significant portion of the total runtime
(26% in our experimental setup), thereby imposing a substantial additional
computational overhead, moreover, the model can only be updated once within a
window, which is not conducive to edges demanding frequent updates.

TABLE III: Ablation experimental performance of different modules of EdgeSync. Method | EdgeSync | AMS | EdgeSync/STF | EdgeSync/TF | EdgeSync/F
---|---|---|---|---|---
Accuracy | 0.7210 | 0.6880 | 0.6840 | 0.6950 | 0.7108

####  IV-B3 Impact of number of cameras

Fig 3 further demonstrates the effect of the number of cameras on the overall
accuracy. Since both No Adaptation and One-Time Adaptation methods do not have
continuous tuning during task execution, they do not need to compete for
occupying the cloud resources, the accuracies of these two methods change
slightly due to the differences between the video data in the cameras; in
contrast, the overall accuracies of the AMS, Ekya, and EdgeSync show a
decreasing trend with the increase of the number of cameras, this is because
when the number of cameras increases, each lightweight model retraining
frequency and the overall training time decreases, as a result, the model at
the edge has low adaptability to its current video content. Compared with the
other two methods, The accuracy of EdgeSync decreases more slowly than the
other two methods(1.2% less than AMS and 2.2% less than Ekya) when the number
of cameras increases, this distinction arises from the fact that, unlike Ekya
and AMS, EdgeSync autonomously decides when to halt the training process. This
adaptive approach increases the frequency of model updates by reducing
training time while ensuring adequate training duration. Additionally,
EdgeSync leverages retraining samples containing the most informative
gradients, enhancing the current lightweight model’s suitability for the
prevailing video distributions.

Figure 5: Sensitivity analysis of filter percentage parameters.

####  IV-B4 Offline hyper-parameters profiling performance.

To demonstrate the effectiveness of the offline hyper-parameters profiling
method, we compare it with the online profiling method. In this validation
experiment, online selection uses the Tree-structured Parzen Estimators
algorithm[4] to optimize the learning rate, momentum, and l2 penalty term, we
set its number of samples parameter to 5, 10, and 20, this parameter
represents the repeated times to run the search algorithm. figure 4 Compares
online dynamic profilings of training hyperparameters with offline profiling
of fixed hyperparameter validate accuracy results and their spent time. We
observe that if number of samples are too small, it performs worse than
offline profiling, a long search process is required if we want to obtain
better hyper-parameters, when number of samples parameter is 10, The accuracy
from online profiling is very small is similar with offline method(0.1% higher
than offline profiling over 31 consecutive windows), however, each online
profiling consumes an additional 100 seconds, which has a significant negative
impact on the speed of model updates.

####  IV-B5 Effectiveness of Each module.

In this experiment, we systematically incorporated each method individually to
assess if there was an improvement in accuracy. Specifically, EdgeSync/STF
denotes EdgeSync without model selection and filtering samples at the edges,
it sets a fixed update time, which is the same as the window averaging time of
the EdgeSync method in this experiment; EdgeSync/TF denotes EdgeSync without
dynamic training time and filtering samples at the edges; EdgeSync/F
represents EdgeSync without filtering samples at the edges. As shown in Table
LABEL:table_abecl EdgeSync/STF demonstrates a decrease in accuracy of
approximately 3.6% compared to EdgeSync, and a slight decrease 0.4% compared
to AMS. This observation suggests that solely diminishing the model update
time in AMS compromises accuracy, indicating that the model updates in AMS are
prone to overfitting, leading to a significant decline in accuracy when there
is a shift in the distribution of frames. EdgeSync/TF achieves around 0.5%
higher accuracy than EdgeSync/STF, which indicates the importance of profiling
edge models to select the most impacted one before retraining, EdgeSync/F
achieves around 1% higher accuracy than EdgeSync/TF, demonstrating the
effectiveness of dynamically deciding update time during runtime. EdgeSync/F
demonstrates a decrease in accuracy of approximately 1% compared to EdgeSync
since they missing The impact degree of different samples for current model.

####  IV-B6 Influence of upload percentage $k$.

figure 5 presents the overall accuracy and single training time across various
sample filtering percentages. To mitigate potential accuracy inflation arising
from increased model update frequency, we conducted experiments using a fixed
model update time of 100 seconds, akin to AMS with a filter sampler. Results
indicate that the baseline accuracy (unfiltered samples) is equivalent to that
achieved with a 30% sample filtering. However, when a small number of samples,
such as 20% of the data, is selected, performance suffers due to insufficient
training data—a necessity for regular deep learning processes. As the volume
of training data increases, model performance improves. The highest overall
accuracy is observed at a filter ratio of 0.7, with a subsequent decrease
beyond this point. This decline may be attributed to the retention of some
interfering samples, weakening the training impact, especially considering the
limited capacities of a lightweight model. This underscores that sample
filtering not only reduces the training time for individual models but also
fosters training consistency, leading to enhanced overall prediction accuracy.

##  V Conclusion

In this article, we introduce an efficient framework customized for real-time
video analysis, wherein a remote server is employed to perpetually train and
stream model updates to the edge device. Our solution aims to minimize model
update latency while enhancing the quality of continuous training.
Specifically, we devise a a sample filtering module, which intelligently
selects samples based on both reliability and temporal significance. Moreover,
we introduce a training management module that leverages the aggregated
characteristics of samples over time to identify the model most likely to
yield improvements, dynamically adjusting the model training duration in
response to real-time demands. Through a series of rigorous experimental
analyses and ablation studies, we validate the efficacy of our approach,
demonstrating notable improvements in both model performance and training
efficiency.

##  Appendix A Proof of the First Zonklar Equation

Appendix one text goes here.

## Appendix B

Appendix two text goes here.

## Acknowledgment

The authors would like to thank…

## References

  * [1] Ganesh Ananthanarayanan, Paramvir Bahl, Peter Bodík, Krishna Chintalapudi, Matthai Philipose, Lenin Ravindranath, and Sudipta Sinha.  Real-time video analytics: The killer app for edge computing.  computer, 50(10):58–67, 2017.
  * [2] Jihwan Bang, Heesu Kim, YoungJoon Yoo, Jung-Woo Ha, and Jonghyun Choi.  Rainbow memory: Continual learning with a memory of diverse samples.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8218–8227, 2021.
  * [3] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Network dissection: Quantifying interpretability of deep visual representations.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541–6549, 2017.
  * [4] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl.  Algorithms for hyper-parameter optimization.  Advances in neural information processing systems, 24, 2011.
  * [5] Romil Bhardwaj, Zhengxu Xia, Ganesh Ananthanarayanan, Junchen Jiang, Yuanchao Shu, Nikolaos Karianakis, Kevin Hsieh, Paramvir Bahl, and Ion Stoica.  Ekya: Continuous learning of video analytics models on edge compute servers.  In 19th USENIX Symposium on Networked Systems Design and Implementation (NSDI 22), pages 119–135, 2022.
  * [6] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto.  Parameter-free online test-time adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8344–8353, June 2022.
  * [7] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi.  Contrastive test-time adaptation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 295–305, 2022.
  * [8] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo.  Diffusiondet: Diffusion model for object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830–19843, 2023.
  * [9] Boris Chidlovskii, Stephane Clinchant, and Gabriela Csurka.  Domain adaptation in the absence of source domain data.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 451–460, 2016.
  * [10] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  * [11] Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  IEEE transactions on pattern analysis and machine intelligence, 44(7):3366–3385, 2021.
  * [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2020.
  * [14] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In International conference on machine learning, pages 1126–1135. PMLR, 2017.
  * [15] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger.  On calibration of modern neural networks.  In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR, 2017.
  * [16] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris.  Spottune: transfer learning through adaptive fine-tuning.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4805–4814, 2019.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin.  Learning a unified classifier incrementally via rebalancing.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 831–839, 2019.
  * [19] Ziwei Ji, Justin Li, and Matus Telgarsky.  Early-stopped neural networks are consistent.  Advances in Neural Information Processing Systems, 34:1805–1817, 2021.
  * [20] Junchen Jiang, Ganesh Ananthanarayanan, Peter Bodik, Siddhartha Sen, and Ion Stoica.  Chameleon: scalable adaptation of video analytics.  In Proceedings of the 2018 conference of the ACM special interest group on data communication, pages 253–266, 2018.
  * [21] Guoliang Kang, Lu Jiang, Yi Yang, and Alexander G Hauptmann.  Contrastive adaptation network for unsupervised domain adaptation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4893–4902, 2019.
  * [22] Mehrdad Khani, Pouya Hamadanian, Arash Nasr-Esfahany, and Mohammad Alizadeh.  Real-time video inference on edge devices via adaptive model streaming.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4572–4582, 2021.
  * [23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al.  Overcoming catastrophic forgetting in neural networks.  Proceedings of the national academy of sciences, 114(13):3521–3526, 2017.
  * [24] Jian Liang, Dapeng Hu, and Jiashi Feng.  Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation.  In International conference on machine learning, pages 6028–6039. PMLR, 2020.
  * [25] Wei Lin, Muhammad Jehanzeb Mirza, Mateusz Kozinski, Horst Possegger, Hilde Kuehne, and Horst Bischof.  Video test-time adaptation for action recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22952–22961, 2023.
  * [26] Mingsheng Long, Han Zhu, Jianmin Wang, and Michael I Jordan.  Unsupervised domain adaptation with residual transfer networks.  Advances in neural information processing systems, 29, 2016.
  * [27] David Lopez-Paz and Marc’Aurelio Ranzato.  Gradient episodic memory for continual learning.  Advances in neural information processing systems, 30, 2017.
  * [28] Ravi Teja Mullapudi, Steven Chen, Keyi Zhang, Deva Ramanan, and Kayvon Fatahalian.  Online model distillation for efficient video inference.  In Proceedings of the IEEE/CVF International conference on computer vision, pages 3573–3582, 2019.
  * [29] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert.  icarl: Incremental classifier and representation learning.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001–2010, 2017.
  * [30] Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell.  Progressive neural networks.  arXiv preprint arXiv:1606.04671, 2016.
  * [31] Gobinda Saha, Isha Garg, and Kaushik Roy.  Gradient projection memory for continual learning.  In International Conference on Learning Representations, 2020.
  * [32] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [33] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim.  Continual learning with deep generative replay.  Advances in neural information processing systems, 30, 2017.
  * [34] James Seale Smith, Junjiao Tian, Shaunak Halbe, Yen-Chang Hsu, and Zsolt Kira.  A closer look at rehearsal-free continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2409–2419, 2023.
  * [35] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt.  Test-time training with self-supervision for generalization under distribution shifts.  In International conference on machine learning, pages 9229–9248. PMLR, 2020.
  * [36] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In International conference on machine learning, pages 6105–6114. PMLR, 2019.
  * [37] Huiyu Wang, Aniruddha Kembhavi, Ali Farhadi, Alan L Yuille, and Mohammad Rastegari.  Elastic: Improving cnns with dynamic scaling policies.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2258–2267, 2019.
  * [38] Zhenyi Wang, Li Shen, Tiehang Duan, Donglin Zhan, Le Fang, and Mingchen Gao.  Learning to learn and remember super long multi-domain task sequence.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7982–7992, 2022.
  * [39] Ziyi Wang, Songyu Zhang, Jing Cheng, Zhixiong Wu, Zhen Cao, and Yong Cui.  Edge-assisted adaptive configuration for serverless-based video analytics.  In 2023 IEEE 43rd International Conference on Distributed Computing Systems (ICDCS), pages 248–258. IEEE, 2023.
  * [40] Yang Xiao, Vincent Lepetit, and Renaud Marlet.  Few-shot object detection and viewpoint estimation for objects in the wild.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3090–3106, 2022.
  * [41] Shipeng Yan, Jiangwei Xie, and Xuming He.  Der: Dynamically expandable representation for class incremental learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2021.
  * [42] Mu Yuan, Lan Zhang, Xuanke You, and Xiang-Yang Li.  Packetgame: Multi-stream packet gating for concurrent video inference at scale.  In Proceedings of the ACM SIGCOMM 2023 Conference, pages 724–737, 2023.
  * [43] Runhao Zeng, Qi Deng, Huixuan Xu, Shuaicheng Niu, and Jian Chen.  Exploring motion cues for video test-time adaptation.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 1840–1850, 2023.
  * [44] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017.

|  Michael Shell Biography text here.
---|---
John Doe Biography text here.
---
Jane Doe Biography text here.
---


# A Personalized Household Assistive Robot that Learns and Creates New
Breakfast Options through Human-Robot Interaction

Ali Ayub1∗, Chrystopher L. Nehaniv1, and Kerstin Dautenhahn1 This research was
undertaken, in part, thanks to funding from the Canada 150 Research Chairs
Program and thanks to an International Research and Partnership Grant at
University of Waterloo.1University of Waterloo, Waterloo, Ontario N2L 3G1,
Canada$\\{$* a9ayub, cnehaniv<EMAIL_ADDRESS>
###### Abstract

For robots to assist users with household tasks, they must first learn about
the tasks from the users. Further, performing the same task every day, in the
same way, can become boring for the robot’s user(s), therefore, assistive
robots must find creative ways to perform tasks in the household. In this
paper, we present a cognitive architecture for a household assistive robot
that can learn personalized breakfast options from its users and then use the
learned knowledge to set up a table for breakfast. The architecture can also
use the learned knowledge to create new breakfast options over a longer period
of time. The proposed cognitive architecture combines state-of-the-art
perceptual learning algorithms, computational implementation of cognitive
models of memory encoding and learning, a task planner for picking and placing
objects in the household, a graphical user interface (GUI) to interact with
the user and a novel approach for creating new breakfast options using the
learned knowledge. The architecture is integrated with the Fetch mobile
manipulator robot and validated, as a proof-of-concept system evaluation in a
large indoor environment with multiple kitchen objects. Experimental results
demonstrate the effectiveness of our architecture to learn personalized
breakfast options from the user and generate new breakfast options never
learned by the robot.

##  I Introduction

With a rapid increase in the aging population worldwide [1, 2], research is
being conducted to develop autonomous robots that can assist older adults in
their homes. These assistive robots are being designed for various roles, such
as caretakers, cleaning robots, and home assistants [3, 4, 5, 6]. To create
robots that can assist users with household tasks, the robots will first need
to learn the preferences of the users related to the assistive tasks. For
example, for the task of setting up a table for breakfast, the robot must
first learn the different kinds of breakfasts that the user likes. Further,
after learning the user preferences, the robot must find creative ways to
perform the assistive tasks, because performing the same task every day can
become boring for the user. For example, setting up the same breakfast option
for the user over multiple days could become boring and the user might want to
try new things. Therefore, in this paper, our goal is to develop a
computational architecture that can allow a household assistive robot to learn
different breakfast options from its user, use the learned knowledge to set up
a table for breakfast, and also create new breakfast options for the user.

For a household assistive robot to perform tasks, it needs the semantic
knowledge of the household i.e. objects (e.g. bowl, spoon) and related
contexts (e.g. kitchen). The robot must also be able to reason on the semantic
knowledge to perform tasks using the objects in the household. Extensive
research has been conducted in recent years to create semantic reasoning
architectures for performing assistive tasks in household environments [7, 8].
Most of these works use a pre-specified knowledge base to perform household
tasks. However, in the real world, different users can have different
preferences about the tasks that they need assistance with. Therefore, for
such cases, we need to develop personalized household robots [9] that can
learn about the tasks that the users need assistance with, from the users.
Research has also been conducted on creativity for robots. Most research in
this field has been on developing cognitive architectures for social robots to
create new artistic drawings [10], or for humanoid robots to perform creative
dance moves [11, 12]. However, these works are not directly applicable to
household assistive robots for completing tasks in creative ways.

Figure 1: Our complete architecture for learning and setting up breakfast
options in a household. Sensory inputs from the Fetch robot are processed
through the perceptual system and encoded into latent variables, which are
stored in the episodic memory during the learning phase, and stored in STM to
track breakfasts eaten by the user over multiple days. The breakfast creation
module can use the data in the episodic memory to create new breakfast
options. The task planner can plan lower-level commands for the robot’s
actuators to set up a table for breakfast. The wide, dark blue line indicates
that all three outputs from the perceptual system are passed on to the task
planner. Circled numbers show the flow of information in the architecture,
with pink-colored numbers for the learning process and yellow-colored numbers
for the breakfast setup. Processes that run in parallel are tagged with the
same number.

In this paper, we develop a cognitive architecture that allows a robot to
learn different breakfast options using the objects in the household from its
user, set up the learned breakfast options on a table upon request from the
user, and create new breakfast options for the user over the long term. The
architecture allows the robot to interact with its user using a graphical user
interface (GUI) and learn different breakfast options. Inspired by the dual
memory theory of mammalian memory [13], the breakfast options taught by the
user, grounded in the processed sensory data of the robot, are stored in the
long-term episodic memory. The architecture also keeps track of different
breakfasts eaten by the user over multiple days and stores them in short-term
memory (STM). The architecture can access the learned knowledge from the
episodic memory and plan lower-level actuator commands for the robot to set up
a table for the learned breakfasts. The architecture can further reason on the
knowledge stored in the episodic memory to generate a semantic knowledge graph
which can be used to create new breakfast options. The user can ask the robot
to set up a previously learned breakfast or create a new breakfast option
through the GUI. We integrate the proposed architecture on the Fetch mobile
manipulator robot [14] and test it in a large indoor space with 9 common
kitchen objects. Experimental results confirm that the robot can accurately
learn different breakfast options from the user and set them up on a table.
The results also show that the robot can create various new breakfast options
that were never observed by the robot in its experience in the household
context.

##  II Related Work

Socially assistive robots have been developed in recent years that can be
interactive meal partners for older adults in long-term care homes [15, 16].
These robots, however, only interact with older adults to suggest different
meal options and do not physically perform the task of setting up the table
for a meal. Various cognitive architectures have been developed that can use
the semantic knowledge of a household environment and physically perform tasks
in the household, such as fetching an object, setting up a table for
breakfast, cleaning a table [7, 8, 17]. Although these robots can perform
different tasks in a household environment, they perform only a pre-programmed
set of tasks, and they do not adapt to the preferences of their users. For
example, the mobile manipulator robot in [7] can set up a table for only one
type of breakfast. This can also get boring for the users if the robot sets up
the same breakfast every single day over multiple weeks. In such cases, the
robot must create new breakfast options for its users.

Research for developing creative robots has been limited to creating artistic
drawings or dancing robots. For example, Augello et al. [10] develop a
cognitive architecture for social robots that can create a new drawing while
collaborating with a human. Infantino et al. [11] and Manfre et al. [12]
develop cognitive architectures to enable creativity in humanoid robots so
that they can dance in pleasant manners. These works, however, are not
applicable to household assistive robots that can perform household tasks in
creative ways. Research has also been conducted on developing cognitive
architectures that can allow social robots to stimulate creativity in children
[18, 19]. These architectures, however, do not allow a robot to be creative
but rather stimulate creativity in children.

With the advent of deep learning, generative adversarial networks (GANs) have
been developed that can generate new data the model never learned [20, 21,
22]. These networks can learn general semantic representations about different
household contexts (e.g. bedroom) from a large amount of training data, and
then generate new images that were never seen by the model. One of the main
limitations of these models is that they can generate many random images which
do not belong to any context, such as creating random images that do not look
like a bedroom context. Therefore, they cannot be applied to make assistive
robots creative, as the robot would make many mistakes, which can hurt the
trust of its user towards the robot [23]. Further, GANs also require a large
amount of training data to learn, which might be infeasible in real-world
situations where the robot learns from the supervision provided by its users.
Real users (especially older adults) would be unwilling to provide hundreds
and thousands of examples of a single task to teach the robot. In this paper,
we use Gaussian processes [24] as generative models to create new breakfast
options, as these models have been shown to work with limited data [25].

##  III Contextual Memory System for a Creative Robot

Figure 1 shows our cognitive architecture for a creative breakfast setting
robot. Different computational modules in the architecture were integrated
using ROS on the Fetch mobile manipulator robot. Note that all the modules are
stand-alone, therefore they can be reused as blocks in different frameworks.
These modules are described below:

###  III-A Robot’s Sensors

The Fetch mobile manipulator robot was used for this project [14]. Fetch
consists of a mobile base and a 7 DOF arm. The robot also contains an RGB
camera, a depth sensor and a Lidar sensor. These sensors can be used for 3D
perception, slam mapping, and obstacle detection in the robot’s environment.
In our architecture, the mobile base, the 7 DOF arm, and all three sensors are
used for perception, manipulation, mapping, and navigation in an indoor
environment.

###  III-B Perceptual System

The perceptual system of the architecture takes an RGB image and point cloud
data as input from the robot’s sensors, and parses this data into separate
objects. We use the YOLOv2 object detector [26] for the detection of objects
in the RGB images. The 2D bounding boxes from YOLO are converted into 3D
coordinates using the point cloud data. We collected $\sim$5000 images of 9
household objects used in our experiments and trained the YOLO object detector
on the collected data. The perceptual system, thus, parses the input images
and outputs the object categories, 2D bounding boxes and 3D coordinates for
all the objects in the image.

###  III-C Memory Encoding

The data obtained from the robot’s sensors or the perceptual system must be
encoded into a low dimensional feature space (also called a latent variable),
before it can be used to reason about the entities in the world (e.g. objects
in the household). In this paper, we encode the processed sensory inputs by
the perceptual system, using conceptual spaces [27, 28]. In cognitive science,
a Conceptual Space is a metric space in which entities are characterized by
quality dimensions. Conceptual spaces have mostly been used in cognitive
science for category learning, where the dimensions of a latent variable (LV)
in a conceptual space represent the category features. In this paper, we use a
conceptual space LV to represent different breakfast setups (such as {cereal,
milk, bowl, spoon} make a breakfast setup), where the features of the LV
represent the collection of objects in the breakfast setup represented by the
LV. Further, as each breakfast setup contains food items such as cereal, milk,
etc and utensils such as spoon, bowl, etc, we also encode this information
about the objects in another LV. We term this LV, a food-context LV to
differentiate it from the object LV for the breakfast options. This
information can help the architecture generate creative breakfast setups
(Section III-F).

###  III-D Short-Term Memory (STM)

Once an input image of a breakfast setup is encoded into a latent variable, it
is stored in the short-term memory (STM) of the architecture. The size ($k$)
of STM is set as a hyper-parameter to allow the architecture to store encoded
images for a certain number of days. Once STM is full, data stored from
earlier days is removed to make room for more data.

STM tracks the breakfast eaten by the user over multiple days. Using the data
stored in STM, the architecture can suggest new breakfast options that the
user has not eaten in previous days. Formally, let’s consider there are $n$
number of breakfast options stored in the episodic memory as LVs
$X=\\{x_{1},x_{2},...,x_{n}\\}$. Over the course of $k$ (hyperparameter in
STM) number of days, the user eats different breakfast options, where
$M=\\{m_{1},m_{2},...,m_{n}\\}$ represents the total number of times each of
the $n$ breakfast options was eaten by the user. From this set, the robot can
find the breakfast options that have been least eaten by the user over $k$
days as $\arg\min M$ and set it up on the table. If multiple breakfast options
were eaten the least number of times, then the robot randomly chooses one of
these breakfast options.

###  III-E Episodic Memory

The episodic memory stores different breakfast options taught by the robot’s
user. As different users can have different breakfast preferences, it is not
possible to store a general set of breakfast options. Therefore, the robot
must learn about these preferences by interacting with the user.

In our architecture, a user can initiate a learning session using a GUI
(details in Section III-H) and provide examples of different breakfast setups.
The robot captures the breakfast setups as images using its sensors. The
perceptual system (Section III-B) processes the training images which are then
encoded into latent variables (Section III-C). The encoded LVs (both object
LVs and food-context LVs) are then stored in the episodic memory, which can be
accessed later to set up a table for breakfast.

###  III-F Creating New Breakfast Options

The user can also ask the robot to surprise them (see Figure 3) by creating a
new breakfast option that the user never taught the robot i.e. such a
breakfast option does not exist in the episodic memory. We define a creative
breakfast as a new combination of food and utensil items that were never
directly learned by the robot from the user. To achieve this, we use the
object LVs stored in the episodic memory to find the mean $\mu$ and covariance
matrix $\Sigma$ for a Gaussian distribution in a Gaussian process. We generate
a pseudo-LV111The sampled LVs are termed as pseudo-LVs because they are not
real LVs learned from the user. after sampling the Gaussian distribution.
However, the pseudo-LV can be the same as one of the object LVs stored in the
episodic memory i.e. it is not a new breakfast option. Therefore, if the
pseudo-LV is the same as any of the object LVs in the episodic memory, we
continue to resample the Gaussian distribution until we get a pseudo-LV that
is different from the object LVs in the episodic memory.

The new pseudo-LV, however, can be an invalid breakfast setup. For example,
{cereal, milk, spoon} is an invalid breakfast setup as it does not contain any
container (such as a bowl) to pour cereal and milk. To fix such cases, we find
the conditional relationships among various objects that are used in different
breakfast setups stored as LVs in the episodic memory. Using these conditional
relationships we infer logic-based rules to generate a knowledge graph, which
can be used to fix invalid breakfast setups.

Figure 2: An example of logic-based knowledge graph generation using the three
breakfast options learned from a user. Digits next to visualizations
correspond to the rule generation process for each food object. For Apple and
Banana, we show the final probabilities only. Note that the logic-based rules
are learned solely from the data shown by the user to the robot, therefore,
some of the rules could be imperfect or unconventional, such as the
requirement of Apple and Banana with each other. (Bottom) An example of fixing
the generated breakfast option using the logic-based rules from the knowledge
graph.

We use the food-context LVs in the episodic memory to determine the dependency
of different food items on a combination of other food items and utensils. To
achieve this, let’s consider $n$ LVs in the episodic memory, and consider a
food object represented by dimension $i$ in the LVs. For each $i$th food item,
we consider all the breakfast setups (say $r$ LVs) where this food item
exists. Among the $r$ LVs, we first calculate the probability
${\rm{P}}(i|no\\_utensil)$, i.e. if the $i$th food item does not require a
utensil to be present in a breakfast setup. If ${\rm{P}}(i|no\\_utensil)>0$,
there is at least one breakfast setup where the food item is not accompanied
by a utensil, therefore the food item can be a part of a breakfast setup
without a utensil. Otherwise, if ${\rm{P}}(i|no\\_utensil)=0$, the food item
requires at least one utensil. In this case, we go through all $r$ LVs to find
different combinations of utensils that the food item depends on, as a food
item could depend on multiple utensils, e.g. cereal would depend on a spoon
and a bowl. For this, let’s consider that there are a total of $m$ utensils
present in the $r$ LVs. For each $j$th utensil present in the $r$ LVs, we find
the conditional probability ${\rm{P}}(j|l)$ with all the other
$l=\\{1,...,m\\}$ utensils in the $r$ LVs. ${\rm{P}}(j|l)$ represents the
probability that $j$th utensil exists given that $l$th utensil exists in the
same LV. ${\rm{P}}(j|l)$ is determined as follows:

${\rm{P}}(j|l)=\frac{\sum_{q=1}^{r}z_{q}^{j}\mbox{ such that
}z_{q}^{j}z_{q}^{l}>0}{\sum_{q=1}^{r}z_{q}^{l}\mbox{ such that }z_{q}^{l}>0},$
(1)

where $z_{q}^{l}$ represents the value of $l$th utensil item in the $q$th
food-context LV. If ${\rm{P}}(j|l)=1$, utensil $j$ must exist when utensil $l$
exists in an LV accompanied by the $i$th food item. As a result, we get an
$m\times m$ matrix representing the dependency of utensils on other utensils
that accompany the $i$th food item in $r$ LVs. Using this dependency matrix,
we find all the utensil items that are independent of other utensils or that
are interdependent with other utensils i.e. for two utensils $j$ and $l$,
${\rm{P}}(j|l)$=1 and ${\rm{P}}(l|j)$=1. The resulting set represents
combinations of different utensils that the $i$th food item depends on. These
sets are then used to generate a logic-based knowledge graph based on
$is\\_required$ relationships (see Figure 2 for an example). Note that the
food item requires only one of the dependent utensil combinations to be
present in the breakfast setup, not all the combinations. For example, milk
must either be accompanied by a cup for drinking or {bowl, spoon} in a cereal
breakfast. Figure 2 shows a simple example of generating a logic-based
knowledge graph from the learned breakfast options in memory.

After finding the dependencies on utensils, the same process is repeated to
determine if a food item depends on other food items. After this process, we
can find a combination of objects (foods or utensils) that each food item in
the LVs depends on for valid breakfasts. We do not find a separate list of
dependent objects for utensils as these items are only needed to accompany the
food items in breakfast setups. Note that the knowledge graph is generated
based on the breakfast options taught by the user, so the dependency rules
encoded in the graph are personalized to the user. Experimental results in
Section IV confirm this.

Using the logic-based knowledge graph, we can determine if a feature dimension
in a pseudo-LV satisfies its dependency on other items. If a feature dimension
is not accompanied by its dependent items, we manually add the dependent items
in the pseudo-LV (see Figure 2 for an example). Finally, after the dependency
check, the pseudo-LV is decoded using the inverse of the procedure in Section
III-C to get the objects in the new breakfast option. The object names/labels
are then passed on to the task planner.

###  III-G Task Planner

The task planner gets the decoded breakfast option from the creativity module
(Section III-F), and plans lower-level actions to be taken by the robot to set
up a table for breakfast. The task planner passes lower-level commands to the
mobile base and the arm of the robot to move and fetch objects from the
kitchen to the dining table.

Figure 3: The graphical user interface (GUI) used to interact with the robot.
The head camera output of the Fetch robot with objects detected through YOLO
is in the window to the right. The window to the left shows three buttons that
can be used to teach the robot a new breakfast option, ask the robot to bring
a previously learned breakfast option and ask the robot to create a new
breakfast option.

###  III-H Graphical User Interface

A simple graphical user interface (GUI) is integrated with the architecture to
allow the robot to communicate with the user. The GUI allows the user to
initiate a teaching session with the robot where the user can show the robot
different breakfast setups on a table. The user physically places the set of
objects in a breakfast setup on the table in front of the robot’s camera (see
Figure 3). The user can provide the name for the breakfast option by typing it
in a textbox. The robot captures the breakfast data using the RGB camera and
the depth sensor and then encodes and stores the breakfast option in the
episodic memory (Section III-E).

The GUI also allows the user to ask the robot to set up a table for breakfast.
The user can type in the name of the breakfast that they want, ask the robot
to set up the table for breakfast without typing any particular breakfast name
or ask the robot to surprise them by creating a new breakfast option. After
getting the input from the user, the architecture can use a combination of all
the modules to allow the robot (Section III-G) to set up the table for
breakfast.

##  IV Experiments

In this section, we first describe the experimental setup and the
implementation details. We then describe two experiments to evaluate the
performance of our architecture for learning different breakfast options from
the user, setting them up on the table, and creating new breakfast options.
For all the experiments reported in this section, the experimenters take the
role of a user.

TABLE I: Results of learning and setting up 7 breakfast options by the robot over 15 runs. Accuracy represents the ratio of the number of times a breakfast was correctly set up and the number of times it was chosen. STM shows the number of times a breakfast option was chosen by tracking data in short-term memory, without explicitly being asked by a user. LE represents the learning error. Breakfast Options |  Accuracy |  STM |  LE
---|---|---|---
milk, cup |  2/3 |  1 |  0
milk, cup, banana |  3/3 |  0 |  0
milk, cereal, spoon, bowl |  1/2 |  1 |  0
banana, milk, cereal, spoon, bowl |  1/1 |  0 |  0
honey, milk, cereal, spoon, bowl |  1/2 |  0 |  0
honey, milk, cup |  1/2 |  1 |  0
apple, orange, banana |  1/2 |  0 |  0

###  IV-A Experimental Setup

We use the Fetch robot [14] and its associated ROS packages for all the
experiments. We performed experiments in a large indoor space where we set up
the kitchen and the dining area with realistic household objects. The indoor
space is mapped using the Lidar sensor on the Fetch robot and an existing SLAM
algorithm available from Fetch Robotics. Navigation in the environment was
achieved using ROS packages provided by Fetch Robotics. Common household
items/objects belonging to 9 categories (see Table II for a list of graspable
objects) are placed on three tables in the kitchen. Out of the 9 objects, 3
(Banana, Bowl, and Spoon) were not graspable by the robot. Therefore, for
breakfast setups that required these 3 objects, the user had to fetch the
objects themselves. Manipulation of objects (pick and place) was achieved
using ROS packages for gripper, arm, and torso control provided by Fetch
Robotics. The RGB camera and depth sensors on the Fetch robot were used for
visual sensing of the environment. RGB images from the camera are passed
through the perception module of the architecture which uses YOLOv2 [26] to
detect and localize objects in the images (see Section III-B for details).

For all the experiments (unless mentioned otherwise), the user (experimenter)
first teaches the Fetch robot different breakfast options on the dining table
using the GUI (Section III-H). The robot learns the breakfast options and
stores them in episodic memory. As the user would eat breakfast once every
day, we can ‘simulate’ multiple days by asking the robot to set up a table for
breakfast multiple times a day. For the short-term memory (STM) in the
architecture, we set the hyper-parameter $k$ to 5 days. Examples of teaching
breakfast options to the robot and testing the robot to set up known and new
breakfast options are shown in the supplementary video.

###  IV-B Experiment 1: Setting Up Known Breakfast Options

In this experiment, we tested if the robot can learn breakfast options from
the user and then set up the learned breakfast options when asked by the user.
We taught the robot 7 different breakfast options as shown in Table I. Figure
4 shows examples of 2 out of 7 breakfast options learned by the robot. The
robot was then asked to set up a table for breakfast 15 times. In each of the
15 turns (except for the 5th, 10th, and 15th run), the user typed the name of
the breakfast in the GUI to ask the robot to set up the table for a particular
breakfast. We randomly chose a breakfast option to be typed in each turn. For
the 5th, 10th, and 15th turn, the user did not type any breakfast name,
therefore the robot used the data stored in STM to choose the least eaten
breakfast to set up. For each breakfast setup, the robot moved all the
graspable objects in the breakfast setup from the kitchen to the dining table.
On average, it took the robot $\sim$4 minutes to set up a breakfast option on
the table.

Figure 4: Examples of two different breakfast options learned and set up by the Fetch robot. TABLE II: Results of setting up 7 breakfast options in 15 runs in terms of perceptual errors (PE), manipulation errors (ME), and grasping errors (GE) for graspable objects. Each column represents the ratio between the number of errors for manipulation of an object and the total number of times the object occurred in 15 runs. Object |  PE |  ME |  GE
---|---|---|---
milk |  1/12 |  1/12 |  0
cup |  2/8 |  2/8 |  0
cereal |  0 |  1/5 |  0
apple |  0 |  0 |  0
orange |  1/2 |  0 |  1/2
honey |  1/4 |  2/4 |  0

Table I shows the results of setting up 7 breakfast options learned by the
robot. All the breakfast options were learned correctly by the robot, and
there was no learning error. The robot was able to correctly set up breakfast
options in 10 out of 15 runs. As each breakfast setup required multiple
objects, failing to fetch even a single object would result in an incorrect
breakfast setup. Most of the breakfast setup failures happened because of a
single object in the breakfast setup (more details below). There were three
runs when the robot was asked to set up a breakfast option using STM. The
robot correctly chose one of the least-eaten breakfast options in all three
runs.

Table II shows the results of the experiment in 15 runs, with three different
kinds of errors for each graspable object used in the 7 breakfast setups. The
most common error type was the manipulation error (ME), which occurred because
of two reasons: (1) the motion planner could not find a path to reach the
goal, (2) the perceptual system provided an incorrect pose estimate to pick
the object (perceptual error (PE)). There were no object detection failures
during the experiment because even if the robot failed to detect an object, it
moved its head up and down until it found the correct object. Therefore, all
of the perceptual errors happened during the 3D pose estimation of objects.
Finally, there was only one grasping error for Orange. The robot’s arm was not
low enough and because the orange is round it was not captured in the gripper
of the robot. Objects with sharper edges, such as Milk did not face this
issue. These results confirm that our architecture can allow a robot to learn
most breakfast options from the user and set up the learned breakfast options
on a table.

###  IV-C Experiment 2: Creating New Breakfast Options

####  IV-C1 Experiment with a Robot

In this experiment, we tested the ability of our architecture to allow a robot
to create new breakfast options that were never learned by the robot. The
experimental setup was the same as in experiment 1. The robot was started with
the same 7 breakfast options in the beginning as in experiment 1. After that,
we tested the robot 5 times to create and set up new breakfast options.

Table III shows the five new breakfast options created by the robot. All five
breakfast setups were valid setups because each food object was accompanied by
the correct set of utensils. Two out of five breakfast options generated by
the Gaussian process were invalid. For example, breakfast option 2 had bowl
missing, and breakfast option 5 had cup missing. However, these objects were
added by the architecture using the logic-based rules encoded in the knowledge
graph for the food items (Section III-F). Finally, note that there was no
learning error (LE) encountered for these breakfast options because they were
not learned from any example provided by the user to the robot. The values for
perceptual and manipulation errors were consistent with the previous
experiments.

Breakfast Options |  Valid? |  LE |  PE |  ME
---|---|---|---|---
milk, banana, honey, cup |  Yes |  0 |  1/1 |  1/1
apple, milk, cereal, spoon, bowl |  Yes |  0 |  0 |  1/1
apple, honey, milk, cereal, spoon, bowl |  Yes |  0 |  1/1 |  1/1
milk, cereal, bowl, cup, spoon |  Yes |  0 |  0 |  0
apple, milk, banana, orange, cup |  Yes |  0 |  0 |  0
TABLE III: Five breakfast options created by the robot that were not taught by
the user. LE, PE, and ME represent the learning error, perceptual error, and
manipulation error, respectively.

####  IV-C2 Simulated Experiments

To further evaluate the effectiveness of our breakfast creativity algorithm,
we tested the architecture in simulation to create 50 breakfast options. Note
that in this case the architecture was only asked to suggest the breakfast
option and the robot did not physically set up the generated breakfast option
on the table. Out of the 50 breakfast options, 27 were the same setups as the
ones stored in the episodic memory and were thus discarded. Out of the other
23 options, 7 were invalid options generated by the Gaussian process. However,
these invalid options were corrected using the logic-based knowledge graph for
the food items. Overall, out of the 23 new breakfast options, 6 were
duplicates, so there were 17 distinct new options. These results confirm that
our architecture can allow the robot to create and set up new breakfast
options that were not learned by the robot. Further, our architecture was able
to create more than double the breakfast options (17) it had learned by
interacting with the user (7). However, the robot cannot generate a
significantly large number of distinct breakfast options when learning from a
few examples.

We further test our approach on a larger scale with a total of 25 objects, an
initial set of 20 breakfasts and ask the creativity module to generate 200
breakfast options. Out of the 200 generated breakfasts, 65 were the same as
the ones stored in the episodic memory and were therefore discarded. For the
rest of the 135 breakfasts, 113 were invalid options but they were corrected
by the logic-based knowledge graph for the food items. Finally, out of the 113
new breakfast options 36 were duplicates. Therefore, the architecture was able
to generate 99 distinct new breakfast options from only 20 initial breakfast
setups. These results confirm the scalability of our approach to larger
datasets learned over the long term.

Finally, we tested our approach with some unconventional breakfast setups. For
example, we added a breakfast setup {cereal, bowl} as some users might eat
cereal without any milk. Other examples of unconventional setups were
{peanut_butter, bowl, spoon}, {yogurt, spoon}, etc.. For this experiment, we
had a total of same 25 objects as in the previous experiment, and 12 breakfast
setups where 6 of the breakfast setups were unconventional. The creativity
module generated 50 breakfast options, with 25 out of 50 being distinct new
options. Interestingly, we noticed that the creative breakfast setups followed
the dependency of food items learned through the data of the initial
breakfasts. For example, the creativity module generated setups such as,
{apple, cereal, bowl, spoon, yogurt, peanut_butter} where cereal is not
accompanied by milk. These results confirmed the ability of our architecture
to personalize to their users’ preferences even when creating new breakfast
options. These results also show the effectiveness of our unique combination
of data-driven learning, logic-based reasoning, and human-robot interaction.

##  V Conclusions

This paper has presented an architecture for learning and setting up different
breakfast options for the user. The architecture can also create new breakfast
options that were never taught by the user. Extensive proof-of-concept system
evaluations on a Fetch mobile manipulator robot demonstrate the ability of our
architecture to allow a robot to accurately learn multiple breakfast options
from the user and then set them up on a table upon request. The results also
confirm the ability of the architecture to be able to track previously eaten
breakfasts by the user to suggest new breakfasts, and even create multiple
breakfast options that were never learned by the robot. We hope that this work
will lead to designing more effective personalized household robots that can
interact with, learn and provide long-term assistance to older adults in their
own homes to support independent living.

##  VI Limitations and Future Work

For all the experimental evaluations, the experimenter performed the role of
the user. In the future, we hope to conduct a user study with real
participants to investigate the usability of the system in real-world
household environments. Further, the robot showed promising results with the
chosen hyperparameter value for $k$ in the STM. However, we hope to perform
more experiments in the future to analyze the effect of this hyperparameter on
the choice of breakfast options.

There were some objects that the robot struggled with when setting up
different breakfasts, particularly because of the 3D pose estimation of
objects. However, designing robust pose estimation and manipulation algorithms
for complex household objects was out of the scope of this work. In the
future, we hope to explore these limitations to scale up our approach to more
realistic household environments.

## References

  * [1] J. Iriondo and J. Jordan, “Older people projected to outnumber children for first time in U.S. history,” 2018. [Online]. Available: https://www.perkinslawpc.com/images/Older˙People˙Projected˙to˙Outnumber˙Children˙for˙First˙Time˙in˙U.S.˙History.pdf
  * [2] J. Fuss, “Our aging population—a serious problem for canada,” 2020. [Online]. Available: https://www.fraserinstitute.org/blogs/our-aging-population-a-serious-problem-for-canada
  * [3] M. J. Matarić, “Socially assistive robotics: Human augmentation versus automation,” _Science Robotics_ , vol. 2, no. 4, 2017.
  * [4] A. Ayub, C. L. Nehaniv, and K. Dautenhahn, “Don’t forget to buy milk: Contextually aware grocery reminder household robot,” in _IEEE International Conference on Development and Learning (ICDL)_. IEEE, 2022, pp. 299–306.
  * [5] L. Petrecca, “How Robot Caregivers Will Help an Aging U.S. Population,” Oct. 2018. [Online]. Available: https://www.aarp.org/caregiving/home-care/info-2018/new-wave-of-caregiving-technology.html
  * [6] J. Shah, A. Ayub, C. L. Nehaniv, and K. Dautenhahn, “Where is my phone? towards developing an episodic memory model for companion robots to track users’ salient objects,” in _Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction_ , 2023, pp. 621–624.
  * [7] G. Kazhoyan, S. Stelter, F. K. Kenfack, S. Koralewski, and M. Beetz, “The robot household marathon experiment,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 9382–9388.
  * [8] S. Liu, G. Tian, Y. Zhang, M. Zhang, and S. Liu, “Service planning oriented efficient object search: A knowledge-based framework for home service robot,” _Expert Systems with Applications_ , vol. 187, p. 115853, 2022.
  * [9] K. Dautenhahn, “Robots we like to live with?! - a developmental perspective on a personalized, life-long robot companion,” in _13th IEEE International Workshop on Robot and Human Interactive Communication (RO-MAN)_ , 2004, pp. 17–22.
  * [10] A. Augello, I. Infantino, A. Manfré, G. Pilato, and F. Vella, “Analyzing and discussing primary creative traits of a robotic artist,” _Biologically Inspired Cognitive Architectures_ , vol. 17, pp. 22–31, 2016\.
  * [11] I. Infantino, A. Augello, A. Manfré, G. Pilato, and F. Vella, “Robodanza: Live performances of a creative dancing humanoid,” in _Proceedings of the Seventh International Conference on Computational Creativity_ , 2016, pp. 388–395.
  * [12] A. Manfré, A. Augello, G. Pilato, F. Vella, and I. Infantino, “Exploiting interactive genetic algorithms for creative humanoid dancing,” _Biologically Inspired Cognitive Architectures_ , vol. 17, pp. 12–21, 2016\.
  * [13] J. L. Mcclelland, B. L. Mcnaughton, and R. C. Oreilly, “Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory.” _Psychological Review_ , vol. 102, no. 3, p. 419–457, 1995.
  * [14] M. Wise, M. Ferguson, D. King, E. Diehr, and D. Dymesich, “Fetch and freight: Standard platforms for service robot applications,” in _IJCAI, Workshop on Autonomous Mobile Service Robots_ , 2016.
  * [15] D. McColl and G. Nejat, “A Socially Assistive Robot That Can Monitor Affect of the Elderly During Mealtime Assistance1,” _Journal of Medical Devices_ , vol. 8, no. 3, 07 2014.
  * [16] ——, “Meal-time with a socially assistive robot and older adults at a long-term care facility,” _Journal of Human-Robot Interaction_ , vol. 2, no. 1, pp. 152–171, 2013.
  * [17] Z. Wang, G. Tian, and X. Shao, “Home service robot task planning using semantic knowledge and probabilistic inference,” _Knowledge-Based Systems_ , vol. 204, p. 106174, 2020.
  * [18] P. Alves-Oliveira, S. Gomes, A. Chandak, P. Arriaga, G. Hoffman, and A. Paiva, “Software architecture for yolo, a creativity-stimulating robot,” _SoftwareX_ , vol. 11, p. 100461, 2020.
  * [19] M. Elgarf, G. Skantze, and C. Peters, “Once upon a story: Can a creative storyteller robot stimulate creativity in children?” in _Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents_ , 2021, pp. 60–67.
  * [20] C. Zhang, Y. Xu, and Y. Shen, “Decorating your own bedroom: Locally controlling image generation with generative adversarial networks,” _arXiv preprint arXiv:2105.08222_ , 2021.
  * [21] D. Mateja and A. Heinzl, “Towards machine learning as an enabler of computational creativity,” _IEEE Transactions on Artificial Intelligence_ , vol. 2, no. 6, pp. 460–475, 2021.
  * [22] A. Ayub and A. R. Wagner, “EEC: Learning to encode and regenerate images for continual learning,” in _International Conference on Learning Representations (ICLR)_ , 2021.
  * [23] P. Robinette, A. Howard, and A. Wagner, “Effect of robot performance on human–robot trust in time-critical situations,” _IEEE Transactions on Human-Machine Systems_ , vol. 47, no. 4, pp. 425–436, 2017.
  * [24] A. Papoulis, _Probability, random variables, and stochastic processes_. Mc Graw Hill, 1991.
  * [25] A. Ayub and A. R. Wagner, “What am i allowed to do here?: Online learning of context-specific norms by pepper,” in _International Conference on Social Robotics_. Springer, 2020, pp. 220–231.
  * [26] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [27] P. Gärdenfors, _Conceptual Spaces: The Geometry of Thought_. MIT press, 2004.
  * [28] I. Douven and P. Gärdenfors, “What are natural concepts? A design perspective,” _Mind & Language_, vol. 35, no. 3, pp. 313–334, 2020.