Dongseong Hwang, Changwan Ryu, Khe Chai Sim

# Edit Distance based RL for RNNT decoding

###### Abstract

RNN-T is currently considered the industry standard in ASR due to its
exceptional WERs in various benchmark tests and its ability to support
seamless streaming and longform transcription. However, its biggest drawback
lies in the significant discrepancy between its training and inference
objectives. During training, RNN-T maximizes all alignment probabilities by
teacher forcing, while during inference, it uses beam search which may not
necessarily find the maximum probable alignment. Additionally, RNN-T's
inability to experience mistakes during teacher forcing training makes it more
problematic when a mistake occurs in inference. To address this issue, this
paper proposes a Reinforcement Learning method that minimizes the gap between
training and inference time. Our Edit Distance based RL (EDRL) approach
computes rewards based on the edit distance, and trains the network at every
action level. The proposed approach yielded SoTA WERs on LibriSpeech for the
600M Conformer RNN-T model.

Index Terms: speech recognition, RNN-T, reinforcement learning, Actor-critic,
teacher-forcing, edit distance

##  1 Introduction

The Recurrent Neural Network-Transducer (RNN-T) [1] has demonstrated
significant success in both academic and industrial settings for automatic
speech recognition (ASR) [2, 3, 4, 5]. This can be attributed to several
factors, including the availability of multiple public benchmarks that have
established the RNN-T model as state-of-the-art (SoTA) for ASR [6, 7, 8, 9],
such as LibriSpeech [10], SpeechStew [11], and Multi-lingual LibriSpeech [12].
Additionally, the RNN-T model offers seamless support for streaming ASR [13]
and longform utterances [14], which are crucial requirements for many real-
world applications. In comparison, other popular models such as the
Connectionist temporal classification (CTC) [15] and attention-based models
[16] have limitations. Specifically, CTC models generally exhibit higher word
error rates (WER) than RNN-T models [8], while attention-based models are
challenging to support streaming requirements and require non-trivial
modifications to support longform inputs [17].

Despite its advantages in streaming and longform support, the RNN-T model is
not without its limitations. Unlike attention-based ASR models, the RNN-T
model requires teacher-forcing during training, which means that only ground
truth labels are used regardless of the predicted output. As a result, the
RNN-T model lacks experience in recovering from errors during training. This
is in contrast to the scheduled sampling approach [18, 16] used by attention-
based models, where ground truth tokens are replaced by predicted output
tokens randomly. However, the RNN-T model's training objective is to maximize
the probabilities of all possible alignments, which is incompatible with
scheduled sampling. The CTC model exhibits similar limitations to the RNN-T
model.

An additional limitation of the RNN-T model is the significant discrepancy
between its training objective and inference method. While beam search is the
standard approach for ASR inference, there is no guarantee that it will
identify the highest log likelihood hypothesis. The RNN-T model's training
objective is to maximize the probability of all possible alignments, which
differs substantially from the beam search approach. In contrast, the cross-
entropy (CE) loss utilized by attention-based ASR models maximizes the token-
level probability, which aligns more closely with the beam search algorithm.

Minimum Word Error Rate Training (MWER) training was proposed as a solution to
the problems caused by exposure bias due to teacher-forcing and the
discrepancies between the training objective and inference method [19]. MWER
is commonly employed in industry as a crucial step in production models,
typically following RNN-T pretraining. Nevertheless, this training approach is
a sentence-level policy gradient [20], which results in poor sample
efficiency.

We introduce a novel reinforcement learning (RL) technique for RNN-T decoding.
Our research offers two noteworthy contributions:

  1. 1.

We present a functional RL algorithm for the RNN-T model that generates a
training signal for each token rather than for the entire sentence. Our
proposed approach leads to notable improvements in the SoTA performance on
LibriSpeech WERs.

  2. 2.

We propose a novel reward engineering technique that utilizes edit distance to
proficiently train both emission and blank actions within RNN-T models,
thereby achieving direct minimization of the Word Error Rate (WER) metric.

##  2 Related work

The automatic speech recognition (ASR) problem is widely recognized to exhibit
exposure bias as a result of teacher forcing and the objective gap between
training and inference. Notable adaptations of reinforcement learning (RL)
have been proposed for ASR, including the Minimum Word Error Rate (MWER)
training [19] approach, which is an ASR version of the REINFORCE algorithm
[20], the most basic RL algorithm introduced in 1992. MWER works by generating
the top k hypotheses using beam search and then providing a reward to
hypotheses with a better than average WER and a penalty to those with a worse
than average WER. MWER has training signal at the sentence level, meaning that
all tokens in the same hypothesis are rewarded or penalized together. As WER
of SoTA RNN-T models is already less than $10\%$, many of the tokens in the
hypotheses are correct but receive an unfair penalty, which can hinder model
training. For this reason, both the REINFORCE algorithm and MWER are known to
suffer from poor sample efficiency.

In 1999, actor-critic algorithms [21] were introduced as a solution to the
issue of poor sample efficiency. This algorithm estimates the value of every
action sequence and compute the policy gradient in equation (4) for every
action. This approach provides a training signal for each action, resulting in
improved sample efficiency. However, a new challenge arises: how to estimate
the value. This question has been a major focus of research in RL since the
introduction of actor-critic algorithms.

The Optimal Completion Distillation (OCD) method [22] provides an innovative
solution to the value estimation problem. By leveraging the ability to compute
the edit distance for every token, OCD is able to calculate the exact Q-value
[23] instead of relying on Q-value estimation. Although OCD works well for
character tokens, it is not feasible to apply this method to subword units due
to the high computational cost of assigning a Q-value to each subword unit per
action. Additionally, if the hypothesis predicts an incorrect subword unit,
computing the Q-value for all subword units on the incorrect action becomes a
non-trivial problem.

There was a previous attempt to adapt a gradient policy with token-level
reward to attention-based ASR in [24]. Our work builds upon this previous
study but focuses on RNN-T [1] model. While the earlier paper assigned
negative rewards to only the last token, we identified this approach as
problematic. Instead, we propose assigning negative rewards to every incorrect
action and also propose a value assignment method for blank tokens.

##  3 Methods

###  3.1 Training

Reinforcement learning (RL) generates data via a behavior policy and
subsequently computes policy gradient through reward and target policy [20,
21]. In our study, the target policy is a Recurrent Neural Network-Transducer
(RNN-T) [1] model that generates a softmax distribution of subword IDs (i.e.,
WordPiece [25]), while the behavior policy is a beam search algorithm based on
the RNN-T model. We configure the behavior policy's beam search algorithm to
emulate ASR inference time, as our training objective is to optimize the ASR
inference process itself.

The training procedure for our model consists of two stages. Firstly, we
conduct regular training of RNN-T model. Once a converged checkpoint is
obtained, we proceed to further fine-tune the model using the RL objective.

Figure 1: The schematic diagram illustrates how beam search generates
hypotheses. Incorrect subwords are highlighted in red and assigned negative
rewards, while correct ones receive positive rewards. Figure 2: This schematic
diagram depicts the process of computing the value for the RL loss, based on
the hypothesis 'helo whyld' generated by beam search decoding.

In the context of our study, the process of beam search involves the use of
labels, ${\mathbf{y}}_{U}$, and audio features, ${\mathbf{x}}_{T}$, to
generate a hypothesis, $\tilde{{\mathbf{y}}}_{U}$, and an associated action
sequence, $\hat{{\mathbf{y}}}_{\hat{U}}$, where
$({\mathbf{y}},\phi)\in\hat{{\mathbf{y}}}$ and $\hat{u}$ represents the index
of the action sequence. This process is illustrated in Figure 2.

Once the hypothesis, $\tilde{{\mathbf{y}}}_{U}$, is obtained, we can compute
the reward, $\bm{r}_{U}$, using the edit distance metric. Specifically, we
define the error, $e_{u}$, as the increase in edit distance resulting from a
given action. If an action does not contribute to an increase in edit
distance, we assign a positive value, $r_{p}$, as the reward. The specific
value of the hyper parameter $r_{p}$ can vary, and we found that $r_{p}=0.1$
performed effectively in the experimental results presented in Section 5.1.

$\bm{r}_{u}=\begin{cases}-e_{u},&\text{if }e_{u}>0\\\
r_{p},&\text{otherwise}\end{cases}$ (1)

In Figure 2, we present a TPU-friendly approach to computing rewards, which
involves a scatter operation to distribute subword IDs into their constituent
characters, followed by computation of the edit distance and error at the
character level. The errors are then gathered into token-level values, which
are used to compute the reward.

Once the reward, $\bm{r}_{u}$, is obtained, we can compute the corresponding
value, $\bm{V}_{\hat{u}}$, using a discount factor, $\gamma$. The reward,
$\bm{r}_{u}$, is assigned only to emission actions (i.e.,
$y_{k}\in{\mathbf{y}}$), whereas the value, $\bm{V}_{\hat{u}}$, is assigned to
both emission and blank actions (i.e.,
$\hat{y_{k}}\in\hat{{\mathbf{y}}}=({\mathbf{y}},\phi)$), which together
constitute the action space. It is noteworthy that the value computed based on
edit distance offers training signals for both emission and blank actions. It
naturally incentivizes or disincentivizes blank actions associated with a good
or bad emission action, respectively. In our experiments presented in Section
5.1, we found that setting $\gamma=0.95$ provided effective results.

$\displaystyle\displaystyle\bm{V}_{\hat{u}}$
$\displaystyle=\bm{r}_{u}+\gamma\bm{V}_{\hat{u}+1}$ (2)

Using the action sequence, $\hat{{\mathbf{y}}}_{\hat{U}}$, and the
corresponding value, $\bm{V}_{\hat{u}}$, we can compute the policy gradient
[26]. In equation (4),
$P_{\theta}(\hat{\mathbf{y}}_{\hat{u}}|\hat{\mathbf{y}}_{\hat{u}-1},{\mathbf{x}}_{T})$
refers to the probability of subword ID by the RNN-T model.

$\displaystyle\nabla_{\theta}J(P_{\theta})=\mathbb{E}_{\tau\sim
P_{\theta}}[\sum^{\hat{U}}_{\hat{u}=0}\nabla_{\theta}\log
P_{\theta}(\hat{\mathbf{y}}_{\hat{u}}|\hat{\mathbf{y}}_{\hat{u}-1},{\mathbf{x}}_{T})V_{\hat{u}}]$
(3)

An approximation for the expectation can be obtained by calculating the sample
mean. We gather a collection of trajectories denoted as
$\mathcal{D}=\\{\tau_{i}\\}_{i=1,...,N}$ where each trajectory is obtained by
beam search using the policy $P_{\theta}$. The estimation of the policy
gradient can be achieved using equation (4), where the number of trajectories
in $\mathcal{D}$ (denoted as $|\mathcal{D}|$ or $N$) corresponds to the number
of top k hypotheses obtained through beam search.

$\displaystyle\nabla_{\theta}J(P_{\theta})\approx\frac{1}{|\mathcal{D}|}\sum_{\tau\in\mathcal{D}}\sum^{\hat{U}}_{\hat{u}=0}\nabla_{\theta}\log
P_{\theta}(\hat{\mathbf{y}}_{\hat{u}}|\hat{\mathbf{y}}_{\hat{u}-1},{\mathbf{x}}_{T})V_{\hat{u}}$
(4)

The high variance of the gradient estimator is a significant challenge for
policy gradient methods [27]. This issue stems, in part, from the challenge of
assigning value to the actions that influenced future rewards. This difficulty
mainly arises due to the high level of noise in the estimated value, making it
challenging to determine whether a given action is beneficial or detrimental.
Notably, this problem has received significant attention in the field of RL
[26, 28, 27]. In contrast, our proposed method does not have this challenge by
computing the value based on edit distance, which enables us to determine
precisely whether a given action is advantageous or disadvantageous. Our
reward proposal assigns negative values to incorrect actions, which is a
relatively uncommon in RL situations.

Since $\bm{V}_{\hat{u}}$ is a value that is detached from the neural network
via stop gradient in equation (4), we can compute the RL loss using the
following formulation:

$\displaystyle\displaystyle\mathcal{L}_{\text{RL}}$
$\displaystyle=-J(P_{\theta})=\frac{1}{N}\sum_{\tau\in\mathcal{D}}\sum^{\hat{U}}_{\hat{u}=0}-\log
P_{\theta}(\hat{\mathbf{y}}_{\hat{u}}|\hat{\mathbf{y}}_{\hat{u}-1},{\mathbf{x}}_{T})V_{\hat{u}}$
(5)

In order to promote stability during training, we incorporate the RL loss with
the RNN-T loss. In our study, we found that setting the weighting factor
$\lambda=0.5$ provided effective results in Section 5.1.

$\displaystyle\displaystyle\mathcal{L}_{\text{total}}$
$\displaystyle=\lambda\mathcal{L}_{\text{RL}}+\mathcal{L}_{\text{RNN-T}}$ (6)

Algorithm 1 outlines all the necessary steps for computing the RL loss.

Algorithm 1 Calculate the Edit distance RL loss

input audio features, ${\mathbf{x}}_{T}$ and labels, ${\mathbf{y}}_{U}$.

1\. Perform beam search to decode the top k hypotheses, as shown in Figure 1.

2\. Compute the reward per token based on the edit distance using equation
(1).

3\. Compute the value per action using equation (2) as shown in Figure 2.

4\. Calculate the RL loss, $\mathcal{L}_{\text{RL}}$, using equation (5).

###  3.2 Inference

Beam search is a commonly used technique for Automatic Speech Recognition
(ASR) models. In our approach, we directly optimize the behavior policy during
inference, which necessitates the use of beam search. Consequently, we use
beam search during the RNN-T decoding process. Our beam search configuration
is set with a beam expansion size of 5 and a top-k value of 4.

##  4 Experiments

###  4.1 Experiment Setup

The Lingvo [29] open-source machine learning framework was employed for all
experiments conducted in this study.

####  4.1.1 Model Architecture

In this study, we employed the W2v-BERT Conformer XL model ($0.6$B) introduced
in [6]. The audio input feature is defined as log mel, comprising 80
dimensions, with a stride of $10ms$ and a window of $25ms$. For the labels,
$1k$ WordPiece subword units [25] were utilized. The RNN-T model functions as
a policy network, with its actions being the subword IDs.

The audio encoder is composed of $24$ Conformer blocks [30] with a model
dimension of $1024$. The self-attention layer is made up of $8$ heads with
$128$ left and right context length, and the convolution kernel size is $5$.
The decoder consists of a 2-layer LSTM label encoder with $2048$ units
projected down to $640$ output units, and a joint network with a single feed-
forward layer with $640$ units. The model has a total of $0.6$B weights.

####  4.1.2 Data

The LibriSpeech dataset [10], which comprises 960 hours of audio and
corresponding human transcriptions, is used for all of our experiments such as
both RNN-T training and RL training.

####  4.1.3 Training procedure

In this study, the baseline model underwent a pre-training phase as described
in the W2v-BERT paper [6], consisting of a two-step process: firstly, W2v-BERT
pretraining using the Libri-Light dataset [31] (60,000 hours), and secondly,
RNN-T finetuning using the LibriSpeech dataset [10] (960 hours). The model was
trained using a batch size of 256 and was found to converge at 15k steps. Once
the model has converged, we then further finetune it using the RL loss defined
in equation (6). The RL finetune was found to converge at an additional 7k
steps. The RNN-T pretraining was conducted on 16 TPU V3 cores [32] for a
duration of half day, followed by RL finetuning on 32 TPU V3 cores for an
additional half day.

###  4.2 Experiment results

We first train the RNN-T baseline model (B0), and then perform further
finetuning using the RL objective (E1). For comparison, we also perform
finetuning using the MWER [19] objective (E2). The EDRL method utilizes an
action-level policy gradient, while MWER utilizes a sequence-level policy
gradient. The results in Table 1 show that our RL method (E1) outperforms the
RNN-T baseline (B0), whereas our RNN-T baseline already outperforms the WER
reported for W2v-BERT [8] using only the LibriSpeech dataset.

In the W2v-BERT paper [8], self-training was further conducted via Noisy
Student Training [33] on the unlabeled Libri-Light dataset [31] of 60,000
hours, resulting in the achievement of a SoTA WER. Our RL model attains a WER
that is similar to the SoTA WER, without leveraging self-training on extensive
unlabeled data.

In contrast, even after an extensive hyperparameter search, MWER [19] was not
able to achieve better WERs than the RNN-T baseline in our setup.

To the best of our knowledge, this is the first RL algorithm that achieves
SoTA performance on the LibriSpeech dataset using an RNNT model.

Table 1: The table displays LibriSpeech WERs without language model (LM) fusion, and demonstrates that our proposed EDRL method outperforms the RNN-T baseline. Model | WER
---|---
dev | dev-other | test | test-other
B0 (RNN-T pretrain) | ${1.4}$ | ${2.7}$ | ${1.5}$ | ${2.7}$
E1 (EDRL, ours) | ${1.4}$ | 2.6 | 1.4 | 2.6
E2 (MWER [19]) | ${1.5}$ | ${2.8}$ | ${1.5}$ | ${2.9}$
W2v-BERT XL [8] | ${1.5}$ | ${2.9}$ | ${1.5}$ | ${2.9}$
Self-training on 60k [8] | ${1.3}$ | ${2.6}$ | ${1.4}$ | ${2.7}$

##  5 Discussion

###  5.1 Ablation study

As we propose new algorithm, we conduct the extensive ablation study.

####  5.1.1 Negative reward

We put forth a negative reward function denoted by $-e_{u}$ in equation (1).
The function implies that each subword ID receives a distinct negative reward
ranging from 1 to the length of the subword. To address concerns surrounding
varying rewards, we assigned a constant reward of -1 for incorrect subword
IDs. However, this approach led to model divergence as it exploited the reward
gap. Specifically, the model began to hallucinate at the end of the utterance
since the maximum negative reward was constrained to -1. Thus, rather than
utilizing ad-hoc reward engineering, we chose to enable the model to undergo
training based on raw edit distance.

####  5.1.2 Positive reward

We conducted a sweep of positive rewards, denoted by $r_{p}$, across the
values (1, 0.9, 0.7, 0.5, 0.3, 0.1, 0.05, 0.01). Our findings indicate that a
positive reward value of $r_{p}=0.1$ demonstrated optimal performance.

####  5.1.3 Discount factor

We found that the discount factor, alongside the learning rate, is one of the
most crucial hyper parameters. We conducted a sweep of discount factor values,
denoted by $\gamma$, ranging from 0 to 0.999, with the specific values (0,
0.1, 0.5, 0.9, 0.93, 0.94, 0.95, 0.96, 0.98, 0.99). Our findings suggest that
a discount factor value of $\gamma=0.95$ produced the best performance.

Table 2: The table presents a comparison of the average LibriSpeech WERs for different discount factors, denoted by $\gamma$. $\gamma$ | Avg WER
---|---
RNN-T | ${2.04}$
$0.0$ | ${2.03}$
$0.1$ | ${2.05}$
$0.5$ | ${2.04}$
$0.9$ | ${2.04}$
$0.93$ | ${2.04}$
$0.94$ | ${2.03}$
$0.95$ | 2.01
$0.96$ | ${2.04}$
$0.98$ | ${2.04}$
$0.99$ | ${2.04}$

####  5.1.4 RL loss weight

We performed a sweep of RL loss weight values, denoted by $\lambda$ in
equation 6, ranging from 0.003 to 1, with the specific values (0.003, 0.1,
0.5, 1). Our results suggest that an RL loss weight of $\lambda=0.5$ produced
the best performance.

####  5.1.5 RNN-T loss weight

We performed a sweep of RNN-T loss weight values, while keeping the RL loss
weight fixed at 0.5, across the range of (0.0, 0.1, 0.2, 0.5, 1.0). Our
results suggest that an RNN-T loss weight of $1.0$ produced the best
performance.

If the RNN-T loss weight is set to $0.0$, the WERs increase to $100\%$ due to
deletion errors, indicating that the RNN-T model generates too many blank
actions. Without the RNN-T loss as guidance, it is challenging for our RL
proposal to differentiate between emission and blank actions. Consequently, we
treat the RL loss as an auxiliary loss to the RNN-T loss, with the RL loss
weight set to $\lambda=0.5$ and the RNN-T loss weight set to $1.0$.

The approach employed by InstructGPT (i.e., ChatGPT) [34] for utilizing RL
loss as an auxiliary loss is analogous. The corresponding weights assigned to
cross-entropy (CE) and reinforcement learning (RL) are 27.8 and 1,
respectively. The self-contained RL objective for the sequence-to-sequence
model is a topic of potential investigation for future research.

###  5.2 Limitation

The motivation behind this study is to narrow the gap between the WER and the
Oracle WER, which represents the best WER among the top-k beam search
hypotheses. As shown in Table 3, the Oracle WER is significantly better than
the regular WER. Therefore, if we can select the oracle hypothesis from the
top-k hypotheses, it would be a significant improvement over the SoTA WERs.
The previous ranking approach [35] involves using audio features, joint
features, labels, and hypotheses as input to rank the top-k hypotheses.
However, this approach only yields marginal improvements in WERs and is not
readily adaptable to streaming and long-form ASR. In this study, we attempt to
address this challenge using RL, but our improved results are still far from
the Oracle WERs.

Table 3: The table displays LibriSpeech WERs and Oracle WER after RNN-T training Model | WER
---|---
dev | dev-other | test | test-other
RNN-T WER | ${1.4}$ | ${2.7}$ | ${1.5}$ | ${2.7}$
Oracle WER | ${0.55}$ | ${1.3}$ | ${0.56}$ | ${1.4}$

##  6 Conclusions

In this paper, we proposed a novel approach for improving the performance of
RNN-T models on speech recognition tasks using reinforcement learning (RL).
Our approach involves directly optimizing beam search for inference time,
which leads to better performance compared to the RNN-T baseline and other
existing approaches. Our approach represents the first successful utilization
of RL for the LibriSpeech dataset, and it has achieved SoTA WERs. Our findings
suggest that RL has the potential to enhance the accuracy of RNN-T models in
speech recognition tasks, paving the way for further research in this area.

## References

  * [1] A. Graves, ``Sequence transduction with recurrent neural networks,'' in _International Conference on Machine Learning: Representation Learning Workshop_ , 2012.
  * [2] T. N. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S.-y. Chang, W. Li, R. Alvarez, Z. Chen _et al._ , ``A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 6059–6063.
  * [3] B. Li, A. Gulati, J. Yu, T. N. Sainath, C.-C. Chiu, A. Narayanan, S.-Y. Chang, R. Pang, Y. He, J. Qin _et al._ , ``A better and faster end-to-end model for streaming asr,'' in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 5634–5638.
  * [4] A. Narayanan, R. Prabhavalkar, C.-C. Chiu, D. Rybach, T. N. Sainath, and T. Strohman, ``Recognizing long-form speech using streaming end-to-end models,'' in _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_. IEEE, 2019, pp. 920–927.
  * [5] D. Hwang, K. C. Sim, Z. Huo, and T. Strohman, ``Pseudo label is better than human label,'' in _INTERSPEECH 2022_ , 2022.
  * [6] Y. Zhang, J. Qin, D. S. Park, W. Han, C.-C. Chiu, R. Pang, Q. V. Le, and Y. Wu, ``Pushing the limits of semi-supervised learning for automatic speech recognition,'' _arXiv preprint arXiv:2010.10504_ , 2020.
  * [7] Q. Xu, A. Baevski, T. Likhomanenko, P. Tomasello, A. Conneau, R. Collobert, G. Synnaeve, and M. Auli, ``Self-training and pre-training are complementary for speech recognition,'' in _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2021, pp. 3030–3034.
  * [8] Y.-A. Chung, Y. Zhang, W. Han, C.-C. Chiu, J. Qin, R. Pang, and Y. Wu, ``W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training,'' _arXiv preprint arXiv:2108.06209_ , 2021.
  * [9] Y. Zhang, D. S. Park, W. Han, J. Qin, A. Gulati, J. Shor, A. Jansen, Y. Xu, Y. Huang, S. Wang _et al._ , ``Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition,'' _IEEE Journal of Selected Topics in Signal Processing_ , 2022.
  * [10] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, ``Librispeech: an asr corpus based on public domain audio books,'' in _IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2015, pp. 5206–5210.
  * [11] W. Chan, D. Park, C. Lee, Y. Zhang, Q. Le, and M. Norouzi, ``Speechstew: Simply mix all available speech recognition data to train one large neural network,'' _arXiv preprint arXiv:2104.02133_ , 2021.
  * [12] V. Pratap, Q. Xu, A. Sriram, G. Synnaeve, and R. Collobert, ``Mls: A large-scale multilingual dataset for speech research,'' _arXiv preprint arXiv:2012.03411_ , 2020.
  * [13] C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina _et al._ , ``State-of-the-art speech recognition with sequence-to-sequence models,'' in _2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2018, pp. 4774–4778.
  * [14] Z. Lu, Y. Pan, T. Doutre, L. Cao, R. Prabhavalkar, C. Zhang, and T. Strohman, ``Input length matters: An empirical study of rnn-t and mwer training for long-form telephony speech recognition,'' _arXiv preprint arXiv:2110.03841_ , 2021.
  * [15] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber, ``Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,'' in _Proceedings of the 23rd international conference on Machine learning_ , 2006, pp. 369–376.
  * [16] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, ``Listen, attend and spell: A neural network for large vocabulary conversational speech recognition,'' in _2016 IEEE international conference on acoustics, speech and signal processing (ICASSP)_. IEEE, 2016, pp. 4960–4964.
  * [17] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, ``Robust speech recognition via large-scale weak supervision,'' _arXiv preprint arXiv:2212.04356_ , 2022.
  * [18] S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer, ``Scheduled sampling for sequence prediction with recurrent neural networks,'' _Advances in neural information processing systems_ , vol. 28, 2015.
  * [19] R. Prabhavalkar, T. N. Sainath, Y. Wu, P. Nguyen, Z. Chen, C.-C. Chiu, and A. Kannan, ``Minimum word error rate training for attention-based sequence-to-sequence models,'' in _2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2018, pp. 4839–4843.
  * [20] R. J. Williams, ``Simple statistical gradient-following algorithms for connectionist reinforcement learning,'' _Reinforcement learning_ , pp. 5–32, 1992.
  * [21] V. Konda and J. Tsitsiklis, ``Actor-critic algorithms,'' _Advances in neural information processing systems_ , vol. 12, 1999.
  * [22] S. Sabour, W. Chan, and M. Norouzi, ``Optimal completion distillation for sequence learning,'' _arXiv preprint arXiv:1810.01398_ , 2018.
  * [23] C. J. Watkins and P. Dayan, ``Q-learning,'' _Machine learning_ , vol. 8, pp. 279–292, 1992.
  * [24] A. Tjandra, S. Sakti, and S. Nakamura, ``End-to-end speech recognition sequence training with reinforcement learning,'' _IEEE Access_ , vol. 7, pp. 79 758–79 769, 2019.
  * [25] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey _et al._ , ``Google's neural machine translation system: Bridging the gap between human and machine translation,'' _arXiv preprint arXiv:1609.08144_ , 2016.
  * [26] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, ``High-dimensional continuous control using generalized advantage estimation,'' in _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2016.
  * [27] C. Wu, A. Rajeswaran, Y. Duan, V. Kumar, A. M. Bayen, S. Kakade, I. Mordatch, and P. Abbeel, ``Variance reduction for policy gradient with action-dependent factorized baselines,'' _arXiv preprint arXiv:1803.07246_ , 2018.
  * [28] H. Van Hasselt, A. Guez, and D. Silver, ``Deep reinforcement learning with double q-learning,'' in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.
  * [29] J. Shen, P. Nguyen, Y. Wu, Z. Chen, M. X. Chen, Y. Jia, A. Kannan, T. Sainath, Y. Cao, C.-C. Chiu _et al._ , ``Lingvo: a modular and scalable framework for sequence-to-sequence modeling,'' _arXiv preprint arXiv:1902.08295_ , 2019\.
  * [30] A. Gulati, J. Qin, C. Chiu, N. Parmar, Y. Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y. Wu, and R. Pang, ``Conformer: Convolution-augmented transformer for speech recognition,'' in _Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Shanghai, China, 25-29 October 2020_ , H. Meng, B. Xu, and T. F. Zheng, Eds. ISCA, 2020, pp. 5036–5040. [Online]. Available: https://doi.org/10.21437/Interspeech.2020-3015
  * [31] J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Q. Xu, P.-E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen _et al._ , ``Libri-light: A benchmark for asr with limited or no supervision,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7669–7673.
  * [32] N. P. Jouppi, C. Young, N. Patil, D. Patterson, G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden, A. Borchers _et al._ , ``In-datacenter performance analysis of a tensor processing unit,'' in _Proceedings of the 44th annual international symposium on computer architecture_ , 2017, pp. 1–12.
  * [33] D. S. Park, Y. Zhang, Y. Jia, W. Han, C.-C. Chiu, B. Li, Y. Wu, and Q. V. Le, ``Improved noisy student training for automatic speech recognition,'' _arXiv preprint arXiv:2005.09629_ , 2020.
  * [34] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray _et al._ , ``Training language models to follow instructions with human feedback,'' _arXiv preprint arXiv:2203.02155_ , 2022.
  * [35] E. Variani, T. Chen, J. Apfel, B. Ramabhadran, S. Lee, and P. Moreno, ``Neural oracle search on n-best hypotheses,'' in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7824–7828.


# User profile-driven large-scale multi-agent learning from demonstration in
federated human-robot collaborative environments

Georgios Th. Papadopoulos1, Asterios Leonidis1, Margherita Antona1 and
Constantine Stephanidis12
<EMAIL_ADDRESS>1Institute of Computer Science
(ICS), Foundation for Research and Technology - Hellas (FORTH), GR-70013,
Heraklion, Crete, Greece2Computer Science Department (CSD), University of
Crete (UoC), GR-70013, Heraklion, Crete, Greece

###### Abstract

Learning from Demonstration (LfD) has been established as the dominant
paradigm for efficiently transferring skills from human teachers to robots. In
this context, the Federated Learning (FL) conceptualization has very recently
been introduced for developing large-scale human-robot collaborative
environments, targeting to robustly address, among others, the critical
challenges of multi-agent learning and long-term autonomy. In the current
work, the latter scheme is further extended and enhanced, by designing and
integrating a novel user profile formulation for providing a fine-grained
representation of the exhibited human behavior, adopting a Deep Learning
(DL)-based formalism. In particular, a hierarchically organized set of key
information sources is considered, including: a) User attributes (e.g.
demographic, anthropomorphic, educational, etc.), b) User state (e.g. fatigue
detection, stress detection, emotion recognition, etc.) and c)
Psychophysiological measurements (e.g. gaze, electrodermal activity, heart
rate, etc.) related data. Then, a combination of Long Short-Term Memory (LSTM)
and stacked autoencoders, with appropriately defined neural network
architectures, is employed for the modelling step. The overall designed scheme
enables both short- and long-term analysis/interpretation of the human
behavior (as observed during the feedback capturing sessions), so as to
adaptively adjust the importance of the collected feedback samples when
aggregating information originating from the same and different human
teachers, respectively.

##  I Introduction

Learning from Demonstration (LfD) constitutes the most dominant paradigm for
robot programming through direct interaction with humans [1]. It relies on the
fundamental principle of robots acquiring new skills by learning to imitate a
(human) teacher [2]. LfD has so far been successfully associated with multiple
aspects of robotics technology, including human-robot interaction, machine
learning, machine vision and motor control [3][4]. The main advantageous
characteristics that have led to the widespread adoption of LfD in multiple
and diverse application cases are [5]: a) It enables robot programming by non-
expert users, b) It allows time-efficient learning, c) It enables adaptive
robotic behaviors and d) It facilitates the robots to operate in complex and
time-varying environments.

With respect to the different types of human demonstration means considered,
LfD approaches generally fall into three main categories [2]:

  * •

Kinesthetic teaching, where the human teacher provides demonstrations by
physically moving the robot through the desired motions [6];

  * •

Teleoperation, where an external input (e.g. joystick, graphical user
interface or other means) is used to guide the robot [7];

  * •

Passive observation, where the robot passively observes the exhibited user
behavior for learning [8].

A critical factor for the success of any LfD scheme, while also in close
relation with the particularities of the selected application case, concerns
the adopted methodology for refining the robot learned policies. Different
types of approaches have been introduced so far, including: a) Reinforcement
learning, where a trial and error methodology is adopted to learn a policy to
solve a given problem [9], b) Optimization, where the optimal solution is
searched based on given criteria [10], c) Transfer learning, where knowledge
of a task or a domain is used to enhance the learning procedure for another
task [11], d) Apprenticeship learning, where the desired performance is
modeled through the use of a set of demonstrated samples that serve as a
template [12], e) Active learning, where the robot decides when to ask for an
optimal response from a (human) expert to a given state and to use these
active samples to improve its policy [13] and f) Structured predictions, where
robotic actions are considered as a sequence of dependent predictions [14].

Although extensive research efforts have been devoted in the LfD field over
the past years, crucial research challenges, namely multi-agent learning [15]
and long-term autonomy [16], still remain to be reliably addressed. Achieving
the latter will act as a tremendous facilitator for enabling the wide-spread
use of robots in large-scale, open and complex environments; this is the
typical case when considering e.g. industrial manufacturing scenarios. The
above need to be also investigated in conjunction with typical challenges in
Human-Robot Interaction (HRI) schemes, like developing appropriate user
interfaces, variance in human performance, variability in knowledge across
human subjects, learning from noisy/imprecise human input, learning from very
large or very sparse datasets, incremental learning, etc.

The current work adopts the Federated Learning (FL) conceptualization that has
very recently been introduced for designing large-scale LfD human-robot
collaborative environments [17], aiming at providing reliable solutions, among
others, to the current critical challenges of multi-agent learning and long-
term autonomy. In particular, the cognitive architecture of [17] is further
elaborated and enhanced, by designing and integrating a novel user profile
formulation for estimating a fine-grained representation of the exhibited
human behavior. The introduced user model follows the Deep Learning (DL)-based
formalism; hence, rendering the overall approach end-to-end learnable.
Specifically, a set of key information sources that are hierarchically
organized is considered, including: a) User attributes concerning each human
teacher’s background (e.g. demographic, anthropomorphic, educational, etc.),
b) User state information that encodes particular human behavioral
characteristics (e.g. fatigue detection, stress detection, emotion
recognition, etc.) and c) Psychophysiological measurements that convey
critical details about the user’s mental state, personality and idiosyncrasy
(e.g. gaze, electrodermal activity, heart rate, etc.). Then, a combination of
Long Short-Term Memory (LSTM) and stacked autoencoders, with appropriately
defined neural network architectures, is employed for the modelling step and
for producing a concrete user profile representation. The overall designed
scheme supports both short- and long-term analysis/interpretation of the human
behavior (as observed during the feedback capturing sessions), so as to
adaptively modulate the importance of the collected feedback samples when
aggregating information originating from the same and different human
teachers, respectively.

The remainder of the paper is organized as follows: Section II describes
challenges and open issues currently present in the LfD learning field.
Additionally, Section III outlines the already introduced cognitive
architecture for FL-based multi-agent human-robot collaborative learning.
Moreover, Section IV details the designed DL-based user profile modelling and
knowledge aggregation approach. Finally, conclusions are drawn in Section V.

##  II Open issues in LfD learning

LfD has been shown to lead to significant advances in multiple and
significantly diverse operational scenarios, involving the use of both
manipulator [18][19] and mobile [20][21] robots. Regarding the targeted goal
of the overall LfD procedure, this can be categorized to different levels of
abstraction [2], including the learning of: a) Policies, i.e. the estimation
of a function that generates the desired behavior, b) Cost and reward
functions, where the ideal behavior is considered to stem out from the
optimization of a hidden function, c) Plans, i.e. high level structured
schemes, composed of several sub-tasks or primitive actions, and d) Multiple
outcomes simultaneously, by jointly modeling complex behaviors at multiple
levels of abstraction.

As already mentioned in Section I, despite the extensive research efforts
devoted by the LfD community, key technological challenges and open issues
still remain to be robustly addressed, including, among others, the following
ones [2][15][5][22]:

  * •

To involve in a more robust way a broad number of teachers with different
styles of and possibly conflicting demonstrations;

  * •

To transfer skills across multiple agents, including multiple and diverse
types of robots;

  * •

To simultaneously learn multiple complex tasks, while storing and reusing
prior knowledge at large scale;

  * •

To robustly implement incremental learning schemes;

  * •

To reinforce the generalization ability;

  * •

To model compound tasks;

  * •

To implement multi-agent imitation learning schemes;

  * •

To operate in realistic, dynamic, time-varying and complex environments.

Towards providing a reliable solution to the above aspects, a novel cognitive
architecture for multi-agent LfD robotic learning has recently been introduced
[17], targeting to enable the efficient deployment of open, scalable and
expandable robotic systems in large-scale and complex environments. In
particular, the designed architecture capitalizes on the recent advances in
the Artificial Intelligence (AI) (and especially the DL field), by
establishing a FL-based framework for incarnating a multi-human multi-robot
collaborative learning environment (as will be discussed in Section III),
going far beyond other literature approaches investigating relatively
straight-forward implementations of FL schemes (without incorporating the
human factor in the learning loop) in specific robotic tasks (e.g. autonomous
navigation [23], motion planning [24], visual perception [25], etc.).

However, the conventional FL mechanism, which is a purely data-driven scheme
and typically employs a simple ‘Federated Averaging’ operator for knowledge
aggregation [26], lacks theoretical guarantees for convergence under realistic
settings [27], i.e. when non-independent-and-identically-distributed (non-IID)
data are available at the various network nodes. The latter assumption is
highly likely to hold when considering human demonstration data (as required
in the LfD scheme). In this context, the current work elaborates the multi-
agent cognitive architecture conceptualization of [17], by introducing
comprehensive DL-grounded user profile modelling and corresponding
sophisticated knowledge aggregation strategies (explained in Section IV).

##  III FL-based multi-agent human-robot collaborative learning

###  III-A Fundamental conceptualization

According to [17], a multi-agent cognitive architecture for LfD learning is
introduced that is grounded on the fundamental mechanism of the FL paradigm,
while a high-level representation is provided in Fig. 1. In particular, a
global AI model W (that bears the targeted robot policy to be learned), which
is aimed to be collaboratively trained and shared among a network of human-
robot pairs, is initially constructed using proxy data (stored either offline
or at a central node). Model W materializes a cognitive process [28][29]
underpinning the robotic task of interest, e.g. sensing, navigation,
manipulation, control, human-robot interaction, etc. Subsequently, the model
is made available to the network and downloaded by each node (i.e. every
employed robot). Every node encapsulates a local database that is used to
estimate improved updates of the global model parameters (e.g. using
conventional Stochastic Gradient Descent (SGD) for the case of Neural Network
(NN)-based AI modules), without making the local (federated) data available to
the network (i.e. being a by-definition privacy-aware method). The computed
local parameter updates (denoted $\Delta\textbf{W}^{l}$ in Fig. 1) are
asynchronously transmitted back to the central node (using encrypted
communication), where an aggregation mechanism is responsible for combining
them and periodically producing a new version of the global model. The overall
process is iterative, i.e. continuously estimating improved versions of the
global model. It needs to be highlighted that each local computational node
can maintain a customized version of the global model, while using the locally
stored data for fine-tuning purposes. Moreover, in the current work a single
robot policy (unique global model W) is estimated for simplicity purposes,
while the original version of the cognitive architecture of [17] considers the
multi-task learning scenario.

Figure 1: High-level representation of the employed cognitive architecture for
LfD learning.

###  III-B Constituent entities

The considered cognitive AI architecture is composed of the following real-
world reference entities: a) The robotic platform and b) The human. In order
to enable the perception and the collection of critical information about the
surrounding environment and the exhibited human behavior, two comprehensive
sets of sensing devices are considered; one that can be attached to the robot
(denoted $S^{r}$) and one to the human (denoted $S^{h}$). In particular, the
set of potentially supported types of sensors for each case, which can be
significantly broad and also depends on the particular application scenario,
are defined as follows:

$\displaystyle S^{r}=\\{s^{r}_{i}|i\in[1,I],i\in\mathbb{N},I\in\mathbb{N}\\}$
$\displaystyle=\\{Vision,Light,Temperature,Chemical,Force,$ $\displaystyle
Acoustic,Gas,Motion,Pressure,Position,...\\}$ (1) $\displaystyle
S^{h}=\\{s^{h}_{k}|k\in[1,K],k\in\mathbb{N},K\in\mathbb{N}\\}$
$\displaystyle=\\{Gaze,Heart~{}rate,Stress~{}level,Motion,$ $\displaystyle
Fatigue,Position,EEG,...\\}$ (2)

Regarding the robotic platform, this refers to the actual mechatronic
equipment to be deployed. Depending on the specific operational scenario and
the targeted robotic task (that may cover a large set of possible perception,
cognition, motor and interaction functionalities), multiple types of robots
can be used, supporting different requirements for mobility, positioning,
manipulation, communication, size, payload, etc. The set of available types of
robots is denoted:

$\displaystyle R=\\{r_{j}|j\in[1,J],j\in\mathbb{N},J\in\mathbb{N}\\}$
$\displaystyle=\\{Arm,AGV,Humanoid,UAV,Vehicle,Industrial,...\\}$ (3)

Taking into account the above-mentioned formalisms, a robotic platform $P_{l}$
can be fully specified as follows:

$\displaystyle P_{l}=\\{S^{r}_{l},R_{l}|S^{r}_{l}\subseteq
S^{r},R_{l}\subseteq R\\},l\in[1,L],$ (4)

where $L$ denotes the total number of robotic platforms present in the
examined cognitive environment.

With respect to the human factor, although all types of LfD demonstration
means methodologies (namely kinesthetic teaching, teleoperation and passive
observation, as detailed in Section I) are supported by the cognitive AI
architecture, the adoption of a teleoperation-based approach is considered to
exhibit significant advantageous characteristics. In particular, an intuitive
and sophisticated teleoperation scheme is foreseen that is based on the
combined used of Augmented Reality (AR) visualization mechanisms and
eXplainable AI (XAI) technologies. Regarding the former, AR tools are employed
in order to allow the human user to perform a physical inspection of the robot
exhibited behaviors (and hence to identify malfunctions, hazardous situations,
deviations from desired policies, etc.), while at the same time being
constantly provided with key detailed insights about the AI processes being
applied (e.g. the AI modules being used, their estimated outputs, how specific
decisions are reached, etc.). Concerning XAI methods, such techniques are
adopted in order to provide precise explanations/insights to the human
operator regarding the deviation of the robot behavior from the desired one,
i.e. enabling an in depth inspection of the robot behavior. It needs to be
mentioned that the set of human operators involved in the designed cognitive
architecture is denoted
$H=\\{h_{m}|m\in[1,M],m\in\mathbb{N},M\in\mathbb{N}\\}$.

###  III-C Collective cognitive AI layer

A collective cognitive AI layer operates at the upper level of the designed
architecture and its core functionality is based on the fundamental mechanism
of the FL paradigm, as detailed in Section III-A. The main building blocks of
the cognitive AI layer, their formalism and detailed explanation of their
functionalities are provided in the followings:
Network nodes: Every robotic platform $P_{l}$ corresponds to a network node of
the defined architecture. Each $P_{l}$ stores locally the generated data,
which largely contain information collected from the set of sensors
$S^{r}_{l}$ incorporated by $P_{l}$.
Global AI model: The goal of the overall framework is to collectively create a
global AI model denoted $\textbf{W}\leftarrow f_{1}(S^{r},R)$, underpinning
the implemented cognitive process present in the examined environment, where
$f_{1}(.)$ implies a generalized function or process that defines the exact
NN-based materialization of model W while considering $S^{r}$ and $R$ as input
parameters. However, each network node $P_{l}$ can maintain a local/customized
version of W.
Learning methodology: Regarding the specific methodology to be followed for
refining the robot learned policies (i.e. for updating AI model W), different
options can be investigated (e.g. reinforcement learning, transfer learning,
active learning, etc.), as discussed in Section I. The most suitable selection
depends on the particularities of the application domain and the examined W.
Local parameter updates: Regardless of the particular learning methodology
selected, each robotic platform $P_{l}$ can estimate updates for model W,
using its locally stored data. More specifically, the following local
parameter update mechanism is applied:

$\displaystyle\textbf{W}^{l,m}\leftarrow\textbf{W}^{l,m}-lr^{l}\cdot\nabla\mathcal{L}(\textbf{W}^{l,m},\delta^{l}),$
(5)

where $\textbf{W}^{l,m}$ is the local/customized version of the global model W
with respect to human teacher $h_{m}$, $lr^{l}$ is the local learning rate,
$\nabla$ denotes the gradient of a function, $\mathcal{L}(.)$ represents the
loss function defined for W and $\delta^{l}$ is the locally stored dataset.
Consequently, the local parameter updates, which are iteratively estimated, to
be sent to the central node (aggregator) are computed as follows:

$\displaystyle\Delta\textbf{W}^{l,m}=\textbf{W}^{l,m}-\textbf{W}$ (6)

User profiling: For efficiently initially interpreting and subsequently
incorporating human feedback, in parallel with the $\textbf{W}^{l,m}$
estimation process, an individual user profile $\textbf{Q}^{l,m}\leftarrow
f_{2}(\delta^{l},S^{h}_{m})$ is constructed for every human subject $h_{m}$ at
every node $P_{l}$. The latter is estimated using the appropriate sensorial
data $S^{h}_{m}$ to model the observed human behavior and a generalized
function $f_{2}(.)$ that defines the exact (NN-based) implementation of the
user profile while considering $S^{h}_{m}$ as input parameters. Under the
current conceptualization, DL-based approaches are considered for generating
$\textbf{Q}^{l,m}$, as in [30] and [31], aiming at combining increased
modelling capabilities and easier integration to the designed FL-based
framework.
Global model update: Having computed the local parameter updates
$\Delta\textbf{W}^{l,m}$ and the corresponding user profiles
$\textbf{Q}^{l,m}$ (using locally generated and processed data $\delta^{l}$),
these are sent to the central node (aggregator) so as to periodically produce
updated versions of global model W. According to the conventional FL
mechanism, an updated version $\textbf{W}^{\prime}$ of W is generated on a
regular basis, by applying a simple averaging operator, as follows:

$\displaystyle\textbf{W}^{\prime}=\textbf{W}+lr^{g}\cdot\boldsymbol{\Gamma}$
(7)
$\displaystyle\boldsymbol{\Gamma}=\frac{1}{\Lambda}\sum_{(l,m)}\Delta\textbf{W}^{l,m},$
(8)

where $lr^{g}$ denotes the global learning rate and $\Lambda$ the total number
of received $\Delta\textbf{W}^{l,m}$ updates. For robustly confronting the
observed variance in the behavior of the large number of involved human
teachers $h_{m}$, the designed cognitive architecture (apart from possible
sensorial data $S^{r}_{l}$ pre-processing for invariance incorporation)
encompasses the estimated user profiles $\textbf{Q}^{l,m}$ in the global model
update process, modifying (8) to the following general formalism:

$\displaystyle\boldsymbol{\Gamma}=\Phi(\\{\textbf{Q}^{l,m}\\},\\{\Delta\textbf{W}^{l,m}\\}),$
(9)

where $\Phi(.)$ denotes a generalized function that combines the available
$\textbf{Q}^{l,m}$ and $\Delta\textbf{W}^{l,m}$, estimated at every network
node $P_{l}$. Depending on the particularities of the selected application
domain (e.g. supported W, $S^{r}$, $S^{h}$, $R$, etc.), the following main and
purely data-driven materializations of $\Phi(.)$, which are capable of being
combined, are considered in [17]: a) User weighting, where each human teacher
$h_{m}$ is modulated by a different weight factor based on a similarity metric
from the global user model, b) Parameter weighting, which assigns varying
weights to the parameters of W, based on the degree of correlation among the
parameters of $\textbf{W}^{l,m}$ and $\textbf{Q}^{l,m}$, and c) User
clustering, which considers the generation of multiple instances
$\textbf{W}_{\eta}$ of the global model W, in order to better capture the
heterogeneity of data distributions across the different users $h_{m}$. In the
remainder of this work the above-mentioned ‘User weighting’ conceptualization
is considered and the focus is put on thoroughly defining an efficient
$\textbf{Q}^{l,m}$ formalism (Section IV), although ‘Parameter weighting’ and
‘User clustering’ (as well as any combination of the three alternatives) are
equally applicable. In particular, the ‘User weighting’ approach adaptively
adjusts the contribution of each human teacher $h_{m}$ by a different weight
factor based on his/her exhibited behavior, as follows:

$\displaystyle\boldsymbol{\Gamma}=\frac{1}{E}\sum_{(l,m)}\varepsilon(\textbf{Q}^{l,m})\cdot\Delta\textbf{W}^{l,m}$
$\displaystyle\varepsilon(\textbf{Q}^{l,m})=1/\lVert\textbf{Q}^{g}-\textbf{Q}^{l,m}\rVert$
$\displaystyle E=\sum_{(l,m)}\varepsilon(\textbf{Q}^{l,m}),$ (10)

where $\varepsilon(\textbf{Q}^{l,m})$ denotes the weight factor for each
$h_{m}$ at every $P_{l}$, $\textbf{Q}^{g}$ the corresponding global user model
(e.g. estimated through the same FL-based mechanism used for constructing W)
and $\lVert.\rVert$ a dissimilarity score metric (e.g. Euclidean distance
between the parameters of the involved user models).

Figure 2: Hierarchy of information sources for generating user profile
$\textbf{Q}^{l,m}$.

##  IV DL-based user modelling and knowledge aggregation

Incorporating human feedback constitutes a fundamental part of the LfD
approach and, consequently, of the employed cognitive architecture. However,
the latter poses additional challenges to the problem formulation that need to
be efficiently addressed (e.g. variance in human performance, variability in
knowledge across human subjects, learning from noisy/imprecise human input,
possible conflicts in provided human feedback information, handling of
individuals with different idiosyncrasies, learning from very large or very
sparse datasets, incremental learning, etc.). On the other hand, the strength
of the utilized FL-based mechanism (detailed in Section III) lies on
incorporating a very large number $\Lambda$ of samples and multiple iterative
updates of $\textbf{W}^{\prime}$ that will likely lead to the convergence to
well-performing and robust W models, while the network nodes $P_{l}$
contributing in (8) may be sampled out of the available ones (usually in a
random way). However, combining information ($\Delta\textbf{W}^{l,m}$) related
to different human subjects $h_{m}$ (that presumably exhibit highly diverse
and varying behavior) is in turn very likely to lead the FL process to be
confined to a local maximum or even to a non-convergence of the FL procedure;
hence, jeopardising the overall learning process. To this end, defining a
detailed and robust user profile model $\textbf{Q}^{l,m}$ is of vital
importance for enabling: a) The accurate assessment and interpretation of the
demonstrated behavior of each human teacher $h_{m}$ and, subsequently, b) The
appropriate adjustment of the importance of the provided human feedback from
the same individual or across multiple ones, during the knowledge aggregation
step.

Figure 3: Employed LSTM autoencoder for producing psychophysiological
measurements representation ${\bf{DR}}^{l,m}_{\nu,\rho}$.

In general, the purpose of defining user profile formalisms is to cover
physical (e.g. human actions, physical capabilities, etc.), cognitive (e.g.
intention, personality, etc.) and social (e.g. non-verbal cues, emotions,
etc.) aspects, in order to model and efficiently interpret the exhibited human
activity [32]. So far, the problem has been investigated and assessed from
different disciplines and scientific perspectives, while methodologies that
rely on the use of Machine Learning (ML) techniques and the integration of the
generated models in broader computational/cognitive systems continuously
receives increasing attention. One of the most popular examples of the latter
category of methods constitutes the so called recommender systems (that aim to
predict the preferences of each individual human user) [33][34] and especially
the so-called ‘collaborative filtering’ class [35]. However, the introduced
profiles so far model relatively simple types of human behavior aspects (e.g.
correlations of individual users and different information items [36]).

###  IV-A Considered information sources

Under the proposed conceptualization, a fine-grained multi-level DL-based
computational approach is introduced for modelling user profile
$\textbf{Q}^{l,m}$, grounded on the efficient processing and sophisticated
combination of various heterogeneous and diverse information streams. The
overall goal is to meet the demanding requirements of the LfD setting for
modelling the exhibited user behavior, focusing on the following two main
pillars: a) Accurately interpreting the captured human behavior both in the
short- and the long-term cases and b) Effectively handling the variance of
cross-subject (and often conflicting) information and its efficient merging.
For that purpose, the following (hierarchically organized) types of
information sources are considered for generating $\textbf{Q}^{l,m}$ (as also
illustrated in Fig. 2):

  * •

User attributes: These include valuable and explicitly defined information
concerning each human teacher’s $h_{m}$ background, covering aspects like
demographic (e.g. age, sex, race, origin, etc.), anthropomorphic (e.g. height,
weight, possible disabilities, etc.), educational (e.g. level and types of
education, etc.), work experience (e.g. duration, responsibilities/tasks,
etc.), etc.;

  * •

User state: This corresponds to particular modules/pipelines that
(automatically) detect specific human behavior characteristics, e.g. fatigue
detection, stress detection, emotion recognition, intention prediction, etc.
For their extraction sensorial data types belonging to $S^{r}$ and $S^{h}$ are
used;

  * •

Psychophysiological measurements: These comprise the actual raw sensorial data
information streams captured (supporting data types of both $S^{r}$ and
$S^{h}$ sets) and convey critical details about the user’s mental state,
personality and idiosyncrasy, e.g. heart rate, electroencephalogram (EEG),
motion, facial expressions, gaze, accelerometer, gyroscope, electrodermal
activity, etc.

It needs to be highlighted that the specific information sources (belonging to
each of the above mentioned categories) to be considered depends on the
particularities of the selected application case.

###  IV-B Processing of collected data

According to the formalism of the cognitive architecture for LfD learning
defined in Section III, each human teacher $h_{m}$ can provide, in relation to
a given robotic platform $P_{l}$, feedback information (denoted
$\mu^{l,m}_{\nu}$) regarding the robot exhibited behavior, where $\nu$
($\nu\in[1,N],\nu\in\mathbb{N},N\in\mathbb{N}$) indicates the index of the
particular feedback session (denoted $\xi^{l,m}_{\nu}$) that consists of
$T_{\nu}$ time steps. The user attributes provided for human $h_{m}$, after
being appropriately numerically encoded, are concatenated to form static
feature vector ${\bf{DE}}^{m}$. On the other hand, the outputs of the specific
feature detectors (that summarize the user’s state) for human $h_{m}$ at node
$P_{l}$ during session $\xi^{l,m}_{\nu}$, after also being appropriately
numerically encoded (if needed), are similarly concatenated to produce static
feature vector ${\bf{DS}}^{l,m}_{\nu}$. With respect to the corresponding
psychophysiological measurements captured during session $\xi^{l,m}_{\nu}$, an
individual multi-dimensional observation sequence $OS^{l,m}_{\nu,\rho}$ (of
length $T_{\nu}$ samples) is formed for every considered sensor type implied
by index $\rho$. In order to estimate a static feature vector
${\bf{DR}}^{l,m}_{\nu,\rho}$ (out of each $OS^{l,m}_{\nu,\rho}$ sequence), a
DL-based unsupervised feature representation learning methodology is
considered. Specifically, an individual LSTM autoencoder, similar to the one
described in [37] and whose architecture is illustrated in Fig. 3, is
introduced for each $OS^{l,m}_{\nu,\rho}$, since every $OS^{l,m}_{\nu,\rho}$
is in general related to a different sampling frequency and, hence, consists
of a different number of samples. As can be seen in Fig. 3, the autoencoder
consists of two LSTM models, namely an encoder and a decoder one. The encoder
receives as input a series of feature vectors (i.e. ${\bf{u}}_{1}$,
${\bf{u}}_{2}$, … ${\bf{u}}_{T}$) and updates at each time step its vectorial
fixed-length internal state representation (i.e. ${\bf{h}}_{1}$,
${\bf{h}}_{2}$, … ${\bf{h}}_{T}$), which models/encodes the input sequence
until that time step. During the decoding phase, the second LSTM is
initialized with internal state ${\bf{h}}^{\prime}_{1}\equiv{\bf{h}}_{T}$ and,
through its fundamental sequential modelling mechanism, it recursively
estimates new internal states (i.e. ${\bf{h}}^{\prime}_{1}$,
${\bf{h}}^{\prime}_{2}$, … ${\bf{h}}^{\prime}_{T}$) that correspondingly
produce a respective series of prediction feature vectors (i.e.
${\bf{\widehat{u}}}_{1}$, ${\bf{\widehat{u}}}_{2}$, …
${\bf{\widehat{u}}}_{T}$). The overall goal of the autoencoder is to adjust
its internal parameters so that sequences ${\bf{u}}_{1}$, ${\bf{u}}_{2}$, …
${\bf{u}}_{T}$ and ${\bf{\widehat{u}}}_{1}$, ${\bf{\widehat{u}}}_{2}$, …
${\bf{\widehat{u}}}_{T}$ to be as similar as possible. Vector ${\bf{h}}_{T}$
that characterizes the overall input sequence ${\bf{u}}_{1}$, ${\bf{u}}_{2}$,
… ${\bf{u}}_{T}$ is considered to coincide with ${\bf{DR}}^{l,m}_{\nu,\rho}$
defined above.

###  IV-C Modeling of user profile

The ultimate goal of designing a model of the human user profile
$\textbf{Q}^{l,m}$ (Section III) is to provide a computational scheme that
will simultaneously analyse, cross-correlate and eventually fuse the different
heterogeneous information streams for producing a joint/compact representation
that will efficiently encode/express the salient characteristics of each human
teacher $h_{m}$ (e.g. mental state, idiosyncrasy, preferences, etc.) in an
implicit way. The latter is of paramount importance for interpreting the human
behavior during the knowledge aggregation step, as will be discussed in
Section IV-D.

For creating a robust user profile, a DL-based hierarchical multi-stream
unsupervised feature fusion approach is adopted. In particular, a multi-stream
stacked autoencoder, similar to the one described in [38] and whose general
architecture is illustrated in Fig. 4, is introduced. Each of the $3$ defined
autoencoder streams is in direct accordance to the information sources
definition of Section IV-A. In particular, a stacked autoencoder is a neural
network consisting of several layers of sparse autoencoders, where the output
of each hidden layer is connected to the input of the successive hidden layer.
The designed autoencoder considers a multi-stream architecture, which aims at
initially capturing the correlations among the features of the same stream
(while also leveraging knowledge from the other streams) and subsequently
modelling the inter-dependencies among the different streams, as can been seen
in Fig. 4. Similarly to the LSTM autoencoder (Section IV-B), the goal during
the training process is for the input and output feature vectors to be as
similar as possible. Moreover, the learned multi-modal representation (feature
vector) of the middle/internal encoding layer compactly summarizes the salient
attributes of the input feature vectors.

Figure 4: Designed stacked autoencoder for generating user profile model
$\textbf{Q}^{l,m}$.

Regarding the actual user profile generation process, an individual stacked
autoencoder is considered with respect to every $\textbf{Q}^{l,m}$. The
autoencoder receives as input (with respect to each associated feedback
session $\xi^{l,m}_{\nu}$) the following feature vectors: a) User attributes
vector ${\bf{DE}}^{m}$, b) User state vector ${\bf{DS}}^{l,m}_{\nu}$, and c)
Concatenation of psychophysiological measurement vectors
${\bf{DR}}^{l,m}_{\nu,\rho}$. The estimated state representation of the
encoding layer (denoted ${\bf{HS}}^{l,m}_{\nu}$) comprises a static feature
vector that summarizes the salient behavioral characteristics of human teacher
$h_{m}$ at node $P_{l}$ during session $\xi^{l,m}_{\nu}$. Moreover, the
parameters of the learned stacked autoencoder (or practically only those of
the encoder part) constitute the actual user profile model $\textbf{Q}^{l,m}$
introduced in the FL-based formalization of Section III.

###  IV-D Knowledge aggregation

Having discussed the development of detailed user behavior profiles (Section
IV-C), the challenging issue of analysing and aggregating human knowledge (in
the form of feedback demonstrations in the LfD setting) is investigated in
this section. The overall conceptualization concentrates on two main axes with
respect to the interpretation of the exhibited human behavior: a) Short-term
analysis for combining information cues originating from the same human
teacher and b) Long-term analysis for integrating knowledge derived from
different individuals.

Regarding the short-term analysis, this targets to in principle interpret the
observed human behavior when providing feedback information $\mu^{l,m}_{\nu}$
during session $\xi^{l,m}_{\nu}$. The goal is to adaptively adjust the
importance of each individual feedback sample $\mu^{l,m}_{\nu}$ when updating
the local/personalized model $\textbf{W}^{l,m}$ for human teacher $h_{m}$ at
node $P_{l}$, by taking into account user profile related information
estimated during each session $\xi^{l,m}_{\nu}$. For achieving that, the loss
function $\mathcal{L}(\textbf{W}^{l,m},\delta^{l})$ defined in (5), which is
used for updating local model $\textbf{W}^{l,m}$, is formalized as follows:

$\displaystyle\mathcal{L}(\textbf{W}^{l,m},\delta^{l})=\frac{\sum_{\nu}\omega^{l,m}_{\nu}\cdot\mathcal{L}_{\nu}(\mu^{l,m}_{\nu},\textbf{W}^{l,m})}{\sum_{\nu}\omega^{l,m}_{\nu}}$
$\displaystyle\omega^{l,m}_{\nu}=1/\lVert{\bf{HS}}^{l,m}_{\nu}-{\overline{\bf{HS}}}^{l,m}_{\nu}\rVert,$
(11)

where $\mathcal{L}_{\nu}(\mu^{l,m}_{\nu},\textbf{W}^{l,m})$ is the loss factor
for sample $\mu^{l,m}_{\nu}$, $\omega^{l,m}_{\nu}$ its associated weight
factor, ${\overline{\bf{HS}}}^{l,m}_{\nu}$ the mean value of
${\bf{HS}}^{l,m}_{\nu}$ (calculated while considering all available
$\mu^{l,m}_{\nu}$ samples) and $\lVert.\rVert$ a dissimilarity score metric
(e.g. Euclidean distance). $\omega^{l,m}_{\nu}$ assigns varying importance to
each $\mu^{l,m}_{\nu}$, contrary to the conventional practice of associating
all $\mu^{l,m}_{\nu}$ samples with equal weight (i.e. $\omega^{l,m}_{\nu}=1$).
The motivation behind the latter choice lies on the fundamental consideration
that observed human behaviors ${\bf{HS}}^{l,m}_{\nu}$ that deviate
significantly from the expected personalized norm
${\overline{\bf{HS}}}^{l,m}_{\nu}$ (i.e. outliers) should receive decreased
importance, since it is assumed that they are likely to be caused by
undesirable user profile states (e.g. when in extreme situations of fatigue,
stress, decreased attention, etc.).

With respect to the long-term analysis, the stacked autoencoder computational
model $\textbf{Q}^{l,m}$ described in Section IV-C is used in the ‘User
weighting’ scheme defined in (10), where $\textbf{Q}^{l,m}$ that are similar
to the respective global $\textbf{Q}^{g}$ user profile receive increased
impact. Similarly to the above, the explanation of the latter states that
model updates $\Delta\textbf{W}^{l,m}$ ((6)) should receive greater importance
proportionally to the associated user profiles $\textbf{Q}^{l,m}$ being
similar to the global norm or expected behavior $\textbf{Q}^{g}$. In other
words, human teacher behaviors ($\textbf{Q}^{l,m}$) that deviate significantly
from the global norm ($\textbf{Q}^{g}$) are considered to be due to frequent
undesirable user profile states, which could very likely in turn jeopardize
the convergence of the overall FL-based learning scheme; hence, decreasing
their impact would facilitate and reinforce the robustness of the overall LfD
learning environment. It needs to be highlighted that in the original work of
[17] no specific materialization of $\textbf{Q}^{l,m}$ is provided.

##  V Conclusions

In this paper, the recently introduced Federated Learning (FL)-based
conceptualization for developing large-scale multi-agent Learning from
Demonstration (LfD) human-robot collaborative environments was further
elaborated and enhanced, by incorporating a novel Deep Learning (DL)-based
user profile formulation for estimating a fine-grained representation of the
exhibited human behavior. The overall integrated scheme supports both short-
and long-term analysis/interpretation of the observed human teacher behavior,
in order to adaptively adjust the importance of the collected feedback samples
when aggregating information originating from the same and different human
individuals, respectively. Future work includes the implementation and
evaluation of the designed framework in real-world settings, the development
of more sophisticated user profile models and the incorporation of more
elaborate DL formalisms (e.g. (self-) attention schemes, transformer networks,
etc.).

## ACKNOWLEDGMENTS

The work presented in this paper was supported by the ICS-FORTH internal RTD
Programme ‘Ambient Intelligence and Smart Environments’.

## References

  * [1] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and autonomous systems_ , vol. 57, no. 5, pp. 469–483, 2009.
  * [2] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, 2020.
  * [3] D. Furuta, K. Kutsuzawa, S. Sakaino, and T. Tsuji, “Motion planning with success judgement model based on learning from demonstration,” _IEEE Access_ , vol. 8, pp. 73 142–73 150, 2020.
  * [4] I. A. Seleem, H. El-Hussieny, S. F. Assal, and H. Ishii, “Development and stability analysis of an imitation learning-based pose planning approach for multi-section continuum robot,” _IEEE Access_ , 2020.
  * [5] A. G. Billard, S. Calinon, and R. Dillmann, “Learning from humans,” in _Springer handbook of robotics_. Springer, 2016, pp. 1995–2014.
  * [6] R. Caccavale, M. Saveriano, A. Finzi, and D. Lee, “Kinesthetic teaching and attentional supervision of structured tasks in human–robot interaction,” _Autonomous Robots_ , vol. 43, no. 6, pp. 1291–1307, 2019.
  * [7] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1–8.
  * [8] Y. Liu, A. Gupta, P. Abbeel, and S. Levine, “Imitation from observation: Learning to imitate behaviors from raw video via context translation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 1118–1125.
  * [9] M. Zhang, Z. McCarthy, C. Finn, S. Levine, and P. Abbeel, “Learning deep neural network policies with continuous memory states,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 520–527.
  * [10] R. Cheng and Y. Jin, “A social learning particle swarm optimization algorithm for scalable optimization,” _Information Sciences_ , vol. 291, pp. 43–60, 2015.
  * [11] T. Brys, A. Harutyunyan, M. E. Taylor, and A. Nowé, “Policy transfer using reward shaping,” in _Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems_ , 2015, pp. 181–188.
  * [12] M. Wulfmeier, P. Ondruska, and I. Posner, “Maximum entropy deep inverse reinforcement learning,” _arXiv preprint arXiv:1507.04888_ , 2015.
  * [13] K. Judah, A. Fern, and T. G. Dietterich, “Active imitation learning via reduction to iid active learning,” _arXiv preprint arXiv:1210.4876_ , 2012\.
  * [14] A. Droniou, S. Ivaldi, and O. Sigaud, “Learning a repertoire of actions with deep neural networks,” in _4th International Conference on Development and Learning and on Epigenetic Robotics_. IEEE, 2014, pp. 229–234.
  * [15] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,” _ACM Computing Surveys (CSUR)_ , vol. 50, no. 2, pp. 1–35, 2017.
  * [16] L. Kunze, N. Hawes, T. Duckett, M. Hanheide, and T. Krajník, “Artificial intelligence for long-term robot autonomy: A survey,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 4023–4030, 2018.
  * [17] G. T. Papadopoulos, M. Antona, and C. Stephanidis, “Towards open and expandable cognitive ai architectures for large-scale multi-agent human-robot collaborative learning,” _arXiv preprint arXiv:2012.08174_ , 2020.
  * [18] D. Vogt, S. Stepputtis, S. Grehl, B. Jung, and H. B. Amor, “A system for learning continuous human-robot interactions from human-human demonstrations,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 2882–2889.
  * [19] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa, “Towards robotic feeding: Role of haptics in fork-based food manipulation,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1485–1492, 2019.
  * [20] Y. Pan, C.-A. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots, “Agile autonomous driving using end-to-end deep imitation learning,” _arXiv preprint arXiv:1709.07174_ , 2017.
  * [21] A. Loquercio, A. I. Maqueda, C. R. Del-Blanco, and D. Scaramuzza, “Dronet: Learning to fly by driving,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 2, pp. 1088–1095, 2018.
  * [22] Z. Zhu and H. Hu, “Robot learning from demonstration in robotic assembly: A survey,” _Robotics_ , vol. 7, no. 2, p. 17, 2018.
  * [23] B. Liu, L. Wang, and M. Liu, “Lifelong federated reinforcement learning: a learning architecture for navigation in cloud robotic systems,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 4555–4562, 2019.
  * [24] M. Bretan, S. Oore, S. Sanan, and L. Heck, “Robot learning by collaborative network training: A self-supervised method using ranking,” in _Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems_ , 2019, pp. 1333–1340.
  * [25] W. Zhou, Y. Li, S. Chen, and B. Ding, “Real-time data processing architecture for multi-robots based on differential federated learning,” in _2018 IEEE SmartWorld Congress_. IEEE, 2018, pp. 462–471.
  * [26] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.
  * [27] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” _arXiv preprint arXiv:1907.02189_ , 2019.
  * [28] S. Thermos, G. T. Papadopoulos, P. Daras, and G. Potamianos, “Deep affordance-grounded sensorimotor object recognition,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6167–6175.
  * [29] G. T. Papadopoulos and P. Daras, “Human action recognition using 3d reconstruction data,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 28, no. 8, pp. 1807–1823, 2016.
  * [30] G. Farnadi, J. Tang, M. De Cock, and M.-F. Moens, “User profiling through deep multimodal fusion,” in _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ , 2018, pp. 171–179.
  * [31] H. Liang, “Drprofiling: deep reinforcement user profiling for recommendations in heterogenous information networks,” _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * [32] S. Rossi, F. Ferland, and A. Tapus, “User profiling and behavioral adaptation for hri: a survey,” _Pattern Recognition Letters_ , vol. 99, pp. 3–12, 2017\.
  * [33] R. Mu, “A survey of recommender systems based on deep learning,” _Ieee Access_ , vol. 6, pp. 69 009–69 022, 2018.
  * [34] S. Zhang, L. Yao, A. Sun, and Y. Tay, “Deep learning based recommender system: A survey and new perspectives,” _ACM Computing Surveys (CSUR)_ , vol. 52, no. 1, pp. 1–38, 2019.
  * [35] D. Ji, Z. Xiang, and Y. Li, “Dual relations network for collaborative filtering,” _IEEE Access_ , vol. 8, pp. 109 747–109 757, 2020.
  * [36] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in _Proceedings of the 26th international conference on world wide web_ , 2017, pp. 173–182.
  * [37] N. Srivastava, E. Mansimov, and R. Salakhudinov, “Unsupervised learning of video representations using lstms,” in _International conference on machine learning_. PMLR, 2015, pp. 843–852.
  * [38] J. Gao, P. Li, Z. Chen, and J. Zhang, “A survey on deep learning for multimodal data fusion,” _Neural Computation_ , vol. 32, no. 5, pp. 829–864, 2020.


# Cross-Modality Translation with Generative Adversarial Networks to Unveil
Alzheimer’s Disease Biomarkers

Reihaneh Hassanzadeh1,2, Anees Abrol2, Hamid Reza Hassanzadeh3, and Vince D.
Calhoun1,2 1School of Electrical and Computer Engineering, Georgia Institute
of Technology, Atlanta, GA 30302, USA2Tri-institutional Center for
Translational Research in Neuroimaging and Data Science (TReNDS), Georgia
State University, Atlanta, GA 30303, USA3Courtesy Faculty Appointment, College
of Pharmacy, University of Florida

###### Abstract

Generative approaches for cross-modality transformation have recently gained
significant attention in neuroimaging. While most previous work has focused on
case-control data, the application of generative models to disorder-specific
datasets and their ability to preserve diagnostic patterns remain relatively
unexplored. Hence, in this study, we investigated the use of a generative
adversarial network (GAN) in the context of Alzheimer’s disease (AD) to
generate functional network connectivity (FNC) and T1-weighted structural
magnetic resonance imaging data from each other. We employed a cycle-GAN to
synthesize data in an unpaired data transition and enhanced the transition by
integrating weak supervision in cases where paired data were available. Our
findings revealed that our model could offer remarkable capability, achieving
a structural similarity index measure (SSIM) of $0.89\pm 0.003$ for T1s and a
correlation of $0.71\pm 0.004$ for FNCs. Moreover, our qualitative analysis
revealed similar patterns between generated and actual data when comparing AD
to cognitively normal (CN) individuals. In particular, we observed
significantly increased functional connectivity in cerebellar-sensory motor
and cerebellar-visual networks and reduced connectivity in cerebellar-
subcortical, auditory-sensory motor, sensory motor-visual, and cerebellar-
cognitive control networks. Additionally, the T1 images generated by our model
showed a similar pattern of atrophy in the hippocampal and other temporal
regions of Alzheimer’s patients.

##  I Introduction

Brain imaging techniques provide a multifaceted view of the brain. For
example, structural magnetic resonance imaging (sMRI) illuminates the
structure of the brain [1], functional MRI (fMRI) provides a dynamic picture
of brain activity [2], and positron emission tomography (PET) delves into the
metabolic processes of the brain [3]. Although each of these modalities
contributes profound insights, their integration through multimodal analysis
provides a more comprehensive understanding of the brain [4].

There is, however, a major limitation in conducting a multimodal study on real
datasets: the availability of different modalities at varying scales. That is,
available multimodal datasets often exhibit missing modalities for many
subjects, which can be attributed to a variety of factors such as ease of data
collection and implementation costs associated with each modality. For
example, fMRI requires a subject to remain still for a relatively longer
duration than sMRI and hence be more challenging for infants or individuals
with conditions like Alzheimer’s disease. Thus, existing neuroimaging datasets
often contain fewer samples of particular modalities, requiring efficient
approaches to handle missing data in multimodal analysis.

Conventionally, approaches like mean imputation, downsampling, or zero-filling
[5] have been employed to facilitate multimodal analyses when faced with
incomplete modalities in datasets. While these techniques are relatively
simple to apply, their effectiveness is compromised due to either the omission
of essential data or the inadvertent creation of bias by altering the genuine
distribution of the data. This calls for more advanced approaches that not
only leverage the available data but also ensure a more accurate analysis.

Recently, advanced techniques, such as generative modeling, have emerged as
promising alternatives to complete missing data while not having such
limitations. Previous work developed GAN-based models for modality translation
in PET-sMRI [6, 7], EEG-fMRI [8], multi-contrast sMRI [9, 10, 11], and sMRI-CT
[9, 11]. While most of the existing works on cross-modality transformation
have examined a controlled-case dataset, the application of generative models
to disorder-specific datasets and their ability to capture diagnostic patterns
is more challenging and an area that remains relatively unexplored.

Figure 1: (A) Translation framework. (B) Architecture of generators and
discriminators. Numbers indicate the output channel and linear size.

In this study, we explored the ability of generative models for T1-FNC
translation within the context of Alzheimer’s disease. We employed cycle-GAN
to synthesize FNC and T1 from each other in an unpaired data transition. To
enhance the efficacy of our model, we integrated weak supervision [12] during
the transition when paired data were available (i.e., both modalities were
present). We evaluated our model both quantitatively using established metrics
such as the SSIM [13] and Pearson correlation that measure the accuracy of the
translations and qualitatively through visual assessments of the generated
images against the original. Our findings showed that our model can
successfully transform the tested neuroimaging modalities while preserving
functional and structural brain patterns specific to Alzheimer’s disease.

##  II Method

###  II-A Cross-Modality Transformation Framework

We utilized Cycle-GAN [12], which uses unpaired data, and its objective is to
translate data from a domain X (e.g., FNC maps) into another domain Y (e.g.,
T1 images) and vice versa. Our approach, shown in Fig. 1.A, is an extension to
that model where we use two generators $G_{1}$ and $G_{2}$ that map one
modality into another and two discriminators $D_{1}$ and $D_{2}$ that
differentiate the actual data from the generated ones. Each of the generators
and its corresponding discriminator were trained in an adversarial manner. In
other words, each generator tries to construct samples that resemble the real
data, and its discriminator tries to identify whether a sample is produced by
the generator. We used the least square adversarial loss to train our model as
formulated below:

$\mathcal{L}_{adv}(G_{1},D_{1})=\begin{aligned} &\mathbb{E}_{y\sim
p_{\text{data}}(y)}\left[\left(D_{1}(y)-1\right)^{2}\right]\\\
&+\mathbb{E}_{x\sim
p_{\text{data}}(x)}\left[\left(D_{1}(G_{1}(x))\right)^{2}\right],\end{aligned}$
(1) $\mathcal{L}_{adv}(G_{2},D_{2})=\begin{aligned} &\mathbb{E}_{x\sim
p_{\text{data}}(x)}\left[\left(D_{2}(x)-1\right)^{2}\right]\\\
&+\mathbb{E}_{y\sim
p_{\text{data}}(y)}\left[\left(D_{2}(G_{2}(y))\right)^{2}\right].\end{aligned}$
(2)

Optimizing the adversarial loss alone may lead to mode collapse [14]. To
partially mitigate this issue, the cycle consistency loss has been employed in
literature to regularize generators’ mapping process and enforce cycle
consistency. The cycle-consistency loss is formulated as follows:

$\mathcal{L}_{cyc}(G_{1},G_{2})=\begin{aligned} &\mathbb{E}_{x\sim
p_{\text{data}}(x)}\left[\left\|G_{2}(G_{1}(x))-x\right\|_{1}\right]\\\
&+\mathbb{E}_{y\sim
p_{\text{data}}(y)}\left[\left\|G_{1}(G_{2}(y))-y\right\|_{1}\right].\end{aligned}$
(3)

Additionally, we incorporated weak supervision whenever paired data were
available to achieve a more accurate translation. This supervision was
implemented using an identity loss as stated below:

$\mathcal{L}_{identity}(G_{1},G_{2})=\begin{aligned}
&\sum_{(x,y)\in\mathcal{P}}\left[\left\|G_{1}(x)-y\right\|_{1}\right]\\\
&+\sum_{(x,y)\in\mathcal{P}}\left[\left\|G_{2}(y)-x\right\|_{1}\right],\end{aligned}$
(4)

where $\mathcal{P}$ is the set of paired data.

Overall, the final objective loss becomes:

$\mathcal{L}(G_{1},G_{2},D_{1},D_{2})=\begin{aligned}
&\mathcal{L}_{adv}(G_{1},D_{1})\\\ &+\mathcal{L}_{adv}(G_{2},D_{2})\\\
&+\lambda_{1}\mathcal{L}_{cyc}(G_{1},G_{2})\\\
&+\lambda_{2}\mathcal{L}_{identity}(G_{1},G_{2}),\end{aligned}$ (5)

where $\lambda_{1}$ and $\lambda_{2}$ are trade-off hyperparameters
controlling the effect of their corresponding loss in the optimization
process.

Figure 2: (A) Real (first column) and generated (second column) FNC maps and
the group mean differences. (B) Real and generated T1 images and $t$-values.
Numbers in parentheses represent the sample size, and U and L indicate the
upper and lower triangular matrix, respectively. Note that, to compare real
and generated samples, we visualized only the samples for which both FNC and
T1 data were available.

###  II-B Model Architecture

Fig. 1.B provides an overview of the model architectures employed for the
generators and discriminators in our model. The generator $G_{1}$ is designed
to transform 1D-FNC maps, each of size 1378, into 3D-T1 images of dimensions
121x145x121 through a sequence of five 3D-transposed convolution layers, each
followed by batch normalization, and a final convolutional layer. The outputs
of the last layer were then scaled with the tanh activation. We added two
upsampling layers to match the final output size to the real T1 images.
$G_{2}$ maps T1 images into FNC maps through five 3D convolutional layers
followed by batch normalization and max-pooling, as illustrated in the figure.
The outputs of the last convolutional network were transformed into FNC maps
after passing a linear layer and a tanh activation. $D_{1}$ and $D_{2}$
combine several convolutional and linear layers, as illustrated in the figure.

##  III Datasets and Experiments

We used the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset [15],
including a total of 2923 images from 986 subjects. Table I shows the
distribution of samples and subjects across modalities and diagnosis groups.
While the dataset includes 2910 T1 images, only 414 FNC maps are available.

TABLE I: Data Distribution. Diagnosis | # of Images (# of Subjects)
---|---
| FNC | T1 | FNC and T1
CN | 207 (122) | 1446 (414) | 207 (122)
AD | 207 (94) | 1465 (575) | 195 (85)

The T1 data were segmented into tissue probability maps for gray matter, white
matter, and cerebral spinal fluid using SPM12. The gray matter images were
then warped to standard space, modulated and smoothed using a Gaussian kernel
with an FWHM = 10mm. The preprocessed gray matter volume images had a
dimensionality of 121×145×121 in the voxel space, with the voxel size of
1.5×1.5×1.5 $mm^{3}$.

For the resting-state fMRI data sets, we removed the first five time points to
ensure signal equilibrium and adaptation of subjects to scanner noise. Then we
performed slice timing correction and rigid body motion correction using the
SPM toolbox, followed by warping the images into the standard Montreal
Neurological Institute (MNI) template using an echo-planar imaging (EPI)
template and the old SPM12 normalization module. Lastly, the data were
resampled to 3x3x3 $mm^{3}$ isotropic voxels, resulting in image
dimensionality of 53x63x52 in the voxel space, and smoothed using a Gaussian
kernel with a full width at half maximum of 6mm. We also implemented thorough
quality control (QC) on the preprocessed fMRI images to discard the images
that exhibited (1) poor correlation with individual and group data masks, (2)
markedly briefer scan lengths, and (3) high head motion parameters (>3°
rotations and >3mm translations).

To generate FNC maps, the resting-state fMRI data were pre-processed using the
NeuroMark, a fully automated independent component analysis (ICA) pipeline
[16], resulting in 53 components. These components are grouped into seven
network domains, namely subcortical (SC), auditory (AU), visual (VI),
sensorimotor (SM), cognitive control (CC), default mode (DM), and cerebellar
(CB). FNC maps were computed using the Pearson correlation coefficient between
two ICA-estimated components.

We trained our model in a 5-fold cross-validation framework for 300 epochs. We
used a batch size of 32 and an Adam optimizer with an initial learning rate of
0.05, which decayed by 0.9 every five epochs. We used $\lambda_{1}=10$ and
$\lambda_{2}=40$, which were selected from a grid search within the value
range of 1, 5, 10, 20, and 40. We kept the latest 50 generated samples in a
buffer and updated our discriminators using the images in the buffer to reduce
the potential for mode collapse further [17]. We measured SSIM between
generated and actual T1 images and Pearson correlation between generated and
real FNC samples to quantify the accuracy of generated data from our GAN-based
model. Furthermore, we evaluated the generated samples through qualitative
assessments of the generated images against the original and conducted
$t$-test to statistically compare the differences in mean between diagnosis
groups.

##  IV Results

Table II shows the quantitative evaluation between the generated and the
actual data. We achieved an SSIM of $0.894\pm 0.003$ for T1s and a correlation
of $0.707\pm 0.004$ for FNCs. Comparing these metrics across diagnostic
groups, we observed that our model could accurately reproduce data for both AD
and CN individuals.

TABLE II: Quantitative Evaluation Metrics Diagnosis | Pearson-Correlation | SSIM
---|---|---
All Data | 0.707$\pm$ 0.004 | 0.894$\pm$ 0.003
CN | 0.707$\pm$ 0.006 | 0.897$\pm$ 0.003
AD | 0.705$\pm$ 0.01 | 0.892$\pm$ 0.005

Table III shows that the comparison between the proposed method and the
standard cycle-GAN as the baseline, where the identity loss ($\lambda_{2}=0$)
is excluded. Based on the results, our method can statistically significantly
outperform both SSIM ($p$=0.003) and Pearson correlation ($p$=0.0008).

Moreover, we evaluated our results qualitatively through visual assessments of
the generated images against the original and observed similar AD-specific
patterns between the generated and real data. In particular, according to
Figure 2.A, in Alzheimer’s patients compared to CN individuals, we observed
significant increases in functional connectivity of the CB-SM [19] and CB-VI
network pairs and significant decreases in the CB-SC, AU-SM, SM-VI, CB-CC, and
DM-DM [20] network pairs. Additionally, as illustrated in Figure 2.B, the T1
images generated by our model revealed neural loss [21] in the hippocampal and
other temporal regions of Alzheimer’s patients with a similar atrophy pattern
to the real data.

TABLE III: Method Comparison Diagnosis | Pearson-Correlation | SSIM
---|---|---
Proposed Method |  0.707$\pm$ 0.004 |  0.894$\pm$ 0.003
Baseline | 0.69$\pm$ 0.006 | 0.871$\pm$ 0.012

##  V Conclusion

In this study, we explored the ability of generative models for T1-FNC
translation within the context of Alzheimer’s disease. We employed cycle-GAN
to synthesize FNC and T1 from each other, and to enhance the efficacy of our
model, we integrated weak supervision during the transformation in cases where
paired data were available. Our findings suggest that our proposed GAN-based
approach can effectively capture and replicate intricate structural and
functional brain patterns associated with Alzheimer’s disease. As an
interesting future direction and application, one can investigate the
potential of such a generative approach in downstream tasks such as diagnosis
using multimodal data, by filing the missing modalities with the generated
samples.

## References

  * [1] M. T. Vlaardingerbroek and J. A. Boer, Magnetic Resonance Imaging: Theory and Practice. Springer Science & Business Media, 2013.
  * [2] N. K. Logothetis, “What we can do and what we cannot do with fMRI,” in Nature, vol. 453, no. 7197, pp. 869–878, 2008. Nature Publishing Group UK London.
  * [3] M. E. Phelps, “Positron emission tomography provides molecular imaging of biological processes,” in Proceedings of the National Academy of Sciences, vol. 97, no. 16, pp. 9226–9233, 2000. National Acad Sciences.
  * [4] V. D. Calhoun and J. Sui, “Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness,” in Biological Psychiatry: Cognitive Neuroscience and Neuroimaging, vol. 1, no. 3, pp. 230–244, 2016. Elsevier.
  * [5] J. Venugopalan, L. Tong, H. R. Hassanzadeh, and M. D. Wang, “Multimodal deep learning models for early detection of Alzheimer’s disease stage,” in Scientific Reports, vol. 11, no. 1, pp. 3254, 2021. Nature Publishing Group UK London.
  * [6] L. Cai, Z. Wang, H. Gao, D. Shen, and S. Ji, “Deep adversarial learning for multi-modality missing data completion,” in Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1158–1166.
  * [7] X. Gao, F. Shi, D. Shen, and M. Liu, “Task-induced pyramid and attention GAN for multimodal brain image imputation and classification in Alzheimer’s disease,” in IEEE Journal of Biomedical and Health Informatics, vol. 26, no. 1, pp. 36–43, 2021. IEEE.
  * [8] D. Cheng, N. Qiu, F. Zhao, Y. Mao, and C. Li, “Research on the modality transfer method of brain imaging based on generative adversarial network,” in Frontiers in Neuroscience, vol. 15, pp. 655019, 2021. Frontiers Media SA.
  * [9] C. Tiago, S. R. Snare, J. Šprem, and K. McLeod, “A Domain Translation Framework With an Adversarial Denoising Diffusion Model to Generate Synthetic Datasets of Echocardiography Images,” in IEEE Access, vol. 11, pp. 17594–17602, 2023. IEEE.
  * [10] S. U. H. Dar, M. Yurt, L. Karacan, A. Erdem, E. Erdem, and T. Cukur, “Image synthesis in multi-contrast MRI with conditional generative adversarial networks,” in IEEE Transactions on Medical Imaging, vol. 38, no. 10, pp. 2375–2388, 2019. IEEE.
  * [11] W. Yuan, J. Wei, J. Wang, Q. Ma, and T. Tasdizen, “Unified generative adversarial networks for multimodal segmentation from unpaired 3D medical images,” in Medical Image Analysis, vol. 64, pp. 101731, 2020. Elsevier.
  * [12] J. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2223–2232.
  * [13] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” in IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004. IEEE.
  * [14] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb, "Learning from simulated and unsupervised images through adversarial training," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2107–2116.
  * [15] C. R. Jack Jr, M. A. Bernstein, N. C. Fox, P. Thompson, G. Alexander, D. Harvey, B. Borowski, P. J. Britson, J. L. Whitwell, C. Ward, et al., “The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods,” in Journal of Magnetic Resonance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, vol. 27, no. 4, pp. 685–691, 2008. Wiley Online Library.
  * [16] Y. Du, Z. Fu, J. Sui, S. Gao, Y. Xing, D. Lin, M. Salman, A. Abrol, M. A. Rahaman, J. Chen, et al., “NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders,” in NeuroImage: Clinical, vol. 28, pp. 102375, 2020. Elsevier.
  * [17] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial networks," in Communications of the ACM, vol. 63, no. 11, pp. 139–144, 2020. ACM New York, NY, USA.
  * [18] K. Armanious, C. Jiang, S. Abdulatif, T. Küstner, S. Gatidis, and B. Yang, “Unsupervised medical image translation using cycle-MedGAN,” in 2019 27th European Signal Processing Conference (EUSIPCO), 2019, pp. 1–5. IEEE.
  * [19] L. Penalba-Sánchez, P. Oliveira-Silva, A. L. Sumich, and I. Cifre, "Increased functional connectivity patterns in mild Alzheimer’s disease: A rsfMRI study," in Frontiers in Aging Neuroscience, vol. 14, pp. 1037347, 2023. Frontiers.
  * [20] A. Badhwar, A. Tam, C. Dansereau, P. Orban, F. Hoffstaedter, and P. Bellec, "Resting-state network dysfunction in Alzheimer’s disease: a systematic review and meta-analysis," in Alzheimer’s & Dementia: Diagnosis, Assessment & Disease Monitoring, vol. 8, pp. 73–85, 2017. Elsevier.
  * [21] G. M. McKhann, D. S. Knopman, H. Chertkow, B. T. Hyman, C. R. Jack Jr, C. H. Kawas, W. E. Klunk, W. J. Koroshetz, J. J. Manly, R. Mayeux, et al., "The diagnosis of dementia due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease," in Alzheimer’s & Dementia, vol. 7, no. 3, pp. 263–269, 2011. Elsevier.