11institutetext: PRHLT Research Center, Universitat Politècnica de València,
Spain 22institutetext: DW Innovation, Germany 33institutetext: CAISA Lab,
University of Marburg, Germany
33email: {ibarsch<EMAIL_ADDRESS>33email: lucie.flek@uni-
marburg.de

# Multilingual Detection of Check-Worthy Claims
using World Languages and Adapter Fusion††thanks: This paper is accepted at
ECIR 2023.

Ipek Baris Schlicht 1122 0000-0002-5037-2203 Lucie Flek 33 0000-0002-5995-8454
Paolo Rosso 11 0000-0002-8922-1242

###### Abstract

Check-worthiness detection is the task of identifying claims, worthy to be
investigated by fact-checkers. Resource scarcity for non-world languages and
model learning costs remain major challenges for the creation of models
supporting multilingual check-worthiness detection.

This paper proposes cross-training adapters on a subset of world languages,
combined by adapter fusion, to detect claims emerging globally in multiple
languages. (1) With a vast number of annotators available for world languages
and the storage-efficient adapter models, this approach is more cost
efficient. Models can be updated more frequently and thus stay up-to-date. (2)
Adapter fusion provides insights and allows for interpretation regarding the
influence of each adapter model on a particular language.

The proposed solution often outperformed the top multilingual approaches in
our benchmark tasks.

###### Keywords:

Fact-checking Checkworthiness Detection Mutilingual Adapters .

##  1 Introduction

There is an increasing demand for automated tools that support fact-checkers
and investigative journalists, especially in the event of breaking or
controversial news [11, 19]. Identifying and prioritizing claims for fact-
checking, aka. check-worthy (CW) claim detection, is the first task of such
automated systems [9]. This task helps guiding fact-checkers to potentially
harmful claims for further investigation. CW claims, as shown in Table 1, are
verifiable, of public interest and may invoke emotional responses [20]. Most
studies in this area focus on monolingual approaches, predominantly using
English datasets for learning the model. Support for multilingualism has
become an essential feature for fact-checkers who investigate non-English
resources [8].

Table 1: An example of a check-worthy claim from CT21. Those who falsely claim
Vaccines are 100% safe and who laugh at anti-vaxxers will be dependent on
those who refuse to tell the official lies to say that wide-spread vaccination
*is* critical. 23 die in Norway after receiving COVID-19 vaccine: [URL] via
[MENTION]
---

Although transformers achieved competitive results on several multilingual
applications [4, 3] including CW detection [21, 18], there are still two main
challenges in the CW task: (1) Because of the task complexity, there are a few
publicly available datasets in multiple languages. Updating a multilingual
model to detect even recently emerged claims requires data annotation and
timely retraining. Finding fact-checking experts to annotate samples would be
hard, especially in low-resourced languages. (2) Storing standalone copies of
each fine-tuned model for every supported language requires vast storage
capacities. Because of the limited budgets of non-profit organizations and
media institutes, it would be infeasible to update the models frequently.

World Languages (WLs) are languages spoken in multiple countries, while non-
native speakers can still communicate with each other using the WL as a
foreign language. English, Arabic and Spanish are examples of WLs
111https://bit.ly/3eMIZ9q. It would appear that finding expert annotators for
collecting samples in a WL is easier than finding expert annotators for low-
resourced languages.

As a resource-efficient alternative to fully fine-tuning transformer models,
adapters [14, 31] have been recently proposed. Adapters are lightweight and
modular neural networks, learning tasks with fewer parameters and transferring
knowledge across tasks [14, 31, 22] as well as languages [24]. Fine-tuned
adapters require less storage than fully fine-tuned pre-trained models.

In this paper, we propose cross-lingual training of datasets in WLs with
adapters to mitigate resource scarcity and provide a cost-efficient solution.
We first train Task Adapters (TAs) [23] for each WL and incorporate an
interpretable Adapter Fusion (AF) [22] to combine WL TAs for an effective
knowledge transfer among the heterogeneous sources.

Our contributions for this paper are summarized as follows 222We share our
source code at https://bit.ly/3rH6yXu.:

  * •

We extensively analyze the WL AF models on the multilingual CW claim detection
task and evaluate the models on zero-shot learning (i.e., the target languages
were unseen during training). We show that the models could perform better
than monolingual TAs and fully fine-tuned models. They also outperformed the
related best performing methods in some languages. In addition, zero-shot
learning is possible with the WL AF models.

  * •

We construct an evaluation to quantify the performance of the models on claims
about global/local topics across the languages. Our approach for curating the
evaluation set could be reused for the assessment of other multilingual,
social tasks.

  * •

We present a detailed ablation study for understanding the limitations of AF
models and their behavior on the CW task.

##  2 Related Work

###  2.1 Identifying Check-Worthy Claims

Early studies applied feature engineering to machine learning models and
neural network models to identify CW claims in political debates and speeches.
ClaimBuster [13, 12] was one of the first approaches using a Support Vector
Machine (SVM) classifier trained on lexical, affective feature sets. Gencheva
et al. [7] combined sentence-level and contextual information from political
debates for training neural network architectures. Lespagnol et al. [17]
employed information nutritional labels [6] and word embeddings as features.
Vasileva et al. [35] applied multitask learning from different fact-checking
organizations to decide whether a statement is CW. Jaradat et al. [15] used
MUSE word embeddings to support the CW detection task in English and Arabic.

CheckThat! (CT) organized multilingual CW detection tasks since 2020 [2, 21,
18]. They support more languages every year and an increasing number of
multilingual systems have been submitted. Schlicht et al. [28] proposed a
model supporting all languages in the dataset. They used a multilingual
sentence transformer [26] and then fine-tuned the model jointly on a language
identification task. Similarly, Uyangodage et al. [34] fine-tuned mBERT on a
dataset containing balanced samples for each language. Recently, Du et al. [5]
fine-tuned mT5 [37] on the CW detection task, jointly with multiple related
tasks and languages by inserting prompts. All of the listed methods were
limited to the languages they were trained on.

Kartal and Kutlu [16] evaluated mBERT for zero-shot learning in English,
Arabic and Turkish, and observed the performance drop in cross-training. A
more effective method would be required for cross-lingual training.
Furthermore, none of the approaches tackled the resource efficiency issue. In
this paper, we use adapters and train them only on WLs for resource efficiency
and evaluate unseen languages, leveraging AF to understand which target
language in the dataset benefits from transferring knowledge from WLs.

###  2.2 Adapters

Adapters have been successfully applied to pre-trained models for efficient
fine-tuning to various applications. Early studies [14, 22] used adapters for
task adaptation in English. Pfeiffer et al. [22] propose an AF module for
learning to combine TAs as an alternative to multi-task learning. Some
researchers [33, 24, 25] exploit language-specific adapters for cross-lingual
transfer. This paper builds upon the works of [22, 25]. We exploit a cross-
lingual training set to learn TAs and then use AF to combine them effectively
and provide interpretability on which source TAs are efficient on the target
language.

##  3 Datasets

###  3.1 Task Datasets

We looked for multilingual datasets in WLs and other languages for the
experiments. CT21 [29] and CT22 [18] are the only publicly available datasets
that meet this requirement. CT21 includes English, Arabic, Turkish, Spanish,
and Bulgarian samples. It deals mainly with Covid-19 events, except for the
Spanish samples, which focus only on politics. Compared with CT21, CT22 also
includes samples in Dutch. The English, Arabic, Bulgarian, and Dutch samples
in the dataset build on the corpora on Covid-19 [1]. The researchers collected
new samples in Turkish and Spanish. The Spanish samples were augmented with
CT21.

The statistics of the datasets are shown in Table 3. English, Spanish and
Arabic are the WLs contained in the datasets. Both datasets are imbalanced,
i,e. CW samples are under-represented across the languages. Although English
is considered a high-resource language, there are considerably fewer English
samples than samples in other languages. Since some samples of the datasets
could overlap, we conducted our research experiments per dataset.

Table 2: Statistics of CT21 and CT22. |  | CT21 | CT22
---|---|---|---
| Split | Total |  $\%$CW | Total |  $\%$CW
ar | Train | 3444 | 22.15 | 2513 | 38.28
Dev | 661 | 40.09 | 235 | 42.55
Test | 600 | 40.33 | 682 | 35.28
es | Train | 2495 | 8.02 | 4990 | 38.14
Dev | 1247 | 8.74 | 2500 | 12.20
Test | 1248 | 9.62 | 4998 | 14.09
en | Train | 822 | 35.28 | 2122 | 21.07
Dev | 140 | 42.86 | 195 | 22.56
Test | 350 | 5.43 | 149 | 26.17
tr | Test | 1013 | 18.07 | 303 | 4.62
bg | Test | 357 | 21.29 | 130 | 43.85
nl | Test | - | - | 666 | 47.45

Table 3: Statistics of Global and Local Topics |  | CT21 | CT22
---|---|---|---
| Split | Total |  $\%$CW | Total |  $\%$CW
ar | Global | 269 | 43.12 | 116 | 46.55
Local | 40 | 42.5 | - | -
es | Global | 1208 | 9.93 | 148 | 22.97
Local | - | - | 917 | 12.43
nl | Global | - | - | 103 | 56.31
| Local | - | - | 15 | 46.67
en | Global | 349 | 5.44 | 14 | 28.57
tr | Global | 887 | 8.91 | - | -
bg | Global | 356 | 21.35 | 25 | 32

###  3.2 Topical Evaluation Dataset

Some countries are culturally dissimilar or might have different political
agendas, thus CW topics might differ among countries. For example, while
vaccination was a globally CW topic throughout the COVID-19 pandemic, some
COVID-19 myths were believed only by a few communities [30]. An ideal
multilingual system should perform well on global as well as local topics.

Global topics are the topics present in all languages in all datasets, while
local topics are present in only one language. We created dedicated datasets
to evaluate the performance of the WL models for identifying CW claims across
global and local topics. This evaluation dataset contains solely global and
local topics and was created as follows. To learn topics across datasets, we
first translated the datasets into English and encoded them with a sentence
transformer [26]. Second, we learned a topic modeling on the training datasets
in the WLs by using BERTopic [10] to assign topics to test samples in all
languages. Test samples with topics present in all WL languages were added as
presumably global topics to the evaluation set. BERTopic labeled topics that
are unrelated to the learnt topics with -1. To select samples with local
topics, we chose the samples labeled with -1 from the evaluation and applied a
new topic modeling on them. Test samples with topics that were not present in
the test dataset of any other language (i.e., local topics) were added to the
evaluation set for local topics. The statistics of the local and global
evaluation sets are presented in Table 3.

##  4 Methodology

Figure 1: The architecture of the WL+AF models within a transformer. WL+AF+LA
is the setup when the LAs (blocks with the dashed lines) are stacked into the
task adapters.

###  4.1 World Language Adapter Fusion

This section describes the WL+AF models. The WL+AF models are transformers
containing additional modules called adapters. During training, only the
adapters are trained and the weights of pre-trained model are frozen.
Therefore, it is a parameter-efficient approach. We experiment with two types
of WL+AF models. WL+AF is a standard setup for combining WL task adapters.
WL+AF+LA has additional language adapters (LAs) [24] by stacking task
adapters. We illustrate the architecture of the models in Figure 1. The input
of the architecture is a text while the output is a probability score
describing the check-worthiness of the input.

Transformer Encoder. To provide cross-lingual representations, we use
multilingual BERT (mBERT) [4] and XLM-Roberta (XLM-R) [3]333We use the base
version of the model, which consists of 12 layers.. Because the transformers
were previously used by the related studies [34, 1]. Furthermore, there are
publicly available LAs for the transformers. Before encoding text with
transformers, we first clean it from URLs, mentions and retweets, then
truncate or pad it to 128 tokens.
Task Adapter. The annotations might be affected by the different cultural
backgrounds and events across the countries, like the CW claims. Additionally,
some may have been created from journalists following manual fact-checking
procedures while others stem from crowd-sourcing [21, 18] For these reasons,
we treat each WL dataset as a separate task. Then, we obtain TAs by optimizing
on their corresponding WL dataset. TAs consist of one down projection layer
followed by a ReLU activation and one up projection layer [23]. The TA of each
WL is fine-tuned on its corresponding dataset.
Adapter Fusion. To share knowledge from the different TAs for predicting
samples in an unseen language or new topics, we need to combine them
effectively. AF is a method for combining multiple TAs, and mitigating the
common issues of multi-task learning, such as catastrophic forgetting and
complete model retraining for supporting new tasks [22]. AF consists of an
attention module which learns how to combine knowledge from the TAs
dynamically. By analyzing the attention weights of the AF, we could learn
which source task adapter contributes more to test predictions. We combine the
TAs trained on the WLs with an AF. Then, the AF is fine-tuned on mixed
datasets of the WLs to use for the target languages.
Language Adapter. To learn CW claim detection from various cross-lingual
sources, we should ensure that the model does not learn a language
classification task but learns how to identify CW claims. Therefore, we need
to separate language-specific knowledge from task-specific knowledge. A recent
study [24] demonstrated that LAs achieved better results when transferring the
knowledge of a task performed in one source language into another language. To
encode language-specific information into the transformer model and to see the
LAs impact on the performance of the AF models, we use the LAs from Adapter
Hub [23]. The architecture of the LAs is analog to those of the TAs. However,
these LAs were pre-trained on the unlabeled Wikipedia [24] by using a masked
language model objective. While learning task-specific transformations with
TAs, each world LA is stacked on its corresponding TA. The weights of the LAs
are kept frozen, so they are not learned again. During inference of the target
task, the source LA is replaced with the target LA.

###  4.2 Implementation Details

We download the pre-trained transformers from Huggingface [36]. We fine-tune
the TAs on English, Arabic, and Spanish training samples (WLs), and evaluate
the models for their capability of zero-shot learning by testing on the other
languages. As LAs for Arabic, English, Spanish and Turkish samples in the
dataset [24], we download the pre-trained LAs from Adapter Hub [23] for both
transformers. However, a LA for Bulgarian and Dutch is not available in
Adapter Hub, therefore, we use the English LA.

We use the trainers of AdapterHub [23] for both full fine-tuning and adapter
tuning. We set the number of epochs as 10 and train the models with a learning
rate of 1e-4 for CT21 and 2e-5 for CT224442e-5 gives better results on the
development set of CT22, and batch size of 16. We use the best models on the
development set according to the dataset metrics. We repeat the experiments
ten times using different random seeds using an NVIDIA GeForce RTX 2080.

##  5 Baselines

We compare the performance of the WL AF models against various baselines:

  * •

Top performing systems: We chose the top-performing systems on CT21 and CT22
which used a single model for the multilingual CW detection instead of
containing language-specific models. Schlicht et al. [28] is a runner-up
555BigIR is the state of art approach, but there is no associated paper/code
describing the system. system on CT21. The original implementation uses a
sentence transformer and the model was fine-tuned on all the language
datasets. For a fair comparison, we set the language identification task to
WLs and replaced the sentence transformer with mBERT and XLM-R. The state-of-
the-art model on CT22 is based on mT5-xlarge [5], a multi-task text-to-text
transformer. Like Schlicht et al., the model is fine-tuned on all of the
corpora by using multi-task learning. Due to the limited computing resources,
we couldn’t fine-tune this model on WLs alone. We report the results from [5].

  * •

Fully fine tuned (FFT) Transformers (on Single Language): We fine-tune the
datasets on a single language of the WLs to evaluate the efficiency of cross-
lingual learning: AR+FFT, EN+FFT, and ES+FFT. We also add BG+FFT, ES+FFT, and
NL+FFT as the baseline for zero-shot learning.

  * •

Task Adapters (on Single Language): These baselines contain a task adapter
followed by a LA, a widely used setup for cross-lingual transfer learning with
adapters [24]: AR+TA+LA, EN+TA+LA, ES+TA+LA. Comparing our model to these
baselines can help understand whether cross-training with WLs is efficient.

  * •

Other WL Models: We evaluate the AF models against a model containing a task
adapter trained on WLs (WL+TA). In addition to this baseline, to see if we
need a complex fusion method, we use WL+TA+LA+Mean, which takes the average of
the predictions by AR+TA, EN+TA and ES+TA. Finally, we analyze the adapter
tuning on multiple WLs against the fully fine tuning of mBERT: WL+FFT.

##  6 Results and Discussion

Table 4: MAP scores for CT21 and F1 scores for CT22 of the CW detection in WLs. The bold indicates the best score and underline indicates the second best score. Overall the AF models performed well on multiple languages while the performance of other models are sensitive to the characteristics of the training set. |  | CT21 | CT22 |
---|---|---|---|---
|  | ar | es | en | ar | es | en | avg
| Du et al.[5] | - | - | - | 62.8 | 57.1 | 51.9 | -
mBERT | AR+FFT | 50.17 | 15.78 | 6.03 | 55.52 | 17.85 | 42.92 | 31.38
ES+FFT | 41.22 | 20.30 | 6.80 | 37.30 | 54.05 | 45.28 | 34.16
EN+FFT | 51.63 | 15.90 | 10.80 | 40.97 | 22.49 | 44.29 | 31.01
AR+TA+LA | 58.20 | 19.97 | 8.62 | 18.13 | 17.61 | 45.73 | 28.04
ES+TA+LA | 48.07 | 18.93 | 11.06 | 37.30 | 54.05 | 45.73 | 35.86
EN+TA+LA | 50.81 | 40.81 | 21.21 | 56.39 | 24.63 | 12.93 | 34.46
WL+FFT | 47.93 | 51.50 | 13.85 | 51.50 | 63.20 | 39.94 | 44.65
Schlicht et al.[28] | 51.51 | 31.04 | 7.87 | 45.93 | 66.48 | 34.18 | 39.50
WL+TA | 53.77 | 46.58 | 14.44 | 39.54 | 62.69 | 37.19 | 42.37
WL+TA+LA+Mean | 54.89 | 35.72 | 12.96 | 0.00 | 33.21 | 51.03 | 31.30
WL+AF | 55.13 | 46.29 | 16.05 | 36.45 | 64.32 | 39.73 | 42.96
| WL+AF+LA | 55.32 | 46.58 | 15.66 | 39.87 | 64.64 | 37.27 | 43.22
XLM-R | AR+FFT | 43.55 | 12.80 | 5.88 | 38.72 | 6.83 | 41.29 | 24.85
ES+FFT | 41.22 | 15.90 | 6.80 | 40.25 | 21.69 | 43.51 | 28.23
EN+FFT | 43.64 | 10.49 | 5.64 | 31.46 | 64.66 | 45.69 | 33.60
AR+TA+LA | 58.16 | 25.87 | 7.64 | 6.93 | 0.06 | 1.40 | 16.68
ES+TA+LA | 50.27 | 52.51 | 11.53 | 41.24 | 65.45 | 38.79 | 43.30
EN+TA+LA | 56.39 | 24.63 | 12.93 | 12.82 | 0.06 | 28.74 | 22.60
WL+FFT | 47.93 | 13.85 | 6.37 | 44.53 | 64.75 | 50.93 | 38.06
Schlicht et al.[28] | 51.56 | 21.61 | 7.32 | 42.59 | 67.13 | 36.40 | 37.77
WL+TA | 58.02 | 50.76 | 11.53 | 42.91 | 63.36 | 31.69 | 43.05
WL+TA+LA+Mean | 59.32 | 46.11 | 10.49 | 25.32 | 35.55 | 35.28 | 35.35
WL+AF | 58.39 | 49.42 | 13.29 | 39.84 | 65.66 | 46.96 | 45.59
| WL+AF+LA | 58.83 | 47.26 | 16.06 | 35.17 | 65.80 | 43.46 | 44.43
Table 5: MAP for CT21 and F1 for CT22 of the CW detection in zero-shot languages. The bold indicates the best score and underline indicates the second best score. The AF models performed well, even outperformed WL+FFT and Schlicht et al. [28] and some of the monolingual approaches in terms of average score.  |  | CT21 | CT22 |
---|---|---|---|---
|  | tr | bg | tr | bg | nl | avg
| Du et al.[5] | - | - | 17.3 | 61.7 | 64.2 | -
mBERT | BG+FFT | - | 35.64 | - | 57.16 | - | -
TR+FFT | 28.47 | - | 14.66 | - | - | -
NL+FFT | - | - | - | - | 49.80 | -
AR+FFT | 30.14 | 22.12 | 10.71 | 54.84 | 45.55 | 32.67
ES+FFT | 23.17 | 24.15 | 8.19 | 47.67 | 33.82 | 27.4
EN+FFT | 42.80 | 33.20 | 13.57 | 54.58 | 54.32 | 39.69
AR+TA+LA | 58.08 | 26.76 | 8.04 | 57.43 | 58.99 | 41.86
ES+TA+LA | 49.09 | 28.09 | 8.58 | 47.67 | 42.16 | 35.12
EN+TA+LA | 54.16 | 44.98 | 8.19 | 33.92 | 33.82 | 35.01
WL+FFT | 27.61 | 24.29 | 10.90 | 58.53 | 29.03 | 30.07
Schlicht et al.[28] | 27.81 | 24.41 | 7.86 | 51.94 | 29.33 | 28.27
WL+TA | 54.11 | 37.04 | 9.73 | 48.02 | 36.95 | 37.17
WL+TA+LA+Mean | 62.32 | 34.52 | 12.02 | 47.17 | 39.91 | 39.19
WL+AF | 50.46 | 39.80 | 9.63 | 53.55 | 38.76 | 38.44
| WL+AF+LA | 50.94 | 40.27 | 9.73 | 52.75 | 43.07 | 39.35
XLM-R | BG+FFT | - | 24.68 | - | 43.47 | - | -
TR+FFT | 24.57 | - | 18.90 | - | - | -
NL+FFT | - | - | - | - | 58.61 | -
AR+FFT | 23.40 | 21.86 | 11.62 | 45.10 | 38.73 | 28.14
ES+FFT | 23.17 | 24.15 | 9.07 | 62.72 | 31.24 | 30.07
EN+FFT | 22.63 | 21.76 | 18.43 | 49.25 | 45.62 | 31.54
AR+TA+LA | 56.19 | 21.37 | 0.48 | 9.75 | 5.22 | 18.60
ES+TA+LA | 44.98 | 23.86 | 8.04 | 46.72 | 30.52 | 30.82
EN+TA+LA | 58.38 | 41.61 | 15.19 | 8.42 | 25.83 | 29.89
WL+FFT | 27.61 | 24.29 | 14.02 | 63.79 | 36.34 | 33.21
Schlicht et al.[28] | 25.86 | 22.19 | 9.55 | 53.75 | 24.44 | 27.16
| WL+TA | 59.37 | 39.72 | 15.60 | 66.14 | 35.98 | 43.36
| WL+TA+LA+Mean | 63.65 | 32.28 | 9.90 | 31.27 | 19.91 | 31.40
| WL+AF | 57.46 | 46.86 | 12.73 | 59.12 | 40.83 | 43.4
| WL+AF+LA | 61.74 | 41.78 | 17.77 | 63.88 | 37.59 | 44.55
Table 6: F1 scores of the models on global topics for each dataset. Adapter
training is more effective than fully fine-tuning. Although WL+TA outperformed
the AF models in particular languages, at average the AF models performed
better.

CT21 CT22 tr es ar en bg es ar en bg nl avg WL+FFT 0.00 0.00 2.60 0.59 0.10
77.59 52.13 52.05 48.52 38.53 27.21 Schlicht et al.[28] 18.15 17.92 58.84 7.18
19.75 81.81 48.12 28.33 36.55 30.34 34.70 WL+TA+LA 52.37 38.09 61.01 13.58
38.44 77.28 45.47 22.57 41.31 45.09 38.52 WL+AF 45.02 41.57 61.51 13.36 36.23
79.14 44.61 50.96 40.58 43.19 45.62 WL+AF+LA 48.52 37.44 61.79 13.38 28.72
77.95 42.84 44.23 41.09 43.99 44.00

Table 7: F1 scores of the models on local topics for each dataset. The AF models show similar results to WL+TA and outperformed WL+FFT.  | CT21 | CT21 |
---|---|---|---
| ar | es | nl | avg
WL+FFT | 0.93 | 60.64 | 31.11 | 30.89
Schlicht et al.[28] | 46.80 | 64.65 | 14.44 | 41.96
WL+TA+LA | 50.88 | 61.65 | 38.47 | 50.33
WL+AF | 48.15 | 62.01 | 34.38 | 48.18
WL+AF+LA | 46.41 | 63.07 | 41.16 | 50.21
Table 8: Number of training parameters and file size comparisons for the models. mT5 is larger than mBERT and XLM-R. Model | Base Model | Parameters | Model Size
---|---|---|---
WL+FFT | mBERT | 178 M | 711.5 MB
| XLM-R | 278 M | 1.1 GB
Schlicht et al.[28] | mBERT | 179 M | 716.3 MB
| XLM-R | 279 M | 1.1 GB
TA & WL+TA+LA | mBERT | 1.5 M | 6 MB
| XLM-R | 1.5 M | 6 MB
AF | mBERT | 22 M | 87.4 MB
| XLM-R | 22 M | 87.4 MB
LA | mBERT | - | 147.78 MB
| XLM-R | - | 147.78 MB
| mT5 | 3.7 B | 15 GB

In this section, we present and analyze the results of the WL AF(+LA) models.
We compare the models performance at CW detection for (1) WLs, (2) zero-shot
languages and (3) local and global topics. Lastly, we compare the performance
of WL+AF and WL+AF+LA to investigate whether LA is effective in model
performance.

As seen in Table 4, the models trained on single languages are able to perform
well for other WLs if only provided with training sets of considerable size,
or language of the training and test sets are same. Additionally, Schlicht et
al.[28] and WL+FFT were performing well only on CT22, overall, the AF models,
perform well for various languages. Du et al. [5] outperformed the AF models
for Arabic and English samples of CT22, but it underperformed for the Spanish
samples.

As shown in Table 5, the AF models achieve good results on target sets in
zero-shot languages. It shows that the fusion of multiple sources with
adapters could be beneficial in knowledge transfer and is better than the
other fusion method WL+TA+LA+Mean. It is noteworthy that Du et al [5] was
trained on all samples of the training datasets and hence has no zero-shot
learning capacity. Although Du et al. achieved a better performance on the
Dutch samples, the AF models could obtain similar results in other languages.
In terms of resource efficiency, the AF models required less space than WL+FFT
and mT5 for storing new weights, as shown in Table 8, which make them more
suitable than updating mT5 for newsrooms with a limited budget.

We compare the performance of models trained on multiple WLs for identifying
CW claims about global or local topics. We tested this experiment with the
evaluation set described in Section 3.2 in terms of F1 score. We take the
average of the scores of the models coded with mBERT and XLM-R and present
them in Tables 6 and 7, respectively, for global and local topics. Overall,
the AF models performed better than WL+FFT and Schlicht et al.[28] for both
types. However, WL+TA performed similarly to WL+AF+LA in predicting local
statements in Arabic samples in CT21.

Last, we compare the performance of WL+AF and WL+AF+LA to investigate whether
LA is effective in model performance. We computed the Fleiss Kappa scores of
the AF models for each experiment and language. The overall score is 0.63,
which is a moderate agreement. We further investigate the disagreements where
the kappa is below 0.5. The conflicts mainly occurred in the zero shot
languages and English, with the lowest CW samples on both datasets. Since
sometimes WL+AF+LA is better than WL+AF and vice versa, we conclude that LA is
not effective in our experiments. The pre-trained LAs were trained on the
Wikipedia texts [24]. Thus, they might miss the properties of social texts,
which are mostly noisy.

(a) CT21 (b) mBERT (c) XLM-R (d) CT22 (e) mBERT (f) XLM-R

Figure 2: The left images are topical relation graphs of CT21 (top left) and
CT22 (bottom left). In the graphs, the size of the nodes varies by the number
of samples, and the edge thickness depends on the overlapping topics. x-axis
of heatmaps shows task adapters, y-axis shows the test samples in the
different languages. (b), (c) attention heatmaps of mBERT, (e), (f) attention
heatmaps of XLM-R. Topical distribution, and the sample sizes of the training
datasets impact the task adapters’ activations. Especially XLM-R TAs are more
sensitive than mBERT.

Table 9: The CW claims predicted correctly by the WL+TA+AF, the examples are
in Spanish and Bulgarian. The order of the texts for each example: 1)
Visualizations on mBERT, 2) Visualizations on XLM-R 3) the red text is the
translation. The models focus more on GPE (e.g. country names) than the other
entity types. We colorized the claims based on their integrated gradients
[32].

Table 10: The performance of the AF model at predicting entity types in terms of average F1 and the standard deviation. The models could predict GPE more accurately than the others.  | Geo-political Entity | Organization | Number | People
---|---|---|---|---
F1 |  46.83 $\pm$ 17.37 |  40.44 $\pm$ 18.62 |  40.99 $\pm$ 22.52 |  37.88 $\pm$ 18.83

##  7 Further Analysis

In this section, we present further analysis of the AF models. We investigate
AF attentions and then apply an error analysis on the models’ predictions.
Interpretation of the Fusion Attentions. The AF models can provide an
interpretation of which source task adapter might be useful when transferring
the knowledge into the target dataset. This kind of analysis would help a data
scientist at a newsroom on a decision on which WL should be collected for
updating model and managing new resources. To check the AF behavior on
WL+TA+LA+AF, we took the average of the softmax probabilities of the layer of
each task adapter in the fusion layer. The higher probability means the more
useful the task for determining the label [22]. In addition, to correlate the
attention with the source datasets, we created a graph displaying the topical
relationship between source and target sets. In the graph, the nodes are the
monolingual datasets; the edges are the overlapped topics between the source
and target dataset, weighted by the percentage of the samples about the
overlapped topic. The size of nodes are scaled according to sample size.
Figure 2 shows the graph for both datasets and the attention weights of mBERT
and XLM-R task adapters. Topical distributions and source datasets’ size
affect which task adapter activates. XLM-R TAs are more sensitive to the
source data size and topical relationship. For instance, the Spanish tests in
CT22 are weakly connected with the Arabic and English source datasets, and the
Spanish TA of XLM-R has less activation than the mBERT TA.

Table 11: The CW claims that are misclassified by the AF models. The black
texts show claims in Turkish. The red texts are the translations, and the blue
ones are the explanation of the claims.

Bunu hep yazdım yine yazacağım , Bakanların aileleri , annesi - babası tam
kapsamlı sağlık tedavileri buna ( Estetik dahil ) devlet bütcesinden
karşılanıyor da , SMA hastası çocukların tedavisi için niye bir bütçe
oluşturulmuyor [UNK] [UNK] # DevletSMAyıYaşatsın I’ve always written this and
I will write it again. The families of the ministers, their mother-father full
health treatments (including Aesthetics) are covered by the state budget, but
why isn’t a budget created for the treatment of children with SMA [UNK] [UNK]
# Let The State Live Example of a CW claim that requires local context. SMA is
a disease that affects children, and the treatment of SMA is a controversial
issue in Turkey. Koronavirüs salgınında vaka sayısı 30 bin 021 [UNK] e ulaştı
# Corona # COVID # coronavirus The number of cases in the coronavirus epidemic
reached 30 thousand 021 [UNK] # Corona # COVID # coronavirus An example of a
CW claim whose veracity could be changed by time.

Error analysis. Last, we analyze the misclassified/correctly classified
samples by both AF models. As shown in Table 10, we noted that the AF model
focuses on geo-political entities (GPE). The models could categorize the
claims with GPE better than claims containing other type of entities as shown
in Table 10. The importance of the GPE could be learned from the WL corpus
whose CW samples have no negligible amount of these entities (e.g. %76 of CT21
and %77 of CT22 Arabic source datasets are GPE). However, the models cannot
predict the claims requiring local context, especially in the zero-shot
languages. Moreover, the models cannot identify the claims whose veracity can
be changed to not CW by time. Some examples are shown in Table 11.
Training efficiency. We measured the models’ training time for one epoch on
the datasets. The TA training is on average 4 minutes less than the fully fine
tuning. However, the AF training without LAs lasts 3 minutes more, and the
training with LAs 9 minutes more than the training time of WL+FFT which was
approx. 22 minutes. The methods such as AdapterDrop [27] could speed up the AF
training.

##  8 Conclusion and Future Work

In this paper, we investigated the cost efficient cross-training of adapter
fusion models on world languages to detect check-worthiness in multiple
languages. The proposed solution performs well on multiple languages, even on
zero-shot learning. Thanks to adapter fusion, the effectiveness of the
adapters on particular languages was possible.

The attention of some task adapters seems to depend on the topic and sample
distribution in the source dataset. Ensuring a topical balance across world
languages appears to be important. Our error analysis results indicate that
local context is required to detect local claims. We recommend the usage of
background knowledge injection to detect local claims.

In the future, we would like to investigate the injection of background
knowledge in adapters and verify our results in additional domains (e.g. war),
employing more languages such as German and focusing on zero-shot learning.

#### Acknowledgements

We would like to thank the anonymous reviewers, Joan Plepi, Flora Sakketou,
Akbar Karimi and Nico Para for their constructive feedback. The work of Ipek
Schlicht was part of the KID2 project (led by DW Innovation and co-funded by
BKM). The work of Lucie Flek was part of the BMBF projects DeFaktS and
DynSoDA. The work of Paolo Rosso was carried out in the framework of IBERIFIER
(INEA/CEF/ICT/A202072381931 n.2020-EU-IA-0252), XAI Disinfodemics
(PLEC2021-007681) and MARTINI (PCI2022-134990-2).

## References

  * [1] Alam, F., Shaar, S., Dalvi, F., Sajjad, H., Nikolov, A., Mubarak, H., Da San Martino, G., Abdelali, A., Durrani, N., Darwish, K., et al.: Fighting the covid-19 infodemic: Modeling the perspective of journalists, fact-checkers, social media platforms, policy makers, and the society. In: Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 611–649 (2021)
  * [2] Barrón-Cedeño, A., Elsayed, T., Nakov, P., Martino, G.D.S., Hasanain, M., Suwaileh, R., Haouari, F., Babulkov, N., Hamdan, B., Nikolov, A., Shaar, S., Ali, Z.S.: Overview of checkthat! 2020: Automatic identification and verification of claims in social media. In: CLEF. Lecture Notes in Computer Science, vol. 12260, pp. 215–236. Springer (2020)
  * [3] Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, É., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8440–8451 (2020)
  * [4] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers). pp. 4171–4186. Association for Computational Linguistics (2019). https://doi.org/10.18653/v1/n19-1423, https://doi.org/10.18653/v1/n19-1423
  * [5] Du, S.M., Gollapalli, S.D., Ng, S.K.: Nus-ids at checkthat! 2022: Identifying check-worthiness of tweets using checkthat5. Working Notes of CLEF (2022)
  * [6] Fuhr, N., Giachanou, A., Grefenstette, G., Gurevych, I., Hanselowski, A., Järvelin, K., Jones, R., Liu, Y., Mothe, J., Nejdl, W., Peters, I., Stein, B.: An information nutritional label for online documents. SIGIR Forum 51(3), 46–66 (2017). https://doi.org/10.1145/3190580.3190588, https://doi.org/10.1145/3190580.3190588
  * [7] Gencheva, P., Nakov, P., Màrquez, L., Barrón-Cedeño, A., Koychev, I.: A context-aware approach for detecting worth-checking claims in political debates. In: Mitkov, R., Angelova, G. (eds.) Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, Varna, Bulgaria, September 2 - 8, 2017. pp. 267–276. INCOMA Ltd. (2017). https://doi.org/10.26615/978-954-452-049-6_037, https://doi.org/10.26615/978-954-452-049-6_037
  * [8] Ginsborg, L., Gori, P.: Report on a survey for fact checkers on covid-19 vaccines and disinformation. Tech. rep., European Digital Media Observatory (EDMO) (2021), https://cadmus.eui.eu/handle/1814/70917
  * [9] Graves, L.: Understanding the promise and limits of automated fact-checking (2018)
  * [10] Grootendorst, M.: Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794 (2022)
  * [11] Hassan, N., Adair, B., Hamilton, J.T., Li, C., Tremayne, M., Yang, J., Yu, C.: The quest to automate fact-checking. In: Proceedings of the 2015 Computation Journalism Symposium (2015)
  * [12] Hassan, N., Arslan, F., Li, C., Tremayne, M.: Toward automated fact-checking: Detecting check-worthy factual claims by claimbuster. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017. pp. 1803–1812. ACM (2017). https://doi.org/10.1145/3097983.3098131, https://doi.org/10.1145/3097983.3098131
  * [13] Hassan, N., Li, C., Tremayne, M.: Detecting check-worthy factual claims in presidential debates. In: Bailey, J., Moffat, A., Aggarwal, C.C., de Rijke, M., Kumar, R., Murdock, V., Sellis, T.K., Yu, J.X. (eds.) Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19 - 23, 2015. pp. 1835–1838. ACM (2015). https://doi.org/10.1145/2806416.2806652, https://doi.org/10.1145/2806416.2806652
  * [14] Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., Attariyan, M., Gelly, S.: Parameter-efficient transfer learning for NLP. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 2790–2799. PMLR (2019)
  * [15] Jaradat, I., Gencheva, P., Barrón-Cedeño, A., Màrquez, L., Nakov, P.: ClaimRank: Detecting check-worthy claims in Arabic and English. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. pp. 26–30. Association for Computational Linguistics, New Orleans, Louisiana (Jun 2018). https://doi.org/10.18653/v1/N18-5006, https://aclanthology.org/N18-5006
  * [16] Kartal, Y.S., Kutlu, M.: Re-think before you share: A comprehensive study on prioritizing check-worthy claims. IEEE Transactions on Computational Social Systems (2022)
  * [17] Lespagnol, C., Mothe, J., Ullah, M.Z.: Information nutritional label and word embedding to estimate information check-worthiness. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. p. 941–944. SIGIR’19, Association for Computing Machinery, New York, NY, USA (2019). https://doi.org/10.1145/3331184.3331298, https://doi.org/10.1145/3331184.3331298
  * [18] Nakov, P., Barrón-Cedeño, A., Martino, G.D.S., Alam, F., Struß, J.M., Mandl, T., Míguez, R., Caselli, T., Kutlu, M., Zaghouani, W., Li, C., Shaar, S., Shahi, G.K., Mubarak, H., Nikolov, A., Babulkov, N., Kartal, Y.S., Beltrán, J.: The clef-2022 checkthat! lab on fighting the covid-19 infodemic and fake news detection. In: European Conference on Information Retrieval (2022)
  * [19] Nakov, P., Corney, D.P.A., Hasanain, M., Alam, F., Elsayed, T., Barrón-Cedeño, A., Papotti, P., Shaar, S., Martino, G.D.S.: Automated fact-checking for assisting human fact-checkers. In: IJCAI. pp. 4551–4558. ijcai.org (2021)
  * [20] Nakov, P., Da San Martino, G., Elsayed, T., Barrón-Cedeno, A., Míguez, R., Shaar, S., Alam, F., Haouari, F., Hasanain, M., Babulkov, N., et al.: The clef-2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In: European Conference on Information Retrieval. pp. 639–649. Springer (2021)
  * [21] Nakov, P., Da San Martino, G., Elsayed, T., Barrón-Cedeño, A., Míguez, R., Shaar, S., Alam, F., Haouari, F., Hasanain, M., Mansour, W., et al.: Overview of the clef–2021 checkthat! lab on detecting check-worthy claims, previously fact-checked claims, and fake news. In: International Conference of the Cross-Language Evaluation Forum for European Languages. pp. 264–291. Springer (2021)
  * [22] Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., Gurevych, I.: Adapterfusion: Non-destructive task composition for transfer learning. In: Merlo, P., Tiedemann, J., Tsarfaty, R. (eds.) Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021. pp. 487–503. Association for Computational Linguistics (2021). https://doi.org/10.18653/v1/2021.eacl-main.39, https://doi.org/10.18653/v1/2021.eacl-main.39
  * [23] Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., Gurevych, I.: AdapterHub: A framework for adapting transformers. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 46–54. Association for Computational Linguistics, Online (Oct 2020). https://doi.org/10.18653/v1/2020.emnlp-demos.7, https://aclanthology.org/2020.emnlp-demos.7
  * [24] Pfeiffer, J., Vulić, I., Gurevych, I., Ruder, S.: MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 7654–7673. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.617, https://aclanthology.org/2020.emnlp-main.617
  * [25] Pfeiffer, J., Vulić, I., Gurevych, I., Ruder, S.: UNKs everywhere: Adapting multilingual language models to new scripts. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 10186–10203. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.800, https://aclanthology.org/2021.emnlp-main.800
  * [26] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)
  * [27] Rücklé, A., Geigle, G., Glockner, M., Beck, T., Pfeiffer, J., Reimers, N., Gurevych, I.: AdapterDrop: On the efficiency of adapters in transformers. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 7930–7946. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.626, https://aclanthology.org/2021.emnlp-main.626
  * [28] Schlicht, I.B., de Paula, A.F.M., Rosso, P.: UPV at checkthat! 2021: Mitigating cultural differences for identifying multilingual check-worthy claims. In: CLEF (Working Notes). CEUR Workshop Proceedings, vol. 2936, pp. 465–475. CEUR-WS.org (2021)
  * [29] Shaar, S., Hasanain, M., Hamdan, B., Ali, Z.S., Haouari, F., Nikolov, A., Kutlu, M., Kartal, Y.S., Alam, F., Martino, G.D.S., Barrón-Cedeño, A., Míguez, R., Beltrán, J., Elsayed, T., Nakov, P.: Overview of the CLEF-2021 checkthat! lab task 1 on check-worthiness estimation in tweets and political debates. In: CLEF (Working Notes). CEUR Workshop Proceedings, vol. 2936, pp. 369–392. CEUR-WS.org (2021)
  * [30] Singh, K., Lima, G., Cha, M., Cha, C., Kulshrestha, J., Ahn, Y.Y., Varol, O.: Misinformation, believability, and vaccine acceptance over 40 countries: Takeaways from the initial phase of the covid-19 infodemic. Plos one 17(2), e0263381 (2022)
  * [31] Stickland, A.C., Murray, I.: BERT and pals: Projected attention layers for efficient adaptation in multi-task learning. In: ICML. Proceedings of Machine Learning Research, vol. 97, pp. 5986–5995. PMLR (2019)
  * [32] Sundararajan, M., Taly, A., Yan, Q.: Axiomatic attribution for deep networks. In: ICML. Proceedings of Machine Learning Research, vol. 70, pp. 3319–3328. PMLR (2017)
  * [33] Üstün, A., Bisazza, A., Bouma, G., van Noord, G.: UDapter: Language adaptation for truly Universal Dependency parsing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (Empirical Methods in Natural Language Processing). pp. 2302–2315. Association for Computational Linguistics, Online (Nov 2020), https://www.aclweb.org/anthology/2020.emnlp-main.180
  * [34] Uyangodage, L., Ranasinghe, T., Hettiarachchi, H.: Can multilingual transformers fight the COVID-19 infodemic? In: Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021). pp. 1432–1437. INCOMA Ltd., Held Online (Sep 2021), https://aclanthology.org/2021.ranlp-1.160
  * [35] Vasileva, S., Atanasova, P., Màrquez, L., Barrón-Cedeño, A., Nakov, P.: It takes nine to smell a rat: Neural multi-task learning for check-worthiness prediction. In: Mitkov, R., Angelova, G. (eds.) Proceedings of the International Conference on Recent Advances in Natural Language Processing, RANLP 2019, Varna, Bulgaria, September 2-4, 2019. pp. 1229–1239. INCOMA Ltd. (2019). https://doi.org/10.26615/978-954-452-056-4_141, https://doi.org/10.26615/978-954-452-056-4_141
  * [36] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., Rush, A.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. pp. 38–45. Association for Computational Linguistics, Online (Oct 2020). https://doi.org/10.18653/v1/2020.emnlp-demos.6, https://aclanthology.org/2020.emnlp-demos.6
  * [37] Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., Raffel, C.: mt5: A massively multilingual pre-trained text-to-text transformer. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 483–498 (2021)


11institutetext: Knowledge Technology, Department of Informatics, University
of Hamburg
11email: {ozan.oezdemir,matthias.kerzel,cornelius.weber,
<EMAIL_ADDRESS>
www.knowledge-technology.info

# Learning Flexible Translation between Robot Actions and Language
Descriptions

Ozan Özdemir  Matthias Kerzel  Cornelius Weber
Jae Hee Lee  Stefan Wermter

###### Abstract

Handling various robot action-language translation tasks flexibly is an
essential requirement for natural interaction between a robot and a human.
Previous approaches require change in the configuration of the model
architecture per task during inference, which undermines the premise of multi-
task learning. In this work, we propose the paired gated autoencoders (PGAE)
for flexible translation between robot actions and language descriptions in a
tabletop object manipulation scenario. We train our model in an end-to-end
fashion by pairing each action with appropriate descriptions that contain a
signal informing about the translation direction. During inference, our model
can flexibly translate from action to language and vice versa according to the
given language signal. Moreover, with the option to use a pretrained language
model as the language encoder, our model has the potential to recognise unseen
natural language input. Another capability of our model is that it can
recognise and imitate actions of another agent by utilising robot
demonstrations. The experiment results highlight the flexible bidirectional
translation capabilities of our approach alongside with the ability to
generalise to the actions of the opposite-sitting agent.

###### Keywords:

language grounding autoencoders multimodal fusion robot language learning
embodiment.

##  1 Introduction

Learning language involves multiple modalities such as audio, vision and
proprioception. For example, a colour word refers to a visual concept; sensing
the weight of an object is related to the concept of force; the concept of
position such as left and right can be learnt with proprioception and vision.
More modalities can be enumerated that help with learning language but the
essential component of language learning pertains to embodiment (i.e. having a
body and interacting in the environment) [4]. The embodied language learning
or language grounding has recently been a topic of interest at the crossroads
between natural language processing (NLP) and robotics [4, 1, 17, 13].

Figure 1: Our object manipulation scenario: a) an example action (‘push cyan
slowly’) undertaken by NICO [10] b) the same action undertaken by the opposite
agent as seen by NICO.

Inspired by the early language development in children in which interactions
in the environment are paired with language, language grounding approaches
have achieved learning representations, forming of abstractions, sequence-to-
sequence learning and bidirectional learning of action and language. However,
these approaches are not designed to endow a robot with the autonomy to
understand and choose the appropriate action to carry out during an
interaction with a human. They are either designed to carry out a single task
such as recognising an instruction and executing it [7, 17, 16, 13] or they
can handle multiple tasks but they require the task mode in advance to know
what is expected of them [14, 18, 19]. In contrast, a truly autonomous agent
must be able to decide whether to produce language or execute an action
according to the verbal instruction given by its human partner. Therefore,
end-to-end multimodal and multi-task models, which do not require adaptation
to new tasks by the experimenter, are desired.

Figure 2: The abstract architecture with four different tasks: ‘describe’,
‘execute’, ‘repeat action’ and ‘repeat language’. The PGAE architecture
consists of encoders, decoders and a GMU bottleneck. Each column shows our
architecture with the information flow (indicated by the arrows) per task. The
thick green arrows denote the main information flow in the respective signal’s
task.

In this work, we address the problem of flexible bidirectional translation
between robot actions and language. We define flexibility as translating
between the two modalities without having to reconfigure the model for a
specific task during inference. According to our scenario (Fig. 1), we expect
our agent to flexibly translate from action to language and vice versa, viz.
given textual descriptions, joint angle values and visual features, our agent
must either manipulate an object or describe the object manipulation act
carried by itself or the second agent depending on the situation. To this end,
we introduce the paired gated autoencoders (PGAE) architecture for flexible
translation between robot actions and language, realised by our humanoid robot
NICO [10, 9] in the simulation environment. PGAE includes an attention
mechanism in its bottleneck which allows the model to directly exchange
information between the action and language modalities. The attention
mechanism, which is adopted from the gated multimodal units (GMU) [3], acts as
a filter to pick information between the modalities across all dimensions.
Moreover, we signal the task by prepending a phrase to the language input and
ensure that our model recognises the task and is trained accordingly. Thus,
during inference, our model is able to do each of the translation tasks (Fig.
2) according to user input without having to configure the model in advance.
Further, we test a realistic setup in which the NICO robot can describe and
repeat the actions of the opposite-sitting second agent. Our experiment
results show that PGAE performs competitively in terms of translations between
language and action with the previous approaches [18, 19] that implicitly bind
the two modalities and in turn could use only some parts of the network, which
is set a priori according to the given translation task. Our contributions can
be summarised as:

  1. 1.

introducing an end-to-end neural network (NN) architecture that can flexibly
handle various action-language translation tasks during inference, consistent
with the training conditions,

  2. 2.

enabling the robot to recognise and imitate both the self-performed actions
and the actions of an opposite-sitting agent.

##  2 Related Work

Translation between language and action has been a topic of interest: there
are approaches that learn the general mapping between objects and language as
well as attributes like colour, texture and size [7, 17], and there exist
approaches that learn complex manipulation concepts [16, 13]. However, these
approaches can only translate from language to action as their focus is not on
bidirectional translation. Other approaches can translate from action to
language [8, 6]. Only few approaches [14, 2, 18, 1, 19] are capable of
bidirectional translation, i.e. they have the ability to translate a given
action into language as well as to translate a given language description into
an action.

Abramson et al. [1] propose a complex paradigm combining supervised learning,
reinforcement learning (RL) and imitation learning in order to solve the
problem of intelligently interacting in an abstract 3D play environment while
using language. In the environment, two agents communicate with each other as
one agent (setter) asks questions to or instructs the other (solver) that
answers questions and interacts with objects according to a given instruction.
However, the scenario is abstract as the objects are interacted with
unrealistically. Hence, proprioception is not used as the actions are
abstract. Therefore, the transfer of the approach from simulation to the real
world is non-trivial.

Ogata et al. [14] propose an RNN-based model to enable bidirectional
translation between compound sentences and robotic arm motions. Artificial
bias vectors are used to bind the two modalities, which have separate RNNs, to
enable flexible translation between them. Yamada et al. [18] introduce the
paired recurrent autoencoders (PRAE) approach which can bidirectionally
translate between robot actions and language in a one-to-one manner: each
action has exactly one description. Similar to [14], PRAE consists of
independent action and language networks and uses a binding loss to align the
hidden representations of the paired actions and descriptions. PRAE cannot
handle one-to-many mapping between action and language, i.e. when an action
can be described by multiple descriptions. Moreover, it cannot flexibly change
the direction of the translation during inference. For instance, to translate
from action to language, PRAE accepts joint values and visual features through
its action encoder and uses its description decoder alone to output a
description - it practically excludes its description encoder and action
decoder. Inspired by the PRAE architecture, we proposed the paired variational
autoencoders (PVAE) [19] that can translate between a robot action and its
multiple descriptions - we enabled one-to-many translation between actions and
descriptions by utilising the stochastic gradient variational Bayes-based
sampling (SGVB) [12] that randomises the hidden representation space so that
descriptions that are equivalent in meaning are represented tightly together,
whereas those that have different meanings are represented far from each
other. Like PRAE, PVAE employs the binding loss to map descriptions and
actions. Therefore, due to its artificial nature in its multimodality fusion,
PVAE too must be in a certain configuration according to the desired
translation direction. The aim of our work is therefore to lift this
constraint and allow flexible use of the model triggered by a verbally
provided signal.

##  3 Proposed Method: PGAE

Our paired gated autoencoders approach (PGAE) is a bidirectional translation
model between robot actions and language. As can be seen in Fig. 3, PGAE
consists of two autoencoders, namely language and action. It is intended to
associate simple robot actions like pushing a cube on the table with their
corresponding language descriptions. PGAE accepts as input language
descriptions, visual features extracted from images and joint angle values.
PGAE outputs language descriptions and joint angle values conditioned on
visual features. Moreover, PGAE is trained end-to-end with a signal prepended
to the language input indicating the expected output of the training
iteration. Five different signals are randomly chosen during training at each
iteration: ‘describe’, ‘execute’, ‘repeat action’, ‘repeat both’ and ‘repeat
language’.

Figure 3: The architecture of the PGAE model: Language encoder is either an
LSTM or the BERT model. The action encoder and both decoders are LSTMs - we
show unfolded versions of the LSTMs. The bottleneck, where the two streams are
connected, is based on the GMU; $z$ is the gating vector, whilst $h$ is the
shared representation vector.

###  3.0.1 Language Autoencoder

The language autoencoder (AE) accepts as input one-hot encoded words of a
description (or the whole description at once when BERT is used as language
encoder) and produces a description by outputting a word at each time step.
The language AE has an encoder, a decoder and hidden layers (in the
bottleneck) that contribute to the common hidden representations. The language
encoder embeds a description of length $N+1$, $(x_{1},x_{2},...,x_{N+1})$,
including the signal, into the final state $f_{N+1}$ as follows:

$\displaystyle h_{t}^{\text{enc}},c_{t}^{\text{enc}}$
$\displaystyle=\text{LSTM}(x_{t},h_{t-1}^{\text{enc}},c_{t-1}^{\text{enc}})\hskip
14.22636pt(1\leq t\leq N+1),$ $\displaystyle f_{N+1}^{\text{enc}}$
$\displaystyle=[h_{N+1}^{\text{enc}};c_{N+1}^{\text{enc}}],$

where $h_{t}^{\text{enc}}$ and $c_{t}^{\text{enc}}$ are the hidden and cell
state of the LSTM at time step $t$ respectively. $h_{0}^{\text{enc}}$ and
$c_{0}^{\text{enc}}$ are set as zero vectors, whereas $x_{1}$ is the signal
word. The square brackets $[.;.]$ denote the concatenation operation. The
LSTMs we use here and the action encoder and both decoders are a peephole LSTM
[15] following [18, 19]. The language encoder LSTM can also be replaced by a
pretrained language model to recognise unconstrained user instructions.
Specifically, we use the pretrained BERT Base model [5] as the language
encoder. This variation of the model is called PGAE-BERT.

The language decoder autoregressively generates the descriptions word by word
by expanding the shared latent representation vector $h$:

$\displaystyle h_{0}^{\text{dec}},c_{0}^{\text{dec}}$
$\displaystyle=W^{\text{dec}}\cdot h+b^{\text{dec}},$ $\displaystyle
h_{t}^{\text{dec}},c_{t}^{\text{dec}}$
$\displaystyle=\text{LSTM}(y_{t-1},h_{t-1}^{\text{dec}},c_{t-1}^{\text{dec}})\hskip
8.53581pt(1\leq t\leq N-1),$ $\displaystyle y_{t}$
$\displaystyle=\text{soft}(W^{\text{out}}\cdot
h_{t}^{\text{dec}}+b^{\text{out}})\hskip 14.22636pt(1\leq t\leq N-1),$

where soft represents the softmax activation function. $y_{0}$ is the vector
for the symbol indicating the beginning of the sentence, the $<$BOS$>$ tag.

###  3.0.2 Action Autoencoder

The action autoencoder (AE) accepts a sequence of joint angle values and
visual features as input and generates the appropriate joint angle values. It
consists of an encoder, a decoder and latent layers (in the bottleneck) that
contribute to the common latent representations. The action encoder encodes a
sequence of length $M$, $((j_{1},v_{1}),(j_{2},v_{2}),...,(j_{M},v_{M}))$ that
is the combination of joint angles $j$ and visual features $v$. The visual
features are extracted by the channel-separated convolutional autoencoder
(CAE) in advance. The action encoder can be defined as111Some symbols in the
equations coincide with the symbols used for the language AE.:

$\displaystyle h_{t}^{\text{enc}},c_{t}^{\text{enc}}$
$\displaystyle=\text{LSTM}(v_{t},j_{t},h_{t-1}^{\text{enc}},c_{t-1}^{\text{enc}})\hskip
14.22636pt(1\leq t\leq M),$ $\displaystyle f_{M}^{\text{enc}}$
$\displaystyle=[h_{M}^{\text{enc}};c_{M}^{\text{enc}}],$

where $h_{t}^{\text{enc}}$ and $c_{t}^{\text{enc}}$ are the hidden and cell
state of the LSTM at time step $t$. $h_{0}^{\text{enc}}$, $c_{0}^{\text{enc}}$
are set as zero vectors. $f_{M}^{\text{enc}}$ is the final state of the action
encoder.

The action decoder generates the joint angles at each time step by recursively
expanding the shared latent representation vector $h$:

$\displaystyle h_{0}^{\text{dec}},c_{0}^{\text{dec}}$
$\displaystyle=W^{\text{dec}}\cdot h+b^{\text{dec}},$ $\displaystyle
h_{t}^{\text{dec}},c_{t}^{\text{dec}}$
$\displaystyle=\text{LSTM}(v_{t},\hat{\jmath}_{t},h_{t-1}^{\text{dec}},c_{t-1}^{\text{dec}})\hskip
14.22636pt(1\leq t\leq M-1),$ $\displaystyle\hat{\jmath}_{t+1}$
$\displaystyle=\text{tanh}(W^{\text{out}}\cdot
h_{t}^{\text{dec}}+b^{\text{out}})\hskip 14.22636pt(1\leq t\leq M-1),$

where tanh is the hyperbolic tangent activation function and
$\hat{\jmath}_{1}$ is equal to ${j}_{1}$, i.e. joint angle values at the
initial time step. Visual features $v$ are used as in teacher forcing.

###  3.0.3 Bottleneck

The language and action streams connect at the bottleneck, which is situated
between the encoders and decoders of the model. We use a Gated Multimodal Unit
(GMU) to fuse the language and action (joints, images) modalities. Thanks to
its learned gating mechanism, GMU allows our model to flexibly learn multiple
tasks according to the given command such as ‘describe’ or ‘execute’. This
way, our approach works in different translation directions using the whole
model during inference without having to put the model in a specific mode. Our
bottleneck can be defined with the following equations:

$\displaystyle L_{\text{feats}}$
$\displaystyle=f_{N+1},A_{\text{feats}}=f_{M},$ $\displaystyle L_{h}$
$\displaystyle=\text{tanh}(W^{\text{L}}\cdot L_{\text{feats}}+b^{\text{L}}),$
$\displaystyle A_{h}$ $\displaystyle=\text{tanh}(W^{\text{A}}\cdot
A_{\text{feats}}+b^{\text{A}}),$ $\displaystyle z$
$\displaystyle=\sigma(W^{z}\cdot[A_{\text{feats}};L_{\text{feats}}]+b^{z}),$
$\displaystyle h$ $\displaystyle=z\odot A_{h}+(1-z)\odot L_{h},$

where $\sigma$ denotes the sigmoid activation function, whilst tanh stands for
the hyperbolic tangent activation function and $\odot$ is the Hadamard
product. $h$ represents the shared hidden representation vector and is used as
input to both language and action decoders.

###  3.0.4 Signals

Five different signals are used during training. Four of them can be used
during inference. According to the given signal, the input and output of the
model change.

  * •

Describe tells the model to describe the given action sequence, i.e. action-
to-language translation. With this signal, the model accepts as input the
sequence of visual features and joint angle values for the action as well as
the $<$EOS$>$ tag for language. The model is then trained to output the
correct description and the static joint angle values corresponding to the
final time step of the action sequence.

  * •

Execute signals the model to execute the given language description, i.e.
language-to-action translation. With this signal, the model expects to be fed
with the whole description sentence, and joint angle values and visual
features corresponding to the first time step of the action sequence. The
model is then expected to output the joint angle values of the action sequence
from the action decoder and the $<$EOS$>$ from the language decoder.

  * •

Repeat Action is the signal for reconstructing the sequence of joint angle
values. PGAE expects as input the sequence of joint angle values and visual
features for the action encoder and the $<$EOS$>$ tag in addition to the
signal for the language encoder. The action decoder reconstructs the joint
values and the language decoder outputs the $<$EOS$>$ tag.

  * •

Repeat Both signal demands the paired language and action input to be present.
With this signal, PGAE is trained similar to PVAE [19] end-to-end. The
language encoder accepts the full description in addition to the phrase
‘repeat both’, whilst the action encoder accepts the corresponding action
sequence (joint values and visual features). The language decoder and action
decoder outputs the full description and joint angle values correspondingly.
This signal is intended to be used only during training, since it is
implausible to expect the robot to repeat an action and its description at the
same time.

  * •

Repeat Language is used for reconstructing the full description. The full
description and the first time step of the action sequence are fed as input to
the encoders. As output, the language decoder reconstructs the description and
the action decoder outputs the joint angle values of the first time step.

###  3.0.5 Visual Feature Extraction

Following the previous work [19], we employ the channel-separated
convolutional autoeconder architecture (CAE) to extract the visual features
from images captured by the NICO robot. Instead of processing all three
channels together, we train an instance of the CAE for each colour channel
(red, green and blue) – i.e. channel separation. The channel separation
technique has been shown to distinguish between the colours of the objects
more accurately [19]. Each channel of $120\times 160$ RGB images fed into the
channel-separated CAE at a time. The channel-separated CAE consists of a
convolutional encoder, a fully-connected bottleneck and a deconvolutional
decoder. After training for the RGB channels separately, the channel-specific
visual features are extracted from the bottleneck and then concatenated. The
resulting visual features $v$ are used as input to PGAE. For more details of
the channel-separated CAE, please refer to the PVAE paper [19].

###  3.0.6 Loss Function

The overall loss is calculated by adding up the reconstruction losses, i.e.
language loss and action loss. The language loss, $L_{\text{lang}}$, is
defined as the cross entropy loss between input and output words, whereas the
action loss, $L_{\text{act}}$, is defined as the mean squared error (MSE)
between original and predicted joint values:

$L_{\text{lang}}=\frac{1}{N-1}\sum_{t=1}^{N-1}\left(-\sum_{i=0}^{V-1}w^{\left[i\right]}x_{t+1}^{\left[i\right]}\log
y_{t}^{\left[i\right]}\right),$
$L_{\text{act}}=\frac{1}{M-1}\sum_{t=1}^{M-1}\left\|j_{t+1}-\hat{\jmath}_{t+1}\right\|_{2}^{2},$

where $V$ is the vocabulary size, $N$ is the number of words per description,
M is the sequence length for an action trajectory and $w$ is the weight vector
used to counter the imbalance in the frequency of words. The overall loss is
the sum of the language and action loss:

$L_{\text{all}}=\alpha L_{\text{lang}}+\beta L_{\text{act}}$

where $\alpha$ and $\beta$ are weighting factors for language and action terms
in the loss function. In our experiments, we set $\alpha$ and $\beta$ to 1.

###  3.0.7 Training Details

To train PGAE and PGAE-BERT, we first extract visual features using our
channel-separated CAE. The visual features are used to condition the actions
depending on the cube arrangement, i.e., the action execution according to a
given description is dependent on the position of the target cube. Both PGAE
and PGAE-BERT are trained end-to-end. PGAE and PGAE-BERT are trained for 6,000
epochs with the gradient descent algorithm and Adam optimiser [11]. In our
experiments, $h$ has 50 dimensions, $x$ has 28 dimensions, $j$ has 5
dimensions, $N$ is equal to 5 and $M$ is 50 for fast and 100 for slow actions.
We take the learning rate as $10^{-5}$ with a batch size of 6 samples after
determining them as optimal hyperparameters. After a few trials, we have
decided to freeze the weights of BERT during training as fine-tuning it
reduces the performance of our model. Since BERT has millions of parameters,
fine-tuning it leads to overfitting.

##  4 Experiment Results

We train and test our model on the paired robot actions and descriptions
dataset [19] that involves 864 samples of sequences of images, joint values
and textual descriptions. The dataset consists of simple manipulation of two
cubes of different colours on the table by the humanoid NICO robot. The NICO
robot is a child-size humanoid robot with a camera in each of its two eyes.
The dataset was created using the Blender software222https://www.blender.org/.
According to our scenario, using its left arm, NICO manipulates one of the two
cubes on the table for each sample utilising the inverse kinematics solver
provided on Blender. Each sample includes a sequence of first-person view
images and joint angle values from NICO’s left arm alongside a textual
description of the action. In total, the dataset includes 12 distinct actions,
6 cube colours, 288 descriptions and 144 patterns (action-description-cube
arrangement combinations). We slightly vary the 144 patterns six times
randomly in terms of action execution in simulation. Out of 864 samples, we
exclude 216 samples that involve every unique description and action type and
use them as the test set. By carefully selecting the test samples, we ensure
that the combinations of descriptions, action types and cube arrangements in
the test set are not seen during training. For more details on the dataset,
please consult the PVAE paper [19]. Additionally, we introduce a second agent
that does the same actions and include the resulting images from the NICO’s
perspective. We use the visual features extracted from these images as
additional input, and randomly select between them and the images that show
NICO doing the actions. These cases are shown in Table 1, with the ‘-opposite’
suffix for describing or repeating the action of the second agent, and with
the ‘-self’ suffix for describing or repeating the action of NICO. Therefore,
PGAE-self and PGAE-opposite are the same model trained on the same dataset –
the former is tested with self-agent (NICO’s own) actions, while the latter
with the second-agent actions. PGAE and PGAE-BERT are trained on the previous
dataset as PVAE [19]. We test the models on action-to-language, language-to-
action, action-to-action and language-to-language translations as shown in
Table 1.

Table 1: Translation results for test set. Green background denotes the main output of the respective task (e.g., description for the action-to-language translation). As the opposite-sitting agent data is irrelevant for ‘repeat language’ and ‘execute’ tasks, PGAE-self and PGAE-opposite share those results. | Describe | Repeat Lang. | Execute | Repeat Act.
---|---|---|---|---
| Act.→Lang. | Lang.→Lang. | Lang.→Act. | Act.→Act.
Approach | Descr. Acc. | Descr. Acc. | Descr. Acc. | Descr. Acc.
PGAE | 93.05% | 96.30% | 100% | 100%
PGAE-BERT | 94.91% | 99.07% | 100% | 100%
PGAE-self | 80.56% |  |  | 100%
PGAE-opposite | 65.28% | 93.98% | 100% | 100%
Approach | J. Err. (nRMSE) | J. Err. (nRMSE) | J. Err. (nRMSE) | J. Err. (nRMSE)
PGAE | 0.23% | 0.37% | 0.44% | 0.44%
PGAE-BERT | 0.21% | 0.33% | 0.44% | 0.42%
PGAE-self | 0.58% |  |  | 0.89%
PGAE-opposite | 2.40% | 0.73% | 0.79% | 0.80%

###  4.0.1 Action-to-Language Translation

PGAE and its variants use the ‘describe’ signal as input to the language
encoder to translate from action to language. The first result column of Table
1 reports the accuracy of predicted test descriptions for different
approaches. In order for a generated description to be accepted as correct,
all of its words must match the ground truth description according to our
predefined grammar, i.e. action-colour-speed-$<$EOS$>$. Moreover, the second
half of the table shows the normalised root-mean-square error (nRMSE) between
the generated and ground truth joint values. We calculate nRMSE values by
dividing the square root of the MSE (which is used as the action loss
$L_{\text{act}}$) by the observed range of joint values. Accordingly, PGAE is
competitive with the earlier approach, PVAE (achieves 100%, not given in the
table), that has to be set in the specific configuration, which includes using
only the action encoder and language decoder while bypassing the language
encoder and action decoder by avoiding feeding any language input and not
outputting the final joint values. Therefore, PVAE cannot output joint values
when it is configured to be used for action-to-language translation. Both PGAE
and PGAE-BERT, however, recognise the task from the signal in the language
input and generate both the description and the joint values for all different
tasks. Both PGAE and PGAE-BERT achieve near perfect joint value prediction.
Moreover, PGAE-BERT performs slightly better than PGAE in terms of description
accuracy. However, its main advantage over PGAE is that it has the potential
to recognise unconstrained natural language due to the use of a pretrained
language model.

In the setting where we demand the model to describe the action done by the
opposite agent, our model achieves 65% accuracy (PGAE-opposite). It achieves
80% accuracy when describing NICO’s own actions (PGAE-self). Moreover, the
joint value error also increases slightly to over 2% for the case opposite-
agent case, whereas it is comparable with the original approaches for PGAE-
self. The decrease in the action-to-language accuracy is expected as
introducing the second-agent actions makes the problem more challenging, e.g.,
pulling an object by the second agent might be interpreted as pushing the
object by NICO itself. Moreover, this is an extra capability demonstrated by
our approach.

###  4.0.2 Language-to-Action Translation

PGAE and its variants use the ‘execute’ signal prepended to the description
for the translation from language to action. The description accuracy (whether
$<$EOS$>$ is outputted by the language decoder) and nRMSE between predicted
and ground truth joint values for language to action translation are given in
Column ‘Lang.→Act.’. All of the approaches are able to generate near perfect
joint values (less than 1% nRMSE). However, PVAE is not trained to generate
descriptions ($<$EOS$>$ in this case) when it is configured to execute
descriptions (N.A.). This highlights the superiority of using signals as part
of the language input and having a common hidden representation vector over
the artificial use of a loss term to align two separate streams. Training PGAE
with the demonstrations from the opposite-sitting agent does not significantly
affect the action-to-language performance (0.79% for PGAE-self/PGAE-opposite).

###  4.0.3 Language-to-Language & Action-to-Action Translations

PGAE and PGAE-BERT are competitive with PVAE (achieves 100%, not given in the
table) in terms of the description accuracy for language-to-language
translation and slightly better in terms of the joint value prediction for the
action-to-action translation. PVAE does not have the capacity to output joint
values for language-to-language and descriptions for action-to-action
translations, whereas PGAE and PGAE-BERT almost perfectly output the initial
time-step joint values for language-to-language translation and achieves
perfect description accuracy in action-to-action translation. Training PGAE
with the additional opposite-sitting agent demonstrations slightly increases
the joint value error in action-to-action translation for both the actions
executed by NICO (PGAE-self) and by the second agent (PGAE-opposite).

##  5 Conclusion

We have introduced an end-to-end NN approach that can flexibly perform
translation between robot actions and language descriptions in multiple
directions, some of which involving both first-person actions and opposite-
sitting agent actions. By integrating the task signal in the language input,
our approach can recognise the given task and output the suitable descriptions
and joint values during inference. Our approach, PGAE, exhibits competitive
performance in all four translation tasks while having a consistent
configuration across learning and inference. With the additional
demonstrations from a second agent, our model can not only recognise and
imitate its own actions but also the actions of the second agent despite the
challenging nature of the task. To our knowledge, this skill set has not been
modelled by previous approaches. In summary, PGAE can perform various
translation tasks robustly without any change in the use of the architecture
between learning and test time, which the previous approaches lacked, through
its attention-based explicit multimodal fusion mechanism and the insertion of
the task signal to the language input. Furthermore, the realism in our
simulation promises sim-to-real transfer, which we will tackle in the future.
Another avenue is to embed our model into a continuous human-robot dialogue
framework in a closed-loop. Finally, we can utilise RL for more dexterous
object manipulation with diverse ways to execute an action.

## Acknowledgements

The authors gratefully acknowledge support from the German Research Foundation
DFG under Project CML (TRR 169).

## References

  * [1] Josh Abramson, Arun Ahuja, Iain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, et al.  Imitating interactive intelligence.  arXiv preprint arXiv:2012.05672, 2020.
  * [2] Alexandre Antunes, Alban Laflaquiere, Tetsuya Ogata, and Angelo Cangelosi.  A bi-directional multiple timescales LSTM model for grounding of actions and verbs.  In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2614–2621, 2019.
  * [3] John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González.  Gated multimodal networks.  Neural Computing and Applications, 32(14):10209–10228, 2020. https://doi.org/10.1007/s00521-019-04559-1.
  * [4] Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian.  Experience grounds language.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8718–8735. Association for Computational Linguistics, November 2020.
  * [5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In NAACL-HLT (1), 2019.
  * [6] Aaron Eisermann, Jae Hee Lee, Cornelius Weber, and Stefan Wermter.  Generalization in multimodal language learning from simulation.  In Proceedings of the International Joint Conference on Neural Networks (IJCNN 2021), Jul 2021.
  * [7] Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, and Jethro Tan.  Interactively picking real-world objects with unconstrained spoken language instructions.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3774–3781. IEEE, 2018.
  * [8] Stefan Heinrich, Yuan Yao, Tobias Hinz, Zhiyuan Liu, Thomas Hummel, Matthias Kerzel, Cornelius Weber, and Stefan Wermter.  Crossmodal language grounding in an embodied neurocognitive model.  Frontiers in Neurorobotics, 14:52, 2020.
  * [9] Matthias Kerzel, Theresa Pekarek-Rosin, Erik Strahl, Stefan Heinrich, and Stefan Wermter.  Teaching NICO how to grasp: an empirical study on crossmodal social interaction as a key factor for robots learning from humans.  Frontiers in Neurorobotics, 14:28, 2020.
  * [10] Matthias Kerzel, Erik Strahl, Sven Magg, Nicolás Navarro-Guerrero, Stefan Heinrich, and Stefan Wermter.  NICO—Neuro-Inspired COmpanion: A developmental humanoid robot platform for multimodal interaction.  In 2017 26th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), pages 113–120. IEEE, 2017.
  * [11] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR, San Diego, CA, USA, May 7-9, 2015.
  * [12] Diederik P. Kingma and Max Welling.  Auto-encoding variational Bayes.  In Proceedings of International Conference on Learning Representations (ICLR), Banff, AB, Canada, April 14-16, 2014.
  * [13] Corey Lynch and Pierre Sermanet.  Language conditioned imitation learning over unstructured data.  Robotics: Science and Systems, 2021.
  * [14] Tetsuya Ogata, Masamitsu Murase, Jun Tani, Kazunori Komatani, and Hiroshi G. Okuno.  Two-way translation of compound sentences and arm motions by recurrent neural networks.  In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1858–1863, 2007.
  * [15] Haşim Sak, Andrew Senior, and Françoise Beaufays.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling.  In Proceedings of Interspeech 2014, pages 338–342, 2014.
  * [16] Lin Shao, Toki Migimatsu, Qiang Zhang, Karen Yang, and Jeannette Bohg.  Concept2robot: Learning manipulation concepts from instructions and human demonstrations.  In Proceedings of Robotics: Science and Systems (RSS), 2020.
  * [17] Mohit Shridhar, Dixant Mittal, and David Hsu.  INGRESS: Interactive visual grounding of referring expressions.  The International Journal of Robotics Research, 39(2-3):217–232, 2020.
  * [18] Tatsuro Yamada, Hiroyuki Matsunaga, and Tetsuya Ogata.  Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions.  IEEE Robotics and Automation Letters, 3(4):3441–3448, 2018.
  * [19] Ozan Özdemir, Matthias Kerzel, and Stefan Wermter.  Embodied language learning with paired variational autoencoders.  In 2021 IEEE International Conference on Development and Learning (ICDL), pages 1–6, Aug 2021.