# SemEval-2021 Task 4: Reading Comprehension of Abstract Meaning

Boyuan Zheng2 , Xiaoyu Yang1, Yu-Ping Ruan3, Zhenhua Ling3,
Quan Liu3, Si Wei4, Xiaodan Zhu1
1Queen’s University, Kingston, Canada; 2Northeastern University, Shenyang,
China
3University of Science and Technology of China; 4iFlytek Research, Hefei,
China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>This work was performed when Boyuan Zheng visited
Queen’s University.

###### Abstract

This paper introduces the SemEval-2021 shared task 4: Reading Comprehension of
Abstract Meaning (ReCAM). This shared task is designed to help evaluate the
ability of machines in representing and understanding abstract concepts. Given
a passage and the corresponding question, a participating system is expected
to choose the correct answer from five candidates of abstract concepts in a
cloze-style machine reading comprehension setup. Based on two typical
definitions of abstractness, i.e., the imperceptibility and nonspecificity,
our task provides three subtasks to evaluate the participating models.
Specifically, Subtask 1 aims to evaluate how well a system can model concepts
that cannot be directly perceived in the physical world. Subtask 2 focuses on
models’ ability in comprehending nonspecific concepts located high in a
hypernym hierarchy given the context of a passage. Subtask 3 aims to provide
some insights into models’ generalizability over the two types of
abstractness. During the SemEval-2021 official evaluation period, we received
23 submissions to Subtask 1 and 28 to Subtask 2. The participating teams
additionally made 29 submissions to Subtask 3. The leaderboard and competition
website can be found at https://competitions.codalab.org/competitions/26153.
The data and baseline code are available at
https://github.com/boyuanzheng010/SemEval2021-Reading-Comprehension-of-
Abstract-Meaning.

##  1 Introduction

Humans use words with abstract meaning in their daily life. In the past,
research efforts have been exerted to better understand and model abstract
meaning Turney et al. (2011); Theijssen et al. (2011); Changizi (2008); Spreen
and Schulz (1966). Modelling abstract meaning is closely related to many other
NLP tasks such as reading comprehension, metaphor modelling, sentiment
analysis, summarization, and word sense disambiguation.

In the past decade, significant advancement has been seen in developing
computational models for semantics, based on deep neural networks. In this
shared task, we aim to help assess the capability of the state-of-the-art deep
learning models on representing and modelling abstract concepts in a specific
reading comprehension setup.

We introduce SemEval-2021 Task 4, Reading Comprehension of Abstract Meaning
(ReCAM). Specifically, we design this shared task by following the machine
reading comprehension framework Hermann et al. (2015); Onishi et al. (2016);
Hill et al. (2016), in which computers are given a passage ${D_{i}}$ as well
as a human summary ${S_{i}}$ to comprehend. If a model can digest the passage
as humans do, we expect it to predict the abstract word used in the summary,
if the abstract word is masked. Unlike the previous work that requires
computers to predict concrete concepts, e.g., named entities, in our task we
ask models to fill in abstract words removed from human summaries. During the
SemEval-2021 official evaluation period, we received 23 submissions to Subtask
1 and 28 submissions to Subtask 2. The participating teams additionally made
29 submissions to Subtask 3. In this paper, we induce the shared task and
provide a summary for the evaluation.

##  2 Task Description

We organize our shared task based on two typical definitions of abstractness,
named as imperceptibility and nonspecificity in this paper, implemented in
Subtask 1 and Subtask 2, respectively. Subtask 3 further evaluates models’
generalizability over the two definitions of abstractness.

###  2.1 Subtask 1: ReCAM-Imperceptibility

In one definition Turney et al. (2011); Theijssen et al. (2011); Spreen and
Schulz (1966), concrete words refer to things, events, and properties that
humans can directly perceive with their senses, e.g., trees and flowers. In
contrast, abstract words refer to “ideas and concepts that are distant from
immediate perception”, e.g., objective, culture, and economy. In Subtask 1, we
perform reading comprehension on imperceptible abstract concepts, named as
ReCAM-ImPerceptibility. Table 1 shows an example.

Passage |  … Observers have even named it after him, “Abenomics”. It is based on three key pillars of monetary policy to ensure long-term sustainable growth in the world’s third-largest economy, with fiscal stimulus and structural reforms. In this weekend’s upper house elections, ….
---|---
Question |  Abenomics: The @placeholder and the risk.
Answer |  (A) chance (B) prospective (C) government (D) objective (E) threat
Table 1: An example for Subtask 1. The correct answer to the question is
objective.

###  2.2 Subtask 2: ReCAM-NonSpecificity

The second typical definition of abstractness is based on nonspecific concepts
Theijssen et al. (2011); Spreen and Schulz (1966). Compared to specific
concepts such as groundhog and whale, words such as vertebrate are regarded as
more abstract. Our Subtask 2, named as ReCAM-NonSpecificity, is designed based
on this viewpoint. We will discuss how the datasets are constructed in Section
3.

###  2.3 Subtask 3: ReCAM-Cross

In this subtask, participants are asked to submit their predictions on the
test data of Subtask 2, using models trained on the training data of Subtask
1, and vice versa. This subtask aims to demonstrate models’ generalizability
between modelling the two typical definitions of abstractness.

##  3 Data Construction

We develop our multi-choice machine reading comprehension datasets based on
the XSum summarization dataset Narayan et al. (2018). We first locate words
with abstract meaning using our abstractness scorers. Then we perform data
filtering to select our target words to construct our datasets.

###  3.1 The XSum Data

By collecting online articles from the British Broadcasting Corporation (BBC),
Narayan et al. (2018) developed a large-scale text summarization dataset,
XSum, in which each article has a single sentence summary. We developed our
ReCAM dataset based on XSum.

###  3.2 Finding Imperceptible Concepts

#### Abstractness Scorer for Imperceptibility

Following Turney et al. (2011), we use the MRC Psycholinguistic Database
Coltheart (1981), which includes 4,295 words rated with a degree of
abstractness by human subjects, to train our abstractness scorer for
imperceptibility. The rating of the words in the MRC Psycholinguistic Database
ranges from 158 (highly abstract) to 670 (highly concrete). We linearly scale
the rating to the range of 0 (highly abstract) to 1 (highly concrete). The
neural regression model accepts fixed Glove embedding Pennington et al. (2014)
as input and predicts the abstractness rating score between 0 and 1. Our
regression model is a three-layer network that consists of two non-linear
hidden layers with the ReLU activation and a sigmoid output layer. The mean
square error (MSE) is used as the training loss.

To test the regression model’s performance, we randomly split the MRC
Psycholinguistic Database into train and test set with the size of 2,148 and
1,877, respectively. Table 2 shows the final performance of the neural
regression model on the MRC database. We use the Pearson correlation between
ratings predicted by models and original ratings from MRC as the evaluation
metric. We can see that the regression model achieves high correlation
coefficients (the higher, the better), i.e., 0.934 and 0.835, on the training
and test set. The correlations are significant ($p$-values are smaller than
$10^{-5}$), reflecting the quality of our models in finding abstract words.
Note that Turney et al. (2011) report a correlation score of $0.81$ on their
MRC test set. Their training-test split is unavailable, so we run cross-
validation here in our experiment. The scorer can then be used to assign an
imperceptibility score to a word that is not in the MRC Psycholinguistic
Database.

| #samples | Pearson $r$ |  $p$-value
---|---|---|---
train | 2,148 | 0.934 | $p<10^{-5}$
test | 1,877 | 0.854 | $p<10^{-5}$
Table 2: Fitting performance of neural regression model on the MRC database.

Using the abstractness scorer described above, we assign an abstractness value
to each word in summaries and select words with a value lower than 0.35 as the
candidates for our target words (words that will be removed from the summaries
to construct questions). We only consider content words as potential target
words, i.e., nouns, verbs, adjectives, and adverbs. For this purpose, we use
part-of-speech tagging model qi-etal-2018-universal implemented in Stanza Qi
et al. (2020).

###  3.3 Finding Nonspecific Concepts

#### Nonspecificity Scorer

Following the work of Changizi (2008), we assign a nonspecificity score to a
word token based on the hypernym hierarchy of WordNet Miller (1998).
Specifically, the root of the hierarchy is at level 0 and regarded as the most
abstract. The abstractness of a node in the hierarchy is measured by the
maximal length of its path to the root. The hypernym level in WordNet is
between 0 and 17. For each word token in summaries, we use Adapted Lesk
Algorithm Banerjee and Pedersen (2002) to label the sense since the WordNet
hypernym hierarchy works at the sense level. Since a summary sentence may be
short, we concatenate each summary sentence with the corresponding passage for
word sense disambiguation. Built on this, each token, which is labelled with a
sense, receives an abstractness score based on the WordNet hierarchy.

Using the nonspecificity scorer, we assign an nonspecificity value to each
word in summaries and select words with a value smaller than six as the
candidate target words. The targets words will be nouns and verbs since the
hypernym hierarchy in WordNet marciniak-2020-wordnet consists of these two POS
types.

###  3.4 Filtering

We aim to avoid developing simple questions. For example, if a target word
also appears in the passage, it is likely that a model can easily find the
answer without the need to understand the passage in depth.

#### Filtering by Lemmas

We lemmatized passages and summaries. If a lemma appears both in a summary and
the corresponding passage, the lexemes of the lemma will not be considered as
target words. Note that a strict filter may exclude some good candidates for
target words but helps avoid introducing many simple questions.

#### Filtering by Synonyms and Antonyms

For a word in a summary, if a synonym or antonym of the word appears in the
corresponding passage, we will not consider this word to be our target word.
We use WordNet marciniak-2020-wordnet to derive synonyms and antonyms. Instead
of using word sense disambiguation (WSD), for a word $w_{i}$ in a summary, we
use all senses of this word and add all synonyms and antonyms into a pool.
Only if none of the words in the pool appear in the passage, we consider
$w_{i}$ as a candidate target word. Otherwise, we will not use $w_{i}$ to
construct a question for this passage-summary pair.

#### Filtering by Similarity

We further filter words by similarity. For each candidate target word in a
summary and each word in the passage, we calculate similarity and use that to
perform further filtering.

We use 300-dimension GloVe word embedding trained on 840 billion tokens
Pennington et al. (2014). We calculate the cosine similarity between a
candidate target word and a passage word. For contextual embedding, we embed
each sentence in a passage as well as the summary into a context-aware
representation matrix using the BERT-large uncased language model. Then, we
calculate the similarity between each passage token and question token with
the cosine similarity. If the similarity is higher than 0.85, we will not
consider the involved summary words as candidate target words.

###  3.5 Constructing Multiple Choices

We train machine reading comprehension models using the data built so far to
generate four choices for each question. Together with the ground-truth (the
target word identified above and removed from the human summary), we have five
choices/options for each question. In our work, we propose to use three
models, Gated-Attention Reader Hermann et al. (2015), Attentive Model and
Attention Model with Word Gloss to generate the candidate options. Please find
details of the models in Appendix B and Appendix C as well as the training
details in Appendix D.

We adopt the idea of k-fold cross validation to train the above mentioned
three models to generate candidate answer words. Specifically, we split the
data into 4 folds. Each time, we train the baseline models on 3 folds of data
and use the trained models to predict candidate words on the remaining 1-fold
data. With 4-fold iteration, we obtain predication of each model on the entire
data. The performance of the three baseline models are listed in Table 3 for
Subtask 1 and Table 4 for Subtask 2, using several typical retrieval-based
evaluation metrics.

For each target word that has been removed from the corresponding summary
sentence (again, a question is a summary sentence containing a removed target
word), we collect top-10 words predicted by each of the three models. In this
way, we can collect a candidate word pool of 30 predicted word tokens for each
removed target word. To avoid including multiple correct choices for each
question, we adopt synonym and context similarity filtering methods described
in Section 3.4. Specifically we first calculate similarity between the ground-
truth target word and each word type in the pool. We exclude a word type from
the multiple choices if its similarity to the ground-truth is higher than
0.85. In addition, we also exclude synonyms of the ground-truth target word.
For the remaining word tokens in the pool, we select four most frequent word
types (a word type may have multiple tokens in the pool). Together with the
ground-truth word, we obtain five choices for each question.

|  MRR |  R@1 |  R@5 |  R@10
---|---|---|---|---
GAReader |  0.245 |  0.175 |  0.314 |  0.378
AttReader |  0.235 |  0.167 |  0.300 |  0.363
+gloss |  0.179 |  0.123 |  0.227 |  0.276
Table 3: Three baseline models are used to generate candidate multiple choices for Subtask 1. The table shows their performance on the XSum dataset, evaluated with MRRCraswell (2009), Recall@1, Recall@5, and Recall@10. |  MRR |  R@1 |  R@5 |  R@10
---|---|---|---|---
GAReader |  0.343 |  0.268 |  0.422 |  0.484
AttReader |  0.348 |  0.273 |  0.424 |  0.490
+gloss |  0.228 |  0.166 |  0.286 |  0.345
Table 4: Three baseline models are used to generate candidate multiple choices
for Subtask 2. The table shows their performance on the XSum dataset,
evaluated with MRR, Recall@1, Recall@5, and Recall@10.

###  3.6 Further Quality Control

We further make the following efforts to remove noise in the dataset and
improve the datasets’ quality. We observe that up to now, there are mainly two
kinds of noise in our dataset: 1) some target words cannot be inferred solely
based on the corresponding passage; 2) more than one of the multiple choices
are correct answers.

The first issue is mainly related to the property of the XSum dataset, in
which the first sentence of a passage is used as the summary. The second type
of problems are often caused by our automatic generation method. Although we
have applied strict rules in Section 3.4 to handle this, among a small portion
of the resulting data, multiple potentially correct answers still exist in
candidate answers.

To further ensure the quality of our dataset, we invite workers in Amazon
Mechanical Turk to perform further data selection. Each annotator needs to
follow the procedure of Appendix A to answer the question and annotate
relevant information, with which further data selection is applied. To ensure
quality, we only include workers from English-speaking countries and only if
their previous HITs’ approval rates are above 90%. To see more details about
this process, please refer to Appendix E.

###  3.7 ReCAM Data Statistics

Table 5 lists the size of our ReCAM datasets, i.e., numbers of questions. For
example, in total Subtask 2 has 6,186 questions, which are split into
training/development/test subsets.

Dataset | Subtask 1 | Subtask 2 | Total
---|---|---|---
Train | 3,227 | 3,318 | 6,545
Dev | 837 | 851 | 1,688
Test | 2,025 | 2,017 | 4,042
Total | 6,089 | 6,186 | 12,275
Table 5: Size of the ReCAM Dataset.

##  4 Systems and Results

Our shared task received 23 submissions to Subtask 1, 28 submissions to
Subtask 2, and 29 submissions to Subtask 3. We use accuracy as the evaluation
metric for the three subtasks.

In general, most participating teams use pre-trained language models in their
systems such as BERT Devlin et al. (2019), ALBERT Lan et al. (2020),
DistilBERT Sanh et al. (2019), RoBERTa Liu et al. (2019), ELECTRA Clark et al.
(2020), DeBERTa He et al. (2020), XLNet Yang et al. (2019), T5 Raffel et al.
(2020). Data augmentation, external knowledge resources, and/or transfer
learning are additionally used by many teams to further enhance their model
performance.

###  4.1 Subtask 1: ReCAM-Imperceptibility

Table 6 shows all the official submissions and most of them outperform the
baseline model. The baseline used for Subtask 1 is the Gated-Attention (GA)
Reader Dhingra et al. (2017). The GA Reader uses a multi-layer iterated
architecture with a gated attention mechanism to derive better query-aware
passage representation. The motivation behind using GA Reader is to have a
simple comparison between our task and the CNN/Daily Mail reading
comprehension dataset since GA Reader achieves reasonably good performance on
the CNN/Daily Mail reading comprehension dataset.

Note that the last column of the table lists the accuracy (Acc. Cross) for
models trained on the Subtask 2 training data and tested on the Subtask 1
testset. We will discuss those results later in Section 4.3.

Rank |  Team | Acc | Acc. Cross
---|---|---|---
- |  GA Reader | 25.1 | -
1 |  SRC-B-roc | 95.1 | 91.8 ($\downarrow$ 3.3)
2 |  PINGAN-Omini-Sinitic | 93.0 | 91.7 ($\downarrow$ 1.3)
3 |  ECNU-ICA-1 | 90.5 | 88.6($\downarrow$ 1.9)
4 |  tt123 | 90.0 | 86.2($\downarrow$ 3.8)
5 |  cxn | 88.7 | -
6 |  nxc | 88.6 | 74.2($\downarrow$ 14.4)
7 |  ZJUKLAB | 87.9 | -
8 |  IIE-NLP-Eyas | 87.5 | 82.1($\downarrow$ 5.4)
9 |  hzxx1997 | 86.7 | -
10 |  XRJL | 86.7 | 81.8($\downarrow$ 4.9)
11 |  noobs | 86.2 | 78.6($\downarrow$ 7.6)
12 |  godrevl | 83.1 | -
13 |  ReCAM@IITK | 82.1 | 80.7($\downarrow$ 1.4)
14 |  DeepBlueAI | 81.8 | 76.3($\downarrow$ 5.5)
15 |  LRG | 75.3 | 61.8($\downarrow$ 13.5)
16 |  xuliang | 74.7 | -
17 |  Llf1206571288 | 72.8 | -
18 |  Qing | 71.4 | -
19 |  NEUer | 56.6 | 51.8($\downarrow$ 4.8)
20 |  CCLAB | 46.3 | 35.2($\downarrow$ 11.1)
21 |  UoR | 42.0 | 39.4($\downarrow$ 2.6)
22 |  munia | 19.3 | -
23 |  BaoShanCollege | 19.0 | -
Table 6: Official results of Subtask 1 and Subtask 3. Acc is the accuracy of
the models trained on the Subtask 1 training data and tested on the Subtask 1
testset. Acc. cross is the accuracy of models trained on the Subtask 2
training data and tested on the Subtask 1 testset.

The best result in Subtask 1 was achieved by team SRC-B-roc Zhang et al.
(2021) with an accuracy of 0.951. The system was built on a pre-trained
ELECTRA discriminator and it further applied upper attention and auto-
denoising mechanism to process long sequences. The second-placed system,
PINGAN omini-Sinitic Wang et al. (2021), adopted an ensemble of ELECTRA-based
models with task-adaptive pre-training and a mutli-head attention based
multiple-choice classifier. ECNU-ICA-1 Liu et al. (2021) ranked third in this
subtask with a knowledge-enhanced Graph Attention Network and a semantic space
transformation strategy.

Most teams in Subtask 1 utilize pre-trained language models (PLM), like BERT
Devlin et al. (2019), ALBERT Lan et al. (2020), DistilBERT Sanh et al. (2019),
RoBERTa Liu et al. (2019), ELECTRA Clark et al. (2020), DeBERTa He et al.
(2020), XLNet Yang et al. (2019), T5 Raffel et al. (2020). SRC-B-roc Zhang et
al. (2021) conducted an ablation study regarding the performance discrepancy
of different transformers-based pre-training models. They tested BERT, ALBERT,
and ELECTRA by directly fine-tuning the pre-trained LMs on the ReCAM data.
ELECTRA outperforms BERT and ALBERT by large margins, which may be due to the
different learning objectives of these pre-trained models.

Most participating systems performed intermediate task pre-training
Pruksachatkun et al. (2020) for their language models. For example, CNN/Daily
Mail dataset was selected by ZJUKLAB Xie et al. (2021a) to further pre-train
their language models. The CNN/Daily Mail dataset and Newsroom dataset boost
model performance on both Subtask 1 and Subtask 2. Data augmentation methods
are also popular among participants. ZJUKLAB Xie et al. (2021a) performed
negative data augmentation with a language model to leverage misleading words.
IIE-NLP-Eyas Xie et al. (2021b) adopted template-based input reconstruction
methods to augment their dataset and further fine-tuned their language models
based on the dataset.

Most teams also used an ensemble of multiple pre-trained language models to
further enhance model performance. SRC-B-roc Zhang et al. (2021) applied Wrong
Answer Ensemble Kim and Fung (2020) by training the model to learn the correct
and wrong answer separately and ensembled them to obtain the final
predictions. Stochastic Weight Averaging Izmailov et al. (2018) was also
performed across multiple checkpoints in the same run to achieve better
generalization.

In addition, some interesting approaches were additionally used to tackle the
task from different perspectives. PINGAN omini-Sinitic Wang et al. (2021)
turned the original multi-choice task into a masked-sentence classification
task by adding each option to the placeholder. Noise detection methods and
auto denoising methods were further proposed by adding a noise-tolerant loss.
ZJUKLAB Xie et al. (2021a) used label smoothing to encourage the activations
of the penultimate layer. ECNU-ICA-1 Liu et al. (2021) utilized a semantic
space transformation strategy to convert ordinary semantic representations
into abstract representations for classification.

Rank |  Team | Acc. | Acc. Cross
---|---|---|---
- |  GA Reader | 24.3 | -
1 |  PINGAN-Omini-Sinitic | 95.3 | 94.2 ($\downarrow$ 1.1)
2 |  SRC-B-roc | 94.9 | 93.9($\downarrow$ 1.0)
3 |  tt123 | 93.4 | 85.8($\downarrow$ 7.6)
4 |  ECNU-ICA-1 | 93.0 | 92.8($\downarrow$ 0.2)
5 |  cxn | 92.9 | -
6 |  ZJUKLAB | 92.8 | -
7 |  nxc | 92.7 | -
8 |  hzxx1997 | 90.2 | -
9 |  XRJL | 90.0 | 87.6($\downarrow$ 2.4)
10 |  IIE-NLP-Eyas | 89.6 | 84.1($\downarrow$ 5.5)
11 |  ReCAM@IITK | 87.6 | 85.2($\downarrow$ 2.4)
12 |  noobs | 87.1 | 82.4($\downarrow$ 4.7)
13 |  DeepBlueAI | 86.2 | 80.7($\downarrow$ 5.5)
14 |  xuliang | 81.0 | -
15 |  LRG | 77.8 | 65.6($\downarrow$ 12.2)
16 |  Yotta | 71.6 | -
17 |  sayazzad | 68.3 | -
18 |  itanhisada | 67.7 | -
19 |  NEUer | 66.9 | 45.0($\downarrow$ 21.9)
20 |  YaA@JUST | 66.1 | -
21 |  NLP-IIS@UT | 64.4 | -
22 |  CCLAB | 48.1 | 31.8($\downarrow$ 16.3)
23 |  K-FUT | 47.6 | -
24 |  owlmx | 44.8 | 31.0($\downarrow$13.8 )
25 |  UIT-ISE-NLP | 42.0 | 27.3($\downarrow$ 14.7)
26 |  UoR | 39.1 | 34.2($\downarrow$ 4.9)
27 |  Noor | 19.9 | -
28 |  BaoShanCollege | 17.6 | -
Table 7: Official results of Subtask 2 and Subtask 3. Acc is the accuracy (%)
of the models trained on the Subtask 2 training data and tested on the Subtask
2 testset. Acc. Cross is the accuracy(%) of models trained on the Subtask 1
training data and tested on the Subtask 2 testset.

Many teams used external knowledge resources to further improve model
performance. WordNet Fellbaum (1998) was widely used to provide candidate word
definitions. ECNU-ICA-1 Liu et al. (2021) also used ConceptNet5 Speer et al.
(2016) and Graph Neural Network in their systems. To alleviate the noise
induced by incorporating structured knowledge through unimportant edges, they
propose a noise reduction strategy. owlmx used the MRC Psycholinguistic
Database to obtain a measurement of imperceptibility abstractness.

Different pre-processing techniques were proposed in multiple systems. ZJUKLAB
Xie et al. (2021a) used a sliding window to limit input length in training.
PINGAN Omini-Sinitic Wang et al. (2021) used the cycle noisy label detection
algorithm to make models more robust.

Much interesting analysis regarding the failure cases and data distribution
was discussed in several system description papers. XRJL Jiang et al. (2021)
found that for a few questions, common sense knowledge was further needed to
help find the answer. They also pointed out that there were still a few
questions in which multiple candidate choices may serve as appropriate
answers.

###  4.2 Subtask 2: ReCAM-Nonspecificity

In Subtask 2, we received 28 submissions. Table 7 shows the official
leaderboard. The best result in Subtask 2 was achieved by team PINGAN omini-
Sinitic Wang et al. (2021) with an accuracy of 0.953, using a model similar to
the team’s model in Subtask 1. The second-placed team SRC-B-roc Zhang et al.
(2021) also adopted the same model it used in Subtask 1 with a data
augmentation method based on the hypernym hierarchy in WordNet.

In general, the participating teams in Subtask 2 used pre-trained language
models and neural networks similar to those they used in Subtask 1. The main
differences lie in how the participants performed data augmentation and
leveraged external knowledge. For example, in addition to SRC-B-roc Zhang et
al. (2021), the IRG team Sharma et al. (2021) also performed data augmentation
using hypernyms from WordNet.

###  4.3 Subtask 3: Cross-task Performance

In this section, we explore models’ performance across the two types of
definitions of abstractness. Specifically, in this subtask, participants train
their models on the training set of one subtask and test on the testset of the
other subtask. We received 29 submissions in total from the participants.

#### Cross-task performance: Subtask 2-to-1 testing.

We asked participants to test their models trained on the Subtask 2 training
data on the Subtask 1 test data. The results are shown in the last column of
Table 6.

The results we received show that the performance of all systems drops
substantially. For some systems ranking among top 10, the accuracy can
decrease by 5 points (IIE-NLP-Eyas Xie et al. (2021b) and XRJL Jiang et al.
(2021)), or even more (14 points for nxc). Some systems show good
generalization ability in this Subtask 2-to-1 scenario; the performance of
PINGAN-Omini-Sinitic Wang et al. (2021) is only 1.3 point less, which may be
due to the the data augmentation and task adaptive training used in the model.

#### Cross-task Performance: Subtask 1-to-2 Testing.

Participants are asked to test their Subtask 1 systems on the Subtask 2
testset. Details of the results can be seen in the last column of Table 7. All
systems’ performances drop. For example, among the top-10 systems, the
accuracy decreases by 5 points (IIE-NLP-Eyas Xie et al. (2021b)) or 7 points
(tt123).

However, ECNU-ICA-1 Liu et al. (2021) shows a very good generalization ability
in Subtask 1-to-2 testing. PINGAN-Omini-Sinitic Wang et al. (2021), SRC-B-roc
Zhang et al. (2021) and XRJL Jiang et al. (2021)’s systems are rather
consistent in this Subtask 1-to-2 cross testing. Some algorithms they used may
explain the models’ good generalization ability. ECNU-ICA-1’s algorithm of
using knowledge-enhanced Graph Attention Network can provide external
knowledge to the model. The Wrong Answer Ensemble algorithm Kim and Fung
(2020) used in PINGAN-Omini-Sinitic Wang et al. (2021) is a relatively simple
but an effective way of improving model performance and generalization
ability. Also, the Stochastic Weight Averaging algorithm across multiple
checkpoints is effective for better generalization. XRJL Jiang et al. (2021)
retrieves the definitions of candidate answers from WordNet and feeds them to
the model as extra inputs. We also think data augmentation methods contribute
to the generalization ability.

##  5 Related Work

There have been tasks being proposed to evaluate machines’ ability on reading
comprehension, which either require models to find an entity or text span from
the source document as the answer Hermann et al. (2015); Hill et al. (2016);
Onishi et al. (2016); Rajpurkar et al. (2016); Trischler et al. (2017), or
further generate an answer Nguyen et al. (2016); He et al. (2018); Kočiskỳ et
al. (2018). The cloze-style MRC tasks Hermann et al. (2015); Onishi et al.
(2016); Hill et al. (2016) are most similar to ours, in which the missing
words in the cloze questions are entities appearing in source documents.
Unlike previous work, ReCAM questions specifically focus on abstract words
unseen in the corresponding source documents.

In general, multi-choice questions have been widely used as a tool for
language examination to test both humans and machines. In this paper, we
follow the multiple-choice framework for our proposed ReCAM task to evaluate
computers’ ability in comprehending abstract concepts, in which computers are
asked to predict the missing abstract words in human-written summaries.

##  6 Summary

This shared task aims to study the ability of machines in representing and
understanding abstract concepts, based on two definitions of abstractness, the
imperceptibility and nonspecificity, in a specific machine reading
comprehension setup. We provide three subtasks to evaluate models’ ability in
comprehending the two types of abstract meaning as well as their
generalizability. In Subtask 1, the top system achieves an accuracy of 0.951,
and in Subtask 2, an accuracy of 0.953, suggesting the current systems perform
well in the specific setup of our share task. In Subtask 3, we found that in
general the models’ performances dropped in both Subtask 2-to-1 and Subtask
1-to-2 testing. However, some models generalize well, benefiting from
technologies such as data augmentation and task adaptive training. We hope the
shared task can help shed some light on modelling abstract concepts and help
design more challenging tasks in the future.

## References

  * Banerjee and Pedersen (2002) Satanjeev Banerjee and Ted Pedersen. 2002.  An adapted lesk algorithm for word sense disambiguation using wordnet.  In _Computational Linguistics and Intelligent Text Processing, Third International Conference, CICLing 2002, Mexico City, Mexico, February 17-23, 2002, Proceedings_ , volume 2276 of _Lecture Notes in Computer Science_ , pages 136–145. Springer.
  * Bird and Loper (2004) Steven Bird and Edward Loper. 2004.  NLTK: the natural language toolkit.  In _Proceedings of the ACL 2004 on Interactive poster and demonstration sessions_ , page 31. Association for Computational Linguistics.
  * Changizi (2008) Mark A. Changizi. 2008.  Economically organized hierarchies in wordnet and the oxford english dictionary.  _Cogn. Syst. Res._ , 9(3):214–228.
  * (4) Kyunghyun Cho, Bart van Merriënboer Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder–decoder for statistical machine translation.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: pre-training text encoders as discriminators rather than generators.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Coltheart (1981) Max Coltheart. 1981.  The MRC psycholinguistic database.  _The Quarterly Journal of Experimental Psychology_ , 33(4):497–505.
  * Craswell (2009) Nick Craswell. 2009.  Mean reciprocal rank.  In _Encyclopedia of Database Systems_ , pages 1703–1703. Springer.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Dhingra et al. (2017) Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. 2017.  Gated-attention readers for text comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , pages 1832–1846. Association for Computational Linguistics.
  * Fellbaum (1998) Christiane Fellbaum. 1998.  _WordNet: An Electronic Lexical Database_.  Bradford Books.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020.  DeBERTa: Decoding-enhanced BERT with disentangled attention.  _CoRR_ , abs/2006.03654.
  * He et al. (2018) Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018\.  Dureader: a chinese machine reading comprehension dataset from real-world applications.  In _Proceedings of the Workshop on Machine Reading for Question Answering@ACL 2018, Melbourne, Australia, July 19, 2018_ , pages 37–46. Association for Computational Linguistics.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Hill et al. (2016) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016.  The goldilocks principle: Reading children’s books with explicit memory representations.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2018.  Averaging weights leads to wider optima and better generalization.  In _Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018_ , pages 876–885. AUAI Press.
  * Jiang et al. (2021) Yuxin Jiang, Ziyi Shou, Qijun Wang, Hao Wu, and Fangzhen Lin. 2021.  XRJL-HKUST at SemEval-2021 Task 4: Wordnet-enhanced dual multi-head co-attention for reading comprehension of abstract meaning.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Kim and Fung (2020) Hyeondey Kim and Pascale Fung. 2020.  Learning to classify the wrong answers for multiple choice question answering (student abstract).  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 13843–13844. AAAI Press.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kočiskỳ et al. (2018) Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association of Computational Linguistics_ , 6:317–328.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lesk (1986) Michael Lesk. 1986.  Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone.  In _Proceedings of the 5th annual international conference on Systems documentation_ , pages 24–26. ACM.
  * Liu et al. (2021) Pingsheng Liu, Linlin Wang, Qian Zhao, Hao Chen, Yuxi Feng, Xin Lin, and Liang He. 2021.  ECNU_ICA_1 SemEval-2021 Task 4: Leveraging knowledge-enhanced graph attention networks for reading comprehension of abstract meaning.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Miller (1998) George Miller. 1998.  _WordNet: An electronic lexical database_.  MIT press.
  * Moro et al. (2014) Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014.  Entity linking meets word sense disambiguation: a unified approach.  _Transactions of the Association for Computational Linguistics_ , 2:231–244.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 1797–1807. Association for Computational Linguistics.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human generated machine reading comprehension dataset.  In _Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016_ , volume 1773 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Onishi et al. (2016) Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016\.  Who did what: A large-scale person-centered cloze dataset.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2230–2235.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020\.  Intermediate-task transfer learning with pretrained language models: When and why does it work? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 5231–5247. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A Python natural language processing toolkit for many human languages.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100, 000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2383–2392. The Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Sharma et al. (2021) Abheesht Sharma, Harshit Pandey, Gunjan Chhablani, Yash Bhartia, and Tirtharaj Dash. 2021.  LRG at SemEval-2021 Task 4: Improving reading comprehension with abstract words using augmentation, linguistic features and voting.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Speer et al. (2016) Robert Speer, Joshua Chin, and Catherine Havasi. 2016.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _AAAI Conference on Artificial Intelligence_.
  * Spreen and Schulz (1966) Otfried Spreen and Rudolph W. Schulz. 1966.  Parameters of abstraction, meaningfulness, and pronunciability for 329 nouns.  _Journal of Verbal Learning and Verbal Behavior_ , 5(5):459 – 468\.
  * Theijssen et al. (2011) DL Theijssen, H van Halteren, LWJ Boves, and NHJ Oostdijk. 2011.  On the difficulty of making concreteness concrete.
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  Newsqa: A machine comprehension dataset.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP, Rep4NLP@ACL 2017, Vancouver, Canada, August 3, 2017_ , pages 191–200. Association for Computational Linguistics.
  * Turney et al. (2011) Peter D Turney, Yair Neuman, Dan Assaf, and Yohai Cohen. 2011.  Literal and metaphorical sense identification through concrete and abstract context.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_ , pages 680–690. Association for Computational Linguistics.
  * Wang et al. (2021) Ye Wang, Yanmeng Wang, Haijun Zhu, Bo Zeng, Zhenghong Hao, Shaojun Wang, and Jing Xiao. 2021.  PINGAN Omini-Sinitic at SemEval-2021 Task 4: Reading comprehension of abstract meaning.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Xie et al. (2021a) Xin Xie, Xiangnan Chen, Xiang Chen, Yong Wang, Ningyu Zhang, Shumin Deng, and Huajun Chen. 2021a.  ZJUKLAB at SemEval-2021 Task 4: Negative augmentation with language model for reading comprehension of abstract meaning.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Xie et al. (2021b) Yuqiang Xie, Luxi Xing, Wei Peng, and Yue Hu. 2021b.  IIE-NLP-Eyas at SemEval-2021 Task 4: Enhancing plm for recam with special tokens, re-ranking, siamese encoders and back translation.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhang et al. (2021) Jing Zhang, Yimeng Zhuang, and Yinpei Su. 2021.  TA-MAMC at SemEval-2021 Task4: Task-adaptive pretraining and multi-head attention for abstract meaning reading comprehension.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_.

##  Appendix A Annotation Script

##  Appendix B Gated-Attention Reader

The Gated-Attention (GA) Reader Dhingra et al. (2017), the state-of-art model
on CNN/Daily Mail reading comprehension dataset Hermann et al. (2015), is
adapted here in our experiments. The GA Reader uses a multi-layer iterated
architecture with a gated attention mechanism, which is based on
multiplicative interactions between the query embedding and the intermediate
states of a recurrent neural network document reader, to derive better query-
aware passage representation. To apply GA Reader to our ARC task, we input the
news passage $p$ as the document and the processed summary $s$ as the query to
GA Reader.

Specifically, for an input passage $p=[p_{1},p_{2},...,p_{l_{p}}]$ with
$l_{p}$ words and its corresponding summary $s=[s_{1},s_{2},...,s_{l_{s}}]$
with $l_{s}$ words, we first derive their corresponding word embedding
sequence $\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},...,\mathbf{p}_{l_{p}}]$
and $\mathbf{S}=[\mathbf{s}_{1},\mathbf{s}_{2},...,\mathbf{s}_{l_{s}}]$
respectively. Then the GA Reader accepts the $\mathbf{P}$ and $\mathbf{S}$ as
inputs and return the hidden states
$\mathbf{H}_{p}=[\mathbf{h}_{1}^{p},\mathbf{h}_{2}^{p},...,\mathbf{h}_{l_{p}}^{p}]$
and
$\mathbf{H}_{s}=[\mathbf{h}_{1}^{s},\mathbf{h}_{2}^{s},...,\mathbf{h}_{l_{s}}^{s}]$
as the sequential representation for passage $p$ and summary $s$ respectively.
As for the final prediction process, we do not adopt the operations in Dhingra
et al. (2017) because in ARC the answer words are unseen in the corresponding
passage, however, GA Reader in Dhingra et al. (2017) tries to select a entity
word in the passage as the final prediction since their target answer word
appears in the passage. So we redesign the part of prediction.

First, the corresponding representation of “ _@placeholder_ ” in
$\mathbf{H}_{s}$, denoted as $\mathbf{h}_{q}^{s}$ ($q$ is the position index
of _@placeholder_ in summary $s$), is used as the final vector representation
for summary $s$. For the final vector representation $\mathbf{p}$ for passage
$p$, a bilinear attention between $\mathbf{h}_{q}^{s}$ and $\mathbf{H}_{p}$ is
used for its derivation:

$\displaystyle e_{i}$
$\displaystyle={\mathbf{h}_{q}^{s}}^{T}\mathbf{W}^{att}\mathbf{h}_{i}^{p},\forall
i\in[1,...,l_{p}]$ (1) $\displaystyle\mathbf{p}$
$\displaystyle=\sum_{i=1}^{l_{p}}\frac{\exp{e_{i}}}{\sum_{j=1}^{l_{p}}\exp{e_{j}}}\mathbf{h}_{i}^{p},$
(2)

We set a token embedding $\mathbf{a}_{t}^{e}$ for each candidate abstractive
word $a_{t}$ ($t\in[1,...,n_{c}]$, $n_{c}$ is the size of candidate set). We
first concatenate the $\mathbf{h}_{p}^{s}$ and $\mathbf{p}$, then use the
bilinear product and softmax to predict the probability distribution over all
$n_{c}$ candidate abstractive words.

$\displaystyle r_{t}$
$\displaystyle=[\mathbf{h}_{q}^{s};\mathbf{p}]^{T}\mathbf{W}_{p}\mathbf{a}_{t}^{e},\forall
t\in[1,...,n_{c}],$ (3) $\displaystyle{o}_{t}$
$\displaystyle={softmax}_{t}(r_{t}),\forall t\in[1,...,n_{c}]$ (4)

in which ${o}_{t}$ represents the probability of predicting the candidate
abstractive word $a_{t}$ as the final answer.

##  Appendix C Attentive Model

Figure 1: The model architecture of the attentive model with word gloss (AMWG)
implemented in this paper. $\bigoplus$ denotes the concatenation of input
vectors. All the encoders are 1-layer bi-directional GRU-RNNs, $\bigotimes$
denotes the weighted sum of vectors.

The word gloss, which defines a word sense meaning, has been mainly used in
word sense disambiguation (WSD) task and its variants Lesk (1986); Moro et al.
(2014). Since the goal of ARC is to predict a word that can summarize
corresponding information from the source passage, which is an abstracting
process, it may be helpful when the gloss, i.e., interpretation of candidate
abstractive words, are provided.

We design an attentive model with word gloss (AMWG) as Figure 1 shows.
Specifically, all the _encoders_ are 1-layer bi-directional recurrent neural
networks (RNNs) with Gated Recurrent Units (GRU) Cho et al. . For an input
news passage $p=[p_{1},p_{2},...,p_{l_{p}}]$ with $l_{p}$ words, we can derive
its hidden states
$\mathbf{H}_{p}=[\mathbf{h}_{1}^{p},\mathbf{h}_{2}^{p},...,\mathbf{h}_{l_{p}}^{p}]$
by sending its word embedding sequence
$\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},...,\mathbf{p}_{l_{p}}]$ to the
_Passage Encoder_. Similarly, we can derive hidden states
$\mathbf{H}_{s}=[\mathbf{h}_{1}^{s},\mathbf{h}_{2}^{s},...,\mathbf{h}_{l_{s}}^{s}]$
for summary $s$ by inputting its word embedding sequence
$\mathbf{S}=[\mathbf{s}_{1},\mathbf{s}_{2},...,\mathbf{s}_{l_{s}}]$ into the
_Summary Encoder_ and hidden states
$\mathbf{H}_{g_{t}}=[\mathbf{h}_{1}^{g_{t}},\mathbf{h}_{2}^{g_{t}},...,\mathbf{h}_{l_{g_{t}}}^{g_{t}}]$
for gloss $g_{t}$ of the candidate word $a_{t}$ by sending its word embedding
sequence
$\mathbf{G}_{t}=[\mathbf{g}_{1}^{t},\mathbf{g}_{2}^{t},...,\mathbf{g}_{l_{g_{t}}}^{t}]$
to the _WordGloss Encoder_.

Similar to Section B, the corresponding representation of “ _@placeholder_ ”,
i.e., $\mathbf{h}_{q}^{s}$, is used as the final vector representation for
summary $s$. And an bilinear attention $f_{att}^{p}(\bullet)$ is applied to
$\mathbf{h}_{q}^{s}$ and $\mathbf{H}_{p}$ as follows:

$\displaystyle e_{i}$
$\displaystyle={\mathbf{h}_{q}^{s}}^{T}\mathbf{W}_{att}^{p}\mathbf{h}_{i}^{p},\forall
i\in[1,...,l_{p}]$ (5) $\displaystyle\alpha_{i}$
$\displaystyle=\frac{\exp{e_{i}}}{\sum_{j=1}^{l_{p}}\exp{e_{j}}},\forall
i\in[1,...,l_{p}]$ (6)

Then $\mathbf{p}$ is derived as the vector representation for passage $p$ by
the weighed sum of $\mathbf{H}_{p}$, which is further concatenated with the
$\mathbf{h}_{q}^{s}$ to form the final summarization vector $\mathbf{v}$:

$\displaystyle\mathbf{p}$
$\displaystyle=\sum_{i=1}^{l_{p}}\alpha_{i}\mathbf{h}_{i}^{p},$ (7)
$\displaystyle\mathbf{v}$
$\displaystyle=concat(\mathbf{p},\mathbf{h}_{q}^{s}),$ (8)

Another attention $f_{att}^{g}(\bullet)$ is applied to $\mathbf{v}$ and
$\mathbf{H}_{g_{t}}$,

$\displaystyle e_{j}$
$\displaystyle={\tanh(\mathbf{W}_{att}^{g}\mathbf{v}+\mathbf{b})}^{T}\mathbf{h}_{j}^{g_{t}},\forall
j\in[1,...,l_{g_{t}}]$ (9) $\displaystyle\beta_{j}$
$\displaystyle=\frac{\exp{e_{j}}}{\sum_{i=1}^{l_{g_{t}}}\exp{e_{i}}},\forall
j\in[1,...,l_{g_{t}}],$ (10)

The following weighted sum of $\mathbf{H}_{g_{t}}$, i.e, $\mathbf{a}_{t}^{g}$,
is derive as the final vector representation for the gloss of candidate word
$a_{t}$:

$\displaystyle\mathbf{a}_{t}^{g}$
$\displaystyle=\sum_{j=1}^{l_{g_{t}}}\beta_{j}\mathbf{h}_{j}^{g_{t}}$ (11)

We also set a token embedding $\mathbf{a}_{t}^{e}$ for each candidate word
$a_{t}$ ($t\in[1,...,n_{c}]$, $n_{c}$ is the size of candidate set), which is
further concatenated with $\mathbf{a}_{t}^{g}$ to build the final
representation $\mathbf{a}_{t}$ for candidate word $a_{t}$. For the final
prediction, we input the summarization vector $\mathbf{v}$ and candidate
representation vector $\mathbf{a}_{t}$ to $f_{pred}(\bullet)$ and apply the
softmax to derive the probability distribution over all $n_{c}$ candidate
abstractive words,

$\displaystyle\mathbf{a}_{t}$
$\displaystyle=concat(\mathbf{a}_{t}^{g},\mathbf{a}_{t}^{e}),$ (12)
$\displaystyle r_{t}$
$\displaystyle=\mathbf{v}^{T}\mathbf{W}_{pred}\mathbf{a}_{t},\forall
t\in[1,...,n_{c}],$ (13) $\displaystyle{o}_{t}$
$\displaystyle={softmax}_{t}(r_{t}),\forall t\in[1,...,n_{c}]$ (14)

in which ${o}_{t}$ gives the probability of predicting the candidate word
$a_{t}$ as the final answer. The word gloss, which defines a word sense
meaning, has been mainly used in word sense disambiguation (WSD) task and its
variants Lesk (1986); Moro et al. (2014). Since the goal of ARC is to predict
a word that can summarize corresponding information from the source passage,
which is an abstracting process, it may be helpful when the gloss, i.e.,
interpretation of candidate abstractive words, are provided.

We design an attentive model with word gloss (AMWG) as Figure 1 shows.
Specifically, all the _encoders_ are 1-layer bi-directional recurrent neural
networks (RNNs) with Gated Recurrent Units (GRU) Cho et al. . For an input
news passage $p=[p_{1},p_{2},...,p_{l_{p}}]$ with $l_{p}$ words, we can derive
its hidden states
$\mathbf{H}_{p}=[\mathbf{h}_{1}^{p},\mathbf{h}_{2}^{p},...,\mathbf{h}_{l_{p}}^{p}]$
by sending its word embedding sequence
$\mathbf{P}=[\mathbf{p}_{1},\mathbf{p}_{2},...,\mathbf{p}_{l_{p}}]$ to the
_Passage Encoder_. Similarly, we can derive hidden states
$\mathbf{H}_{s}=[\mathbf{h}_{1}^{s},\mathbf{h}_{2}^{s},...,\mathbf{h}_{l_{s}}^{s}]$
for summary $s$ by inputting its word embedding sequence
$\mathbf{S}=[\mathbf{s}_{1},\mathbf{s}_{2},...,\mathbf{s}_{l_{s}}]$ into the
_Summary Encoder_ and hidden states
$\mathbf{H}_{g_{t}}=[\mathbf{h}_{1}^{g_{t}},\mathbf{h}_{2}^{g_{t}},...,\mathbf{h}_{l_{g_{t}}}^{g_{t}}]$
for gloss $g_{t}$ of the candidate word $a_{t}$ by sending its word embedding
sequence
$\mathbf{G}_{t}=[\mathbf{g}_{1}^{t},\mathbf{g}_{2}^{t},...,\mathbf{g}_{l_{g_{t}}}^{t}]$
to the _WordGloss Encoder_.

Similar to Section B, the corresponding representation of “ _@placeholder_ ”,
i.e., $\mathbf{h}_{q}^{s}$, is used as the final vector representation for
summary $s$. And an bilinear attention $f_{att}^{p}(\bullet)$ is applied to
$\mathbf{h}_{q}^{s}$ and $\mathbf{H}_{p}$ as follows:

$\displaystyle e_{i}$
$\displaystyle={\mathbf{h}_{q}^{s}}^{T}\mathbf{W}_{att}^{p}\mathbf{h}_{i}^{p},\forall
i\in[1,...,l_{p}]$ (15) $\displaystyle\alpha_{i}$
$\displaystyle=\frac{\exp{e_{i}}}{\sum_{j=1}^{l_{p}}\exp{e_{j}}},\forall
i\in[1,...,l_{p}]$ (16)

Then $\mathbf{p}$ is derived as the vector representation for passage $p$ by
the weighed sum of $\mathbf{H}_{p}$, which is further concatenated with the
$\mathbf{h}_{q}^{s}$ to form the final summarization vector $\mathbf{v}$:

$\displaystyle\mathbf{p}$
$\displaystyle=\sum_{i=1}^{l_{p}}\alpha_{i}\mathbf{h}_{i}^{p},$ (17)
$\displaystyle\mathbf{v}$
$\displaystyle=concat(\mathbf{p},\mathbf{h}_{q}^{s}),$ (18)

Another attention $f_{att}^{g}(\bullet)$ is applied to $\mathbf{v}$ and
$\mathbf{H}_{g_{t}}$,

$\displaystyle e_{j}$
$\displaystyle={\tanh(\mathbf{W}_{att}^{g}\mathbf{v}+\mathbf{b})}^{T}\mathbf{h}_{j}^{g_{t}},\forall
j\in[1,...,l_{g_{t}}]$ (19) $\displaystyle\beta_{j}$
$\displaystyle=\frac{\exp{e_{j}}}{\sum_{i=1}^{l_{g_{t}}}\exp{e_{i}}},\forall
j\in[1,...,l_{g_{t}}],$ (20)

The following weighted sum of $\mathbf{H}_{g_{t}}$, i.e, $\mathbf{a}_{t}^{g}$,
is derive as the final vector representation for the gloss of candidate word
$a_{t}$:

$\displaystyle\mathbf{a}_{t}^{g}$
$\displaystyle=\sum_{j=1}^{l_{g_{t}}}\beta_{j}\mathbf{h}_{j}^{g_{t}}$ (21)

We also set a token embedding $\mathbf{a}_{t}^{e}$ for each candidate word
$a_{t}$ ($t\in[1,...,n_{c}]$, $n_{c}$ is the size of candidate set), which is
further concatenated with $\mathbf{a}_{t}^{g}$ to build the final
representation $\mathbf{a}_{t}$ for candidate word $a_{t}$. For the final
prediction, we input the summarization vector $\mathbf{v}$ and candidate
representation vector $\mathbf{a}_{t}$ to $f_{pred}(\bullet)$ and apply the
softmax to derive the probability distribution over all $n_{c}$ candidate
abstractive words,

$\displaystyle\mathbf{a}_{t}$
$\displaystyle=concat(\mathbf{a}_{t}^{g},\mathbf{a}_{t}^{e}),$ (22)
$\displaystyle r_{t}$
$\displaystyle=\mathbf{v}^{T}\mathbf{W}_{pred}\mathbf{a}_{t},\forall
t\in[1,...,n_{c}],$ (23) $\displaystyle{o}_{t}$
$\displaystyle={softmax}_{t}(r_{t}),\forall t\in[1,...,n_{c}]$ (24)

in which ${o}_{t}$ gives the probability of predicting the candidate word
$a_{t}$ as the final answer.

##  Appendix D Training Details

We train all models using the non-negative log-likelihood as the objective
function. The gloss of candidate words are derived from _WordNet_ using the
NLTK tools Bird and Loper (2004). Specifically, we first lemmatize the
candidate word and use the lemmatized word as the query word for the searching
in _WordNet_. To cope with the semantic ambiguity of words, we just
concatenate the gloss of the first sense in each retrieved POS for the query
word with corresponding POS tag as the deliminator.

Models in our experiments are trained with the following hyperparameter
settings: All word embeddings and token embeddings $\mathbf{a}_{t}^{e}$ have
$300$ dimensions and are initialized with Glove Pennington et al. (2014). The
passage $p$ and summary $s$ share one set of word embeddings, which are fixed
during training. The glosses $\\{g_{t}\\}$ for candidate words $\\{a_{t}\\}$
keep its own word embeddings. The hidden state vectors of all bi-directional
GRU-RNNs in all models have $150$ dimensions. The number of attention hops in
GA Reader is set to $3$. The batch size is set to $32$. The method of Adam
Kingma and Ba (2015) is adopted for optimization with initial learning rate
$1e-03$. A dropout with rate $0.3$ is applied to the input layers for all GRU-
RNN encoders and the final summarization vector $\mathbf{v}$.

##  Appendix E Annotation Selection

To ensure most of our annotation is valid, we select annotations satisfying
the following criteria: a) the average accuracy is higher than 40%; b) both
text spans should not be empty; c) if the difficulty level is rated as easy,
then this data sample should be answered correctly.


../Figures/

# The Door and Drawer Reset Mechanisms: Automated Mechanisms for Testing and
Data Collection

Kyle DuFrene1, Luke Strohbehn1, Keegan Nave1, Ravi Balasubramanian1, and Cindy
Grimm1 1Collaborative Robotics and Intelligent Systems (CoRIS) Institute,
Oregon State University, Corvallis, OR 97331
Thank you to our partners at the UMass Lowell Nerve Center, Brian Flynn, Adam
Norton, and Holly Yanco.
This work was supported in part by the NSF under Grants CCRI 1925715 and RI
1911050.

###### Abstract

Robotic manipulation in human environments is a challenging problem for
researchers and industry alike. In particular, opening doors/drawers can be
challenging for robots, as the size, shape, actuation and required force is
variable. Because of this, it can be difficult to collect large real-world
datasets and to benchmark different control algorithms on the same hardware.
In this paper we present two automated testbeds, the Door Reset Mechanism
(DORM) and Drawer Reset Mechanism (DWRM), for the purpose of real world
testing and data collection. These devices are low-cost, are sensorized,
operate with customized variable resistance, and come with open source
software. Additionally, we provide a dataset of over 600 grasps using the DORM
and DWRM. We use this dataset to highlight how much variability can exist even
with the “same” trial on the “same” hardware. This data can also serve as a
source for real-world noise in simulation environments.

##  I Introduction

Incorporating robots in human environments is a challenging problem that
requires the fusion of many different disciplines. Robots for tasks such as
mopping and vacuuming have successfully been integrated into many homes, but
tasks that require general manipulation have yet to see the same level of
success. Actions that require opening doors and drawers can be particularly
challenging for robots due to door/drawer variation and the required
manipulation fidelity.

In the past few decades, significant progress has been made in control models,
such as impedance control, for opening doors and drawers [1]. However, this
approach requires accurate force-torque sensors located in the manipulator or
end effector. Recently, learning methods are becoming more commonplace in
grasping and manipulation, and have been applied to door opening [2, 3]. These
machine learning methods require large amounts of data to train on. Given the
varying geometry and force required to open different doors and drawers,
training must also include a diverse set of mechanisms [4]. Because of this,
many machine learning approaches use simulated data to train. This,
unfortunately, leads to a loss of real world noise and fidelity in their
datasets [5, 4].

Figure 1: The Drawer Reset Mechanism with a Kinova JACO 7-dof arm. As
pictured, the instrumented knob is attached to the Drawer Reset Mechanism.

To improve robotic manipulation of doors and drawers, there is a need for
repeatable testing mechanisms for benchmarking control algorithms, and
generating datasets. To the best of the authors’ knowledge, there are no
current open-source door and drawer testbeds and accompanying software
packages for this end. In this paper, we propose two testbeds for the
collection of data and benchmarking of door and drawer manipulation tasks, as
well as the accompanying software and datasets.

The Door Reset Mechanism (DORM) is an artificial door which automatically
closes, interfaces with a variety of pull attachments (handles/knobs), and
provides variable resistive forces against opening. Additionally, the DORM has
a variety of sensors to provide force feedback and door angle. The Drawer
Reset Mechanism (DWRM) provides all the functionality of the DORM, except in
the form of a drawer. Each testbed is low cost (under $400), highly
customizable, and sensorized. The built in sensors, combined with external
cameras and data from the manipulator, allow us to quickly create robust
datasets or compare proposed control methods. Each testbed can record, reset,
and be monitored remotely, making large dataset collection feasible and
repeatable.

Paired with the testbeds, our modular software packages perform automatic data
collection and processing, and are compatible with most robotic manipulators
and standard visualization tools, such as RViz [6].

Figure 2: Pictured on the left is the Door Reset Mechanism, on the right is
the Drawer Reset Mechanism. Important parts are labeled, including
electronics, sensors, motors, and pull attachments.

This work presents the design of the testbeds, associated software, and a
dataset to demonstrate the capabilities of the testbeds. With the DORM and
DWRM, we conducted a total of 660 trials with a Kinova Gen 3 robot arm and a
Kinova JACO robot arm. Trials were conducted with an instrumented handle and
knob, a variety of resistance forces (applied by the DORM and DWRM), and
multiple grasp types. The dataset includes video, manipulator joint states,
instrumented pull attachment forces, and trial result for each test.

In this work our contributions include:

  * •

Instrumented hardware platforms for automated trials of opening doors and
drawers, the Door Reset Mechanism (DORM) and Drawer Reset Mechanism (DWRM).

  * •

Software packages for conducing grasp trials and visualization.

  * •

A dataset of over 600 grasps.

Table I: Design considerations for the Door and Drawer Reset Mechanisms, and their corresponding solutions. Design Requirements |  Design Solutions
---|---
Cost |  The DWRM frame is built from 2x4s and plywood, and the DORM from an Ikea cabinet. Custom parts are either 3D printed (PLA), or available at common online retailers. Each can be build for under $400.
Replicability |  All components are readily available at retailers or 3D printed. 3D CAD, wiring diagrams, and software are all provided on the website.
Compatibility with a wide range of manipulators |  Manipulators are prompted by a ROS action client. Any manipulator that be controlled within a custom ROS action server is compatible.
Robust and accurate resetting (closing) |  Simple pulley systems with high-tensile strength string. Error checking with position/orientation sensors.
Variable resistance to opening |  A disk brake on the DWRM provides variable resistance between $0$ and $25$ Newtons of resistance. Electromagenets in the DORM provide between $0$ and $10$ Newtons of resistance.
Position/orientation feedback |  A time-of-flight sensor inside the DWRM measures drawer position and a magnetic rotatory position sensor above the DORM measures orientation of the door.
Compatibility with various pull attachments |  Any pull attachment that can be 3D printed, or attached to a 3D printed base is compatible. The base is attached to the DORM/DWRM with two thumb screws.
Force feedback on pull attachments |  The handle has 12 evenly distributed force-sensitive resistors, the knob has 5. While not precise, they provide accurate models of contact forces on the pull attachments.

All aspects of the testbeds’ designs and dataset discussed in this paper are
open source and are available at https://osurobotics.github.io/Physical-
Robotic-Manipulation-Test-Facility/.

##  II Related Work

Due to the challenges of real world data collection and testing, many existing
benchmarks and testing environments are provided through simulation such as
[7, 3, 8]. Work from Urakami, et al. demonstrates an open source randomized
environment for door opening tasks as well as a baseline agent for
benchmarking purposes [3]. However when transferred to the real world, the
authors’ note performance degradation. Similarly, in [9] tactile sensing is
used during training of their model, and although their sim-to-real
performance is improved, there still exists a gap in performance.

When transferring from simulation to the real world, door and drawer opening
tasks require some form of mechanism for benchmarking. Karayiannidis et al.
and Mochurad et al. used random real doors to benchmark their control
approaches [1, 2]. In both papers, a manual measurement of door angle was
required to determine if a trial was successful or unsuccessful. In other work
such as [10] small mock doors with built-in IMU sensors were used to test and
train their policy in real world trials. However due to constraints on their
real world testbed they were unable to complete all of the tasks done in
simulation. Across all work, there is no definitive benchmarking tool, and the
definition of success can vary from paper to paper.

General grasping and manipulation datasets such as [11] and [12] are useful
for training models, however to the best of the authors’ knowledge, there are
no existing open source, real world grasp datasets for door and drawer
grasping/manipulation.

##  III The Mechanisms

The Door and Drawer Rest Mechanisms are designed to automate the resetting
(closing) of a door and a drawer for grasp and pull testing. As part of the
resetting process, the testbeds return to their closed state and resistance
forces are set. To support many manipulators, we use a state-machine based
interface that triggers resetting, manipulator control, and data-collection
actions. The testbeds also have sensing built-in to collect information on how
fast/far they were opened and where on the handle the grasp applied force.
Design requirements for these features and corresponding solutions are
presented in Table I.

The entire system is broken into into three components: the mechanical design
of the two testbeds, electrical design, and software.

###  III-A Mechanical Design

The Door and Drawer Reset Mechanisms provide the same functionality in
slightly different ways. Both require a certain amount of force to be opened,
and both return to the original state after the grasp and open operations have
completed. Additionally, each testbed interfaces with a modular pull
attachment such as a handle or knob.

####  III-A1 Door

The Door Reset Mechanism is built from a modified EKET cabinet from IKEA. As
the door may experience more extreme forces that it was designed for, 3D
printed braces placed in the corners maintain structural integrity. The door
opens and closes as usual about the hinge location. A high tensile strength
braided Kevlar string is attached to the opposite side of the door and
connected to a 12 volt DC gearmotor attached to the frame. The string is wound
around a spool, which is used to retract the string and close the door. Before
a trial, the sting is spooled out a few inches by the motor with a small
fishing weight maintaining minimal tension.

Three electromagnets mounted to the frame provide variable resistance to the
door opening. Combined, they can provide a resistance of up to 10 Newtons.

The top hinge replaced by a 3D printed hinge with a small magnet mounted in
the top. Above the magnet, an AS5047P magnetic rotatory position sensor is
mounted to the frame. The sensor detects the rotation angle of the magnet,
providing precise orientation data of the door.

A cut-out in the door allows for mounting of various pull attachments, as
discussed later. Additionally, a custom PCB is mounted behind the DORM, and
cut-outs in the rear allow wires to pass inside.

To protect the DORM in event of extreme manipulator movements, the base of the
DORM mounts to the table with magnetic feet. The feet resist larger amounts of
forces in the forward and sideways directions, but forces backwards (into the
door) and excessive forces will dislodge the feet from the table. This
prevents the DORM from tipping over or otherwise being damaged during extreme
movements.

Figure 2 presents a labeled diagram of the DORM.

####  III-A2 Drawer

The Drawer Reset Mechanism’s frame is custom assembly of wood 2x4s and
plywood. Movement of the drawer occurs along drawer slides mounted to the
bottom edge of the box. An L bracket is fixed to the frame of the reset
structure to ensure a full stop of the drawer in case of mechanical failure.
An eye ring screw is fixed to the back of the drawer to allow the reset
mechanism to pull the drawer back to the closed state via a high tensile-
strength braided Kevlar string.

A 1/2” aluminum axle is located behind the drawer, and serves as the main
component of the reset mechanism. The axle is attached to bearings on both
sides, which are then physically pressed into custom 3D printed mounts bolted
to the frame. A 3D printed double spool is fixed to this axle, where one side
of the spool winds the string attached to the drawer. This string remains taut
at all times to prevent tangles and knots. The other side of the spool winds
an identical string in the opposite direction, which then winds around a spool
attached to a NEMA 17 stepper motor. Similar to the door, this string is
unwound before trials to allow the drawer to open, and minimal tension is
maintained with a small fishing weight.

Next to the double spool and also fixed to the aluminum axle is a disc brake,
fixed with custom 3D printed mounts. The brake is screwed to one of the custom
3D printed bearing mounts. This brake allows for variable resistance to be
applied during a drawer pull operation, and is subsequently released when the
reset operation occurs.

A VL53L0X time-of-flight (ToF) sensor is fixed to the back of the drawer
structure and points directly at the back of the drawer for high-accuracy
measurements of drawer location.

The braking force is applied to the disc by pulling the braking cable with a
100:1 geared NEMA 17 stepper motor. A 3D printed brake cable housing is
mounted to the stepper to secure the cable.

During a reset operation, the reset string is wound in towards the stepper and
away from the axle spool. In a similar but opposite manner, the spool
connected to the drawer simultaneously winds in, pulling the drawer back to
its closed state. The reset stepper continues to wind in until the ToF sensor
returns a previously-determined minimum distance value of the drawer. Once
stopped, the reset stepper then reverses direction and unwinds the string,
with a fishing line weight providing minimal tension to prevent tangles.
Introducing slack into the reset string allows subsequent drawer pulls by the
robotic arms so that they only feel the resistance of the applied brake and
the weight of the drawer itself, without the interference of the reset stepper
motor.

Figure 2 presents a labeled diagram of the DWRM.

####  III-A3 Pull Attachments

Figure 3: The knob (left) and handle (right) used as the pull attachments for
the dataset in this paper. Each have a silicone wrapper to protect and
distribute grasping forces to the force sensitive resistors (FSRs). The
silicone on the handle is rolled back to show the FSRs.

Both the door and drawer accept the same pull attachments. For this paper we
focus on a handle and a knob (shown in Figure 3), however the attachment may
be customized to be any shape and vary in size. Two thumbscrews mount the pull
attachment to the testbed, allowing for easy installation and removal.

The handle mimics a handle that may be found on an entry door to a commercial
building. The handle is 3D printed, and has a layer of twelve separate 0.5”
force sensitive resistors (FSR) evenly distributed across the surface. A layer
of silicone is wrapped around the handle to protect the FSRs and distribute
forces.

The knob is similar in size and geometry to a doorknob in a standard
residential home. Similar to the handle, it is 3D printed and has a layer of
five FSRs placed around the knob. A layer of silicone protects the FSRs and
distributes forces.

###  III-B Electrical Design

The Door and Drawer Reset Mechanisms feature custom-designed printable circuit
boards (PCBs) for an organized layout and removable electrical connections.
Power is provided to the PCBs via 5 and 12 volt power supplies. A 3.3 volt
buck converter steps down the 12 volt input power to use with the pull
attachment sensors. All connections to and from the PCBs are either JST type
plugs or screw-terminals for quick and reliable connections.

Both PCBs use the same setup for pull attachment FSRs. Two analog to digital
converters (ADCs) are located on the PCB and connected to the Raspberry Pi
with serial communication. The ADCs analog pins are connected to a 12 pin JST
plug, which ultimately connects to the FSRs through pull-down resistors. A 14
pin cable runs from the 12 pin JST plug and two screw terminals (for power and
ground) to a small breakout PCB located on the pull attachment itself. There,
10 kiloOhm pull-down resistors pull the data pins to ground. A data pin and
voltage is connected to each FSR.

Specific to the DORM, the rotatory magnetic position sensor is connected to
the Raspberry Pi via serial peripheral interface (SPI). Additionally, an
H-bridge motor driver and transistor drive the DC motor and electromagnets,
respectively.

For the DWRM, the ToF communicates with the Raspberry Pi via I2C. Both stepper
motors are driven by microstepper drivers. The microstepper drivers are
connected to general purpose input output (GPIO) pins on the Raspberry Pi.

###  III-C Software

Figure 4: The FlexBE state machine showing trial progression for standard
grasping trials with the DORM and DWRM.

The software used for both Door and Drawer Reset Mechanisms builds off of
previous work in [13], and uses the Robot Operating system (ROS) for
communication and data collection. Each testbed has an onboard Raspberry Pi
which communicates with a central control computer using ROS nodes. Video and
sensor data is automatically recorded and labelled with trial metadata after
each run. To trigger data collection, arm code, and closing operations, the
state machine FlexBE [14] is used. This software framework allows us to easily
switch between arms, grippers, and testbeds.

In addition to our core control software, we also include a visualization
tool. This allows users to watch sensor data live, or play back recorded
rosbag data from any trial. Visualization includes: arm path reconstruction
using an embedded RViz window, sensor plots, and 3D models for activated FSR’s
on the pull attachment. The visualization tool is modular — it can be expanded
to support addition testbeds, manipulators, and sensors. A view of the
interface with an trial from the dataset is shown in Figure 5.

All software and related documentation is available at our website provided in
Section I, and an in-depth look at the software design can be found in our
related work [13].

Figure 5: Our visualisation interface showing a trial. Buttons, drop-downs,
and text boxes allow user inputs to enable or disable certain features. Four
visualization windows are displayed. Top left: a view of the Kinova JACO 2
states, Top right: a 3D model of the handle, Bottom left: a graph of the
drawer distance recorded by the ToF, Bottom right: a plot of FSR data.

##  IV Dataset

In addition to the physical testbeds, we present a dataset with multiple
manipulators, pull attachments, grasps, and resistances.

###  IV-A Dataset Methods

Our goal was to create a dataset of grasps and pulls under diverse conditions
to represent real-world variation and noise. This dataset demonstrates the
capabilities of the testbeds, and can be used for benchmarking or learning
purposes.

The dataset consists of multiple grasps, with both pull attachments, and a
variety of resistances on both the DORM and DWRM. For the DORM, all trials
were conducted with a Kinova Gen3 paired with a RobotIQ 2F-85 gripper. For the
DWRM, all trials were conducted with a Kinova JACO arm with its standard
3-finger gripper. A high-level view of the combinations is presented in Table
II.

Table II: All combinations of grasps presented in the dataset. Mechanism |  Manipulator |  Handle Grasps |  Knob Grasps |  Resistances |  # Trials
---|---|---|---|---|---
Door |  Kinova Gen3 + RobotIQ 2F-85 |  5 |  5 |  0, 5, 10 N |  10
Drawer |  Kinova JACO 2 |  4 |  2 |  0, 7, 10, 15, 20, 25 N |  10
Figure 6: A plot showing the force recorded by FSR 9 on the handle for
multiple 0 and 25 Newton trials of the Drawer Reset Mechanism over the
duration of the pull operation. Of the 10 repeating trials, trials with
significantly longer pull times or outlier force values were filtered out. The
rest of the trials were normalized over time to the start of a sharp rise
(grasp) and a sharp fall (grasp release). FSR 9 is directly under the top
right finger of the Kinova JACO 2 gripper. A picture of one of the trials in
progress is shown to the right.

We selected grasps that represent various common ways a door/drawer may be
opened. For example, for the DORM with the knob, we preformed the following
five grasps:

  * •

Palm horizontal grasp (straight on)

  * •

Fingertip horizontal grasp (straight on)

  * •

Top down horizontal grasp

  * •

Fingertip angled grasp

  * •

Fingertip vertical grasp

Due to grasping force exerted by the RobotIQ gripper for the DORM trials, we
only test successful grasps. For the DWRM, the Kinova JACO fingers are
underactuated and have significantly less grasping power. We focused on
sampling successful grasps, however there were some failures.

###  IV-B Dataset Results

A total of 660 trials were conducted, 300 with the DORM and 360 with the DWRM.
All trials on the DORM were successful, but a small number of fingertip grasps
combined with higher resistance forces applied by the DWRM were failures.

As we collected the dataset in the real world, our dataset includes real world
noise and inconsistencies that are difficult to represent in simulation. For
example, each trial is individually planned with MoveIt, and varies based on
starting position due to tolerances in the joints. As a result, each
manipulator path may have slight variation, and the actual manipulator
(especially the Kinova JACO 2) follows the path with some deviation.
Additionally, the force applied by the disk brake in the DWRM varies slightly
over the course of a trial (similar to how a real world drawer is not
perfectly smooth). The grasp, especially at higher resistance forces and with
the underactuated Kinova JACO 2 manipulator, may shift slightly over the trial
duration.

An example of the FSR data collected as described in Section IV is visualized
in Figure 6. This figure explores the difference in grasp quality of the
handle on the drawer between two different friction resistances using an
identical grasping pose. Notably, most of the force on FSRs is provided simply
by grip strength. Higher resistance forces (the brown lines on the plot)
actually show lower FSR readings over the course of the pull movement. This is
due to the underactuated Kinova JACO 2 gripper slipping on the handle’s
silicone layer, and the redistribution of forces across the handle due to the
silicone.

The dataset includes all valuable data for each trial. This includes the
following information:

  * •

Grasp pose

  * •

Testbed name and pull attachment

  * •

Testbed sensor data (position/orientation of door/drawer)

  * •

Pull attachment force readings

  * •

Manipulator states (position and velocity)

  * •

Rear left and right RGB camera feed

  * •

Wrist mounted RGBD camera feed (Kinova Gen3 only)

##  V Discussion and Conclusions

In this paper we propose two low-cost, open source testbeds for automated
testing and data collection for door and drawer tasks — the Door Reset
Mechanism and Drawer Reset Mechanism. With our real world automated testing
mechanisms, we can emulate a variety of door/drawer resistance forces and
handle morphologies. The testbeds also capture valuable data, including
distance/orientation and forces on the pull attachment. The testbeds are easy
to replicate, and cost less than $400 each. Because of this, the system is a
good benchmarking tool for both hardware and control algorithms. In future
work we look to create standardized benchmarking tests that can be used
alongside our hardware.

Recording data from the testbeds (in addition to manipulator data, video, and
more) is possible with our modular software packages and state machine. We
also include a visualization interface for real time or recorded visual
information about trials. The testbeds and software enable robust, large scale
data collection for training learning models or testing algorithms.

We additionally present a dataset of 660 grasps to demonstrate our testbeds
and provide a resource for training and validation to the robotics community.
The dataset demonstrates the importance of real-world trials — friction,
uneven hardware movements, grasping contacts, and other real world variations
all are difficult to model in their entirety.

All CAD, schematics, and software for the DORM and DWRM, as well as the
accompanying dataset can be found on our website (linked in Section I).

## References

  * [1] Y. Karayiannidis, C. Smith, F. E. V. Barrientos, P. Ögren, and D. Kragic, “An adaptive control approach for opening doors and drawers under uncertainties,” _IEEE Transactions on Robotics_ , vol. 32, no. 1, pp. 161–175, 2016.
  * [2] L. Mochurad, Y. Hladun, Y. Zasoba, and M. Gregus, “An Approach for Opening Doors with a Mobile Robot Using Machine Learning Methods,” _Big Data and Cognitive Computing_ , vol. 7, no. 2, p. 69, Apr. 2023. [Online]. Available: https://www.mdpi.com/2504-2289/7/2/69
  * [3] Y. Urakami, A. Hodgkinson, C. Carlin, R. Leu, L. Rigazio, and P. Abbeel, “DoorGym: A Scalable Door Opening Environment And Baseline Agent,” May 2022, arXiv:1908.01887 [cs]. [Online]. Available: http://arxiv.org/abs/1908.01887
  * [4] O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” _J. Mach. Learn. Res._ , vol. 22, no. 1, jan 2021.
  * [5] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-real transfer in deep reinforcement learning for robotics: a survey,” in _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_ , 2020, pp. 737–744.
  * [6] H. R. Kam, S.-h. Lee, T. Park, and C.-h. Kim, “Rviz: a toolkit for real domain data visualization,” _Telecommunication Systems_ , vol. 60, no. 2, pp. 337–345, 10 2015.
  * [7] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3019–3026, 2020.
  * [8] Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, and Y. Zhu, “robosuite: A Modular Simulation Framework and Benchmark for Robot Learning,” Nov. 2022, arXiv:2009.12293 [cs]. [Online]. Available: http://arxiv.org/abs/2009.12293
  * [9] Z. Ding, Y.-Y. Tsai, W. W. Lee, and B. Huang, “Sim-to-Real Transfer for Robotic Manipulation with Tactile Sensory,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Sep. 2021, pp. 6778–6785, iSSN: 2153-0866.
  * [10] S. Gu, E. Holly, T. Lillicrap, and S. Levine, “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. Singapore, Singapore: IEEE, May 2017, pp. 3389–3396. [Online]. Available: http://ieeexplore.ieee.org/document/7989385/
  * [11] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen, “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection,” _CoRR_ , vol. abs/1603.02199, 2016.
  * [12] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016, pp. 3406–3413.
  * [13] K. DuFrene, K. Nave, J. Campbell, R. Balasubramanian, and C. Grimm, “The grasp reset mechanism: An automated apparatus for conducting grasping trials,” 2023\. [Online]. Available: https://osurobotics.github.io/Physical-Robotic-Manipulation-Test-Facility/docs/Papers/grasp-reset-mechanism.html
  * [14] P. Schillinger, “Flexbe behavior engine.” [Online]. Available: http://philserver.bplaced.net/fbe/