# Identifying Intensity of the Structure and Content in Tweets and the
Discriminative Power of Attributes in Context with Referential Translation
Machines

Ergun Biçici
AI Enablement
Huawei Türkiye R&D Center
Istanbul, Turkey
orcid.org/0000-0002-2293-2031
<EMAIL_ADDRESS>
bicici.github.io

###### Abstract

We use referential translation machines (RTMs) to identify the similarity
between an attribute and two words in English by casting the task as machine
translation performance prediction (MTPP) between the words and the attribute
word and the distance between their similarities for Task 10 with stacked RTM
models. RTMs are also used to predict the intensity of the structure and
content in tweets in English, Arabic, and Spanish in Task 1 where MTPP is
between the tweets and the set of words for the emotion selected from WordNet
affect emotion lists. Stacked RTM models obtain encouraging results in both.

##  1 Introduction

SemEval-2018 Apidianaki et al. (2018) contained two prediction tasks that
referential translation machine (RTM) models Biçici (2017a, b); Biçici and Way
(2015) can be applied to enable new modeling capabilities and provide new
results for comparison. The trace of the shared information with tweets or
social media to the audiences’ world can be approached by the intensity of the
content and structure used in text with Task 1 Mohammad et al. (2018). The
effectiveness of attributes to semantically separate two words from each other
is the goal in Task 10 Paperno et al. (2018), which can be used to improve
natural language understanding systems. Table 1 lists the number of instances
in the provided datasets. We use RTMs to model both tasks.

Figure 1: RTM depiction: parfda selects interpretants close to the data using
corpora; two MTPPS use interpretants, training data, and test data to generate
features in the same space; learning and prediction use these features as
input. Spheres are for feature spaces.

RTMs use parfda Biçici (2016a) to select both parallel and monolingual
interpretants, data close to the task instances selected specifically for the
task, to derive features measuring the closeness of the test sentences to the
training data, the difficulty of translating them, and to identify translation
acts between any two data sets using machine translation performance
prediction system (MTPPS) Biçici et al. (2013); Biçici (2022) to build
prediction models. Interpretants provide context and text for feature
derivation to link translation source and target and training and test sets.
RTMs are applicable in different domains and tasks and in both monolingual and
bilingual settings. Figure 1 explains RTMs’ model building process.

##  2 Predicting the Discriminative Power of Attributes

Capturing discriminative attributes task (Task 10) Paperno et al. (2018) is
looking at whether an attribute (e.g. red) can be used to discriminate between
two other words (e.g. apple and banana) to complement semantic similarity
efforts. The answer to whether the attribute can be used to discriminate the
two words can be useful for semantic similarity with contextual dependency.
The task is posed as a binary classification task:
$f(w_{1},w_{2},a)\rightarrow 0\;?\;1$. The target value to predict is either
$0$ or $1$ and shows whether the attribute can be used for discrimination for
the given context. Evaluation metric is $F_{1}$. RTMs are used by casting the
task as MTPP between the words and the attribute and building predictors that
use the distance between the predictions. We assume that the discriminative
power increase when the attribute is similar to words with significant
difference.

|  |  | English | Arabic | Spanish
---|---|---|---|---|---
train | Task 1 | emotion | 7102 | 3376 | 4544
dev | Task 1 | emotion | 1464 | 661 | 793
test | Task 1 | emotion | 71816 | 1563 | 2616
train | Task 1 | valence | 1181 | 932 | 1566
dev | Task 1 | valence | 449 | 138 | 229
test | Task 1 | valence | 17874 | 730 | 648
train | Task 10 |  | 17521 |  |
dev | Task 10 |  | 2722 |  |
test | Task 10 |  | 2340 |  |

Task 1 | emotion | English | Arabic | Spanish
---|---|---|---|---
train | anger | 1701 | 877 | 1166
train | fear | 2252 | 882 | 1166
train | joy | 1616 | 728 | 1058
train | sadness | 1533 | 889 | 1154
dev | anger | 388 | 150 | 193
dev | fear | 389 | 146 | 202
dev | joy | 290 | 224 | 202
dev | sadness | 397 | 141 | 196
test | anger | 17939 | 373 | 627
test | fear | 17923 | 372 | 618
test | joy | 18042 | 448 | 730
test | sadness | 17912 | 370 | 641

Table 1: Number of instances in the provided data. About $95\%$ of the Task 1
emotion English test sets are not evaluated for the competition rankings and
1002, 986, 1105, and 975 instances were actually scored. Figure 2: RTM with
stacked combined prediction use a combined model to obtain feature
representations and predictions for both $w_{1}\rightarrow a$ and
$w_{2}\rightarrow a$, which are processed before additional learning and
prediction.

The stacked RTM model with combined prediction step (Figure 2) use the same
model to predict the MTPP similarity of $w_{1}\rightarrow a$ and of
$w_{2}\rightarrow a$ where the data is collected such that the first row for
each attribute is for $w_{1}\rightarrow a$ and the second is for
$w_{2}\rightarrow a$. Stacking is used to build higher level models using
predictions from base prediction models where they can also use the
probability associated with the predictions Ting and Witten (1999). The
combined model in Figure 2 is adding the predictions as additional feature. We
obtain an RTM representation vector for each of the instances by using the
derived features and the combination of the prediction scores along with 5
additional features:

$\displaystyle\hat{y}_{1}$ $\displaystyle\hat{y}_{2}$
$\displaystyle|\hat{y}_{1}-\hat{y}_{2}|$
$\displaystyle(\hat{y}_{1}+\hat{y}_{2})/2$
$\displaystyle\sqrt{\hat{y}_{1}*\hat{y}_{2}}$ (1)

After this filtering step, we run another learning and prediction on the
concatenation of the features from both rows and the additional features.

Figure 3: RTM with stacked separate predictions use two different learning
steps to obtain feature representations and predictions for either
$w_{1}\rightarrow a$ or $w_{2}\rightarrow a$, which are processed before
additional learning and prediction.

The stacked RTM model with separate prediction steps use separated feature
sets with an initial prediction with each (Figure 3). Using separate
predictors for each word need not achieve better performance than using a
single predictor since the rows of words we compare need not belong to
different groups due to their ordering in the dataset and the words may be
only randomly positioned. However, both predictions can be useful as coming
from two weaker predictors since half of the training instances were used to
train each. The benefit of using separate feature sets is further
specialization of $w_{1}\rightarrow a$ and $w_{2}\rightarrow a$ models with
feature selection and partial-least squares from Section 4. As we see in Table
4, separated feature sets with two separate learning and prediction steps
improve the performance.

The architectures of Figure 2 and Figure 3 are general enough to be useful for
Task 1 as well and when we want to make use of the differences between the
predictions for the same instances.

##  3 Predicting the Intensity of the Structure and Content in Tweets

Affect in tweets 111www.twitter.com task (Task 1) Mohammad et al. (2018) is
about predicting the intensity of the emotion expressed for tweets within
sadness, joy, fear, or anger emotion categorizations or the valance
(sentiment). The emotion within tweets is about how the tweeter wrote and
valence or sentiment is about what the tweeter wrote. Intensity scores are
obtained with best-worst scaling (bws) Mohammad and Kiritchenko (2018), which
counts only the number of times a tweet is labeled as best or worst among 4
tweets where each is annotated by multiple workers. The scores are obtained
with the percentage of counts scaled to $[-1,1]$, which is later scaled to
$[0,1]$ for the task. bws can decrease the annotation effort to obtain the set
of binary comparisons used to obtain reliably agreed labels.222bws is similar
to inter-annotator agreement (IAA) $\tau$, which is in $[-1,1]$ and calculated
as IAA $\tau=(C-D)/(C+D)$, where C is the number of concordant pairs and D is
the number of discordant pairs. IAA $\tau$ is the same measure as “Kendall’s
$\tau$ with ties penalized” used in Bojar et al. (2013) (Task1.2), where it
was used for measuring the correlation between quality estimation systems and
human rankings. Oracle METEOR evaluation achieved $\tau=0.23$ for ranking in
2013’s quality estimation task (Task 1.2, see Bojar et al. (2013)). We tried a
randomized IAA (RIAA) $\tau$ QTLeap, DFKI (2013–2016) where the calculations
are similar to IAA $\tau$, but all ties were converted into _better_ and
_worse_ ranking randomly to obtain more robust results by distributing the
ambiguous counts. This also helps normalization of the counts such that their
sum are same for different models. A corresponding randomized bws would
randomly assign the remaining pairwise comparison counts, where for 4 tweets
only a single comparison would be left ambiguous, and the count can be
distributed evenly.

We model the task as MTPP of the tweets to the emotions to answer questions
like “to what degree is this tweet showing the emotion of”. Since a single
emotion word need not provide enough context for semantic discrimination, we
use sets of words for each emotion that express the same meaning using a
subset of the WordNet affect emotion lists Strapparava and Valitutti (2004).
The lexicon used for English is in Appendix A. We obtained their translations
to Arabic and Spanish using web translation sites 333e.g. translate.google.com
or www.bing.com/translator to obtain the corresponding lexicon. We use the
whole set of words corresponding to the emotion instead of the emotion word to
translate to. For valence intensity tasks, we used both of the sets of words
from emotions joy and sadness as a single sentence to translate to and we also
used them as separate rows using Figure 2 and Figure 3 where the tweet’s MTPP
is predicted to the sets of words of either joy or sadness (Table 10). We
participate in the regression tasks of Task 1 to predict the emotion and
valence intensity in Arabic, English, and Spanish tweets. The official
evaluation metric is Pearson’s correlation. 444The program for evaluation is
at https://github.com/felipebravom/SemEval_2018_Task_1_Eval

Task |  | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---|---
English | Task 1 | emotion | anger | 0.104 | 0.1744 | 1.121 | 0.3849 | 1.106
English | Task 1 | emotion | fear | 0.063 | 0.169 | 1.156 | 0.3869 | 1.155
English | Task 1 | emotion | joy | 0.266 | 0.1859 | 1.193 | 0.3849 | 1.293
English | Task 1 | emotion | sadness | 0.233 | 0.1673 | 1.114 | 0.3543 | 1.165
English | Task 1 | emotion | ALL | 0.168 | 0.1745 | 1.147 | 0.3781 | 1.182
Arabic | Task 1 | emotion | anger | 0.196 | 0.1407 | 0.988 | 0.3112 | 0.886
Arabic | Task 1 | emotion | fear | 0.125 | 0.1445 | 1.01 | 0.3152 | 0.904
Arabic | Task 1 | emotion | joy | 0.316 | 0.1431 | 0.949 | 0.4076 | 0.826
Arabic | Task 1 | emotion | sadness | 0.238 | 0.1452 | 0.99 | 0.3238 | 0.89
Arabic | Task 1 | emotion | ALL | 0.229 | 0.1434 | 0.981 | 0.3428 | 0.871
Spanish | Task 1 | emotion | anger | 0.211 | 0.171 | 1.007 | 0.3796 | 0.957
Spanish | Task 1 | emotion | fear | 0.35 | 0.1612 | 0.923 | 0.4543 | 0.837
Spanish | Task 1 | emotion | joy | 0.389 | 0.1596 | 0.9 | 0.471 | 0.838
Spanish | Task 1 | emotion | sadness | 0.415 | 0.1491 | 0.896 | 0.3968 | 0.833
Spanish | Task 1 | emotion | ALL | 0.34 | 0.1602 | 0.928 | 0.4273 | 0.858
ALL | Task 1 | emotion | ALL | 0.216 | 0.1641 | 1.041 | 0.3878 | 1.021
English | Task 1 | valence |  | 0.161 | 0.2642 | 1.458 | 0.5155 | 1.674
Arabic | Task 1 | valence |  | 0.264 | 0.1921 | 0.958 | 0.5595 | 0.919
Spanish | Task 1 | valence |  | 0.272 | 0.1665 | 0.962 | 0.5302 | 0.85
ALL | Task 1 | valence |  | 0.117 | 0.2141 | 1.154 | 0.5333 | 1.185
English | Task 10 | combined | -0.045 | 0.5034 | 1.018 | 1.0954 | 0.994
|  | $F_{1}$ |  |  |  |
English | Task 10 | 0.47 |  |  |  |
Table 2: Competition results on the test set. | T1 EI-en | T1 EI-ar | T1 EI-es | T1 V-en | T1 V-ar | T1 V-es | T10
---|---|---|---|---|---|---|---
ranks | 44 | 13 | 10 | 35 | 13 | 10 | 9
out of | 50 | 15 | 17 | 39 | 15 | 15 | 9
Table 3: RTM ranks at SemEval-2018. Task |  | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---|---
English | Task 1 | emotion | anger | 0.245 | 0.1689 | 1.086 | 0.3619 | 1.074
English | Task 1 | emotion | fear | 0.05 | 0.1526 | 1.044 | 0.413 | 0.952
English | Task 1 | emotion | joy | 0.028 | 0.1641 | 1.053 | 0.438 | 0.963
English | Task 1 | emotion | sadness | 0.004 | 0.1566 | 1.043 | 0.4064 | 0.944
English | Task 1 | emotion | ALL | 0.2245 | 0.1734 | 1.14 | 0.367 | 1.1693
Arabic | Task 1 | emotion | anger | 0.209 | 0.1413 | 0.992 | 0.3093 | 0.878
Arabic | Task 1 | emotion | fear | 0.173 | 0.1444 | 1.01 | 0.3112 | 0.905
Arabic | Task 1 | emotion | joy | 0.377 | 0.1417 | 0.94 | 0.4126 | 0.805
Arabic | Task 1 | emotion | sadness | 0.269 | 0.1442 | 0.983 | 0.3291 | 0.872
Arabic | Task 1 | emotion | ALL | 0.2543 | 0.1428 | 0.9771 | 0.344 | 0.8545
Spanish | Task 1 | emotion | anger | 0.183 | 0.1706 | 1.005 | 0.3807 | 0.927
Spanish | Task 1 | emotion | fear | 0.398 | 0.1607 | 0.92 | 0.4548 | 0.846
Spanish | Task 1 | emotion | joy | 0.298 | 0.1676 | 0.945 | 0.4856 | 0.843
Spanish | Task 1 | emotion | sadness | 0.405 | 0.1513 | 0.909 | 0.3943 | 0.838
Spanish | Task 1 | emotion | ALL | 0.324 | 0.1627 | 0.9426 | 0.4311 | 0.8568
English | Task 1 | valence |  | 0.1326 | 0.1884 | 1.0399 | 0.525 | 0.9791
Arabic | Task 1 | valence |  | 0.2981 | 0.1879 | 0.9366 | 0.5637 | 0.8482
Spanish | Task 1 | valence |  | 0.2152 | 0.1684 | 0.973 | 0.5399 | 0.8317
Table 4: RTM results on the test set after the competition. Task |  | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---|---
English | Task 1 | emotion | anger | 0.245 | 0.1689 | 1.086 | 0.3619 | 1.074
English | Task 1 | emotion | fear | 0.05 | 0.1526 | 1.044 | 0.413 | 0.952
English | Task 1 | emotion | joy | 0.028 | 0.1641 | 1.053 | 0.438 | 0.963
English | Task 1 | emotion | sadness | 0.004 | 0.1566 | 1.043 | 0.4064 | 0.944
English | Task 1 | emotion | ALL | 0.2245 | 0.1734 | 1.14 | 0.367 | 1.1693
Arabic | Task 1 | emotion | anger | 0.209 | 0.1413 | 0.992 | 0.3093 | 0.878
Arabic | Task 1 | emotion | fear | 0.177 | 0.1443 | 1.009 | 0.311 | 0.905
Arabic | Task 1 | emotion | joy | 0.377 | 0.1416 | 0.939 | 0.4123 | 0.805
Arabic | Task 1 | emotion | sadness | 0.271 | 0.144 | 0.982 | 0.329 | 0.871
Arabic | Task 1 | emotion | ALL | 0.2562 | 0.1428 | 0.9765 | 0.3439 | 0.8541
Spanish | Task 1 | emotion | anger | 0.183 | 0.1706 | 1.004 | 0.3807 | 0.926
Spanish | Task 1 | emotion | fear | 0.398 | 0.1607 | 0.921 | 0.4551 | 0.846
Spanish | Task 1 | emotion | joy | 0.297 | 0.1677 | 0.946 | 0.4857 | 0.842
Spanish | Task 1 | emotion | sadness | 0.405 | 0.1514 | 0.91 | 0.3944 | 0.837
Spanish | Task 1 | emotion | ALL | 0.3235 | 0.1627 | 0.9427 | 0.4313 | 0.8565
English | Task 1 | valence |  | 0.1332 | 0.1888 | 1.042 | 0.5254 | 0.9842
Arabic | Task 1 | valence |  | 0.3023 | 0.1874 | 0.9344 | 0.5628 | 0.8471
Spanish | Task 1 | valence |  | 0.2149 | 0.1684 | 0.973 | 0.5399 | 0.8321
Table 5: RTM results on the test set after the competition with symbolic grounding and scaling of predictions. Task |  | $r$ | MAE | RAE | MAER | MRAER | rMAER | rMRAER
---|---|---|---|---|---|---|---|---
without | MIX |  |  |  |  |  |  |  |  |
English | Task 1 | emotion | ALL | 0.2106 | 0.1755 | 1.1536 | 0.3687 | 1.1936 | 0.0795 | 0.4261
Arabic | Task 1 | emotion | ALL | 0.256 | 0.1428 | 0.9771 | 0.3442 | 0.852 | 0.0515 | 0.2447
Spanish | Task 1 | emotion | ALL | 0.3125 | 0.163 | 0.9445 | 0.432 | 0.8605 | 0.0564 | 0.2315
English | Task 1 | valence |  | 0.1203 | 0.1841 | 1.0164 | 0.5569 | 0.9296 | 0.0615 | 0.2367
Arabic | Task 1 | valence |  | 0.225 | 0.1927 | 0.9608 | 0.5702 | 0.8984 | 0.0615 | 0.1952
Spanish | Task 1 | valence |  | 0.2216 | 0.1683 | 0.9724 | 0.5416 | 0.8357 | 0.0533 | 0.187
with | MIX |  |  |  |  |  |  |  |  |
English | Task 1 | emotion | ALL | 0.2351 | 0.1693 | 1.1129 | 0.3648 | 1.1204 | 0.0722 | 0.3857
Arabic | Task 1 | emotion | ALL | 0.2543 | 0.1428 | 0.9771 | 0.344 | 0.8545 | 0.051 | 0.2482
Spanish | Task 1 | emotion | ALL | 0.324 | 0.1627 | 0.9426 | 0.4311 | 0.8568 | 0.0548 | 0.2298
English | Task 1 | valence |  | 0.1326 | 0.1884 | 1.0399 | 0.525 | 0.9791 | 0.0674 | 0.2808
Arabic | Task 1 | valence |  | 0.2981 | 0.1879 | 0.9366 | 0.5637 | 0.8482 | 0.0604 | 0.198
Spanish | Task 1 | valence |  | 0.2169 | 0.1683 | 0.9726 | 0.5396 | 0.8313 | 0.052 | 0.19
with | MIX | symbolic | grounding |  |  |  |  |  |  |
English | Task 1 | emotion | ALL | 0.2105 | 0.1738 | 1.1423 | 0.3675 | 1.1705 | 0.0768 | 0.4116
Arabic | Task 1 | emotion | ALL | 0.2546 | 0.1429 | 0.9772 | 0.3441 | 0.8542 | 0.051 | 0.248
Spanish | Task 1 | emotion | ALL | 0.324 | 0.1627 | 0.9427 | 0.4313 | 0.8563 | 0.0547 | 0.2293
English | Task 1 | valence |  | 0.129 | 0.1893 | 1.045 | 0.5274 | 0.9891 | 0.0688 | 0.2826
Arabic | Task 1 | valence |  | 0.3023 | 0.1875 | 0.9345 | 0.5632 | 0.847 | 0.0619 | 0.2002
Spanish | Task 1 | valence |  | 0.2171 | 0.1683 | 0.9726 | 0.5397 | 0.831 | 0.0521 | 0.1901
with | new MIX | symbolic | grounding |  |  |  |  |  |  |
English | Task 1 | emotion | ALL | 0.2355 | 0.1692 | 1.112 | 0.3648 | 1.1187 | 0.0719 | 0.3848
Arabic | Task 1 | emotion | ALL | 0.255 | 0.1428 | 0.977 | 0.3441 | 0.8528 | 0.0526 | 0.247
Spanish | Task 1 | emotion | ALL | 0.3236 | 0.1627 | 0.9427 | 0.4311 | 0.8567 | 0.0543 | 0.2296
English | Task 1 | valence |  | 0.1324 | 0.1881 | 1.0381 | 0.5251 | 0.9751 | 0.0665 | 0.2783
Arabic | Task 1 | valence |  | 0.2983 | 0.1879 | 0.9366 | 0.5637 | 0.848 | 0.0602 | 0.1978
Spanish | Task 1 | valence |  | 0.2175 | 0.1683 | 0.9725 | 0.5396 | 0.8308 | 0.0519 | 0.1899
with | fold MIX | symbolic | grounding |  |  |  |  |  |  |
English | Task 1 | emotion | ALL | 0.2355 | 0.1692 | 1.1119 | 0.3647 | 1.1185 | 0.0717 | 0.3844
Arabic | Task 1 | emotion | ALL | 0.2555 | 0.1428 | 0.9771 | 0.344 | 0.8533 | 0.0526 | 0.249
Spanish | Task 1 | emotion | ALL | 0.324 | 0.1627 | 0.9426 | 0.4311 | 0.8568 | 0.055 | 0.2299
English | Task 1 | valence |  | 0.1321 | 0.1881 | 1.0381 | 0.5249 | 0.9752 | 0.0676 | 0.279
Arabic | Task 1 | valence |  | 0.2997 | 0.1877 | 0.9358 | 0.5638 | 0.8475 | 0.0597 | 0.1972
Spanish | Task 1 | valence |  | 0.2169 | 0.1683 | 0.9724 | 0.5397 | 0.831 | 0.0521 | 0.1898
Table 6: RTM results on the test set without symbolic grounding and scaling of
predictions.

##  4 RTM Models and Results

The RTM models used predictions from machine learning models including ridge
regression (RR), k-nearest neighors (KNN), support vector regression (SVR),
AdaBoost Freund and Schapire (1997), and extremely randomized trees (TREE)
Geurts et al. (2006) in combination with feature selection (FS) Guyon et al.
(2002) and partial least squares (PLS) Wold et al. (1984). We use averaging of
scores from different models for robustness Biçici (2017b). Model
implementations use scikit-learn. 555http://scikit-learn.org/ We optimize
$\lambda$ for RR, k for KNN, $\gamma$, C, and $\epsilon$ for SVR, minimum
number of samples for leaf nodes and for splitting an internal node for TREE,
the number of features for FS, and the number of dimensions for PLS. We use
500 estimators in the TREE model and also for AdaBoost along with exponential
loss. We use grid search for SVR. We evaluate with Pearson’s correlation
($r$), mean absolute error (MAE), relative absolute error (RAE), MAER (mean
absolute error relative), and MRAER (mean relative absolute error relative)
Biçici and Way (2015). We use $7$-fold cross-validation performance on the
training set to rank models.

$1$&$2$-gram wrec
---
$1$&$2$&$3$-gram wGM
$1$&$2$-gram wGM
$1$&$2$-gram w$F_{1}$
$1$-gram wGM
Table 7: Top $5$ features selected for Task 10 translation logprobability bpw
---
word alignment (1 - WER)
word alignment $F_{1}$ score
$3$gram w$F_{1}$
sentence number of characters
Table 8: Top $5$ features selected for Task 1 Spanish

Interpretants are selected from the corpora distributed by the translation
task of WMT17 WMT (2017) and they consist of monolingual sentences used to
build the LM and parallel sentence pair instances used by MTPPS to derive the
features. We built RTM models using:

  * •

300 thousand sentences for training data

  * •

5 million sentences for LM

RTMs generate features for the training and the test set to map both to the
same space where the total number of features in Task 1 becomes 492 and Task
10 becomes 117. The difference is due to the smaller context the attribute
word provides and most of the sentence-level features become not useful
including the sentence structure parsing features or word alignment features.

Setting | $F_{1}$ | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---|---
combined | 0.5169 | 0.035 | 0.4735 | 0.958 | 1.1397 | 0.95
separate | 0.5006 | 0.014 | 0.4761 | 0.963 | 1.0317 | 0.939
Table 9: Task 10 test set results with translation similarity distance
modeling.

RTM results in the Task 1
666https://competitions.codalab.org/competitions/17751 and Task 10
777https://competitions.codalab.org/competitions/17326 competitions are in
Table 2 and ranks are in Table 3 Mohammad et al. (2018). 8 of the results
obtain MRAER larger than 1 suggesting more work towards these tasks or
subtasks. weight # models combine top # models’ predictions Biçici (2017b).
The predictions for Task 10 were transformed to binary classes by thresholding
with $0.5$ and obtains $0.47$ $F_{1}$.

Task 1 | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---
English | tweet $\rightarrow$ $V_{joy}\cup V_{sadness}$ | 0.1271 | 0.1973 | 1.0892 | 0.5248 | 1.0739
| tweet $\rightarrow$ $V_{joy}$ | 0.2121 | 0.1922 | 1.061 | 0.5334 | 1.0907
| tweet $\rightarrow$ $V_{sadness}$ | 0.2869 | 0.2027 | 1.1188 | 0.4552 | 1.1848
| combined | 0.2536 | 0.1891 | 0.943 | 0.56 | 0.8583
| separate | 0.2779 | 0.2477 | 1.3671 | 0.4883 | 1.5812
| separate+combined | 0.269 | 0.1973 | 1.0892 | 0.5298 | 1.1598
| separate+combined | 0.3555 | 0.176 | 0.9714 | 0.4542 | 0.9547 (MRAER)
Arabic | tweet $\rightarrow$ $V_{joy}\cup V_{sadness}$ | 0.2398 | 0.1921 | 0.9576 | 0.5486 | 0.8903
| tweet $\rightarrow$ $V_{joy}$ | 0.2127 | 0.1929 | 0.9616 | 0.565 | 0.8897
| tweet $\rightarrow$ $V_{sadness}$ | 0.243 | 0.19 | 0.9471 | 0.5554 | 0.8649
| combined | 0.2618 | 0.1896 | 0.9454 | 0.5805 | 0.8497
| separate | 0.1985 | 0.194 | 0.9673 | 0.5804 | 0.8968
| separate+combined | 0.1985 | 0.194 | 0.9673 | 0.5804 | 0.8968
Spanish | tweet $\rightarrow$ $V_{joy}\cup V_{sadness}$ | 0.2288 | 0.1679 | 0.97 | 0.5356 | 0.822
| tweet $\rightarrow$ $V_{joy}$ | 0.3764 | 0.1596 | 0.9218 | 0.5061 | 0.8442
| tweet $\rightarrow$ $V_{sadness}$ | 0.3259 | 0.1623 | 0.9376 | 0.5051 | 0.8657
| combined | 0.3096 | 0.1634 | 0.944 | 0.5228 | 0.8363
| separate | 0.3685 | 0.1601 | 0.9252 | 0.4986 | 0.8611
| separate+combined | 0.3331 | 0.1627 | 0.9397 | 0.5102 | 0.875
Table 10: Task 1 valence test set results with separate learning. Combined
learning has twice the amount of training data and test data is also doubled
accordingly.

The top $5$ features selected for Task 10 are listed in Table 7. $1$&$2$gram
w$F_{1}$ is $F_{1}$ score over $1$-gram and $2$-gram features with recall
computed according to the sum of the likelihood of observing them among
$1$-grams or $2$-grams correspondingly (wrec) and precision computed according
to all corresponding counts in $n$-grams. wGM is weighted geometric mean of
the arguments of $F_{1}$. The features enable linking $w_{1}$ and $a$ and
$w_{2}$ and $a$ and $8$ of the top $10$ features use $n$-gram features, which
makes sense for linking words and we observe this even semantically for Task
10. The top $5$ features selected for Task 1 are listed in Table 8. bpw is
bits per word. WER is word error rate.

Task 1 | $r$ | MAE | RAE | MAER | MRAER
---|---|---|---|---|---
Task | Model |  | $a(x_{1}-x_{2})+b$
English | combined |  |  |  |  |
| separate |  |  |  |  |
Arabic | combined | 0.044 | 0.2 | 0.997 | 0.6069 | 0.876
| separate | -0.005 | 0.201 | 1.002 | 0.6096 | 0.884
Spanish | combined | 0.056 | 0.1735 | 1.003 | 0.5539 | 0.841
| separate | -0.05 | 0.175 | 1.011 | 0.5547 | 0.855
Task | Model |  | $a(x_{1}+x_{2})/2+b$
English | combined |  |  |  |  |
| separate |  |  |  |  |
Arabic | combined | 0.238 | 0.1908 | 0.951 | 0.5817 | 0.857
| separate | 0.24 | 0.1907 | 0.951 | 0.5778 | 0.854
Spanish | combined | 0.34 | 0.1641 | 0.948 | 0.5262 | 0.82
| separate | 0.385 | 0.1603 | 0.926 | 0.5134 | 0.82
Table 11: Task 1 valence test set results with separate learning and
prediction combinations.

##  5 Experiments After the Challenge

We recalculated our results using 200 thousand training instances for Task 1,
which also helps comparison with previous results Biçici and Way (2015) and an
advanced IBM2 alignment model after the challenge. Table 4 lists the new
results obtained after the challenge. The number of results with MRAER larger
than 1 decreased to 5. The MRAER obtained by RTMs in STS in 2016 is $0.73$
Biçici (2016b) and in quality estimation task for English to German in 2017 is
$0.8$ Biçici (2017a). Translation similarity distances alone can be useful to
identify the discriminative power of attributes Table 9. The predictions for
Task 10 were transformed to binary classes by thresholding with optimized
thresholds on the training set. Prediction differences are used from the
combined model or the separate model. Table 10 show that valence can be better
predicted with MTPP towards the translation of the vocabulary of sadness
rather than using the union of contrastive vocabulary sets of joy and sadness.
Separate learning also improve the performance for English and Spanish valence
prediction. Translation similarity distances are also used for Task 1 in Table
10 where instead of learning a threshold to optimize for $F_{1}$ score, we fit
a linear model, $ax+b$, to the distance between predictions.

###  5.1 Symbolic Grounding of Score Thresholds

We symbolically ground the score thresholds with respect to the statistics of
the training set scores and use the corresponding linear mathematical equation
in the form of $a\mu+b\sigma+c$ on the test set statistics to obtain a
relative test set threshold. We observe up to $1\%$ improvements in the
prediction performance in Task 10 (Section 5).

###  5.2 Symbolic Grounding of Prediction Score Statistics

We symbolically ground the predictions with respect to the statistics of the
training set scores and use the corresponding linear mathematical equation on
the test set statistics to obtain relative predictions on the test set. We
observe up to $\ldots\%$ improvements in the prediction performance (Section
5).

Metric | ranking error | $r_{S}$
---|---|---
r | 0.0564 | 0.6614
MAE | 0.1167 | 0.2997
RAE | 0.1167 | 0.2997
MAER | 0.1518 | 0.0892
MRAER | 0.0947 | 0.4318
rMAER | 0.0893 | 0.464
rMRAER | 0.0869 | 0.4785
Table 12: Ranking errors and $r_{S}$ with different metrics.

###  5.3 Ranking Results

We would like to obtain robust sortings on the training set so that the
sortings are reflected on the performance sorting on the test set. We compare
the ranking differences that incur when we choose a metric to rank both the
training and test set results. Table 12 compares the performance metrics we
use using $r_{S}$, Spearman’s correlation, and ranking errors, which compute
the squared error of normalized ranking differences (Equation 2).

$rankError=\sum_{i}(\texttt{rank}(i,\mbox{train})/n-\texttt{rank}(i,\mbox{test})/n)^{2}$
(2)

$r_{P}$ is Pearson’s correlation and $r_{S}$ is calculated using the same
formula of $r_{P}$ but using rankings where tied ranks are assigned the mean
rank Kokoska and Zwillinger (2000). Equation 3 provides an approximate result
when there are ties. $r_{S}$ provides higher-level information than $r_{P}$
since we use the rankings obtained.

$\displaystyle r_{P}$ $\displaystyle=$
$\displaystyle\frac{\displaystyle\sum_{i=1}^{n}(\hat{y_{i}}-\bar{\hat{y}})(y_{i}-\bar{y})}{\sqrt{\displaystyle\sum_{i=1}^{n}(\hat{y_{i}}-\bar{\hat{y}})^{2}}\sqrt{\displaystyle\sum_{i=1}^{n}(y_{i}-\bar{y})^{2}}}$
$\displaystyle r_{S}$ $\displaystyle\simeq$ $\displaystyle
1-\frac{6\sum_{i=1}^{n}d_{i}^{2}}{n(n^{2}-1)}$ (3)

Both $r_{S}$ and $r_{P}$ are invariant to scaling and we can use Equation 6
Rodgers and Nicewander (1988) for standardized $\hat{y}$ and $y$ where
$xy=\frac{1}{2}(x^{2}+y^{2}-(x-y)^{2})$ equivalence and
$\sum_{i}\hat{y_{i}}=1$ and $\sum_{i}y_{i}=1$ are used. Since $-1\leq r\leq
1$, the variance of the difference is in $[0,4]$. Similarly,
$r_{P}=\frac{1}{2}s^{2}_{\hat{\textbf{y}}+\textbf{y}}-1$, and thus correlation
and $s^{2}_{\hat{\textbf{y}}-\textbf{y}}$ or
$s^{2}_{\hat{\textbf{y}}+\textbf{y}}$ can be used to determine each other
Rodgers and Nicewander (1988). Equation 3 is obtained using Equation 4 and
assuming that ranks are distinct.

$\displaystyle r_{P}$ $\displaystyle=$
$\displaystyle\frac{\displaystyle\sum_{i=1}^{n}\hat{y_{i}}y_{i}}{\sqrt{\displaystyle\sum_{i=1}^{n}\hat{y_{i}}^{2}}\sqrt{\displaystyle\sum_{i=1}^{n}y_{i}^{2}}}\;\;\mbox{(for
standardized input)}$ $\displaystyle=$
$\displaystyle\displaystyle\frac{1}{n}\sum_{i=1}^{n}\hat{y_{i}}y_{i}$
$\displaystyle=$
$\displaystyle\displaystyle\frac{1}{2n}\sum_{i=1}^{n}(\hat{y_{i}}^{2}+y_{i}^{2}-(\hat{y_{i}}-y_{i})^{2})$
(4) $\displaystyle=$ $\displaystyle\displaystyle
1-\frac{1}{2n}\sum_{i=1}^{n}(\hat{y_{i}}-y_{i})^{2}$ (5) $\displaystyle=$
$\displaystyle\displaystyle 1-\frac{1}{2}s^{2}_{\hat{\textbf{y}}-\textbf{y}}$
(6)

$\displaystyle\mbox{MAER}(\hat{\textbf{y}},\textbf{y})$ $\displaystyle=$
$\displaystyle\frac{\displaystyle\sum_{i=1}^{n}\frac{|\hat{y_{i}}-y_{i}|}{\lfloor|y_{i}|\rfloor_{\epsilon}}}{n}$
(7) $\displaystyle\mbox{MRAER}(\hat{\textbf{y}},\textbf{y})$ $\displaystyle=$
$\displaystyle\frac{\displaystyle\sum_{i=1}^{n}\frac{|\hat{y_{i}}-y_{i}|}{\lfloor|\bar{y}-y_{i}|\rfloor_{\epsilon}}}{n}$
(8) $\displaystyle\mbox{rMAER}(\hat{\textbf{y}},\textbf{y})$ $\displaystyle=$
$\displaystyle\frac{1}{n}\displaystyle\sum_{i=1}^{n}\frac{|\hat{y_{i}}-y_{i}|}{\lfloor|y_{i}|\rfloor_{\epsilon}}f_{<0,\epsilon}\left(\frac{(\hat{y_{i}}-\bar{\hat{y}})(y_{i}-\bar{y})}{\sigma_{\hat{y}}\sigma_{y}\lfloor|y_{i}|\rfloor_{\epsilon}^{2}}\right)$
(9) $\displaystyle f_{<0,\epsilon}(x)$ $\displaystyle=$
$\displaystyle\begin{cases}\lfloor x\rfloor_{\epsilon}&\text{if }x\geq 0\\\
\lfloor-2x\rfloor_{\epsilon}&\text{if }x<0\end{cases}$ (10)

MAER and MRAER are capped from below888We use $\lfloor\;.\;\rfloor_{\epsilon}$
to cap the argument from below to $\epsilon$. with
$\epsilon=\mu_{|\hat{\textbf{y}}-\textbf{y}|}/2$, which is the measurement
error and it is estimated as half of the mean absolute error or deviation of
the predictions from the target mean. $\epsilon$ represents half of the score
step with which a decision about a change in measurement’s value can be made.
$\epsilon$ is similar to half of the standard deviation, $\sigma$, of the data
but over absolute differences. For discrete target scores,
$\epsilon=\frac{\textit{step size}}{2}$.

We use standardized relative correlation measure Biçici (2016b). rMAER
combines correlation with MAER using a reciprocal function of negative values,
$f$. rMRAER is defined similar to rMAER.

##  6 Conclusion

Referential translation machines obtain automatic prediction of semantic
similarity using MTPP. We presented encouraging results with stacked RTM
models for predicting the intensity of the structure and content in text with
our novel MTPP modeling for translation to WordNet emotion lists and the
discriminative power of attributes using stacked RTM models. Our results also
enable comparisons of prediction results of RTMs in different natural language
processing tasks Biçici (2017c).

## References

  * WMT (2017) 2017\.  _Second Conf. on Machine Translation_. Copenhagen, Denmark.
  * Apidianaki et al. (2018) Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, and Marine Carpuat. 2018.  12th int. workshop on semantic evaluation (semeval-2018).  In _12th Int. Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, USA.
  * Biçici (2016a) Ergun Biçici. 2016a.  ParFDA for instance selection for statistical machine translation.  In _Proc. of the First Conf. on Machine Translation (WMT16)_ , pages 252–258, Berlin, Germany.
  * Biçici (2016b) Ergun Biçici. 2016b.  RTM at SemEval-2016 task 1: Predicting semantic similarity with referential translation machines and related statistics.  In _SemEval-2016: Semantic Evaluation Exercises - Inter. Workshop on Semantic Evaluation_ , pages 758–764, San Diego, CA, USA.
  * Biçici (2017a) Ergun Biçici. 2017a.  Predicting translation performance with referential translation machines.  In _Proc. of the Second Conf. on Machine Translation (WMT17)_ , pages 540–544, Copenhagen, Denmark.
  * Biçici (2017b) Ergun Biçici. 2017b.  RTM at SemEval-2017 task 1: Referential translation machines for predicting semantic similarity.  In _11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 194–198, Vancouver, Canada.
  * Biçici (2017c) Ergun Biçici. 2017c.  RTM Performance Prediction Results.  Comparison of RTM results for various prediction tasks.
  * Biçici (2022) Ergun Biçici. 2022.  Machine translation performance prediction system: Optimal prediction for optimal translation.  _Springer Nature Computer Science_ , 3.
  * Biçici et al. (2013) Ergun Biçici, Declan Groves, and Josef van Genabith. 2013.  Predicting sentence translation quality using extrinsic and language independent features.  _Machine Translation_ , 27(3-4):171–192.
  * Biçici and Way (2015) Ergun Biçici and Andy Way. 2015.  Referential translation machines for predicting semantic similarity.  _Language Resources and Evaluation_ , pages 1–27.
  * Bojar et al. (2013) Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013.  Findings of the 2013 Workshop on Statistical Machine Translation.  In _Eighth Workshop on Statistical Machine Translation_ , pages 1–44, Sofia, Bulgaria.
  * Freund and Schapire (1997) Yoav Freund and Robert E Schapire. 1997.  A decision-theoretic generalization of on-line learning and an application to boosting.  _Journal of Computer and System Sciences_ , 55(1):119–139.
  * Geurts et al. (2006) Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006.  Extremely randomized trees.  _Machine Learning_ , 63(1):3–42.
  * Guyon et al. (2002) Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002.  Gene selection for cancer classification using support vector machines.  _Machine Learning_ , 46(1-3):389–422.
  * Kokoska and Zwillinger (2000) Stephen Kokoska and Daniel Zwillinger. 2000.  _CRC standard probability and statistics tables and formulae_.  Taylor and Francis, Hoboken, NJ.
  * Mohammad et al. (2018) Saif M. Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018.  SemEval-2018 Task 1: Affect in Tweets.  In _12th Int. Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, USA.
  * Mohammad and Kiritchenko (2018) Saif M. Mohammad and Svetlana Kiritchenko. 2018.  Understanding emotions: A dataset of tweets to study interactions between affect categories.  In _11th Edition of the Language Resources and Evaluation Conf._ , Miyazaki, Japan.
  * Paperno et al. (2018) Denis Paperno, Alessandro Lenci, and Alicia Krebs. 2018.  SemEval-2018 Task 10: Capturing discriminative attributes.  In _12th Int. Workshop on Semantic Evaluation (SemEval-2018)_ , New Orleans, USA.
  * QTLeap, DFKI (2013–2016) QTLeap, DFKI. 2013–2016.  Quality translation by deep language engineering approaches. Grant no: 610516.
  * Rodgers and Nicewander (1988) Joseph Lee Rodgers and W. Alan Nicewander. 1988.  Thirteen ways to look at the correlation coefficient.  _The American Statistician_ , 42(1):59–66.
  * Strapparava and Valitutti (2004) Carlo Strapparava and Alessandro Valitutti. 2004.  WordNet-Affect: An affective extension of WordNet.  In _4th Int. Conf. on Language Resources and Evaluation_ , pages 1083–1086. ELRA.
  * Ting and Witten (1999) Kai Ming Ting and Ian H. Witten. 1999.  Issues in stacked generalization.  _Journal of Artificial Intelligence Research_ , 10:271–289.
  * Wold et al. (1984) S. Wold, A. Ruhe, H. Wold, and III Dunn, W. J. 1984.  The collinearity problem in linear regression. the partial least squares (pls) approach to generalized inverses.  _SIAM Journal on Scientific and Statistical Computing_ , 5:735–743.

##  Appendix A English Lexicon used from WordNet Affect Emotion Lists

English lexicon used for identifying discriminative attributes (Task 10) as a
subset of the WordNet affect emotion lists Strapparava and Valitutti (2004)
available at http://web.eecs.umich.edu/~mihalcea/affectivetext/. 999The
original lexicon was made available freely for research purposes. The lexicon
in this section is for the publication and demonstration.

anger
---
torment pissed frustrated crucify pout grudge umbrageous incensed spitefulness
execration exacerbate choleric infuriating displease teasing aggression
misology maliciously riled ire baffled spiteful pestered enragement rag
misanthropy misogynism harried grudging annoyance abominate displeased furious
irritating wrothful nettlesome detest vexation hatred indignantly angry
offense vindictiveness despising revengefully disdain belligerent
murderousness stung misanthropical lividly madness irritation belligerently
misogynic begrudge abhor temper indignation infuriated anger discouraged
grievance contemn bitterness vexatious rancour resentment hate exasperating
malign covetous enraged envious enviously jealous irascibility loathing
displeasingly gravel odium grasping infuriate sulkiness outrage annoyed mad
covetously vindictively belligerence prehensile enviousness scene harassment
murderously enfuriate wrathful hateful displeasure loathe scorn maliciousness
execrate venom enviably pique roiled heartburning malicious tantrum irritated
envy frustration nettle harassed spleen brood enviable offend animosity
despiteful maddening sore balked aggressive enmity vexed vex jealously
resentfully angered galling pestering vindictive hackles umbrage bothersome
score gall misanthropic vexing persecute aggressiveness indignant amuck
oppress rancor jealousy annoying resentful covetousness frustrating outraged
tantalize devil greedy hostilely wrathfully covet exasperation frustrate
hostility misogyny incense rage choler malevolently malignity malevolence
vengefulness huffiness malevolent furiously chafe pesky begrudging huffy
angrily harass wroth despise fury rile avaricious malefic bother abomination
spite nettled antagonism aggravated hatefully aggravate hostile belligerency
annoy maddened provoked exasperate malice plaguey vengefully displeasing
lividity stew pestiferous evil wrath irritate infuriation grizzle aggravation
abhorrence misoneism fit
fear
atrocious anxiously diffidently coldheartedness cruelty frightening hesitantly
chill dreadfully scarily panic foreboding hideous panicky hesitancy timidness
hesitance cower intimidation ruthlessness dreaded alarm suspense intimidate
chilling dreadful horrify diffidence alert fearful presentiment creep
apprehension hysterical uneasily anxious bashfully horribly timid apprehensive
ugly hardheartedness cringe unkind browbeaten horrifying fawn shadow
unassertiveness horrifyingly dread awfully intimidated frightful afraid
apprehensiveness outrageous premonition panicked shy scary frighten
hysterically heartlessly pitilessness shyly frighteningly fearfulness
diffident monstrously dismay timorous fear horrified timorousness unassertive
fright cruelly scared consternation affright apprehensively dire timidity
heartlessness heartless timidly scarey hysteria alarmed horrific cowed presage
shyness mercilessness awful trepidation hideously unsure scare bullied
frightened horridly fearsome crawl creeps direful terrified fearfully
suspensive shuddery

sadness
---
disconsolate sorrow guilt dolorous demoralization weight dispirit remorse
oppression heartbreaking sorrowfully drear grievously ruefulness sorry
oppressed despondently despondency wretched woefulness dreary desolation
bereft downcast attrition forlornly sadness plaintive pitiable woebegone
penitently pitying desolate heartsickness demoralize dispiritedness dysphoria
depressive loneliness persecuted remorsefully sadly dark joylessly regretful
despairingly tyrannical suffering sadden ruthfulness distressed melancholic
sad bored discouraged disheartening regret penitentially plaintively heartache
forlorn tearfulness lachrymose ruefully cheerlessness forlornness gloomily
oppressive contrite shamefaced heartrending drab plaintiveness melancholy
gloomful mournfulness penance brokenheartedness saddening repentant dysphoric
depress mourning heartsick repentantly heavyhearted doleful aggrieve
contritely hapless rue contrition downheartedness depression tyrannous piteous
pitiful uncheerful demoralized laden tearful grievous woeful gloomy penitence
godforsaken repent penitent rueful pathetic miserably dolefully bad
oppressiveness dismay repentance cheerless dingy heavyheartedness joyless grim
grief persecute bereaved grieving grieve disconsolateness lamentably
unhappiness guilty oppress helplessness weeping misery oppressively deject
hangdog heartbreak demoralising woefully depressed uncheerfulness contriteness
dispirited joylessness disheartened poor gloominess depressing remorseful down
unhappy low weepiness mournful deplorably blue glum despondent harass gloom
downhearted dolefulness mournfully demoralizing woe downtrodden compunction
shamed sorrowing dolourous misfortunate dismal sorrowful glooming
sorrowfulness cheerlessly dispiriting
joy
kick identification amicably anticipation cheerfully favourably eager close
gratifying suspenseful triumphant captivation beaming protectiveness devoted
zest ardour satisfy exhilarating occupy preference exciting happily penchant
compatibly jubilancy tenderness near exultingly approved satisfactory merry
becharm crush admire joy exult jolly admiration adoration complacent pleased
eagerly anticipate fondness regard kindhearted proudly happiness entrance
rapport enamoredness affectional keenness closeness praiseworthily hilariously
festive affect appreciated exultant elation exhilaration protective prideful
gloat impress liking unworried pride friendly soothe move thrill gratify
enthusiastically catch gleefully joyous excitement teased titillate
gleefulness joyousness uproarious goodwill gaiety great worship worry glad
favorably comfortably taste exhilarate perkiness uplift empathy satisfying
hearten attachment comfort gratifyingly beguile commendable fulfil affection
tender fulfillment titillation sunny jocularity stimulating warmth concern
cheer enthralling joyfully comfortableness entranced approving mirthfully
content comforting jubilantly predilection jocund console euphoria
satisfactorily merrily satisfied partiality strike kindly gladfulness enjoy
solace good fascination love zealous sympathetically benevolent fulfilment
affectionateness devotedness gladdened riotously buoyancy benevolently
enthralled adorably beneficent hilarity amorous warmhearted likable weakness
belonging carefree avidness carefreeness affective captivate barrack recreate
sympathetic protectively euphoriant urge friendliness jubilant comfortable
benefic intimacy admirably enthusiasm respect sympathy esteem bang emotive
enchant approval charm favour satisfyingly amative fascinating gloating fancy
brotherlike hilarious loyalty lovingly capture bewitching satisfiable
beneficed compatibility lovesome identify beneficially zeal cheery revel
favourable jubilance fulfill happy lovingness gayly brotherhood jollity
congratulate complacence satisfaction lightsomeness romantic intoxicate
joyously triumph charitable devotion heart ebullient benevolence exhilarated
exuberance approve contented captivated exuberant euphoric exultantly festal
rush cheerful endearingly tickle jubilation complacency gladsomeness
enthusiastic delighted triumphal joyful affectionate gloatingly charmed
eagerness empathic amicability amatory flush kid charge screaming rejoice
relish entrancing fondly expansively exuberantly contentment inspire fond like
suspensive likeable amicable triumphantly expectancy


# Open Arms: Open-Source Arms, Hands & Control

David Hanson
Hanson Robotics Ltd
<EMAIL_ADDRESS>
Alishba Imran
Hanson Robotics Ltd
<EMAIL_ADDRESS>
Gerardo Morales
Hanson Robotics Ltd
<EMAIL_ADDRESS>
Vytas Krisciunas
Hanson Robotics Ltd
<EMAIL_ADDRESS>
Aditya Sagi
Indian Institute of Science
<EMAIL_ADDRESS>
Aman Malali
ARTPARK
<EMAIL_ADDRESS>
Rushali Mohbe
ARTPARK
<EMAIL_ADDRESS>
Raviteja Upadrashta
ARTPARK
<EMAIL_ADDRESS>

###### Abstract

Open Arms is a novel open-source platform of realistic human-like robotic
hands and arms hardware with 28 Degree-of-Freedom (DoF), designed to extend
the capabilities and accessibility of humanoid robotic grasping and
manipulation. The Open Arms framework includes an open SDK and development
environment, simulation tools, and application development tools to build and
operate Open Arms. This paper describes these hands’ controls, sensing,
mechanisms, aesthetic design, and manufacturing and their real-world
applications with a teleoperated nursing robot. From 2015 to 2022, we have
designed and established the manufacturing of Open Arms as a low-cost, high
functionality robotic arms hardware and software framework to serve both
humanoid robot applications and the urgent demand for low-cost prosthetics.
Using the techniques of consumer product manufacturing, we set out to define
modular, low-cost techniques for approximating the dexterity and sensitivity
of human hands. To demonstrate the dexterity and control of our hands, we
present a novel Generative Grasping Residual CNN (GGR-CNN) model that can
generate robust antipodal grasps from input images of various objects in real-
time speeds ($\sim$22ms). We achieved state-of-the-art accuracy of 92.4% using
our model architecture on a standard Cornell Grasping Dataset, which contains
a diverse set of household objects.

##  1 Introduction

Today, multi-DOF robot hands can already accomplish various tasks that only
humans can perform otherwise. However, the development of low-cost human-like
robot hands and the precise manipulation control of everyday objects with such
hands remains a challenge.

Most existing robotic hands are costly, difficult to maintain because of
tendon-driven actuators, or do not provide precise control. The TWENDY-ONE
Hand [1] has stable manipulation skills, but its usage is limited in
industrial applications due to the high development and maintenance cost of
highly sophisticated actuators and sensors. Similarly, the multi-fingered
Shadow Dexterous Hand can accomplish complex tasks [2], but costs about
$100,000, and the tendon-driven actuators make the maintenance difficult. A
more cost-effective hand is the Wonik’s Allegro Hand [3], which is relatively
low-cost ($15,000) but does not provide precise position or force control.
Open Bionics developed the Hero arm [4] as a lower-cost 3D printed alternative
to contemporary prosthetics but these prosthetics lack intelligent controls.
We note opportunities to improve on these existing projects by adding
additional naturalism to the design, with improved actuation, sensing,
intelligent controls, and scalable consumer-grade manufacturing to reduce
costs while achieving high quality.

In this paper, we describe how our novel robotic arms can be designed such
that the fabrication of the robotic hand is fast, the cost of the modification
and maintenance is cheap, and the control of the robotic hand is accurate. We
present the design and construction of novel robotic arms as a new platform,
integrating new hardware design and the controls of the hands and arms using a
cognitive AI framework involving GGR-CNN for accurate grasping.

The Open Arms include novel robot arms built with a humanlike mechanical
configuration and aesthetic, with 28 DoF, touch sensors, and series elastic
actuators. The arms and hands have many improvements such as retraction and
elevation of the shoulders, a reduction in its weight, as well as more grip
strength that can even lift a 600g bottle, pressure sensors in the fingertips
and position sensors in the finger joints, giving you the option to receive
feedback when manipulating objects.

The main contributions of this paper can be summarized as follows:

  * •

The Open Arms is a robotics framework that pushes forward realistic design
with improved actuation, sensing, controls, and scalable manufacturing methods
to reduce the costs of 28 DoF human-like robotic hands and arms.

  * •

The platform includes an open SDK and development environment to build,
customize, and operate Open Arms.

  * •

We propose a Generative Grasping Residual CNN (GGR-CNN) model for the hands
that predicts, plans, and performs antipodal grasps for the objects in the
camera’s field of view.

  * •

We evaluate our grasping model on the publicly available Cornell grasping
dataset and achieve state-of-the-art accuracy of 92.4

  * •

We demonstrate the real-world use of our arms and hands by testing and
outlining the design of a teleoperated nursing robot.

The resulting framework is an integral part of the Sophia Utility Platform
(SUP). The uses of the work extend across domains and include arts, social
human-robot interaction, and targeting more general co-bot applications. These
results are available to test on the broadly released Sophia Utility Platform
(SUP), including Open Arms, for users to try and extend.

##  2 Background

###  2.1 Open Arms, Part of Sophia Utility Platform: An Integrative Platform
for Embodied Cognition

The Open Arms were developed as part of our latest designed framework for
human-like embodied cognition, called Sophia Utility Platform (SUP).

While some cognitive robotics and AI frameworks exist, such as iCub [5], SOAR
[6], ACT-R [7], and Open Cog [8], none provide the full, integrated
capabilities needed for the pursuit of humanlike embodied cognition, including
low-cost mass-manufactured standard hardware for hands and arms, expressive
face, open interfaces, and a diversity of features and extensions. The SUP
provides many features, including an expressive face, Open Arms, a mobile
base, and a feature-rich AI SDK on a new mass-manufacturable hardware and
software framework for cognitive robotics. With SUP, we strive to architect a
humanlike agent with embodiment and cognitive AI tools for an agent to learn
its environment using cutting-edge multimodal learning methods in vision,
speech, grasping and manipulation, locomotion, and social interactions with
human users. We combine various neural frameworks with a rules-based
controller governing an ensemble verbal and non-verbal dialogue model that is
original to us. Using basic emotional parameters to drive the agent’s goal
pursuit, making the agent more useful physical actions like the Open Arms and
hands control while also achieving socially intelligible and appealing human-
robot interactions.

##  3 Open Arms Robotic Hardware Design

Open Arms have 28 DoF, and the hardware is designed to simulate the action,
appearance, and capabilities of the human hands and arms. Each arm includes 2
DoF for the shoulder joint (pitch and roll), 1 DoF for the shoulder or elbow
yaw between the shoulder and the arm, 1 DoF for the elbow (pitch), 1 DoF for
the forearm (yaw), and 2 DoF for the wrist (pitch, roll). Each hand includes 2
DoF for the thumb, 1 DoF for the finger spread, and 4 DoF for each of the
other fingers controlled by a single actuator.

The hands and arms have sensing, including force, position, temperature
feedback, compliant series elastic actuators in the fingers, and touch sensing
in the fingertips. The arms were designed to be modular so that the hands and
forearms could also be used in low-cost prosthetics and could be easily
changed and upgraded. The first iteration of the designs was completed in
2014, intended both for research and for mass-production as part of the Sophia
Utility Platform line of social robots for general use, including human-AI
interactions and collaborative robotics (co-botics). Early versions were
released under Creative Commons open license in 2016 and 2017. From 2020 to
2022, we completed improved Beta designs and initiated the manufacturing of
the Open Arms in volume units. These latest versions are now released under
the General Public License and the designs are available for any developer to
use freely. The design files for the arms are attached as supplementary
material.

Figure 1: Servos and design of the Open Arms

###  3.1 Mechanical Design

Mechanically, we developed our robotic arms to suit the character and
collaborative robotic needs. We used the dimensions from a medium-sized
woman’s arms as constraints and quality components integrated within the
robots. We try to have feedback in all the DOF axes where possible. The
principal structure of the arms is made of CNC Aluminum 6061, and complicated
manufactured parts are made of Nylon SLS PA 2200, black colour for aesthetic.
Feetech servos are used for controlling and for feedback. Due to the size
limitation, each hand uses our versions of Feetech servos with Pololu Maestro
Controller 24ch. For feedback, there are pressure sensor pads and
potentiometers in each finger and wrist. The goal of our arms is for
gesticulation when she speaks and for object grasping and manipulation.

The shoulders on the torso have new features: elevation-depression (22° range)
and retraction-protraction (23° range) using 2 Actuonix+L12-30 linear
actuators on each side motors are connected with universal joints. These
characteristics in the shoulders aggregate us with more realistic movements
and better gestures.

The robotic arms have 7 DOF including:

  * •

Shoulder pitch (180° range).

  * •

Shoulder roll (180° range), which are driven by a pulley and timing belt with
a 2:1 ratio using Feetech SM60CL motors each.

  * •

Shoulder yaw (120° range) is driven by gear with a 2.917:1 ratio with Feetech
SM40BL motor.

  * •

Elbow uses the same belt pulley with a 2:1 ratio (range 127°) and Feetech
SM60CL motor.

  * •

Wrist yaw (180° range) uses bevel gears with a 1.8:1 ratio with Feetech SCS40
TTL motor.

  * •

Finally, wrist roll (40° range) and pitch (45° range) are powered by two
Feetech SM40BL motors with universal joints simultaneously.

The hand has 6 DOF, all with small Feetech STS3032 motors to drive the 5
fingers and the thumb roll with metal wires. The hand closes when the motors
pull the wires and opens when they release it along with the help of torsion
springs that are in the joints of each finger.

The way to connect the arm and send the information to the computer is through
the Feetech FE-URT1 converter, which receives communication from the SM40BL
and SM60CL motors connected in series via RS485 protocol and the rest of the
STS3032 and SCS40TTL motors connected in series via TTL protocol.

Figure 2: Servos on our open hands

###  3.2 Arm Axes

The arms use Feetech servos, and the selection of those servos is determined
by the weight, torque and space limitations. The torque of each Axes was
calculated in Solidworks with the center of mass and the weight.

The features of Feetech servos are: Smart actuator with fully integrated DC
Motor + Reduction Gearhead + Encoder + Controller + Driver. Functions include
precise control, PID control, 360 degrees of position control, and high-speed
communication.

###  3.3 Hand Axes

Due to space limitations, there are 3 Feetech servos in hand and 3 more in the
forearm, which control the fingers. For the feedback, each fingertip has an
internal pressure sensor pad measuring 500gr; the sensor is activated through
a Nylon-rubber fingertip cover.

There are 16 10K ohms potentiometers for measuring positions: index, middle,
ring, pinky, thumb fingers, thumb roll, wrist pitch and wrist roll. There are
torsion springs in fingers joints in one direction for safety use.

###  3.4 Electrical

The power of the Arm is 13.8V, stepped down from 24V for SM40BL and SM60CL
motors and power of 6V for STS3032 and SCS40TTL motors, and the communication
is through the Feetech FE-URT1 converter, which receives communication from
the SM40BL and SM60CL motors connected in series via RS485 protocol and the
rest of the STS3032 and SCS40TTL motors connected in series via TTL protocol.
The sensors like pressure sensors and potentiometers are connected through two
24 channel Pololu Maestro-microcontrollers in each forearm.

##  4 Open Arms, Grasping: Generative Grasping Residual CNN (GGR-CNN)

###  4.1 Deep Learning Methods Using CNNs for Grasping

Deep learning techniques have enabled big advancements in robotic grasping for
unknown items. These approaches typically use variations of Convolutional
Neural Network (CNN) architectures designed for object recognition [9, 10],
and in most cases sample and rank grasp candidates individually using a CNN
[11, 12]. Once the best grasp candidate is determined, a robot arm executes
the grasp open-loop which requires very precise control of the robot and a
completely static environment. This results in long computation times in the
order of a second to tens of seconds.

To overcome these challenges, we leverage the Generative Grasping
Convolutional Neural Network (GG-CNN) [13] to directly generate an antipodal
grasp pose and quality measure for every pixel in an input depth image. This
model does not rely on sampling of grasp candidates, but rather directly
generates grasp poses on a pixel-wise basis. However, the original GG-CNN
method achieves an average grasping pose accuracy of only 83.83%. To increase
the accuracy of this model, we integrate residual networks as many state-of-
the-art results have been obtained by using residual networks for deeper
architectures [14, 15]. The best grasp pose is predicted as a grasp rectangle
calculated by choosing the best grasp from multiple grasp probabilities.
Multiple grasp rectangles can be predicted for multiple objects from the
output of GGR-CNN in one shot which leads to faster computation. Through this
approach and improved data augmentation methods, we are able to increase the
accuracy from 73.0% using the GG-CNN to 92.4% on the Cornell grasping dataset
using our GGR-CNN model.

###  4.2 Grasp Representation

Similar to other CNN based grasping methods, the GG-CNN defines its grasps
perpendicular to a 2D axis with the following equation:

$g=(p,\phi,w,q)$ (1)

Here $p=(x,y,z)$ indicates the pose of the gripper as the Cartesian
coordinates $(x,y)$ of the gripper’s centre relative to an image, $\phi$
indicates the rotation of the gripper around the z-axis, $w$ is the width of
the gripper and $q$ is the grasp quality score. The final grasp is selected
based on the q parameter since it represents the success probability of the
grasp and indicates the best grasp.

We detect a grasp from an image $I=R^{n\times h\times w}$ with height $h$ and
width $w$. The final grasping pose has the output:

$\tilde{g}=(s,\tilde{\varphi},\tilde{\omega},q)$ (2)

In equation (2), $s=u,v$ corresponds to the center of grasp pose $g$ in image
coordinates grasping pose, $\tilde{\varphi}$ is the rotation angle in camera’s
frame of reference which is represented as a value in the range
$[-\frac{\pi}{2},\frac{\pi}{2}]$, $\tilde{\omega}$ is the required width in
image coordinates and $q$ is the same scalar which represents the grasp
quality score. $q$ is indicated as a value between 0 and 1 where a value
closer to 1 indicates a greater chance of grasp success.

To execute on a grasp pose $\tilde{g}$ which is obtained in the image space on
a robot, the grasping pose can be converted into the grasping pose g defined
in the world coordinate system:

$g=t_{RC}(t_{CI}(\tilde{g}))$ (3)

Where, $t_{CI}$ is a transformation that converts image space into camera’s 3D
space using the intrinsic parameters of the camera, and $t_{RC}$ converts from
the image coordinate system into the robot space in the world using the camera
pose calibration value.

We can scale this to define multiple grasps in an image using the following:

$G=(\Phi,W,Q)\in R^{3\times H\times W}$ (4)

$\Phi$, $W$, $Q$ represent the three images of grasp angle, grasp width and
grasp quality score calculated at every pixel of an image, respectively.

##  5 Neural Network Architecture: Generative Grasping Residual CNN

Figure 3: The GGR-CNN architecture includes the encoder, decoder and output of
model which can be seen from the last transpose convolutional layer.

###  5.1 Grasp Dectection Pipeline

The depth image is first cropped to a square and scaled to suit the input of
the GG-CNN. The depth image is inpainted to obtain a depth representation. The
400 × 400 processed input image is fed into the GGR-CNN. The model generates
three images as grasp angle, grasp width, and grasp quality score as the
output. From these three output images, the grasp rectangles are then
inferred.

###  5.2 Architecture

The full architecture can be seen in Figure 3. The input image is fed to the
GGR-CNN, where it is passed through 4 convolutional layers, followed by 6
residual layers and then passed through convolution transpose layers to
generate four images. These four output images include the grasp quality
score, required angle (cos 2$\Phi$ and sin 2$\Phi$ ), and the required width
of the end effector. The GGR-CNN’s first layer is 400 × 400 layer with 16
filters. The second layer is a 200 × 200 layer with 32 filters, followed by a
third 100 × 100 layer with 64 filters and a fourth layer that is 50 × 50 layer
with 128 filters. The output of the convolutional layer is then fed into 6
residual layers. Using residual layers enables us to better learn the identity
functions by using skip connections. After passing the image through these
convolutional and residual layers the size of the image is reduced to 50 × 50.
To still retain spatial features of the image, we up-sample the image by using
the convolution transpose operation. This allows us to obtain the same size of
the image at the output as the input size.

##  6 Evaluation

###  6.1 Dataset

One of the most common publicly available antipodal grasping dataset used to
benchmark results is the Cornell Grasping Dataset [16]. The Cornell Grasping
Dataset includes 885 RGB-D images of various real objects, with 5110 human-
labelled positive and 2909 negative grasps. The ground truth data has several
grasp rectangles representing grasping possibilities for each object. In
general, this is a fairly small dataset to use for training on the GGR-CNN
model. To increase the size of the dataset and the accuracy of our model, we
augment the original Cornell Grasping Dataset using random crops, rotations
and zooms to create over 9000 depth images and the associated grasp map.
Through augmentation, we can integrate over 63k grasp examples.

###  6.2 Dataset Metric for Grasping Detection

We detect a grasp using the rectangle representation. Based on this, a grasp
is successfully detected when the difference between the grasp orientation of
the predicted grasp rectangle and the ground truth rectangle is less than 30◦.
Alongside this, the IoU score is between the ground truth grasp rectangle and
the predicted grasp rectangle is more than 25%. Since the model outputs an
image-based grasp representation, the value corresponding to each pixel in the
output image is mapped to its equivalent rectangle representation.

###  6.3 Cornell Grasping Results

To evaluate the performance of our model on the Cornell Dataset, we have a
cross-validation setup. We do an image-wise (IW) split, where the training set
and the validation set do not share the same image and an object-wise (OW)
data split, where the training set and the validation set do not share any
images from the same object.

Table 1: Results Authors | Model | Accuracy (%) - IW | Accuracy (%) - OW | Speed (ms)
---|---|---|---|---
Lenz [17] | SAE, struct. reg. | 73.9 | 75.6 | 1350
Redmon [18] | AlexNet, MultiGrasp | 88.0 | 87.1 | 76
Kumra [14] | ResNet-50x2 | 89.2 | 88.9 | 103
Morrison [13] | GG-CNN | 73.0 | 69.0 | 19
Zhou [15] | FCGN, ResNet-101 | 97.7 | 96.6 | 117
Asif [19] | GraspNet | 90.2 | 90.6 | 24
Our model | GGR-CNN | 92.4 | 91.1 | 22

##  7 Application: Open Arms in Teleoperated Nursing Robot Use

In one experiment involving applications of the Open Arm project, we
collaborated on the control of the arms with another team for a teleoperated
nursing robot that we named Asha. The team developed an avatar system, through
which a teleoperator can see, hear and interact with a remote environment.

###  7.1 Teleoperation System Architecture

Figure 4: Teleoperation System Architecture

Audio and Video Feed to Operator \- The operator is equipped with a HTC Vive
Pro Eye Virtual Reality (VR) headset which receives a video stream of the
remote scene. The video stream is recorded by a Zed mini stereo camera that is
mounted on Asha’s head. The VR headset enables the operator to perceive depth
information and provides an immersive experience. A pair of headphones worn by
the operator receive the audio stream recorded by a microphone array on the
avatar side.

Facial Expression Tracking \- A facial tracker module is mounted on the VR
headset that determines the state of the lips/mouth of the operator. In
addition to this, the VR headset measures the state of the eyes of the
operator. The combination of this data is used to construct a facial
expression that is then decomposed into a sequence of motor angles for each of
the face motors on Asha.

Avatar audio and Lip Control \- The operator’s speech is recorded using a
microphone and then directly played back on the audio speakers of Asha.
Parallely, viseme information is extracted from the processed speech and used
to control the lip motors of the Asha to provide the appropriate lip sync.

Avatar Arms and Head Control \- In order to control the anthropomorphic Open
Arms of Asha, the operator wears three HTC Vive VR trackers (two of them
mounted on the back of each wrist and the third mounted on the operator’s
chest). The trackers provides an estimate of the pose of the operator’s arm. A
suitably designed Inverse Kinematics (IK) engine maps the operator’s arm pose
to a corresponding arm pose for Asha. The VR headset also serves to estimate
the head pose of the operator which can be used to control Asha’s head through
the neck motors.

Haptic Feedback and Finger Tracking \- The Bebop gloves are used to provide a
haptic response to the operator. It is possible to simulate haptic responses
ranging from a light touch to a hard press, the strength of which is
determined by the signals recorded by pressure sensors mounted on Asha’s
fingertips. The gloves are also equipped with flex sensors on each finger that
records the degree to which a finger has curled. This allows for granular
control of each individual finger and provides the ability to manipulate a
variety of objects.

Navigation \- Asha is mounted on a differential drive Unmanned Ground Vehicle
(UGV) to enable navigating the remote scene. Navigation commands are provided
by the operator through a three pedal foot interface system (Logitech G27).

###  7.2 Inverse Kinematics Solvers for Accurate Teleoperation of Arms

To be able to do useful tasks, the robot will have to track the arm movements
of a remote operator accurately and smoothly. This can be done in two ways: 1)
Replicate joint motions of the operator on the robot, 2) track the operator’s
hand movement and ensure the hand (i.e., the end-effector) of the robot
follows the motion accurately.

Replicating the joint motions of the operator on the robot is possible in this
particular case since Asha is anthromorphic in its build. Our approach is to
track the movement of the operator’s hand in 3D space by using a motion
capture system and then using an inverse kinematics solver to derive the joint
angles from the end-effector pose (position and orientation). The motion
capture system that we are using is the HTC Vive. A tracker is placed on the
back of the operator’s palm and its motion is tracked within a pre-configured
play area.

###  7.3 Analytical vs Numerical Solvers

Inverse kinematics solvers use either analytical solvers or numerical solvers
to solve for the joint angles given the end-effector pose.

Analytical solvers such as IKFast [21] generate analytical expressions to map
the joint angles to the end-effector poses. The expressions can be derived
once and then subsequently used to solve the IK problem. Using these
analytical expressions, the solutions can be obtained within a few
microseconds which is orders of magnitude faster than numerical approaches.
However, this solver requires that one joint in a 7-DoF manipulator be set as
free so that there are only 6 unknown joint angles (which equals the 6 DoF of
the end-effector).

Numerical solvers, such as Kinematics and Dynamics Library (KDL) [22] and
TRAC_IK [23] use numerical optimization techniques to solve the inverse
kinematics problem. These solvers can be orders of magnitude slower than the
analytical solvers, but they provide other benefits. Numerical solvers can
handle joint limits more gracefully by providing a best-fit solution that is
close to the desired output, while analytical solvers do not provide an output
if the constraints are not met. This is important because the range of motion
of the joints of the operator need not always match the range of motion of the
joints of the robot.

###  7.4 Evaluating Numerical Solvers: KDL vs. TRACK_IK

Two numerical solvers were evaluated for this work, namely, KDL and TRAC_IK.
The performance of the solvers was compared based on the solve rate. The solve
rate is defined as the ratio of the number of successful solutions to the
total number of test inputs given to a solver (expressed in percentage). A
forward kinematics solver was used to find end-effector poses for random input
joint angles and this end-effector pose was then provided as input to the
inverse kinematics solvers. TRAC_IK was able to find solutions for more than
99.8% of all inputs whereas KDL was only able to solve 96% of the test inputs.
Based on these results TRAC_IK was chosen as the IK solver in this work.

###  7.5 Improving Solve Rate

A few methods can be used to improve the solve rate of the IK solver. Using a
seed solution as an input to the solver helps to find the solution faster and
it also ensures that the new solution is closer to the older one thus ensuring
smooth arm motion of the robot.

Using a two-stage approach improves the solve rate as the solvers will be able
to find solutions easily when the tolerances are loose. In case the solver is
unable to find solutions in stage 2 (with tighter tolerances), then the
solution from stage 1 is sent to the robot. Although it is preferred to have
solutions as close as possible to the commanded output, sending out
approximate solutions without skipping commands ensures that the robot arms
move smoothly.

##  8 Conclusion

Open Arms is a modular, open-source robotic arm and hand toolkit with
mechanical capabilities comparable to the human hand and arm. The Open Arms
platform has proven valuable and robust as an experimental platform designed
to be mass-manufactured at a low cost. Our system includes touch sensors,
force and position feedback in all joints, and fully integrated hardware with
various cognitive AI controls, including our novel GGR-CNN model for accurate
grasping control.

We hope that ongoing experiments on our social robot platform and robust
simulation tools may produce new opportunities for experiments in embodied
cognition and help progress towards more generalized intelligence in machines.

##  9 References

[1] S. Funabashi, A. Schmitz, T. Sato, S. Somlor, and S. Sugano, “Versatile
in-hand manipulation of objects with different sizes and shapes using neural
networks,” in 2018 IEEE-RAS 18th International Conference on Humanoid Robots
(Humanoids), Nov 2018, pp. 1–9.

[2] OpenAI, “Learning dexterous in-hand manipulation,” CoRR, vol.
abs/1808.00177, 2018. [Online]. Available: http://arxiv.org/abs/1808.00177.

[3] SimLab. Allegro hand overview, 2016. [Online; accessed 25-May-2020].

[4] Scott, C. (2018). Open Bionics Introduces the Hero Arm: First-Ever
Medically Approved 3D Printed Bionic Arm. https://3dprint.com/208598/open-
bionics-hero-arm/.

[5] Parmiggiani, Alberto & Maggiali, Marco & Natale, Lorenzo & Nori, Francesco
& Schmitz, Alexander & Tsagarakis, Nikos & Santos-Victor, José & Becchi,
Francesco & Sandini, Giulio & Metta, Giorgio. (2012). The Design of the iCub
Humanoid Robot. International Journal of Humanoid Robotics.
9.10.1142/S0219843612500272.

[6] Laird, John & Kinkade, Keegan & Mohan, Shiwali & Xu, Joseph. (2012).
Cognitive Robotics Using the Soar Cognitive Architecture. AAAI Workshop -
Technical Report.

[7] Trafton, Greg & Hiatt, Laura & Harrison, Anthony & Tanborello, Frank &
Khemlani, Sangeet & Schultz, Alan. (2013). ACT-R/E: An embodied cognitive
architecture for human-robot interaction. Journal of Human-Robot Interaction.
2. 30-55. 10.5898/JHRI.2.1.Trafton.

[8] Brooks, Rodney & Breazeal, Cynthia & Marjanovic, Matthew & Scassellati,
Brian & Williamson, Matthew. (2002). The Cog Project: Building a Humanoid
Robot. Lecture Notes in Artificial Intelligence. 1562.
10.1007/3-540-48834-0_5.

[9] Lerrel Pinto and Abhinav Gupta. Supersizing selfsupervision: Learning to
grasp from 50k tries and 700 robot hours. In Proc. of the IEEE International
Conference on Robotics and Automation (ICRA), pages 3406–3413, 2016.

[10] Joseph Redmon and Anelia Angelova. Real-Time Grasp Detection Using
Convolutional Neural Networks. In Proc. of the IEEE International Conference
on Robotics and Automation (ICRA), pages 1316–1322, 2015.

[11] Ian Lenz, Honglak Lee, and Ashutosh Saxena. Deep learning for detecting
robotic grasps. The International Journal of Robotics Research (IJRR),
34(4-5):705–724, 2015.

[12] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan,
Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg. Dex-Net 2.0: Deep Learning to
Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics. In
Robotics: Science and Systems (RSS), 2017.

[13] Douglas Morrison, Peter Corke, and Jurgen Leitner. Learning robust, real-
time, reactive robotic grasping. The International Journal of Robotics
Research, page 0278364919859066, 2019.

[14] Sulabh Kumra and Christopher Kanan. Robotic grasp detection using deep
convolutional neural networks. In 2017 IEEE/RSJ International Conference on
Intelligent Robots and Systems (IROS), pages 769–776. IEEE, 2017.

[15] Xinwen Zhou, Xuguang Lan, Hanbo Zhang, Zhiqiang Tian, Yang Zhang, and
Narming Zheng. Fully convolutional grasp detection network with oriented
anchor box. In 2018 IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), pages 7223–7230. IEEE, 2018.

[16] Yun Jiang, Stephen Moseson, and Ashutosh Saxena. Efficient grasping from
rgbd images: Learning using a new rectangle representation. In Robotics and
Automation (ICRA), 2011 IEEE International Conference on, pages 3304–3311.
IEEE, 2011

[17] Ian Lenz, Honglak Lee, and Ashutosh Saxena. Deep learning for detecting
robotic grasps. The International Journal of Robotics Research,
34(4-5):705–724, 2015.

[18] Joseph Redmon and Anelia Angelova. Real-time grasp detection using
convolutional neural networks. In 2015 IEEE International Conference on
Robotics and Automation (ICRA), pages 1316–1322. IEEE, 2015.

[19] Umar Asif, Jianbin Tang, and Stefan Harrer. Graspnet: An efficient
convolutional neural network for real time grasp detection for low-powered
devices. In IJCAI, pages 4875–4882, 2018.

[20] Demonstration of Our Baseline Teleoperation System:
https://drive.google.com/file/d/1EqFkXizNBdBass1-jc26gTMhr8gtqECd/view?usp=sharing

[21] Diankov, Rosen. "Automated construction of robotic manipulation
programs." (2010).

[22] Smits, R." KDL: Kinematics and Dynamics Library",
https://www.orocos.org/kdl

[23] Beeson, Patrick & Ames, Barrett. (2015). TRAC-IK: An Open-Source Library
for Improved Solving of Generic Inverse Kinematics.
10.1109/HUMANOIDS.2015.7363472.