# BEEP! Korean Corpus of Online News Comments
for Toxic Speech Detection

Jihyung Moon*,$\dagger$,1, Won Ik Cho*,2, Junbum Lee3
Department of Industrial Engineering1,
Department of Electrical and Computer Engineering and INMC2,
Graduate School of Data Science3,
Seoul National University, Seoul
<EMAIL_ADDRESS>

###### Abstract

Toxic comments in online platforms are an unavoidable social issue under the
cloak of anonymity. Hate speech detection has been actively done for languages
such as English, German, or Italian, where manually labeled corpus has been
released. In this work, we first present 9.4K manually labeled entertainment
news comments for identifying Korean toxic speech, collected from a widely
used online news platform in Korea. The comments are annotated regarding
social bias and hate speech since both aspects are correlated. The inter-
annotator agreement Krippendorff’s alpha score is 0.492 and 0.496,
respectively. We provide benchmarks using CharCNN, BiLSTM, and BERT, where
BERT achieves the highest score on all tasks. The models generally display
better performance on bias identification, since the hate speech detection is
a more subjective issue. Additionally, when BERT is trained with bias label
for hate speech detection, the prediction score increases, implying that bias
and hate are intertwined. We make our dataset publicly available and open
competitions with the corpus and benchmarks.

00footnotetext: Both authors contributed equally to this
manuscript.00footnotetext: This work was done after the graduation.

##  1 Introduction

Online anonymity provides freedom of speech to many people and lets them speak
their opinions in public. However, anonymous speech also has a negative impact
on society and individuals (Banks, 2010). With anonymity safeguards,
individuals easily express hatred against others based on their superficial
characteristics such as gender, sexual orientation, and age (ElSherief et al.,
2018). Sometimes the hostility leaks to the well-known people who are
considered to be the representatives of targeted attributes.

Recently, Korea had suffered a series of tragic incidents of two young
celebrities that are presumed to be caused by toxic comments (Fortin, 2019;
McCurry, 2019a, b). Since the incidents, two major web portals in Korea
decided to close the comment system in their entertainment news aggregating
service (Yeo, 2019; Yim, 2020). Even though the toxic comments are now
avoidable in those platforms, the fundamental problem has not been solved yet.

To cope with the social issue, we propose the first Korean corpus annotated
for toxic speech detection. Specifically, our dataset consists of 9.4K
comments from Korean online entertainment news articles. Each comment is
annotated on two aspects, the existence of social bias and hate speech, given
that hate speech is closely related to bias (Boeckmann and Turpin-Petrosino,
2002; Waseem and Hovy, 2016; Davidson et al., 2017). Considering the context
of Korean entertainment news where public figures encounter stereotypes mostly
intertwined with gender, we weigh more on the prevalent bias. For hate speech,
our label categorization refers that of Davidson et al. (2017), namely hate,
offensive, and none.

The main contributions of this work are as follows:

  * •

We release the first Korean corpus manually annotated on two major toxic
attributes, namely bias and hate111https://github.com/kocohub/korean-hate-
speech.

  * •

We hold Kaggle competitions222www.kaggle.com/c/korean-gender-bias-
detection333www.kaggle.com/c/korean-bias-detection444www.kaggle.com/c/korean-
hate-speech-detection and provide benchmarks to boost further research
development.

  * •

We observe that in our study, hate speech detection benefits the additional
bias context.

##  2 Related Work

The construction of hate speech corpus has been explored for a limited number
of languages, such as English (Waseem and Hovy, 2016; Davidson et al., 2017;
Zampieri et al., 2019; Basile et al., 2019), Spanish (Basile et al., 2019),
Polish (Ptaszynski et al., 2019), Portuguese (Fortuna et al., 2019), and
Italian (Sanguinetti et al., 2018).

For Korean, works on abusive language have mainly focused on the qualitative
discussion of the terminology (Hong, 2016), whereas reliable and manual
annotation of the corpus has not yet been undertaken. Though profanity
termbases are currently available555https://github.com/doublems/korean-bad-
words666https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-
Otherwise-Bad-Words, term matching approach frequently makes false predictions
(e.g., neologism, polysemy, use-mention distinction), and more importantly,
not all hate speech are detectable using such terms (Zhang et al., 2018).

In addition, hate speech is situated within the context of social bias
(Boeckmann and Turpin-Petrosino, 2002). Waseem and Hovy (2016) and Davidson et
al. (2017) attended to bias in terms of hate speech, however, their interest
was mainly in texts that explicitly exhibit sexist or racist terms. In this
paper, we consider both explicit and implicit stereotypes, and scrutinize how
these are related to hate speech.

##  3 Collection

We constructed the Korean hate speech corpus using the comments from a popular
domestic entertainment news aggregation platform. Users had been able to leave
comments on each article before the recent overhaul (Yim, 2020), and we had
scrapped the comments from the most-viewed articles.

In total, we retrieved 10,403,368 comments from 23,700 articles published from
January 1, 2018 to February 29, 2020. We draw 1,580 articles using stratified
sampling and extract the top 20 comments ranked in the order of Wilson score
(Wilson, 1927) on the downvote for each article. Then, we remove duplicate
comments, single token comments (to eliminate ambiguous ones), and comments
composed with more than 100 characters (that could convey various opinions).
Finally, 10K comments are randomly selected among the rest for annotation.

We prepared other 2M comments by gathering the top 100 sorted with the same
score for all articles and removed with any overlaps regarding the above 10K
comments. This additional corpus is distributed without labels, expected to be
useful for pre-training language models on Korean online text.

Figure 1: A sample comment from the online news platform. It is composed of
six parts: written date and time, masked user id, content, the number of
replies, and the number of up/down votes (from top left to bottom right).

##  4 Annotation

The annotation was performed by 32 annotators consisting of 29 workers from a
crowdsourcing platform DeepNatural AI777https://app.deepnatural.ai/ and three
natural language processing (NLP) researchers. Every comment was provided to
three random annotators to assign the majority decision. Annotators are asked
to answer two three-choice questions for each comment:

  1. 1.

What kind of bias does the comment contain?

     * •

Gender bias, Other biases, or None

  2. 2.

Which is the adequate category for the comment in terms of hate speech?

     * •

Hate, Offensive, or None

They are allowed to skip comments which are too ambiguous to decide. Detailed
instructions are described in Appendix A. Note that this is the first
guideline of social bias and hate speech on Korean online comments.

###  4.1 Social Bias

Since hate speech is situated within the context of social bias (Boeckmann and
Turpin-Petrosino, 2002), we first identify the bias implicated in the comment.
Social bias is defined as a preconceived evaluation or prejudice towards a
person/group with certain social characteristics: gender, political
affiliation, religion, beauty, age, disability, race, or others. Although our
main interest is on gender bias, other issues are not to be underestimated.
Thus, we separate bias labels into three: whether the given text contains
gender-related bias, other biases, or none of them. Additionally, we introduce
a binary version of the corpus, which counts only the gender bias, that is
prevalent among the entertainment news comments.

The inter-annotator agreement (IAA) of the label is calculated based on
Krippendorff’s alpha (Krippendorff, 2011) that takes into account an arbitrary
number of annotators labeling any number of instances. IAA for the ternary
classes is 0.492, which means that the agreement is moderate. For the binary
case, we obtained 0.767, which implies that the identification of gender and
sexuality-related bias reaches quite a substantial agreement.

###  4.2 Hate Speech

Hate speech is difficult to be identified, especially for the comments which
are context-sensitive. Since annotators are not given additional information,
labeling would be diversified due to the difference in pragmatic intuition and
background knowledge thereof. To collect reliable hate speech annotation, we
attempt to establish a precise and clear guideline.

We consider three categories for hate speech: hate, offensive but not hate,
and none. As socially agreed definition lacks for Korean888Though a government
report is available for the Korean language (Hong, 2016), we could not reach a
fine extension to the quantitative study on online spaces., we refer to the
hate speech policies of Youtube ; Facebook ; Twitter . Drawing upon those, we
define hate speech in our study as follows:

  * •

If a comment explicitly expresses hatred against individual/group based on any
of the following attributes: sex, gender, sexual orientation, gender identity,
age, appearance, social status, religious affiliation, military service,
disease or disability, ethnicity, and national origin

  * •

If a comment severely insults or attacks individual/group; this includes
sexual harassment, humiliation, and derogation

However, note that not all the rude or aggressive comments necessarily belong
to the above definition, as argued in Davidson et al. (2017). We often see
comments that are offensive to certain individuals/groups in a qualitatively
different manner. We identify these as offensive and set the boundary as
follows:

  * •

If a comment conveys sarcasm via rhetorical expression or irony

  * •

If a comment states an opinion in an unethical, rude, coarse, or uncivilized
manner

  * •

If a comment implicitly attacks individual/group while leaving rooms to be
considered as freedom of speech

The instances that do not meet the boundaries above were categorized as none.
The IAA on the hate categories is $\alpha$ = 0.496, which implies a moderate
agreement.

##  5 Corpus

(%) | Hate | Offensive | None | Sum (Bias)
---|---|---|---|---
Gender | 10.15 | 4.58 | 0.98 | 15.71
Others | 7.48 | 8.94 | 1.74 | 18.16
None | 7.48 | 19.13 | 39.08 | 65.70
Sum (Hate) | 25.11 | 32.66 | 41.80 | 100.00

Table 1: Distribution of the annotated corpus.

#### Release

From the 10k manually annotated corpus, we discard 659 instances that are
either skipped or failed to reach an agreement. We split the final dataset
into the train (7,896), validation (471), and test set (974) and released it
on the Kaggle platform to leverage the leaderboard system. For a fair
competition, labels on the test set are not disclosed. Titles of source
articles for each comment are also provided, to help participants exploit
context information.

#### Class distribution

Table 1 depicts how the classes are composed of. The bias category
distribution in our corpus is skewed towards none, while that of hate category
is quite balanced. We also confirm that the existence of hate speech is
correlated with the existence of social bias. In other words, when a comment
incorporates a social bias, it is likely to contain hate or offensive speech.

(a) BERT predictions

(b) BERT predictions with bias label

Figure 2: Confusion matrix on the model inference of hate categories.

##  6 Benchmark Experiment

###  6.1 Models

We implemented three baseline classifiers: character-level convolutional
neural network (CharCNN) (Zhang et al., 2015), bidirectional long short-term
memory (BiLSTM) (Schuster and Paliwal, 1997), and bidirectional encoder
representations from Transformer (BERT) (Devlin et al., 2018) based model. For
BERT, we adopt KoBERT999https://github.com/SKTBrain/KoBERT, a pre-trained
module for the Korean language, and apply its tokenizer to BiLSTM as well. The
detailed configurations are provided in Appendix B, and we additionally report
the term matching approach using the aforementioned profanity terms to compare
with the benchmarks.

###  6.2 Results

F1 | Bias (binary) | Bias (ternary) | Hate
---|---|---|---
Term Matching | - | - | 0.195
CharCNN | 0.547 | 0.535 | 0.415
BiLSTM | 0.302 | 0.291 | 0.340
BERT | 0.681 | 0.633 | 0.525
BERT (+ bias) | - | - | 0.569

Table 2: F1 score of benchmarks on the test set. Note that the term matching
model checks the presence of hate or offensiveness. Therefore, in this case,
we combine hate and offensive into a single category, turning the original
ternary task into binary.

Table 2 depicts F1 score of the three baselines and the term matching model.
The results demonstrate that the models trained on our corpus have an
advantage over the term matching method. Compared with the benchmarks, BERT
achieves the best performance for all the three tasks: binary and ternary bias
identification tasks, and hate speech detection. Each model not only shows
different performances but also presents different characteristics.

#### Bias detection

When it comes to the gender-bias detection, the task benefits more on CharCNN
than BiLSTM since the bias label is highly correlated with frequent gender
terms (e.g., he, she, man, woman, …) in the dataset. It is known that CharCNN
well captures the lexical components that are present in the document.

However, owing to that nature, CharCNN sometimes yields results that are
overly influenced by the specific terms which cause false predictions. For
example, the model fails to detect bias in “What a long life for a GAY” but
guesses “I think she is the prettiest among all the celebs” to contain bias.
CharCNN overlooks GAY while giving a wrong clue due to the existence of female
pronouns, namely she in the latter.

Similar to the binary prediction task, CharCNN outperforms BiLSTM on ternary
classification. Table 3 demonstrates that BiLSTM hardly identifies gender and
other biases.

F1 | Gender | Others | None | Bias (ternary)
---|---|---|---|---
CharCNN | 0.519 | 0.259 | 0.826 | 0.535
BiLSTM | 0.055 | 0.000 | 0.819 | 0.291
BERT | 0.693 | 0.326 | 0.880 | 0.633

Table 3: Detailed results on macro-F1 of Bias (ternary)

BERT detects both biases better than the other models. From the highest score
obtained by BERT, we found that rich linguistic knowledge and semantic
information is helpful for bias recognition.

We also observed that all the three models barely perform well on others
(Table 3). To make up a system that covers the broad definition of other bias,
it would be better to predict the label as the non-gender bias. For instance,
it can be performed as a two-step prediction: the first step to distinguish
whether the comment is biased or not and the second step to determine whether
the biased comment is gender-related or not.

#### Hate speech detection

For hate speech detection, all models faced performance degradation compared
to the bias classification task, since the task is more challenging.
Nonetheless, BERT is still the most successful, and we conjecture that hate
speech detection also utilizes high-level semantic features. The significant
performance gap between term matching and BERT explains how much our approach
compensates for the false predictions mentioned in Section 2.

Provided bias label prepend to each comment as a special token, BERT exhibits
better performance. As illustrated in Figure 2, additional bias context helps
the model to distinguish offensive and none clearly. This implies our
observation on the correlation between bias and hate is empirically supported.

##  7 Conclusions

In this data paper, we provide an annotated corpus that can be practically
used for analysis and modeling on Korean toxic language, including hate speech
and social bias. In specific, we construct a corpus of a total of 9.4K
comments from online entertainment news service.

Our dataset has been made publicly accessible with baseline models. We launch
Kaggle competitions using the corpus, which may facilitate the studies on
toxic speech and ameliorate the cyber-bullying issues. We hope our initial
efforts can be supportive not only to NLP for social good, but also as a
useful resource for discerning implicit bias and hate in online languages.

## Acknowledgments

We greatly thank Hyunjoong Kim for providing financial support and Sangwoong
Yoon for giving helpful comments.

## References

  * Banks (2010) James Banks. 2010.  Regulating hate speech online.  _International Review of Law, Computers & Technology_, 24(3):233–239.
  * Basile et al. (2019) Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. 2019\.  Semeval-2019 task 5: Multilingual detection of hate speech against immigrants and women in twitter.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 54–63.
  * Boeckmann and Turpin-Petrosino (2002) Robert J Boeckmann and Carolyn Turpin-Petrosino. 2002.  Understanding the harm of hate crime.  _Journal of social issues_ , 58(2):207–225.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  In _Eleventh international aaai conference on web and social media_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * ElSherief et al. (2018) Mai ElSherief, Vivek Kulkarni, Dana Nguyen, William Yang Wang, and Elizabeth Belding. 2018.  Hate lingo: A target-based linguistic analysis of hate speech in social media.  In _Twelfth International AAAI Conference on Web and Social Media_.
  * (7) Facebook.  Facebook’s policy on hate speech.  https://www.facebook.com/communitystandards/hate_speech.  Accessed: 2020-04-19.
  * Fortin (2019) Jacey Fortin. 2019.  Sulli, south korean k-pop star and actress, is found dead.  _New York Times_.
  * Fortuna et al. (2019) Paula Fortuna, João Rocha da Silva, Leo Wanner, Sérgio Nunes, et al. 2019\.  A hierarchically-labeled portuguese hate speech dataset.  In _Proceedings of the Third Workshop on Abusive Language Online_ , pages 94–104.
  * Hong (2016) Seong Soo Hong. 2016.  _Hate speech: Survey and Regulations_.  National Human Rights Commission of the Republic of Korea.
  * Krippendorff (2011) Klaus Krippendorff. 2011.  Computing krippendorff’s alpha-reliability.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  _arXiv preprint arXiv:1808.06226_.
  * McCurry (2019a) Justin McCurry. 2019a.  K-pop singer goo hara found dead aged 28.  _The Guardian_.
  * McCurry (2019b) Justin McCurry. 2019b.  K-pop under scrutiny over ’toxic fandom’ after death of sulli.  _The Guardian_.
  * Ptaszynski et al. (2019) Michal Ptaszynski, Agata Pieciukiewicz, and Paweł Dybała. 2019.  Results of the poleval 2019 shared task 6: First dataset and open shared task for automatic cyberbullying detection in polish twitter.  _Proceedings ofthePolEval2019Workshop_ , page 89.
  * Sanguinetti et al. (2018) Manuela Sanguinetti, Fabio Poletto, Cristina Bosco, Viviana Patti, and Marco Stranisci. 2018.  An italian twitter corpus of hate speech against immigrants.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE Transactions on Signal Processing_ , 45(11):2673–2681.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * (19) Twitter.  Twitter’s policy on hate speech.  https://help.twitter.com/en/rules-and-policies/hateful-conduct-policy.  Accessed: 2020-04-19.
  * Waseem and Hovy (2016) Zeerak Waseem and Dirk Hovy. 2016.  Hateful symbols or hateful people? predictive features for hate speech detection on twitter.  In _Proceedings of the NAACL student research workshop_ , pages 88–93.
  * Wilson (1927) Edwin B Wilson. 1927.  Probable inference, the law of succession, and statistical inference.  _Journal of the American Statistical Association_ , 22(158):209–212.
  * Yeo (2019) Junsuk Yeo. 2019.  Kakao suspends online comments for entertainment articles after sulli’s death.  _The Korea Herald_.
  * Yim (2020) Hyunsu Yim. 2020.  Why naver is finally shutting down comments on celebrity news.  _The Korea Herald_.
  * (24) Youtube.  Youtube’s policy on hate speech.  https://support.google.com/youtube/answer/2801939?hl=en.  Accessed: 2020-04-19.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  Semeval-2019 task 6: Identifying and categorizing offensive language in social media (offenseval).  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 75–86.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in neural information processing systems_ , pages 649–657.
  * Zhang et al. (2018) Ziqi Zhang, David Robinson, and Jonathan Tepper. 2018.  Detecting hate speech on twitter using a convolution-gru based deep neural network.  In _European semantic web conference_ , pages 745–760. Springer.

##  Appendix A Annotation Guideline

###  A.1 Existence of social bias

The first property is to note which social bias is implicated in the comment.
Here, social bias means hasty guess or prejudice that ‘a person/group with a
certain social identity will display a certain characteristic or act in a
biased way’. The three labels of the question are as follows.

  1. 1.

Is there a gender-related bias, either explicit or implicit, in the text?

     * •

If the text includes bias for gender role, sexual orientation, sexual
identity, and any thoughts on gender-related acts (e.g., “Wife must be
obedient to her husband’s words”, or “Homosexual person will be prone to
disease.”)

  2. 2.

Are there any other kinds of bias in the text?

     * •

Other kinds of factors that are considered not gender-related but social bias,
including race, background, nationality, ethnic group, political stance, skin
color, religion, handicaps, age, appearance, richness, occupations, the
absence of military service experience101010Frequently observable in Korea,
where the military service is mandatory for males., etc.

  3. 3.

A comment that does not incorporate the bias

###  A.2 Amount of hate, insulting, or offense

The second property is how aggressive the comment is. Since the level of
“aggressiveness” depends on the linguistic intuition of annotators, we set the
following categorization to draw a borderline as precise as possible.

  1. 1.

Is strong hate or insulting towards the article’s target or related figures,
writers of the article or comments, etc. displayed in a comment?

     * •

In the case of insulting, it encompasses an expression that can severely harm
the social status of the recipient.

     * •

In the case of hate, it is defined as an expression that displays aggressive
stances towards individuals/groups with certain characteristics (gender role,
sexual orientation, sexual identity, any thoughts on gender-related acts,
race, background, nationality, ethnic group, political stance, skin color,
religion, handicaps, age, appearance, richness, occupations, the absence of
military service experience, etc.).

     * •

Additionally, it can include sexual harassment, notification of offensive
rumors or facts, and coined terms for bad purposes or in bad use, etc.

     * •

Just an existence of bad words in the document does not always fall into this
category.

  2. 2.

Although a comment is not as much hateful or insulting as the above, does it
make the target or the reader feel offended?

     * •

It may contain rude or aggressive contents, such as bad words, though not to
the extent of hate or insult.

     * •

It can emit sarcasm through rhetorical questions or irony.

     * •

It may encompass an unethical expression (e.g., jokes or irrelevant questions
regarding the figures who passed away).

     * •

A comment conveying unidentified rumors can belong to this category.

  3. 3.

A comment that does not incorporate any hatred or insulting

##  Appendix B Model Configuration

Note that each model’s configuration is the same for all tasks except for the
last layer.

###  B.1 CharCNN

For character-level CNN, no specific tokenization was utilized. The sequence
of Hangul characters was fed into the model at a maximum length of 150. The
total number of characters was 1,685, including ‘[UNK]’ and ‘[PAD]’ token, and
the embedding size was set to 300. 10 kernels were used, each with the size of
[3,4,5]. At the final pooling layer, we used a fully connected network (FCN)
of size 1,140, with a 0.5 dropout rate Srivastava et al. (2014). The training
was done for 6 epochs.

###  B.2 BiLSTM

For bidirectional LSTM, we had a vocab size of 4,322, with a maximum length of
256. We used BERT SentencePiece tokenizer Kudo and Richardson (2018). The
width of the hidden layers was 512 (=$256\times 2$), with four stacked layers.
The dropout rate was set to 0.3. An FCN of size 1,024 was appended to the
BiLSTM output to yield the final softmax layer. We trained the model for 15
epochs.

###  B.3 BERT

For BERT, a built-in SentencePiece tokenizer of KoBERT was adopted, which was
also used for BiLSTM. We set a maximum length at 256 and ran the model for 10
epochs.


* [67] Bennett, C.H. 1982 The thermodynamics of computation. Int. J. Theor. Phys. 21, 905–940.
  * [68] Fuchs, C. A.; Schack, R. 2013 Quantum-Bayesian coherence. Rev. Mod. Phys. 85, 1693–1715.
  * [69] Mermin, N. D. 2019 Making better sense of quantum mechanics. Rep. Preg. Phys. 82, 012002.
  * [70] Rovelli, C. 1996 Relational quantum mechanics. Int. J. Mod. Phys. 35, 1637–1678.
  * [71] Pearl, J. 1988 Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. San Mateo CA: Morgan Kaufmann.
  * [72] Clark A. 2017 How to knit your own Markov blanket: Resisting the second law with metamorphic minds. In (T. Wetzinger and W. Wiese, eds.) _Philosophy and Predictive Processing_ 3, 19pp. Frankfurt am Mainz Mind Group.
  * [73] Fields, C.; Marcianò, A. 2020 Markov blankets are general physical interaction surfaces. Phys. Life Rev. 33, 109–111.
  * [74] Bell, J. S. 1964 On the Einstein–Podolsky–Rosen paradox. Physics 1, 195–200.
  * [75] Emary, C.; Lambert, N.; Nori, F. 2014 Leggett–Garg inequalities. Rep. Prog. Phys. 77, 039501.
  * [76] Kochen, S.; Specker, E.P. 1967 The problem of hidden variables in quantum mechanics. J. Math. Mech. 17, 59–87.
  * [77] Mermin, D. 1993 Hidden variables and the two theorems of John Bell. Rev. Mod. Phys. 65, 803–815.
  * [78] Dzharfarov, E.N.; Kon, M. 2018 On universality of classical probability with contextually labeled random variables. J. Math. Psych. 85, 17–24.
  * [79] Horsman C; Stepney S; Wagner RC; Kendon V. 2014 When does a physical system compute? Proc. R. Soc. A 470, 20140182.
  * [80] Smith, J. E.; Nair, R. 2005 The architecture of virtual machines. IEEE Comp. 38(5), 32–38.
  * [81] Barr, M. 1979 *-Autonomous Categories, with an Appendix by Po Hsiang Chu; Lecture Notes in Mathematics 752; Springer: Berlin, Germany.
  * [82] Pratt, V. 1999 Chu spaces. In School on Category Theory and Applications (Coimbra 1999); Volume 21 of Textos Mat. Sér. B; University of Coimbra: Coimbra, Portugal, pp. 39–100.
  * [83] Pratt, V. 1999 Chu spaces from the representational viewpoint. Ann. Pure Appl. Log. 96, 319–333.
  * [84] Baars, B. J.;Franklin, S. 2003 How conscious experience and working memory interact. Trends in Cognitive Science 7, 166–172.
  * [85] Dehaene, S.; Naccache, L. 2001 Towards a cognitive neuroscience of consciousness: Basic evidence and a workspace framework. Cognition 79, 1–37.
  * [86] Mashour, G. A.;Roelfsema, P.; Changeux, J.-P.; Dehaene, S. 2020 Conscious processing and the Global Neuronal Workspace hypothesis. Neuron 105, 776–798.
  * [87] Allwein, G.; Moskowitz, I.S.; Chang, L.-W. 2004 A New Framework for Shannon Information Theory. Technical Report A801024 Naval Research Laboratory, Washington, DC, USA, 17p.
  * [88] Barwise, J. 1997 Information and impossibilities. Notre Dame Journal of Formal Logic 38(4), 488–515.
  * [89] Cherniak, E. 1991 Bayesian networks without tears. AI Magazine 12(4), 50–63.
  * [90] Prakash, C.; Fields, C.; Hoffman, D. D.; Prentner, R.; Singh, M. 2020 Fact, fiction, and fitness. Entropy 22, 514.
  * [91] Fields C, Friston K, Glazebrook JF, Levin M 2021 A free energy principle for generic quantum systems. In review (available as arxiv:2112.15242 [quant-ph]).
  * [92] Lyon, P. 2015 The cognitive cell: Bacterial behavior reconsidered. Front. Microbiol. 6, 264.
  * [93] Micali, G.; Endres, R.G. 2016 Bacterial chemotaxis: Information processing, thermodynamics, and behavior. Curr. Opin. Microbiol. 30, 8–15.
  * [94] Levin, M. 2012 Morphogenetic fields in embryogenesis, regeneration, and cancer: Non-local control of complex patterning. Biosystems 109, 243–261.
  * [95] Levin, M.; Pezzulo, G.; Finkelstein, J. M. 2017 Endogenous bioelectric signaling networks: Exploiting voltage gradients for control of growth and form. Annu. Rev. Biomed. Eng. 19, 353–387.
  * [96] Srivastava, P.; Kane, A.; Harrison, C.; Levin, M. 2021 A meta-analysis of bioelectric data in cancer, embryogenesis, and regeneration. Bioelectricity 3, 42–67.
  * [97] Wuichet, K.; Cantwell, B. J.; Zhulin, I. B. 2010 Evolution and phyletic distribution of two-component signal transduction systems. Curr. Opin. Microbiol. 13, 219–225.
  * [98] Basieva, I; Khrennikov, A; Ohya, M; Yamato, O. 2011 Quantum-like interference effect in gene expression: glucose-lactose destructive interference. Syst. Synth. Biol. 5, 59–68.
  * [99] Inada, T; Kimata, K; Aiba, H. 1996 Mechanism responsible for glucose-lactose diauxie in Eschericha coli challenge to the cAMP model. Genes Cell 1, 293–301.
  * [100] Loh, K. M.; van Amerongen, R.; Nusse, R. 2016 Generating cellular diversity and spatial form: Wnt signaling and the evolution of multicellular animals. Dev. Cell. 38, 643–655.
  * [101] Kolch, W. 2005 Coordinating ERK/MAPK signaling through scaffolds and inhibitors. Nat. Rev. Mol. Cell Biol. 6, 827–838.
  * [102] Schwanbeck, R.; Martini, S.; Bernoth, K.; Just, U. 2011 The notch signaling pathway: molecular basis of cell context dependency. Eur Cell Biol 90(6-7), 572–581.
  * [103] Guo, X.; Wang, X.-F. 2009 Signaling cross-talk between TGF-$\beta$/BMP and other pathways. Cell Res. 19, 71–88.
  * [104] Hunter, T. 2000 Signaling and beyond. Cell 100, 113–127.
  * [105] Adamska, M. 2015 Developmental signalling and emergence of animal multicellularity. In: Evolutionary Transitions to Multicellular Life, edited by Ruiz-Trillo I, Nedelcu AM. Dordrecht: Springer, pp. 425–450
  * [106] Fields, C.; Levin, M. 2018 Multiscale memory and bioelectric error correction in the cytoplasm-cytoskeleton-membrane system. WIRES Syst. Biol. Med. 10, e1410.
  * [107] Hipólito, I; Ramstead, M. J. D.; Convertino, L.; Bhat. A.; Friston, K.; Parr, T. 2021 Markov blankets in the brain. Neuroscience and Biobehavioral Reviews 125, 88–97.
  * [108] Palacios, E. R.; Razi, A.; Parr, T.; Kirchoff, M.; Friston, K. 2020 On Markov blankets and hierarchical self-organization. J. Theoretical Biology 486, 110089.
  * [109] Peters, A.; McEwen, B. S.; Friston, K. 2017 Uncertainty and stress: why it causes diseases and how it can be mastered by the brain. Progress in Neurobiology 156, 164–188.
  * [110] Spruston, N. 2008 Pyramidal neurons: Dendritic structure and synaptic integration. Nat. Rev. Neurosci. 9, 206–221.
  * [111] Rasia-Filho, A. A.; Guerra, K. T. K.; Vásquez, C. E.; Dall’Oglio, A.; Reberger, R.; Jung, C. R.; Calcagnotto, M. E. 2021 The subcortical-allocortical-neocortical continuum for the emergence and morphological heterogeneity of pyramidal neurons in the human brain. Front. Synapt. Neurosci. 13, 616607.
  * [112] Lloyd, S. 2000 Ultimate physical limits to computation. Nature 406, 1047–1054.
  * [113] Attwell, D.; Laughlin S. B. 2001 An energy budget for signaling in the grey matter of the brain. J. Cereb. Blood Flow Metab. 21, 1133–1145
  * [114] Sengupta, B.; Stemmler, M. B.; Friston, K. J. 2013 Information and efficiency in the nervous system: A synthesis. PLoS Comput Biol 9(7), e1003157.
  * [115] Georgiev, D; Kolev, S; Cohen, E; Glazebrook, JF. 2020 Computational capacity of pyramidal neurons in the cerebral cortex. Brain Research 1748, 147069.
  * [116] Deutsch, D. 2002 The structure of the multiverse. Proc. R. Soc. A 458, 2911–2923.
  * [117] Atiyah, M. 1988 Topological quantum field theory. Pub. Math. IHÈS 68, 175–186.
  * [118] Marcianò, A.; Chen, D.; Fabrocini, F.; Fields, C.; Greco, E.; Gresnigt, N.; Jinklub, K.; Lulli, M., Terzidis, K.; Zappala, E. 2021 Deep neural networks as the semi-classical limit of quantum neural networks. Preprint arxiv:2007.00142v2 [cond-mat.diss-nn]. https://arxiv.org/abs/2007.00142
  * [119] Eyal, G.; Verhoog, M. B.; Testa-Silva, G.; Deitcher, Y.; Benavides-Piccione, R.; DeFelipe, J.; de Kock, C. P.J.; Mansvelder, H. D.; Segev, I. 2018 Human cortical pyramidal neurons:from spines to spikes via models. Front. Cellular Neurosci. 12, 181.
  * [120] Galloni, A. R.; Laffere, A.; Rancz, E. 2020 Apical length governs computational diversity of layer 5 pyramidal neurons. eLife 9, e55761.
  * [121] Major, G; Larkum, M. E.; Schiller, J. 2013 Active properties of neocortical pyramidal neuron dendrites. Annu. Rev. Neurosci. 36, 1–24.
  * [122] Shipp, S. 2007 Structure and function of the cerebral cortex. Curr. Biol. 17, R443–R449.
  * [123] Shipp, S. 2016 Neural elements for predictive coding. Front. Psychol. 7, 1792.
  * [124] Swanson, L. W.; Lichtman, J. W. 2016 From Cajal to connectome and beyond. Annu. Rev. Neurosci. 39, 197–216.
  * [125] Vélez-Fort, M.; Rousseau, C. V.; Niedworok, C. J.; Wickersham, I. R.; Rancz, E. A.; Brown, A. P. Y.; Strom, M.; Margrie, T. W. 2014 The stimulus selectivity and connectivity of Layer Six principal cells reveals cortical microcircuits underlying visual processing. Neuron 83, 1431–1443.
  * [126] De Nardo, L. A.; Berns, D. C.; DeLoach, K.; Luo, L. 2015 Connectivity of mouse somatosensory and prefrontal cortex examined with trans-synaptic tracing. Nat. Neurosci. 18, 1687–1697.
  * [127] Harris, K. D.; Shepherd, G. M. G. 2015 The neocortical circuit: Themes and variations. Nat. Neurosci. 18, 170–181.
  * [128] Mennerick S, Zorumsky CF. 2000 Neural activity and survival in the developing nervous system. Mol. Neurobiol. 22, 41–54.
  * [129] Faust TE, Gunner G, Schafer DP. 2021 Mechanisms governing activity-dependent synaptic pruning in the developing mammalian CNS. Nature Rev. Neurosci. 22, 657–673.
  * [130] Deitcher Y.; Eyal G.; Kanari L.; et al. 2007 Comprehensive morpho-electronic analysis shows 2 distinct classes of L2 and L3 pyramidal neurons in human temporal cortex. Cereb. Cortex 27, 5398–5414.
  * [131] Mohan, H.; Verhoog, M. B.; Doreswamy, K. K.; et al. 2015 Dendritic and axonal architecture of individual pyramidal neurons across layers of adult human neocortex. Cereb. Cortex 25, 4839–4853.
  * [132] Beaulieu-Laroche, L; Toloza, E. H. S.; van der Goes, M. S. et al. 2018 Enhanced dendritic compartmentalization in human cortical neurons. Cell 175, 643–651.
  * [133] Nielsen, M. A.; Chuang, I. L. 2000 Quantum Computation and Quantum Information. New York, Cambridge University Press.
  * [134] Luo, L. 2021 Architectures of neuronal circuits. Science 373, eabg7285.
  * [135] Johansson, C.; Lansner, A. 2007 Towards cortex sized artificial neural systems. Neural Networks 20, 48–61.
  * [136] Chen, R.; Cohen, L. G.; Hallett, M. 2002 Nervous system reorganization following injury. Neuroscience 111, 761–773.
  * [137] Demekas, D.; Parr, T.; Friston, K. J. 2020 An ivestigation of the free energy principle for emotional recognition. Frontiers in Computational Neuroscience 14, 30.
  * [138] Levin, M. 2019 The computational boundary of a “self”: Developmental bioelectricity drives multicellularity and scale-free cognition. Front. Psychol. 10, 1688.
  * [139] Levin, M. 2014 Endogenous bioelectrical networks store non-genetic patterning information during development and regeneration. J. Physiol. 592, 2295–2305.
  * [140] Levin, M. 2021 Bioelectric signaling: Reprogrammable circuits underlying embryogenesis, regeneration, and cancer. Cell 184, 1971–1989.
  * [141] Bates, E. 2015 Ion channels in development and cancer. Annu. Rev. Cell. Devel. Biol. 31, 231–247.
  * [142] Harris, M. P. 2021 Bioelectric signaling as a unique regulator of development and regeneration. Development 148, dev180794.
  * [143] Mathews, J.; Levin, M. 2018 The body electric 2.0: Recent advances in developmental bioelectricity for regenerative and synthetic bioengineering. Curr. Opin. Biotechnol. 52, 134–144.
  * [144] Levin, M. 2021 Life, death, and self: Fundamental questions of primitive cognition viewed through the lens of body plasticity and synthetic organisms. Biochem. Biophys. Res. Commun. 564, 114–133.
  * [145] Vandenberg, L. N.; Adams, D. S.; Levin, M. 2012 Normalized shape and location of perturbed craniofacial structures in the Xenopus tadpole reveal an innate ability to achieve correct morphology. Devel. Dyn. 241, 863–878.
  * [146] Pinet, K.; McLaughlin, K. A. 2019 Mechanisms of physiological tissue remodeling in animals: Manipulating tissue, organ, and organism morphology. Devel. Biol 451, 134–145.
  * [147] Pinet, K.; Deolankar, M.; Leung, B.; McLaughlin, K. A. 2019 Adaptive correction of craniofacial defects in pre-metamorphic Xenopus laevis tadpoles involves thyroid hormone-independent tissue remodeling. Development 146, dev175893.
  * [148] Emmons-Bell, M.; Durant, F.; Hammelman, J.; Bessonov, M.; Volpert, V.; Morokuma, J.; Pinet, K.; Adams, D. S.; Pietak, A.; Lobo, D.; Levin, M. 2015 Gap junctional blockade stochastically induces different species-specific head anatomies in genetically wild-type Girardia dorotocephala flatworms. Int. J. Mol. Sci. 16, 27865–27896.
  * [149] Kriegman, S.; Blackiston, D.; Levin, M.; Bongard, J. 2020 A scalable pipeline for designing reconfigurable organisms. Proc. Natl. Acad. Sci. USA 117, 1853–1859.
  * [150] Blackiston, D.; Lederer, E.; Kriegman, S.; Garnier, S.; Bongard, J.; Levin, M. 2021 A cellular platform for the development of synthetic living machines. Sci. Robot. 6, eabf1571.
  * [151] Pai, V. P.; Pietak, A.; Willocq, V.; Ye, B.; Shi, N.-Q.; Levin, M. 2018 HCN2 Rescues brain defects by enforcing endogenous voltage pre-patterns. Nat. Comms. 9, 998.
  * [152] Pai, V. P.; Lemire, J. M.; Paré, J.-F.; Lin, G.; Chen,Y.; Levin, M. 2015 Endogenous gradients of resting potential instructively pattern embryonic neural tissue via Notch signaling and regulation of proliferation. J. Neurosci 35, 4366–4385.
  * [153] Vandenberg, L. N.; Morrie, R. D.; Adams, D. S. 2011 V-ATPase-dependent ectodermal voltage and pH regionalization are required for craniofacial morphogenesis. Devel. Dyn. 240, 1889–1904.
  * [154] Pezzulo, G.; Lapalme, J.; Durant, F.; Levin, M. 2021 Bistability of somatic pattern memories: Stochastic outcomes in bioelectric circuits underlying regeneration. Phil. Proc. R. Soc. B 376, 20190765.
  * [155] Durant F.; Morokuma, J.; Fields, C.; Williams, K.; Adams, D. S.; Levin, M. 2017 Long-term, stochastic editing of regenerative anatomy via targeting endogenous bioelectric gradients. Biophys. J. 112, 2231–2243.
  * [156] Oviedo, N. J.; Morokuma, J.; Walentek, P.; Kema, I. P.; Gu, M. B.; Ahn, J.-M.; Hwang, J. S.; Gojobori, T.; Levin, M. 2010 Long-range neural and gap junction protein-mediated cues control polarity during planarian regeneration. Devel. Biol. 339, 188–199.
  * [157] Durant, F.; Bischof, J.; Fields, C.; Morokuma, J.; LaPalme, J.; Hoi, A.; Levin, M. 2019 The role of early bioelectric signals in the regeneration of planarian anterior/posterior polarity. Biophys. J. 116, 948–961.
  * [158] Koshland, D. E. 1983 The bacterium as a model neuron. Trends Neurosci 6, 133–137.
  * [159] Prindle, A.; Liu, J.; Asally, M.; Ly, S.;Garcia-Ojalvo, J.; Süel, G. M. 2015 Ion channels enable electrical communication in bacterial communities. Nature 527, 59–63.
  * [160] Pezzulo, G.; Levin, M. 2015 Re-membering the body: Applications of computational neuroscience to the top-down control of regeneration of limbs and other complex organs. Integr. Biol. (Cambridge) 7, 1487–1517.
  * [161] Watson, R. A.; Buckley, C. L.; Mills, R.; Davies, A. 2010 In Artificial Life Conference XII. (Odense, Denmark, 2010), pp. 194–201.
  * [162] Biswas, S.; Manicka, S.; Hoel, E.; Levin, M. 2021 Gene regulatory networks exhibit several kinds of memory: Quantification of memory in biological and random transcriptional networks. iScience 24, 102131.
  * [163] Emmons-Bell, M., Durant, F.; Tung, A.; Pietak, A.; Miller, K.; Kane, A.; Martyniuk, C. J.; Davidian, D.; Morokuma, J.; Levin, M. 2019 Regenerative adaptation to electrochemical perturbation in planaria: A molecular analysis of physiological plasticity. iScience 22, 147–165.
  * [164] Jacob, E. B.; Aharonov, Y.; Shapira, Y. 2004 Bacteria harnessing complexity. Biofilms 1, 239–263.
  * [165] Goel, P.; Mehta, A. 2013 Learning theories reveal loss of pancreatic electrical connectivity in diabetes as an adaptive response. PLoS One 8, e70366.
  * [166] Zoghi, M. 2004 Cardiac memory: Do the heart and the brain remember the same? J. Interv. Card. Electrophysiol. 11, 177–182.
  * [167] Chakravarthy, S. V.; Ghosh, J. 1997 On Hebbian-like adaptation in heart muscle: A proposal for ‘cardiac memory’. Biol. Cybern. 76, 207–215.


# A Multimodal LSTM for Predicting
Listener Empathic Responses Over Time

Zhi-Xuan Tan†, Arushi Goel†, Thanh-Son Nguyen†, and Desmond C. Ong
A*STAR Artificial Intelligence Initiative, Agency for Science, Technology and
Research, Singapore † Z.X.T, A.G. and T-S.N. contributed equally to this
work.This work was supported by the A*STAR Human-Centric Artificial
Intelligence Programme (SERC SSF Project No. A1718g0048).

###### Abstract

People naturally understand the emotions of—and often also empathize
with—those around them. In this paper, we predict the emotional valence of an
empathic listener over time as they listen to a speaker narrating a life
story. We use the dataset provided by the OMG-Empathy Prediction Challenge, a
workshop held in conjunction with IEEE FG 2019. We present a multimodal LSTM
model with feature-level fusion and local attention that predicts empathic
responses from audio, text, and visual features. Our best-performing model,
which used only the audio and text features, achieved a concordance
correlation coefficient (CCC) of .29 and .32 on the Validation set for the
Generalized and Personalized track respectively, and achieved a CCC of .14 and
.14 on the held-out Test set. We discuss the difficulties faced and the
lessons learnt tackling this challenge.

##  I Introduction

People display emotions all the time in daily life, but more than that, are
also sensitive to the emotions of those around them [7, 8], and often “feel”
what other people are feeling. Watching someone cry with happiness, even
through a television or movie screen, would make most of us feel emotional as
well [17, 15]. Such _empathic_ responses are a crucial component of our daily
functioning, but have been relatively under-studied in the affective computing
literature.

Research in building artificial socially-intelligent agents have mainly
focused on the person experiencing (and expressing) emotions. Consider Andrew,
who is relating an emotional life story to Lisa. Andrew is smiling broadly,
tripping over his words, hands gesturing wildly, as he describes his travels
in an exotic country. Much of the work in emotion recognition aims to predict
Andrew’s emotions from one or more modalities: what he is saying, how he is
saying it, what his facial expressions convey.

By contrast, the focus in this paper, based on the OMG-Empathy Prediction
Challenge 2019, is not on recognizing emotions in the person expressing them
(Andrew). Rather, we aim to recognize the empathic responses of others (Lisa)
who are listening to—and _empathizing with_ —the emoter. Lisa is engaged with
Andrew’s story, and can almost feel the same excitement that he has while
recounting the story. Here, we present several models that aim to learn Lisa’s
empathic responses from both Andrew and Lisa’s behaviour and the content of
Andrew’s story, with a challenge dataset of several Actors narrating stories
to several Listeners (see Methods).

There has been almost no work in affective computing on predicting empathic
responses, so we briefly review work in multimodal emotion recognition,
especially using multimodal deep learning [12, 16]. We drew inspiration from
several papers tackling similar challenges. First, Wöllmer et al. [14]
proposed a system based on Long Short Term Memory (LSTM) long-range temporal
context modeling on the Audio/Visual Emotion Challenge (AVEC) dataset. Second,
a hybrid network with visual attention [4] was the winning submission for
group-level emotion recognition in the Emotion Recognition in the Wild
(EmotiW) Challenge 2018. Third, the OMG-Emotion Behaviour Challenge 2018 [1]
produced several models, such as a bidirectional LSTM [10] and utterance-level
sentiment regression [2], that predict a speaker’s valence from short one-
minute videos.

In this paper, we propose a multimodal LSTM model with a local attention layer
for predicting the valence ratings of the listener in the OMG-Empathy Dataset.
Our models are available as a public GitHub
repository111https://github.com/desmond-ong/cheem-omg-empathy.

##  II Methods

###  II-A Dataset

We use the OMG-Empathy Dataset released with this challenge, which consists of
semi-scripted interactions between pairs of individuals, one of whom is an
actor, and the other, a naïve participant (or “listener”). The actor recounts
a fictional autobiographical story to the listener. In total, there were 4
actors, each of whom told 2 stories, such as “talking about a childhood
friend”, or “I had a bad flight experience”. Ten participants were recruited
as listeners, and each listener listened to every story. This gives a total of
80 videos, each lasting on average 5 minutes 12 seconds. The dataset was split
into Training (4 stories, 40 videos), Validation (1 story, 10 videos), and
Test (3 stories, 30 videos) sets.

After each session, listeners were shown a video of their interactions, and
were asked to rate using a joystick how they themselves felt as the actor was
telling the story (i.e., their empathic responses). These annotations were
continuous in time, and ranged from -1 (negative) to 1 (positive).

###  II-B Data Augmentation

Given the limited amount of training data, we opted to split each $5$-min
training video into five $1$-min segments, augmenting the number of training
sequences from $40$ to $200$, and allowing us to better train our models using
batched gradient descent. While this breaks the standard assumption that
training examples are conditionally independent given the generating process,
we found empirically that this was made up for by the increase in training
data. The effectiveness of this approach suggests that temporal dependencies
over the $1$-min time-scale are not crucial for predicting empathy.

###  II-C Feature Extraction

####  II-C1 Audio Features

We used openSMILE v2.3.0 [3] with the accompaning emobase configuration file
to extract 990 low-level acoustic features for every 1-second window.

####  II-C2 Text Features

We used YouTube’s automatic subtitling to extract transcripts along with start
and end timings of individual “utterances”222An “utterance” is not necessarily
a complete sentence.. We used 300-dimensional GloVe word embeddings [11] to
obtain features per utterance by averaging across the embeddings of its words.
For synchronization with other modalities, the utterance feature was repeated
for each 1-second window within the timespan of the utterance, with the most
recent utterance chosen if a window overlapped with multiple utterances.

####  II-C3 Visual Features

We extracted the fully-connected 4096-dimensional feature embeddings from the
pre-trained VGG Face CNN descriptor models [9]. The features are extracted for
every frame from the listener’s face and then averaged over a period of
1-second (25 frames). We also extracted the actor’s facial features, but found
that adding actor features to our models usually hurt their performance.

###  II-D Long Short Term Memory (LSTM) Models

Our LSTM model architecture is shown in Fig. 1. We first extract multimodal
features for each 1-second window, compress them to a 128-dimensional
embedding for each modality and concatenate the feature vectors (feature-level
fusion). These feature vectors are then fed into a LSTM recurrent neural
network [5], allowing information from previous time points to influence the
hidden layer representation at time $t$. During training, we ran the LSTM over
the 1-min video segments described in the Data Augmentation section, while at
validation and test time, we used the entire video.

Figure 1: Model architecture. We extract multimodal features $x_{t}$ for every
1-second window, which is input into an LSTM. (We omit the inner LSTM
operations for brevity). We introduce a local attention layer that weights
contributions from $h_{t}$ up to $h_{t-2}$ using attention weights
$a_{t,0},\ldots,a_{t,2}$. The final output layer is a regression to predict
the listener’s valence rating.

Next, we introduced a local attention layer [6] which weights contributions
from a small local window $[t-(W-1),t]$ around the current time-point $t$. The
attention layer outputs a context vector $c_{t}$ by computing a weighted sum
over the hidden states of the LSTM within the window:

$c_{t}=\sum_{i=0}^{W-1}a_{t,i}\cdot h_{t-i}$ (1)

Here, $a_{t,\cdot}$ is a vector of $W$ attention weights, $h_{t}$ is the
hidden state of the LSTM, and $W=3$ seconds is the window size, which was
selected through cross-validation. The attention weights are computed as a
function $a_{t,\cdot}=f_{\phi}(x_{t})$ of the multimodal inputs $x_{t}$, where
$f_{\phi}(\cdot)$ is parameterized by the weights $\phi$ of a multi-layer
perceptron with a 128-dimensional hidden layer. The model is thus able to
learn which hidden states over the recent past matter most when predicting the
valence. Finally, the 512-dimensional context vector $c_{t}$ is fed into a
128-dimensional layer, then into a linear regression that predicts the
listener’s valence rating.

We experimented with multiple models using the same basic architecture,
varying the input modalities and other design choices. For our text-only model
(T), we opted to run our LSTM over sequences of utterances instead of 1-second
windows, and found that predicting the change in the valence, rather than the
valence itself, led to less over-fitting and better performance. For the text-
only (T) and vision-only (V) models, we also omitted the attention layer,
finding that it did not improve performance. In total, we evaluated 7
different LSTM models: unimodal (A: audio, T: text, V: visual), bimodal (AT,
AV, TV), and trimodal (ATV).

##  III Results

To evaluate our results, we use the Concordance Correlation Coefficient (CCC)
between the model’s predictions and participants’ self-reported empathic
responses. The CCC is similar to the Pearson correlation coefficient $\rho$,
in that it measures the agreement between two vectors, where $+1$ corresponds
to perfect agreement. Unlike $\rho$, the CCC also takes into account the
difference between the means of both vectors, penalizing bias in the model’s
predictions.

###  III-A Cross-Validation Results

In our initial experiments, we realized that the stimuli were very
heterogeneous—the same model can perform very well on some stories, and very
poorly on other stories. Thus, we also prioritized finding models (modality-
combinations and hyperparameters) that generalize well across different
stories.

To do this, we performed leave-one-out cross-validation across the provided
stories, training on four of the stories and computing the CCC for the held-
out story. We then computed the mean and standard deviation of the CCC across
stories, reported in Table I, to evaluate the models’ robustness to variations
in the training data. The first three models listed are unimodal LSTMs, the
next four are multimodal LSTMs that employ early feature fusion (as in Fig.
1).

TABLE I: CCC values cross-validated across stories Model | CCC
---|---
Validation Story | Mean $\pm$ SD
1 | 2 | 4 | 5 | 8
A | .22 | -.10 | .14 | .15 | .00 | .08 $\pm$ .12
T | .19 | .00 | .27 | .13 | .12 |  .14 $\pm$ .10
V | .04 | .08 | -.14 | .10 | .19 | .05 $\pm$ .12
AT | .26 | -.04 | .11 | .12 | .05 | .10 $\pm$ .10
AV | .18 | -.22 | .16 | .15 | .01 | .05 $\pm$ .15
TV | .02 | -.04 | .01 | .07 | .16 | .04 $\pm$ .07
ATV | .19 | -.08 | .15 | .18 | .10 | .11 $\pm$ .10

Interestingly, most of our models performed best when validated on the
original Validation set (Story 1), but very poorly when validated on Story 2,
which often resulted in negative CCC values. As a result, the mean cross-
validated CCC was substantially lower than the CCC for the best story, and the
standard deviation tends to be almost as large as, if not larger than, the
mean. While disappointing, given the consistency of this trend across models,
one hypothesis is that Story 2 is an outlier in comparison to the remaining
four stories. The only model that performed reasonably on Story 2 was the
visual-only model (V), but unfortunately it performed poorly on Stories 1 and
4 instead.

Another notable trend was that increasing the number of modalities did not
reliably improve performance across stories. The audio-text model (AT)
performed best on the original Validation set, but performed worse on cross-
validation than the text-only model (T). The audio-text-visual model (ATV)
performed better on cross-validation than AT, but worse on the original
Validation set.

###  III-B Generalized Track

Informed by our cross-validation results, our three submissions for the
Generalized Empathy track are as follows:

[G1]: | AT | Best model on Validation Set (Story 1)
---|---|---
[G2]: | T | Best model on cross-validation
[G3]: | ATV | Best multimodal model on cross-validation

We selected models using these criteria based on different assumptions about
the representativeness of the training set. For [G1], we chose the best-
performing model (AT) on the original Validation set (Story 1). For [G2], we
chose the model with the best cross-validation performance; this happened to
be the text-only model (T). Finally, for [G3], we chose the best-performing
multimodal model (ATV) to see if multimodality would improve generalization.
We note that even though the modality combinations in [G2] and [G3] were
selected via leave-one-out cross-validation, the submission models are still
only trained on the Training set.

The validation CCC of our submissions can be compared against the challenge
baseline in the second column of Table II. We also report the standard
deviation across videos as an error estimate. For submission purposes, we used
early-stopping to determine the best epoch for each model on the original
validation set (Story 1). This explains the difference from the Story 1 column
in Table I, which shows the results for the epoch that gives the best cross-
validated CCC. It can be seen that our models substantially outperform the
baseline provided by the organizers, with the audio-text model [G1] achieving
a CCC of $.29$ with a standard deviation of .12 (calculated across the videos
in the Validation set).

###  III-C Personalized Track

To address the Personalized Track of the challenge, which requires the model
to predict the empathic responses of specific listeners, we took our
Generalized models [G1–G3] as pre-trained seed models, and fine-tuned them for
each listener to produce models [P1–P3]. Fine-tuning meant training only on
videos from one specific listener for an additional 250 epochs, with early-
stopping to select the best epoch. This fine-tuning yielded ten personalized
models, one for each of ten listeners. The CCC values on the Validation set
(averaged across listeners) are displayed in the fourth column of Table II. As
expected, fine-tuning to each listener improved upon the generalized models,
with the ATV model improving from a CCC of $.23$ to $.28$. Similar to the
Generalized track, our Personalized models also substantially outperformed the
provided baseline on the Validation set.

TABLE II: CCC values for Submitted Models (Mean $\pm$ Std. Dev.) Model | CCC
---|---
Generalized | Personalized
Valid. | Test | Valid. | Test
Baseline | .11 | – | .09 | –
[G1/P1] AT |  .29 $\pm$ .12 |  .14 $\pm$ .19 |  .32 $\pm$ .11 |  .14 $\pm$ .20
[G2/P2] T | .19 $\pm$ .11 | .11 $\pm$ .21 | .20 $\pm$ .12 | .07 $\pm$ .20
[G3/P3] ATV | .23 $\pm$ .12 | .07 $\pm$ .15 | .28 $\pm$ .13 | .07 $\pm$ .19
Figure 2: Output of [G3] on Subject 2’s ratings on the Validation Story. The
model tracks the ratings well during the latter half of the video.

###  III-D Test Performance

Unfortunately, our models did not perform as well on the Test set, as shown in
the ‘Test columns of Table II. Our best Generalized and Personalized models
([G1] and [P1]) both achieved a CCC of $.14$. Nonetheless, this still
outperformed the validation baselines, and fell within our estimated error
bounds in the last column of Table I. Much of this decrease in performance can
be traced to poor performance on Story 7 in the test set, which had CCC values
of $.02$, -$.07$ and -$.05$ for models [G1] to [G3] respectively. As was
indicated by our poor cross-validation performance on Story 2, inter-story
variation can make robust empathy prediction a difficult task. This also
explains the increase in standard deviation over videos in the test set—our
models were not robust enough for the videos in Story 7. The personalized
models did not outperform the generalized models despite fine-tuning,
suggesting that fine-tuning led to over-fitting rather than personalization to
each listener.

##  IV Discussion

When empathizing with someone as they tell a story, perhaps the most important
information that influences one’s responses is the _content of their story_.
This intuition was borne out in our experiments, which showed that models that
include the text modality do better than models without. Indeed, we only
scratched the surface of the textual information available: We used word-level
features, ignoring word order in each utterance. These design choices were
compromises given the time and data constraints of the Challenge—transcripts
were not provided, so we used YouTube-generated subtitles that were not
segmented by speaker or sentence—which in turn may have hurt the performance
of our models. Given more time, a more principled approach would account for
the syntactic structure of each utterance, as well as model the fact that
utterances in this dataset occur as a dialogue. Segmenting the text and audio
modalities by speaker would also have allowed separate affect models to be
built for actor and listener, which might have better captured the influence
of the actor upon the listener’s empathic responses.

Another challenge we encountered was the heterogeneity of the stories. First,
the stories all express very different emotions, and there may not have been
enough examples for our models to learn generalizable features. Second, as
noted in the Challenge description, each story was told by a particular actor
intentionally expressing a particular personality trait. This resulted in
considerable variance in model performance across stories. For example, on the
Test set, our models did extremely well on Story 6 (e.g., [G2]: .26) but
poorly on Story 7 ([G2]: -.07).

One final challenge of siginficance was the nature of the ratings and the
scale used. See Fig. 2 as an example: Participants’ ratings were highly
discontinuous, with large, sharp changes. Our models tended to make smaller
changes more frequently, generally capturing the trend but not the larger
discontinuities and extremities. Future work could adapt the model
architecture to deal with sharper changes, perhaps by predicting both the
rating and the rate of change in the rating (i.e. the kinematics of valence),
or by modelling valence changes as event-driven [13].

In conclusion, much work remains to be done in building models that can reason
about an individual’s emotions, but also the emotions of others interacting
with that individual. There are many challenges to be solved, from the
complexity of rich narrative stimuli to dealing with time-series data. We hope
that this challenge will inspire future research into modelling empathic
responses over time, and will pave the way for building applications like
empathic robots and digital personal assistants.

## References

  * [1] P. Barros, N. Churamani, E. Lakomkin, H. Sequeira, A. Sutherland, and S. Wermter.  The OMG-Emotion Behavior Dataset.  In 2018 International Joint Conference on Neural Networks (IJCNN), pages 1408–1414. IEEE, 2018.
  * [2] D. Deng, Y. Zhou, J. Pi, and B. E. Shi.  Multimodal utterance-level affect analysis using visual, audio and text features.  arXiv preprint arXiv:1805.00625, 2018.
  * [3] F. Eyben, F. Weninger, F. Gross, and B. Schuller.  Recent developments in opensmile, the munich open-source multimedia feature extractor.  In Proceedings of the 21st ACM International Conference on Multimedia, pages 835–838. ACM, 2013.
  * [4] X. Guo, B. Zhu, L. F. Polanía, C. Boncelet, and K. E. Barner.  Group-level emotion recognition using hybrid deep models based on faces, scenes, skeletons and visual attentions.  In Proceedings of the 2018 on International Conference on Multimodal Interaction, pages 635–639. ACM, 2018.
  * [5] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Computation, 9(8):1735–1780, 1997.
  * [6] T. Luong, H. Pham, and C. D. Manning.  Effective approaches to attention-based neural machine translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, 2015.
  * [7] D. C. Ong, J. Zaki, and N. D. Goodman.  Affective cognition: Exploring lay theories of emotion.  Cognition, 143:141–162, 2015.
  * [8] D. C. Ong, J. Zaki, and N. D. Goodman.  Computational models of emotion inference in theory of mind: A review and roadmap.  Topics in Cognitive Science, 2018.
  * [9] O. M. Parkhi, A. Vedaldi, and A. Zisserman.  Deep face recognition.  In British Machine Vision Conference, 2015.
  * [10] S. Peng, L. Zhang, Y. Ban, M. Fang, and S. Winkler.  A Deep Network for Arousal-Valence Emotion Prediction with Acoustic-Visual Cues.  In arxiv, 2018.
  * [11] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014.
  * [12] S. Poria, E. Cambria, R. Bajpai, and A. Hussain.  A review of affective computing: From unimodal analysis to multimodal fusion.  Information Fusion, 37:98–125, 2017.
  * [13] Z. Qin and C. R. Shelton.  Event detection in continuous video: An inference in point process approach.  IEEE Transactions on Image Processing, 26(12):5680–5691, 2017.
  * [14] M. Wöllmer, M. Kaiser, F. Eyben, B. Schuller, and G. Rigoll.  Lstm-modeling of continuous emotions in an audiovisual affect recognition framework.  Image and Vision Computing, 31(2):153–163, 2013.
  * [15] J. D. Wondra and P. C. Ellsworth.  An appraisal theory of empathy and other vicarious emotional experiences.  Psychological Review, 122(3):411, 2015.
  * [16] A. Zadeh, R. Zellers, E. Pincus, and L.-P. Morency.  Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages.  IEEE Intelligent Systems, 31(6):82–88, 2016.
  * [17] J. Zaki.  Empathy: A motivated account.  Psychological Bulletin, 140(6):1608, 2014.


# Evaluating the Effectiveness of Corrective Demonstrations and a Low-Cost
Sensor for Dexterous Manipulation
††thanks: *These authors contributed equally to this work.

Abhineet Jain  Georgia Institute of Technology
Atlanta, USA
<EMAIL_ADDRESS>Jack Kolb*
J.M. Abbess IV  Georgia Institute of Technology
Atlanta, USA
<EMAIL_ADDRESS>Georgia Institute of Technology
Atlanta, USA
<EMAIL_ADDRESS>Harish Ravichandar  Georgia Institute of Technology
Atlanta, USA
<EMAIL_ADDRESS>
###### Abstract

Imitation learning is a promising approach to help robots acquire dexterous
manipulation capabilities without the need for a carefully-designed reward or
a significant computational effort. However, existing imitation learning
approaches require sophisticated data collection infrastructure and struggle
to generalize beyond the training distribution. One way to address this
limitation is to gather additional data that better represents the full
operating conditions. In this work, we investigate characteristics of such
additional demonstrations and their impact on performance. Specifically, we
study the effects of corrective and randomly-sampled additional demonstrations
on learning a policy that guides a five-fingered robot hand through a pick-
and-place task. Our results suggest that corrective demonstrations
considerably outperform randomly-sampled demonstrations, when the proportion
of additional demonstrations sampled from the full task distribution is larger
than the number of original demonstrations sampled from a restrictive training
distribution. Conversely, when the number of original demonstrations are
higher than that of additional demonstrations, we find no significant
differences between corrective and randomly-sampled additional demonstrations.
These results provide insights into the inherent trade-off between the effort
required to collect corrective demonstrations and their relative benefits over
randomly-sampled demonstrations. Additionally, we show that inexpensive
vision-based sensors, such as LeapMotion, can be used to dramatically reduce
the cost of providing demonstrations for dexterous manipulation tasks. Our
code is available at https://github.com/GT-STAR-Lab/corrective-demos-
dexterous-manipulation.

###### Index Terms:

learning from demonstrations, reinforcement learning, dexterous manipulation

##  I Introduction

Dexterous manipulation often involves the use of high degree-of-freedom robots
to manipulate objects. Representative dexterous manipulation tasks include
relocating objects, picking up arbitrarily shaped objects, and sequential
interactions with articulated objects (e.g. unlatching and opening a door).
Indeed, factors such as high-dimensional state space and complex interaction
dynamics make these tasks particularly challenging to automate. Classical
control methods, that have proven valuable for a variety of manipulation
problems, are hard to recruit for dexterous manipulation due to the amount of
manual effort required to design controllers in high-dimensional spaces.

Prior work has found success in dexterous manipulation by using self-
supervised learning in simulation and transferring learned policies to real
robots [1]. Others have utilized demonstrations to improve reinforcement
learning [2]. However, these approaches either do not generalize well beyond
small workspaces covered during training, or require long training times for
highly-specialized tasks.

|  |
---|---|---
Figure 1:  We investigate the use of a LeapMotion sensor (left) to capture
demonstrations of human hand motion (center) that can help a simulated Adroit
robot hand (right) learn manipulation skills.

In this paper, we explore how additional demonstrations that represent the
full operating conditions affect performance of the relocation task (described
in Fig. 2) in the full operating space. Approaches like demonstration-
augmented policy gradient (DAPG) use demonstrations for randomly-sampled start
and goal states from a restrictive training distribution. However, we find
this approach does not generalize well to task instances in the full operating
space. We investigate the effects of corrective versus randomly-sampled
additional demonstrations, expecting that corrective demonstrations to areas
of policy failure can benefit performance more than randomly-sampled
demonstrations.

In addition, we explore the feasibility of using a vision-based sensor to
record demonstrations for dexterous manipulation tasks. For everyday robots to
learn novel manipulation tasks from non-technical humans, it will be useful to
reduce the hardware cost of providing demonstrations. The original DAPG paper
uses a CyberGlove III [3] wearable sensor glove to record human hand
trajectories. However, the CyberGlove III costs around US$13,000. We instead
leverage LeapMotion, a US$90 sensor which uses stereo vision to track hand
joint positions, and compare the performance of resulting DAPG policies. Our
findings validate that the granularity of vision-based joint tracking is
effective for training high-performing policies for the relocation task.

##  II Related Work

Previous works explore learning dexterous manipulation from demonstrations,
using deep neural networks [4] or reinforcement learning techniques [2] [5]
[6] to learn task-specific policies. Some contributions learn actuator outputs
from raw color and depth information using deep neural networks [4], while
others use demonstrations to aid trajectory exploration and learn system
dynamics by utilizing linear regression with GMM priors [5]. DAPG uses
demonstrations to pre-train a policy and then applies reinforcement learning
to improve it. Another contribution aims to learn from suboptimal experts,
introducing a policy iteration algorithm called Relative Entropy Q-Learning
(REQ), along with an effective exploration technique that intertwines the
policy’s actions with the demonstrations [6].

A state-of-the-art for solving a Rubik’s cube with a robot hand [1] uses
automatic domain randomization to generate a distribution of environments to
train on. This work uses meta-learning utilizing LSTMs and the asymmetric
actor-critic algorithm for training. While the authors claim that their
approach transfers well onto real robots, their algorithm is time-intensive,
and uses highly engineered reward functions and specialized robot hardware.
These characteristics make their work difficult to reproduce and utilize for
general dexterous manipulation tasks.

In our work, we build upon DAPG which uses a small number of expert
demonstrations to train policies. However, DAPG policies struggle to
generalize beyond the demonstration area. To address this, related work has
explored autonomously learning the reward function for tasks instead of human-
engineering them [7]. Although their work claims to learn better reward
functions than GAIL [8] and Adversarial IRL [9], their learned reward function
does not improve upon the performance of the original paper on their set of
dexterous manipulation tasks. We approach improving DAPG through a human-robot
interaction perspective – when access to the full operating space is granted
for recording demonstrations, we explore whether strategic demo collection can
improve policy performance.

Our work is inspired by DAgger [10], a classical learning from demonstration
algorithm that integrates online corrections to mitigate the covariate shift
between training and testing. A recent work explores mitigating covariate
shift by measuring the induced covariate shift and directly adjusting for it
[11]. It uses expert demonstrations that potentially visit all states that the
policy will visit, making it infeasible for the dexterous manipulation case.
Another work utilizes data augmentation for generating more sample
trajectories, along with a correction neural network to preserve the success
of these augmented trajectories for adversarial imitation [12]. Our approach
aims to utilize the benefits of DAgger without the disadvantages of online
corrections. We collect multiple corrective demonstrations at once and retrain
using the combined set of original and corrective demonstrations.

Figure 2:  The relocation task in MuJoCo involves picking up a blue ball from
a tabletop and carrying it to a green goal region.

##  III Demo-Augmented Policy Gradient (DAPG)

DAPG learns dexterous manipulation via Deep-RL by combining demonstration-
based learning with reinforcement learning. DAPG initially trains a policy
using behavior cloning, and then tunes the policy using natural policy
gradient with a modified loss term, weighing more on the RL policy with
increased training iterations. DAPG’s loss term penalizes trajectories that
are further from the demonstrations. This exploration learns smooth human-like
trajectories from a small number of demonstrations, making the algorithm
feasible for real world applications.

The original DAPG paper uses expert demonstrations captured using a CyberGlove
III [3], a wearable glove embedded with 18-22 sensors. The authors find that
their approach results in a high accuracy on four different dexterous
manipulation tasks: relocate a ball, open a door, hammer a nail, and orient a
pen in-hand. For each of these tasks, the original paper uses 25
demonstrations and trains for 100 iterations. In our paper, we only consider
the relocation task.

The relocation task shown in Fig. 2 requires an agent to grasp and relocate a
blue ball from a tabletop to a green region. In the original DAPG paper, task
initializations are sampled from a restrictive region on the table top (width
of 0.3m). In a pilot study using the authors’ CyberGlove demonstrations, we
find that a policy trained solely in the restrictive space for 100 training
iterations results in a $>$95% success rate in the restrictive space, but a
69% success rate in the full operating space (width of 0.5m). When the RL
agent is provided access to train in the full operating space, it requires
more than 200 training iterations to learn a policy with $>$95% success rate.
Our work aims to reduce the tradeoff between higher sample efficiency and
lower performance for the relocation task.

##  IV Methods

In this section, we describe the two sets of experiments we conducted, each
aimed to investigate a specific research question (RQ) as described below. The
code for these experiments can be found at https://github.com/GT-STAR-
Lab/corrective-demos-dexterous-manipulation.

Figure 3:  Demo collection setup: The upwards-facing LeapMotion sensor reads
hand joint information, the MuJoCo environment updates the Adroit robot hand
joint angles in real time, and the collection is controlled via an external
device.  Figure 4:  We investigate RQ2 by comparing two conditions: i)
Condition A uses demonstrations with randomly-sampled initial states from the
restrictive space and corrective demonstrations associated with failures from
the full operating space (top); ii) Condition B uses demonstrations with
randomly-sampled initial states from both restrictive and full operating
spaces (bottom).

###  IV-A RQ1: What are the effects of low-cost sensing on learning outcomes?

Our first set of experiments examine how policies trained with demonstrations
from a low-cost vision-based sensor (LeapMotion [13]) compare to policies
trained using a wearable glove. A glove’s ability to produce high quality
demonstrations with on-finger joint tracking makes it advantageous for
collecting human demonstrations for complex dexterous manipulation tasks such
as rotating objects in-hand. In such situations, external cameras may have
difficulty picking up finger movements that are obstructed by the palm or by
other fingers. However, for applications aiming to target broader consumer
audiences or research labs, the CyberGlove III is expensive.

External vision-based sensors can infer all joint poses for relocation or
grasping tasks, potentially reducing hardware costs. To evaluate the
performance difference between on-finger and vision-based sensing, we develop
an interface to convert input from a LeapMotion hand tracker – a US$90 device
which uses stereoscopic vision to estimate hand poses – into demonstrations
accessible by the original DAPG implementation. We limit our investigation to
the relocation task. The task does not obstruct external visual tracking, and
therefore can adequately compare the two sensors. Our demonstration collection
environment is shown in Fig. 3.

The LeapMotion’s low price-point comes with lower quality demonstrations: 1)
The device does not track roll joints for the thumb and little finger,
resulting in two fewer DOFs. 2) The device sometimes has difficulty
distinguishing between the four fingers, resulting in fingers moving or
grasping simultaneously; For the relocation task this behavior is acceptable.
3) The device collects noisy observations due to jittery sensor readings. As
such, RQ1 is indirectly related to the robustness of learning algorithms
against suboptimal demonstrations.

Figure 5:  The restrictive (red) space is twice as large as the full operating
(blue) space, and they do not overlap.

  * H1:

LeapMotion demonstrations will result in policies with rollout success rates
that are comparable to CyberGlove III demonstrations for the relocation task,
at the cost of sample efficiency.

###  IV-B RQ2: What is the utility of corrective demonstrations?

Our second research question is motivated by the practical question of whether
a meaningful improvement can be gained by including corrective demonstrations
in a data set. We evaluate whether the performance of dexterous manipulation
policies for the relocation task can be improved by splitting the training
into two stages, labeled as Condition A:

  * Stage 1: Collecting demonstrations from randomly-sampled start and goal locations in the restrictive space.

  * Stage 2: Appending additional demonstrations collected from failure cases in the full operating space, using a policy trained in the full space using Stage 1 demonstrations.

We compare the policy from Condition A to a policy trained on randomly-sampled
demonstrations from both the restrictive space and the full operating space,
labeled as Condition B. Fig. 5 shows a visual of the tabletop areas for the
restrictive and full operating spaces. Table I details the distributions of
demonstrations from the restrictive space and the full operating space for
each of the five policies we address. The full operating space only
encompasses the area outside the restrictive space, causing corrective
demonstrations to come from state initializations not seen in the original
demonstrations. While all policies have the same total number of
demonstrations, policies with corrective demonstrations require twice the
training time. Thus, RQ2 investigates whether the effort of training two
policies – one with the demonstrations from the restrictive space, one after
collecting corrective demonstrations – results in a tangible performance
improvement over randomly-sampled demonstrations from both spaces. Fig. 4
summarizes our methods, which are formalized below:

Table I: Source of demonstrations for each policy

Policy | Condition | Rest.-Orig. | Full-Random | Full-Corr.
---|---|---|---|---
30-O | – | 30 | – | –
10-O+20-R | B | 10 | 20 | –
10-O+20-C | A | 10 | – | 20
20-O+10-R | B | 20 | 10 | –
20-O+10-C | A | 20 | – | 10

  * •

“Full-Corr.” indicates corrective demonstrations from failures cases of a
policy trained on the original demonstration set “Rest.-Orig.”.

For Condition A, we first collect a set of randomly-sampled demonstrations
$D^{O}$ in the restrictive space, and train a policy $\pi_{0}$ in the full
operating space. On an evaluation set of 1000 initial states from the full
operating space, we roll out $\pi_{0}$ and identify the lowest-performing
failure cases. We then record corrective demonstrations $D^{C}$ for these
failure cases, and combine them with the original demonstrations $D^{O}$ to
train a second policy $\pi_{A}$ in the full operating space. We measure the
rollout success ratio for $\pi_{A}$ using the evaluation set in the full
operating space.

We compare $\pi_{A}$ to the policy from Condition B, $\pi_{B}$, trained using
the original demonstrations $D^{O}$ from the restrictive space, and an
additional set of randomly-sampled demonstrations $D^{R}$ from the full
operating space.

We anticipate that by guiding $D^{C}$ towards weaker areas of $\pi_{0}$, the
resulting $\pi_{A}$ will have an increased coverage of challenging initial
task states in the full operating space, resulting in a higher performance
than $\pi_{B}$. We evaluate a policy’s performance by both its rollout success
ratio and its sample efficiency. Furthermore, we expect all four policies that
include demonstrations from the full operating space to outperform Policy
30-O, by matter of having access to demonstrations from the full operating
space.

  * H2:

When the number of additional demonstrations from the full operating space is
higher than the original demonstrations from the restrictive space, corrective
demonstrations will improve performance compared to randomly-sampled
additional demonstrations.

  * H3:

When the number of additional demonstrations from the full operating space is
lower than the original demonstrations from the restrictive space, corrective
demonstrations will improve performance compared to randomly-sampled
additional demonstrations.

Figure 6:  Rollout success ratios across 800 training iterations show that the
policy trained on 25 CyberGlove III demos (red) converges to 100% at close to
200 iterations, and the policy trained on 25 LeapMotion demos (blue) takes
more than 600 iterations to achieve similar performance. Note that policy
relying only on reinforcement (yellow) makes little progress.  Figure 7:  We
compared performance in terms of rollout success ratios. (Left) shows policies
with a high proportion of demonstrations in the full operating space, while
(Right) shows policies with a low proportion of demonstrations in the full
operating space. For comparison with the policy trained on demonstrations only
from the restrictive space, Policy 30-O is shown on both plots.

##  V Results

###  V-A Evaluating H1: Demonstrations from LeapMotion vs. CyberGlove III

We train a policy for 800 iterations using 25 demonstrations collected from
LeapMotion and compare it with a policy trained for 800 iterations using the
original 25 demonstrations provided by the authors of DAPG. The results are
shown in Table II. We use the same network architecture and hyperparameters as
in the original DAPG paper. We observe that the optimal demonstrations from
CyberGlove III allow DAPG to converge towards 100% success ratio in around 200
iterations whereas it takes almost 600 iterations for the LeapMotion’s policy
to reach a comparable performance. A policy trained without any demonstrations
does not even start learning after 800 iterations. (see Fig. 6).

Table II:  Rollout success ratios in full operating space after 800 iterations  Demos | CyberGlove III | LeapMotion
---|---|---
25 | 99.0% | 98.1%

Outcome: Policies trained on the full operating space using demonstrations
from LeapMotion are able to reach comparable success ratios to policies
trained with CyberGlove III demonstrations, at the cost of sample efficiency.

###  V-B Evaluating H2, H3: Applying corrective demonstrations in the full
operating space

We train five policies based on the distribution in Table I. Each policy is
trained for 800 iterations in order to capture the policy’s full plateau of
rollout success. Table III compares the rollout success ratio of each policy
at different iterations.

Table III:  Policy rollout success ratios  | Rollout Success [%]
---|---
Policy | 200-iter | 400-iter | 600-iter | 800-iter
30-O | 1.1 | 21.2 | 72.6 | 88.0
10-O+20-R | 7.0 | 36.0 | 75.6 | 93.9
10-O+20-C | 66.6 | 92.7 | 97.9 | 99.3
20-O+10-R | 3.2 | 59.4 | 96.0 | 99.2
20-O+10-C | 2.7 | 55.8 | 98.4 | 100.0

We find that all combined policies converge to high rollout success ratio
above 90%. However, we see marked differences in their sample efficiencies, as
shown in Fig. 7. All policies perform better than Policy 30-O and are more
sample efficient, which is expected given their access to demonstrations from
the evaluation set. Furthermore, Policy 10-O+20-C is more sample efficient
than Policy 10-O+20-R and has a 5.4% higher rollout success ratio, confirming
our hypothesis H2. However, we do not find a comparable difference between the
sample efficiencies of Policy 20-O+10-C and Policy 20-O+10-R, perhaps due to
those policies having a low proportion of additional demonstrations from the
full operating space. The findings reject our hypothesis H3.

Outcome: When the proportion of additional demonstrations from the full
operating space is higher than the original demonstrations from the
restrictive space, we find that corrective additional demonstrations improve
the policy’s performance for the relocation task compared to randomly-sampled
additional demonstrations. When the proportion of additional demonstrations is
lower, we find no notable difference between policies learned from Condition A
and Condition B.

##  VI Conclusion

We find that vision-based sensors can be used to collect demonstrations for
the relocation task achieving policies with similar success to policies
resulting from wearable sensors (see Fig. 6), at the cost of sample
efficiency. The result validates our use of the LeapMotion device to collect
demonstrations for the relocation task, reducing the setup cost by
approximately 140x. Our validation supports H1.

We use corrective additional demonstrations to guide policy exploration in the
full operating space, and find that corrective demonstrations improve the
performance and sample efficiency of policies as compared to those trained on
randomly-sampled additional demonstrations. However, this improvement is
limited to more additional demonstrations from the full operating space than
the original demonstrations from the restrictive space. Our investigation
supports H2 and rejects H3. Additionally, the high rollout success ratios of
our policies show that DAPG is resilient to suboptimal demonstrations at the
cost of sample efficiency.

To further investigate RQ1, we plan to improve the correspondence between
LeapMotion and MuJoCo. Our pipeline for recording demonstrations using MuJoCo
leverages the LeapMotion tracker for data collection and a web server to
communicate with MuJoCo. Though this pipeline is functional, some joint
mappings are imperfect, limiting the quality of our collected demonstrations.

##  VII Acknowledgements

We thank Nakul Gopalan for his feedback and encouragement towards publishing
this work.

## References

  * [1] I. A. OpenAI, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas _et al._ , “Solving rubik’s cube with a robot hand,” 2019.
  * [2] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations,” 2018.
  * [3] CyberGlove Systems LLC, “CyberGlove III - CyberGlove Systems LLC,” http://www.cyberglovesystems.com/cyberglove-iii/.
  * [4] T. Zhang, Z. McCarthy, O. Jowl, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation,” _Proceedings - IEEE International Conference on Robotics and Automation_ , pp. 5628–5635, 2018.
  * [5] V. Kumar, A. Gupta, E. Todorov, and S. Levine, “Learning Dexterous Manipulation Policies from Experience and Imitation,” 2016. [Online]. Available: http://arxiv.org/abs/1611.05095
  * [6] R. Jeong, J. T. Springenberg, J. Kay, D. Zheng, Y. Zhou, A. Galashov, N. Heess, and F. Nori, “Learning Dexterous Manipulation from Suboptimal Experts,” pp. 1–20, 2020. [Online]. Available: http://arxiv.org/abs/2010.08587
  * [7] J. Orbik, A. Agostini, and D. Lee, “Inverse reinforcement learning for dexterous hand manipulation,” pp. 1–7, 2021.
  * [8] J. Ho and S. Ermon, “Generative adversarial imitation learning,” _Advances in neural information processing systems_ , vol. 29, pp. 4565–4573, 2016.
  * [9] J. Fu, K. Luo, and S. Levine, “Learning robust rewards with adversarial inverse reinforcement learning,” _arXiv preprint arXiv:1710.11248_ , 2017\.
  * [10] S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in _Proceedings of the fourteenth international conference on artificial intelligence and statistics_. JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [11] J. Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and J. A. Bagnell, “Feedback in imitation learning: The three regimes of covariate shift,” _arXiv preprint arXiv:2102.02872_ , 2021.
  * [12] D. Antotsiou, C. Ciliberto, and T.-K. Kim, “Adversarial imitation learning with trajectorial augmentation and correction,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 4724–4730.
  * [13] LeapMotion, “Ultraleap for Developers,” https://developer.leapmotion.com/.