# Fine-grained Emotion and Intent Learning in Movie Dialogues

Anuradha Welivita, Yubo Xie and Pearl Pu
School of Computer and Communication Sciences
École polytechnique fédérale de Lausanne
Switzerland
<EMAIL_ADDRESS>

###### Abstract

We propose a novel large-scale emotional dialogue dataset, consisting of 1M
dialogues retrieved from the OpenSubtitles corpus and annotated with 32
emotions and 9 empathetic response intents using a BERT-based fine-grained
dialogue emotion classifier. This work explains the complex pipeline used to
preprocess movie subtitles and select good movie dialogues to annotate. We
also describe the semi-supervised learning process followed to train a fine-
grained emotion classifier to annotate these dialogues. Despite the large set
of labels, our dialogue emotion classifier achieved an accuracy of $65\%$ and
was used to annotate 1M emotional movie dialogues from OpenSubtitles. This
scale of emotional dialogue classification has never been attempted before,
both in terms of dataset size and fine-grained emotion and intent categories.
Visualization techniques used to analyze the quality of the resultant dataset
suggest that it conforms to the patterns of human social interaction.

##  1 Introduction

Understanding emotion in human social conversations or chitchat has gained
popularity in the natural language processing community due to its usefulness
in developing human-like conversational agents. Emotions revealed in social
chitchat are rather complex. It has many categories of emotions to distinguish
due to subtle variations present in human emotion. For example, Sadness and
Disappointment are pursued and dealt differently in human conversations. Also,
the listeners’ reaction to emotions is not always a straightforward mirroring
effect of the speakers’ emotions. Rather it can be more neutral and convey a
specific intent, as is evident from the dialogue example in Table 1.

Speaker: |  I’ve been hearing some strange noises around the house at night. (Afraid)
---|---
Listener: |  oh no! That’s scary! What do you think it is? (Neutral: Acknowledging; Questioning)
Speaker: |  I don’t know, that’s what’s making me anxious. (Anxious)
Listener: |  I’m sorry to hear that. (Neutral: Sympathizing)
Table 1: An example dialogue showing that the listener’s reactions to emotions
do not always mirror the speaker’s emotions. Figure 1: Steps for curating the
OSED dataset.

Welivita and Pu Welivita and Pu (2020) have analyzed listener responses in the
EmpatheticDialogues dataset Rashkin et al. (2018) and discovered 9 listener
specific empathetic response intents contained in emotional dialogues:
Questioning; Agreeing; Acknowledging; Sympathizing; Encouraging; Consoling:
Suggesting; Wishing; and Neutral (include other intents with neutral emotion
such as expressing an opinion, advising, and disagreeing). They have
automatically annotated the EmpatheticDialogues dataset Rashkin et al. (2018)
with 32 fine-grained emotions and the 9 empathetic response intents and
discovered frequent emotion-intent exchange patterns in human social
conversations. They observe that this type of dataset tagged with fine-grained
emotions and response intents could train neural chatbots to generate
empathetically appropriate responses conditioned on a selected emotion or
intent. However, for this purpose, a large-scale emotion and intent labeled
dataset is even more desirable. Curating such a dataset is technically
challenging because 1) annotating such a large-scale dataset require human
labor that is costly, and 2) given the fine-granularity of the emotion and
intent labels, the human labeling task is more difficult compared to more
generic Angry-Happy-Sad. As a result, existing manually labeled emotional
dialogue datasets such as IEMOCAP Busso et al. (2008), MELD Poria et al.
(2019), and DailyDialogue Li et al. (2017) are smaller in scale and contain
only a limited set of emotions (emotions derived from basic emotion models
such as the Ekman’s), with simpler dialogue responding strategies, or both.
Also, existing datasets often contain a label Neutral or Other for responses
that do not convey emotion, which introduces vagueness and limits the ability
of automatic agents that use such datasets in learning useful response
strategies.

To fill the above gap, we curate a novel large-scale dialogue dataset, OSED
(OpenSubtitles Emotional Dialogues), containing 1M emotional dialogues from
movie subtitles, in which each dialogue turn is automatically annotated with
32 fine-grained emotions and 9 empathetic response intents. Movie subtitles
well approximate human social conversations and how emotion is handled in
them. It is one of the major sources to learn emotional variations and
corresponding response strategies. To reduce the cost of human labeling and
the complexity of labeling dialogues with fine-grained emotions and intents,
we devise a semi-automated human computation task to collect fine-grained
emotion and intent labels for a small set of movie dialogues (9K). We then
follow a semi-supervised approach to expand the labeled data and train a
dialogue emotion classifier to automatically annotate 1M emotional dialogues.

The process of curating the dataset consists of several stages. First, we
apply automatic turn and dialogue segmentation methods on movie subtitles in
the OpenSubtitles (OS) corpus Lison et al. (2019) and obtain close to 9M
dialogues. After data cleaning and removing duplicates, we reduce its size to
4M. Then, we apply a weak labeler, EmoBERT (a BERT-based sentence-level
classifier) trained on the EmpatheticDialogues dataset Rashkin et al. (2018),
to label utterances in OS dialogues and filter 1M emotional dialogues (OSED
initial). Thirdly, with semi-supervised learning methods, we refine EmoBert
and obtain EmoBert+, a more advanced dialogue emotion classifier trained on OS
dialogues. To evaluate EmoBert+, we compare it with FastText. The former is
more accurate than FastText. Finally, we use EmoBert+ to label dialogues in
OSED initial to obtain the final 1M OSED dataset. We evaluate the quality of
the resultant dataset by visually inspecting the emotion-intent flow patterns
that occur in the dataset and checking if they conform to the patterns of
human social conversations discovered in existing work See et al. (2019);
Welivita and Pu (2020). Figure 1 summarizes the process of creating OSED. The
data curation pipeline we follow substantially reduces the cost of human
labor, while ensuring quality annotations.

Our contributions in this paper are three-fold. 1) We curate a dialogue
dataset, OSED, containing 1M emotional dialogues labeled with 32 fine-grained
emotions and 9 empathetic response intents. Compared to existing dialogue
datasets tagged with emotions, OSED is more general-purpose, significantly
larger, and contains more fine-grained emotions and empathetic response
strategies. 2) We outline the complex pipeline used to derive this dataset and
evaluate the annotation quality using visualization methods. 3) We release our
fine-grained emotion classifier used to annotate the OSED dataset, which can
be used as a general-purpose classifier capable of recognizing fine-grained
emotions and empathetic response intents in social chitchat.

##  2 Literature Review

Dataset |  Labels | No. of | No. of | Publicly
---|---|---|---|---
|  | dialogues | utterances | available
IEMOCAP Busso et al. (2008) |  Joy, Sadness, Anger, Frustrated, Excited, and Neutral | $151$ | $7,433$ | ✓
MELD Poria et al. (2019) |  Joy, Surprise, Sadness, Anger, Disgust, Fear, and Neutral | $1,433$ | $13,708$ | ✓
DailyDialogue Li et al. (2017) |  Joy, Surprise, Sadness, Anger, Disgust, Fear, and Neutral | $12,218$ | $103,607$ | ✓
EmotionLines Hsu et al. (2018) |  Joy, Surprise, Sadness, Anger, Disgust, Fear, and Neutral | $1,000$ | $14,503$ | ✓
EmoContext Chatterjee et al. (2019) |  Joy, Sadness, Anger, and Other | $38,421$ | $115,263$ | ✓
Twitter customer support Herzig et al. (2016) |  Customer emotions: Confusion; Frustration; Anger; Sadness; Happiness; Hopefulness; Disappointment; Gratitude; Politeness; and Agent emotional techniques: Empathy; Gratitude; Apology; Cheerfulness | $2,413$ | $\approx 14,078$ | ✗
Empathetic Dialogues Rashkin et al. (2018); Welivita and Pu (2020) |  32 fine-grained emotions and 9 empathetic response intents: Questioning; Agreeing; Acknowledging; Sympathizing; Encouraging; Consoling; Suggesting; Wishing, Neutral. | $24,850$ | $107,220$ | ✓
OSED |  32 fine-grained emotions and 9 empathetic response intents. | $1M$ | $3,488,300$ | ✓
Table 2: Comparison of emotion annotated dialogue datasets available in the
literature against OSED.

IEMOCAP Busso et al. (2008), MELD Poria et al. (2019), DailyDialogue Li et al.
(2017), EmotionLines Hsu et al. (2018), and EmoContext Chatterjee et al.
(2019) are some existing dialogue datasets with emotion labels. Table 2 shows
a summary of the size and the labels present in these datasets. Even though
these datasets approximate social chitchat quite well, they are limited in
size and are labeled with only a small set of emotions without any response
strategies. Several state-of-the-art dialogue emotion classifiers such as CMN
Hazarika et al. (2018b), ICON Hazarika et al. (2018a), IAAN Yeh et al. (2019),
DialogueRNN Majumder et al. (2019), and DialogueGCN Ghosal et al. (2019) were
proposed based on the above datasets. They report accuracy in the range of
$56-65\%$ on a limited set of emotion labels. Though these dialogue emotion
classifiers report high accuracy, they are incapable of identifying fine-
grained emotions and specific response strategies in human conversations,
which makes us unable to use them for our purpose of curating an automatically
labeled dialogue corpus.

Herzig et al. Herzig et al. (2016) detected both customer emotions and agent
emotional techniques (e.g., Apology, Empathy) in customer support dialogues.
They curated a dialogue dataset from two customer support Twitter accounts and
manually annotated the customer turns with one of 9 emotions and the agent
turns with one of 4 emotional techniques. But emotions expressed by customers
in social media service dialogues are mainly negative (e.g. anger,
frustration). Customer service agents are also instructed to behave in a
certain way, which is different from daily chitchat. Social chitchat can
include a variety of positive and negative emotions and a variety of different
response intents.

The EmpatheticDialogues dataset Rashkin et al. (2018) contains 25K open-domain
dialogues grounded on 32 emotions. The 32 emotions range from basic emotions
derived from biological responses Ekman (1992); Plutchik (1984) to larger sets
of subtle emotions derived from contextual situations Skerry and Saxe (2015).
The dialogues were collected using Amazon Mechanical Turk, instructing the
speakers to start a dialogue picking one of 32 emotions and the listeners to
respond empathetically. A major limitation in this dataset is that individual
dialogue turns are not labeled, and it does not identify specific intents that
listeners use when responding to speakers’ emotions. Welvita and Pu Welivita
and Pu (2020) manually analyzed a subset of the listener turns in
EmpatheticDialogues and identified 9 listener specific response intents. They
developed a sentence level weak labeler using which they annotated the entire
dataset with 32 emotions and 9 empathetic response intents. However, due to
the limited size of EmpatheticDialogues, it is difficult to use this for data-
intensive applications.

To address the above limitations, we curate OSED containing 1M movie dialogues
that approximate social emotional conversation. We label each dialogue turn
with 32 emotions and 9 empathetic response intents using our own dialogue
emotion classifier. Table 2 compares OSED to existing emotion annotated
dialogue datasets in the literature. Our dialogue emotion classifier trained
on movie subtitles achieves state-of-the-art performance despite the large set
of labels.

##  3 Methodology

This section describes the dialogue selection process, design of the human
annotation task, sentence embeddings techniques used to expand human-labeled
dialogues, the architecture, and the semi-supervised training process of the
classifier used to produce the resultant 1M annotated OSED dataset.

###  3.1 Dialogue Curation from Movie Subtitles

The OpenSubtitles 2018 corpus consists of 3.7M movie and TV subtitles. It is
comprised of 3.4B sentences and 22.2B tokens. It is an excellent source to
learn emotional variations in dialogue and corresponding response mechanisms
since movie and TV dialogues well approximate social chitchat and how humans
handle emotional events. However, movie subtitles do not contain an explicit
dialogue turn structure and specific indicators where one dialogue ends and
the next dialogue begins. To overcome the first issue, we reproduced the work
by Lison and Meena Lison and Meena (2016) to build an SVM based classifier
that determines if two consecutive sentences are part of the same dialogue
turn. Our classifier achieved a segmentation accuracy of $76.69\%$, which is
close to the accuracy of $78\%$ that the authors claim. The set of features
that gave the best turn segmentation accuracy are: 1) unigram and bi-gram
features of adjacent sentences after lemmatization; 2) first and final tokens
of adjacent sentences; 3) first and final bi-grams of adjacent sentences; 4)
whether the 2 sentences belong to the same subtitle block or not (boolean); 5)
genre of the movie (Drama, Crime, Musical etc.); 6) sentence density of the
subtitles file (no. of sentences / subtitle duration); and 7) quadratic
combinations of the above features with itself and the rest.

After performing turn segmentation on the OpenSubtitles corpus, we divided the
turns into separate dialogues based on a simple heuristic. If the difference
between the end time of the previous turn and the start time of the current
turn is more than 5 seconds, we take these two turns as belonging to 2
different dialogues. An exception occurs if this timestamp information is
missing in at least one of the turns. In this case, we assume that these two
turns appear in the same subtitle block and consider them as belonging to the
same dialogue. This way, we formed 9M dialogues from the OpenSubtitles corpus
altogether. The choice of 5 sec.s to separate dialogues is explained in
Appendix D.

To further clean the dialogues, we removed character names, repetitive
dialogue turns, turns that start with “previous on…” (monologue at the
beginning of TV episodes), turns with character length less than $2$ or
greater than $100$, turns with an alphabetic proportion less than $60\%$, and
turns with a lot of repetitive tokens. When a dialogue turn was removed, all
the turns following that turn were also removed from the dialogue to maintain
consistency. After that, all the dialogues left with only one turn were
removed from the corpus. This resulted in a cleaned OS dialogue dataset
consisting of 4M dialogues. This dataset is mainly used to train our fine-
grained dialogue emotion classifier.

To filter the most emotional dialogues from the cleaned OS dialogues dataset,
we employed a weak labeler, EmoBERT (a BERT transformer-based sentence level
classifier) trained on 25K situation descriptions from EmpatheticDialogues
Rashkin et al. (2018) tagged with 32 emotion classes, and 7K listener
utterances tagged with 9 empathetic response intents Welivita and Pu (2020).
The classifier had a high top-1 classification accuracy of $65.88\%$. We call
it a weak labeler since it predicts emotion or intent only at sentence-level
and is trained on a different dataset other than OS. We filtered the top 1M
dialogues having the highest label confidence as predicted by this classifier
to form the 1M OSED (initial) dataset. The statistics of the OSED dataset is
given in Table 3.

Total no. of dialogues | $1,000,000$
---|---
Total no. of turns | $2,829,426$
Total no. of tokens | $39,469,825$
Avg. no. of turns per dialogue | $2.83$
Avg. no. of tokens per dialogue | $39.47$
Avg. no. of tokens per turn | $13.95$
Table 3: Statistics of the OSED dataset.

###  3.2 Human Computation

To train a dialogue emotion classifier that can identify both fine-grained
emotions and empathetic response intents, we devised an Amazon Mechanical Turk
(AMT) experiment to collect an initial set of ground truth labels for OS
dialogues. But annotating dialogue turns with one of $41$ labels is a daunting
task. To make the task less exhaustive, we devised a semi-automated approach
using our weak labeler, EmoBERT. By applying EmoBERT on each turn of the
cleaned OS dialogue dataset, we filtered the turns having prediction
confidence $\geq 0.9$, along with their dialogue history. Next, we ranked
these dialogues according to their readability and selected the highest
readable dialogues from each class to be labeled. This is to reduce the time
spent by the workers in having to read long and complicated dialogues. The
steps followed in computing dialogues’ readability are included in Appendix A.
Workers had to select a label from the top-3 predictions made by EmoBERT. If
none of the top-3 predictions matched, they could manually specify the correct
class. If the correct emotion or intent was not present among the $41$ labels
we provide, they could manually enter any new emotion or intent they think is
appropriate. The AMT task’s user interface design is included in Appendix B.

After ranking the dialogues according to readability, we selected the top 250
dialogues in each category for the AMT task. We bundled 15 dialogues in a HIT
with 5 quiz questions that served as checkpoints to evaluate the crowd
workers’ quality. Situation descriptions from the EmpatheticDialogues dataset
for which we already knew the emotion labels were used to formulate the quiz
questions. Finally, we obtained dialogues where we had 2 out of 3 worker
agreement, which resulted in $8,913$ dialogues altogether. Table 4 shows the
results of the AMT task.

Total no. of dialogues |  $10,250$
---|---
No. of dialogues labeled |
with majority vote |  $8,913(86.96\%)$
Inter-annotator agreement (Fleiss’ Kappa) |  $0.46$ (moderate agreement)
Avg. time taken per HIT |  $20.74$ min.
% of times workers got 3/5 |
quiz questions correct |  $77.75\%$
Table 4: AMT task results.

By allowing crowd workers to specify any new emotion or intent class, we were
able to gather suggestions on new intent classes such as Greeting,
Declaration, Commanding, Hesistant, and Accusing that were not already present
among the labels provided. They are listed with corresponding examples in
Appendix C. Since incorporating these new intents requires thorough validation
and a sufficient number of examples from the entire dataset to be grounded as
frequently appearing intents in social chitchat, we leave it as future work.

###  3.3 Distant Learning Using Dialogue Embeddings

To extend the training data obtained from the AMT task, we used the Sentence-
BERT (SBERT) approach proposed by Reimers and Gurevych Reimers and Gurevych
(2019), which uses siamese and triplet network structures to derive
semantically meaningful sentence embeddings that can be compared using cosine-
similarity. Using this approach, we obtained semantically similar dialogues to
those annotated by crowd workers and tagged them with the same class label.
Among several models the authors have proposed, we used the roberta-base-nli-
stsb-mean-tokens model, fine-tuned on the NLI Bowman et al. (2015) and STS
benchmark (STSb) Cer et al. (2017) datasets, since it has a high score in the
STS benchmark, and is more efficient to use than roberta-large.

Before proceeding, we left $20\%$ of the crowd-annotated dialogues, balanced
across all class labels, as testing data to test our classifier’s performance
over different iterations. Then, we followed the following steps to extend the
rest of the dialogues using SBERT. 1) Using the SBERT model, first, we
computed dialogue turn embeddings (each with a vector representation of 768
dimensionalities) for all the turns ($\approx$19M) in the cleaned OS dataset.
2) Then, we calculated dialogue embeddings for both human-annotated and
unlabeled dialogues from the cleaned OS dialogues dataset. For this, we
applied a decaying weight starting from the last turn and took the weighted
average of the turn embeddings of each dialog. We used half decaying, i.e, if
we have a dialogue with turn embeddings $v_{1}$, $v_{2}$, and $v_{3}$, the
final dialogue embedding would be $(4/7)v_{3}+(2/7)v_{2}+(1/7)v_{1}$. 3) Next,
we calculated the cosine-similarity between annotated and unlabeled dialogue
embeddings and ranked the results. 4) Finally, we applied a similarity
threshold and obtained all the unlabeled dialogues with a cosine similarity
that exceeds this threshold and tagged them with the same crowd annotated
class label. Here, we used a threshold of $0.92$.

We could extend the original crowd annotated dialogue dataset by $3,196$ more
dialogues with distantly annotated class labels using the above method. The
selection of parameters, such as the decaying weights, is explained in
Appendix D.

###  3.4 The Fine-grained Emotion Classifier

We use the curated data to train a fine-grained dialogue emotion classifier,
EmoBERT+ (EB+), to annotate the movie dialogues. EB+ consists of a
representation network that uses the BERT architecture, an attention layer
that aggregates all hidden states at each time step, one hidden layer, and one
softmax layer. We use the BERT-base architecture with 12 layers, 768
dimensions, 12 heads, and 110M parameters as the representation network. We
initialize it with weights from the pre-trained language model RoBERTa Liu et
al. (2019), which is proven to have superior performance than pre-trained
BERT. We feed in a dialogue turn along with the preceding context in the
reverse order as input to the representation network. To give more importance
to the dialogue turn for which prediction has to be made and the turns that
immediately precede it, we multiply the token embeddings belonging to each
turn by a decreasing weight factor. As depicted in Figure 2, for a given
token, its input representation is constructed by summing the corresponding
token embedding multiplied by the weighting factor and its position embedding.

Figure 2: EmoBERT+ architecture. Model |  Training data | Precis. | Recall | F1 | Acc.
---|---|---|---|---|---
FastText (base) |  dialogues from AMT (5K) | $00.78$ | $04.73$ | $01.32$ | $06.36$
FastText (1st iter.) |  5K + Similar dialogues from SBERT (3K) | $07.18$ | $10.28$ | $05.65$ | $11.20$
FastText (2nd iter.) |  5K + 3K + Self-labeled dialogues (4K) | $09.76$ | $11.19$ | $07.37$ | $12.04$
FastText (2nd iter., extended) |  5K + 3K + 4K + Similar self-labeled dialogues from SBERT (2K) | 13.23 | 15.06 | 10.89 | 15.53
EB+ (base) |  dialogues from AMT (5K) | $63.33$ | $63.94$ | $63.28$ | 65.17
EB+ (1st iter.) |  5K + Similar dialogues from SBERT (3K) | $63.55$ | $63.54$ | $62.92$ | $64.55$
EB+ (2nd iter.) |  5K + 3K + Self-labeled dialogues (4K) | $63.96$ | $64.27$ | $63.61$ | $64.88$
EB+ (2nd iter., extended) |  5K + 3K + 4K + Similar self-labeled dialogues from SBERT (2K) | 64.11 | 64.59 | 63.86 | $65.00$
Table 5: Performance results of the FastText (baseline) and EB+ classifiers
over the semi-supervised learning iterations. All scores are reported on the
OS test dataset. F1-score reported here is the macro-F1 score.

We split the original crowd annotated dialogues to $60\%$ training, $20\%$
validation, and $20\%$ testing, i.e., $5,390$ dialogues for training, $1,785$
for validation, and $1,777$ for testing and trained several models of EB+
iteratively following a semi-supervised learning approach. First, EB+ base was
trained only on the crowd annotated dialogues from AMT. Next, we trained a
second model, EB+, 1st iteration, by combining the crowd annotated dialogues
with the similar dialogues found using SBERT.

After the 1st iteration, we applied the classifier on unlabeled turns in
cleaned OS dialogues and obtained the top-$100$ high confidence turns in each
category along with their preceding context. It resulted in $4,100$ self-
labeled dialogues, which were incorporated into the training dataset. Next, we
trained EB+, 2nd iteration with the training dataset extended with the self-
labeled dialogues. Finally, another model EB+, 2nd iteration extended was
trained by re-applying the SBERT approach to obtain similar dialogues to the
self-labeled dialogues that were newly incorporated in the 2nd iteration. In
this stage, we could incorporate $2,118$ additional dialogues having cosine-
similarity above a threshold of $0.9$. We trained each model using a batch
size of $256$ for $10$ epochs and obtained the one giving the lowest
classification loss in the validation set. We used $100$ as the maximum length
of input tokens and a learning rate of $2\times 10^{-5}$.

To compare EB+’s performance against a standard baseline, we followed the same
semi-supervised learning steps to train a FastText classifier with default
configuration Joulin et al. (2017). In this, the dialogue turn for which the
prediction has to be made was concatenated with preceding dialogue turns and
fed as one input sequence to the FastText classifier. Finally, this
experiment’s best performing model was used to annotate all the turns in the
OSED dataset automatically.

##  4 Results

The EB+ classifier’s performance results over different iterations on the OS
test dataset having $1,777$ dialogues distributed equally across classes are
denoted in Table 5. The final model EB+, 2nd iteration extended, trained on
14K labeled dialogue turns altogether, performs the best as denoted by
precision, recall, and macro-F1 metrics. Even the accuracy of the final model
does not deviate significantly from previous iterations. It could be noted
that compared to FastText, the semi-supervised learning iterations could only
yield minor improvement in EB+ since it already performs significantly well
when trained only on the 5K crowd-annotated dialogue turns. The performance
improvement in FastText is significant over the semi-supervised learning
iterations; however, it is still notably low compared to EB+. Performance
comparison of the best performing EB+ model with the state-of-the-art dialogue
emotion classifiers is included in Appendix F. A direct comparison with the
state-of-the-art cannot be made since existing classifiers can only predict a
limited number of emotion labels.

##  5 OSED Quality Analysis

(a) Beginning emotion: Joyful

(b) Beginning emotion: Surprised

(c) Beginning emotion: Sad

(d) Beginning emotion: Angry

Figure 3: Emotion-intent flow visualizations in the OSED dataset.

OSED with 1M automatically annotated dialogues could be readily used in
applications such as developing chatbots that could generate controlled
responses conditioned on emotions and intents. This requires an evaluation of
the annotation quality of the resultant dataset. Though the accuracy of the
EB+ classifier speaks for OSED annotation quality, we performed a visual
analysis on the emotion-intent flow patterns in OSED to inspect if they agree
with human social interaction patterns.

Figure 3 shows emotion-intent flow patterns that occur in OSED when the
beginning dialogue emotion is one of joyful, surprised, sad and angry,
respectively. We limit the patterns to 4 dialogue turns for ease of
visualization. Visualizations for further emotions are included in Appendix I.
According to the visualizations, the most frequent intent that immediately
follows a given emotion, irrespective of the emotion being positive or
negative, is questioning. This observation is on par with claims made by See
et al. See et al. (2019) and Welivita and Pu Welivita and Pu (2020). Intents
acknowledging and agreeing also take prominence in turns that follow emotional
dialogue turns. Another important observation is that emotional dialogue turns
are frequently followed by turns containing similar emotions. For example,
joyful followed by emotions excited, joyful, and content, which belong to the
same sector in the Plutchik’s emotion wheel Plutchik (1984). This is a bit
contradictory to observations made by Welivita and Pu Welivita and Pu (2020)
in EmpatheticDialogues, in which they state the listener responses are mostly
neutral and contain specific intents such as questioning, agreeing or
acknowledging. However, our observation here can be justified since, in
EmpatheticDialogues, the listeners were specifically asked to react calmly and
reassuringly, whereas movie dialogues are more natural and dramatic.

##  6 Discussion

This study proposed an emotional dialogue curation pipeline from movie
subtitles resulting in 1M emotional movie dialogues automatically annotated
with 32 emotions and 9 empathetic response intents. The new response intents
such as Greeting, Declaration, Commanding, Hesistant, and Accusing proposed by
the crowd workers, validate the complexity of listeners’ reactions to emotions
and invoke the need to further conduct analysis in this area. The fine-grained
dialogue emotion classifier we utilized in the annotation process could be
used as a general-purpose classifier for human social conversations. An
F1-score of $64\%$ over $41$ class labels is also significant. Even the top-
performing systems in recent EmoContext, EmotionX 2018, and EmotionX 2019
challenges, which also use pre-trained BERT in their model architecture,
report F1-scores in the range $88.5\%$ (EmotionX-IDEA Huang et al. (2019b) on
EmotionPush dataset) - $76\%$ (FigureEight Xiao (2019) on EmoContext dataset).
Since these scores are reported only on a limited number of emotions, the
F1-score of $64\%$ we report in this paper over $41$ emotion and intent
classes is comparable to the performance of the state-of-the-art. The
visualization of emotion-intent exchange patterns further suggests the
resultant OSED dataset conforms to the emotion-intent exchange patterns in
human social conversations and can be utilized in applications such as
developing chatbots that could learn and respond to emotional prompts.

Implications of this work include 1) help in assessing the response quality of
existing social chatbots at a finer granularity, 2) inform social chatbots
about the desirable responses given an emotional prompt, and 3) help in the
design and development of more controllable and interpretable neural chatbots
Xu et al. (2018). There are some limitations to this approach as well, which
include inaccuracies occurring due to automatic turn and dialogue segmentation
and annotation with only 9 empathetic response intents, whereas in reality,
there can be more and even contrasting intents such as Accusation, and
Disagreement.

##  7 Conclusion

In this work, we curated a large-scale dialogue dataset, OSED, comprising of
1M emotional dialogues from movie subtitles. This dataset is more general-
purpose, larger in size, and contains more fine-grained emotion categories and
empathetic response intents than the existing emotional dialogue datasets. To
facilitate annotation, we developed a dialogue emotion classifier capable of
recognizing 32 fine-grained emotions and 9 empathetic response intents with
significant accuracy. It was trained on movie dialogues initially annotated
using human computation and extended using self-labeling, and sentence
similarity approaches. As future work, we intend to extend the taxonomy of
empathetic response intents using new labels discovered during this process
and utilize the OSED dataset to develop a controllable neural chatbot capable
of generating empathetic responses during social chitchat.

> ## References
>
>   * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts,
> and Christopher D. Manning. 2015\.  A large annotated corpus for learning
> natural language inference.  In _Proceedings of the 2015 Conference on
> Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon,
> Portugal. Association for Computational Linguistics.
>   * Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe
> Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and
> Shrikanth S Narayanan. 2008.  Iemocap: Interactive emotional dyadic motion
> capture database.  _Language resources and evaluation_ , 42(4):335.
>   * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-
> Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual
> similarity multilingual and crosslingual focused evaluation.  In
> _Proceedings of the 11th International Workshop on Semantic Evaluation
> (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for
> Computational Linguistics.
>   * Chatterjee et al. (2019) Ankush Chatterjee, Umang Gupta, Manoj Kumar
> Chinnakotla, Radhakrishnan Srikanth, Michel Galley, and Puneet Agrawal.
> 2019.  Understanding emotions in text using deep learning and big data.
> _Computers in Human Behavior_ , 93:309–317.
>   * Ekman (1992) Paul Ekman. 1992.  An argument for basic emotions.
> _Cognition & emotion_, 6(3-4):169–200.
>   * Ghosal et al. (2019) Deepanway Ghosal, Navonil Majumder, Soujanya Poria,
> Niyati Chhaya, and Alexander Gelbukh. 2019.  DialogueGCN: A graph
> convolutional neural network for emotion recognition in conversation.  In
> _Proceedings of the 2019 Conference on Empirical Methods in Natural Language
> Processing and the 9th International Joint Conference on Natural Language
> Processing (EMNLP-IJCNLP)_ , pages 154–164, Hong Kong, China. Association
> for Computational Linguistics.
>   * Hazarika et al. (2018a) Devamanyu Hazarika, Soujanya Poria, Rada
> Mihalcea, Erik Cambria, and Roger Zimmermann. 2018a.  Icon: Interactive
> conversational memory network for multimodal emotion detection.  In
> _Proceedings of the 2018 Conference on Empirical Methods in Natural Language
> Processing_ , pages 2594–2604.
>   * Hazarika et al. (2018b) Devamanyu Hazarika, Soujanya Poria, Amir Zadeh,
> Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018b.
> Conversational memory network for emotion recognition in dyadic dialogue
> videos.  In _Proceedings of the conference. Association for Computational
> Linguistics. North American Chapter. Meeting_ , volume 2018, page 2122. NIH
> Public Access.
>   * Herzig et al. (2016) Jonathan Herzig, Guy Feigenblat, Michal Shmueli-
> Scheuer, David Konopnicki, Anat Rafaeli, Daniel Altman, and David Spivak.
> 2016.  Classifying emotions in customer support dialogues in social media.
> In _Proceedings of the 17th Annual Meeting of the Special Interest Group on
> Discourse and Dialogue_ , pages 64–73.
>   * Hsu et al. (2018) Chao-Chun Hsu, Sheng-Yeh Chen, Chuan-Chun Kuo, Ting-
> Hao Huang, and Lun-Wei Ku. 2018\.  EmotionLines: An emotion corpus of multi-
> party conversations.  In _Proceedings of the Eleventh International
> Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki,
> Japan. European Language Resources Association (ELRA).
>   * Huang et al. (2019a) Chenyang Huang, Amine Trabelsi, and Osmar Zaïane.
> 2019a.  ANA at SemEval-2019 task 3: Contextual emotion detection in
> conversations through hierarchical LSTMs and BERT.  In _Proceedings of the
> 13th International Workshop on Semantic Evaluation_ , pages 49–53,
> Minneapolis, Minnesota, USA. Association for Computational Linguistics.
>   * Huang et al. (2019b) Yen-Hao Huang, Ssu-Rui Lee, Mau-Yun Ma, Yi-Hsin
> Chen, Ya-Wen Yu, and Yi-Shin Chen. 2019b.  Emotionx-idea: Emotion bert–an
> affectional model for conversation.  _arXiv preprint arXiv:1908.06264_.
>   * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and
> Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In
> _Proceedings of the 15th Conference of the European Chapter of the
> Association for Computational Linguistics: Volume 2, Short Papers_ , pages
> 427–431. Association for Computational Linguistics.
>   * Khosla (2018) Sopan Khosla. 2018.  Emotionx-ar: Cnn-dcnn autoencoder
> based emotion classifier.  In _Proceedings of the Sixth International
> Workshop on Natural Language Processing for Social Media_ , pages 37–44.
>   * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao,
> and Shuzi Niu. 2017.  DailyDialog: A manually labelled multi-turn dialogue
> dataset.  In _Proceedings of the Eighth International Joint Conference on
> Natural Language Processing (Volume 1: Long Papers)_ , pages 986–995,
> Taipei, Taiwan. Asian Federation of Natural Language Processing.
>   * Lison and Meena (2016) Pierre Lison and Raveesh Meena. 2016.  Automatic
> turn segmentation for movie & tv subtitles.  In _2016 IEEE Spoken Language
> Technology Workshop (SLT)_ , pages 245–252. IEEE.
>   * Lison et al. (2019) Pierre Lison, Jörg Tiedemann, Milen Kouylekov, et
> al. 2019.  Open subtitles 2018: Statistical rescoring of sentence alignments
> in large, noisy parallel corpora.  In _LREC 2018, Eleventh International
> Conference on Language Resources and Evaluation_. European Language
> Resources Association (ELRA).
>   * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar
> Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin
> Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
> _arXiv preprint arXiv:1907.11692_.
>   * Majumder et al. (2019) Navonil Majumder, Soujanya Poria, Devamanyu
> Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019.
> Dialoguernn: An attentive rnn for emotion detection in conversations.  In
> _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33,
> pages 6818–6825.
>   * Plutchik (1984) Robert Plutchik. 1984.  Emotions: A general
> psychoevolutionary theory.  _Approaches to emotion_ , 1984:197–219.
>   * Poria et al. (2019) Soujanya Poria, Devamanyu Hazarika, Navonil
> Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019.  MELD: A
> multimodal multi-party dataset for emotion recognition in conversations.  In
> _Proceedings of the 57th Annual Meeting of the Association for Computational
> Linguistics_ , pages 527–536, Florence, Italy. Association for Computational
> Linguistics.
>   * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li,
> and Y-Lan Boureau. 2018.  Towards empathetic open-domain conversation
> models: A new benchmark and dataset.  _arXiv preprint arXiv:1811.00207_.
>   * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.
> Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In
> _Proceedings of the 2019 Conference on Empirical Methods in Natural Language
> Processing and the 9th International Joint Conference on Natural Language
> Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association
> for Computational Linguistics.
>   * Schuller et al. (2012) Björn Schuller, Michel Valster, Florian Eyben,
> Roddy Cowie, and Maja Pantic. 2012.  Avec 2012: the continuous audio/visual
> emotion challenge.  In _Proceedings of the 14th ACM international conference
> on Multimodal interaction_ , pages 449–456.
>   * See et al. (2019) Abigail See, Stephen Roller, Douwe Kiela, and Jason
> Weston. 2019.  What makes a good conversation? how controllable attributes
> affect human judgments.  _arXiv preprint arXiv:1902.08654_.
>   * Skerry and Saxe (2015) Amy E Skerry and Rebecca Saxe. 2015.  Neural
> representations of emotion are organized around abstract event features.
> _Current biology_ , 25(15):1945–1954.
>   * Welivita and Pu (2020) Anuradha Welivita and Pearl Pu. 2020.  A taxonomy
> of empathetic response intents in human social conversations.  In
> _Proceedings of the 28th International Conference on Computational
> Linguistics_ , pages 4886–4899.
>   * Xiao (2019) Joan Xiao. 2019.  Figure eight at SemEval-2019 task 3:
> Ensemble of transfer learning methods for contextual emotion detection.  In
> _Proceedings of the 13th International Workshop on Semantic Evaluation_ ,
> pages 220–224, Minneapolis, Minnesota, USA. Association for Computational
> Linguistics.
>   * Xu et al. (2018) Can Xu, Wei Wu, and Yu Wu. 2018.  Towards explainable
> and controllable open domain dialogue generation with dialogue acts.  _arXiv
> preprint arXiv:1807.07255_.
>   * Yeh et al. (2019) Sung-Lin Yeh, Yun-Shao Lin, and Chi-Chun Lee. 2019.
> An interaction-aware attention network for speech emotion recognition in
> spoken dialogs.  In _ICASSP 2019-2019 IEEE International Conference on
> Acoustics, Speech and Signal Processing (ICASSP)_ , pages 6685–6689. IEEE.
>

##  Appendix A Computing the readability of OS dialogues

We followed the following steps in calculating the readability of the
dialogues. The dialogues that scored high in readability were preferred for
the crowd-annotation task since they avoid the overhead of having to read long
and complex dialogues that may exhaust the crowd-worker.

  1. 1.

Build a frequency vocabulary by calculating the token count for all the
dialogues in the cleaned OS dataset.

  2. 2.

For each dialog, aggregate the frequencies of all tokens and take the average
using the following formula, in which $f_{sum}$ is the sum of frequencies of
all tokens, $n_{tokens}$ is the total number of tokens in the dialog, and
$\alpha$ is a constant (set to 87 in our case). The idea behind this is that
difficult to read dialogues contain less frequent words and should result in
less readability.

$f=f_{sum}/(\alpha+n_{tokens})$

  3. 3.

For each dialog, also calculate the percentage of distinct words, say $d$.

  4. 4.

Finally, compute the readability score for each dialogue by taking the
weighted sum of $f$ and $d$. Experimental results showed that the combination
of $f+0.04d$ was giving the best results. We take the combination of both $f$
and $d$ because, if only $f$ is considered, then dialogues that contain a lot
of repetitive tokens can score high in readability, which is undesirable.

##  Appendix B AMT task interfaces

The user interface used to collect labels from the AMT workers is denoted in
Figure 4 and the design of quiz question interface is denoted in Figure 5.

Figure 4: The user interface of the AMT crowd-annotation task. Figure 5: Quiz
question interface design of the AMT crowd-annotation task. Quiz questions
were used to measure the quality of the crowdworkers as well as help them
improve their annotation quality by understanding their mistakes.

##  Appendix C New intent categories discovered in the AMT task

Table 6 lists some of the frequent classes proposed by the crowd workers
during the AMT crowd-annotation task along with corresponding examples.

New category |  Example dialogue
---|---
Greeting |  |  \- Hello, Florelle.
---
Declaration |  |  \- Claire, what ’s happening?
---
\- I saw him. He was here.
Commanding |  |  \- Power of unity!
---
\- Alpha Team, it’s a combined operation. After putting on winter military
gear, come to 464 airport by 21:00 today.
Hesistant |  |  \- Now that’s more like it.
---
\- Where have you been all night?
\- I just … went out for a walk.
Accusing |  |  \- I don’t want anything to do with your fucked-up, exploitative, supplying God knows how many black markets school. OK?
---
-You should be ashamed, making money out of a captive audience.
Table 6: New categories proposed by the crowd workers along with corresponding
examples.

##  Appendix D Choice of hyper-parameters and additional training details
regarding EB+

The choice of 5 seconds to separate dialogues is based on a histogram of time
intervals between adjacent subtitle blocks in the OpenSubtitles corpus, which
is denoted in Figure 6. As it can be observed in the histogram, most of the
time gaps fall below 3 seconds. A clear drop in count was observed between 3-5
seconds. Therefore, we chose 5 seconds as the time interval to separate
dialogues.

Figure 6: Histogram of time intervals between adjacent subtitle blocks in the
OpenSubtitles corpus.

Using decreasing weights for context utterances is based on the intuition that
in human dialogues, more attention is paid to the most recent utterances in
dialogue history. This idea is backed up by time-decay functions used in
neural dialogue understanding approaches See et al. (2019). We conducted an
ablation study with and without using decreasing weights in the model.
Performance of the unweighted models was lower than the performance of
weighted models yielding final F1 scores of $63.44$ and $64.86$ for unweighted
and weighted models, respectively.

All the experiments were conducted on a machine with<EMAIL_ADDRESS>256 GB
RAM, 2x240 GB SSD, and 2xGPU (NVIDIA Titan X Maxwell).

##  Appendix E Distribution of labels in the datasets used to train the EB+
classifier

Figure 7 shows the distribution of emotions and intents in the crowd-annotated
9K dialogues used to train the EB+ classifier, while Figure 8 shows the
distribution of labels extended by considering dialogue similarity using the
Siamese-BERT approach Reimers and Gurevych (2019).

Figure 7: Distribution of emotions and intents in the AMT annotated 9K OS
dialogues. Figure 8: Distribution of emotions and intents in the 2K similar
self-labeled dialogues incorporated in the EB+ 2nd iteration, extended.

##  Appendix F Comparison of the performance of the EB+ classifier with the
state-of-the-art dialogue emotion classifiers

Table 7 compared the performance of the EB+ classifier with several state-of-
the-art dialogue emotion classifiers. However, a direct comparison cannot be
made since existing classifiers are trained to predict only a limited set of
emotion labels.

Classifier |  Dataset |  No. of labels | F1 | Acc.
---|---|---|---|---
AR Khosla (2018) |  EmotionLines |  4 Emotion labels | $-$ | Friends: $62.50$
|  dataset Hsu et al. (2018) |  |  | EmotionPush: $62.48$
CMN Hazarika et al. (2018b) |  IEMOCAP dataset Busso et al. (2008) |  6 Emotion labels | $56.13$ | $56.56$
ICON Hazarika et al. (2018a) |  |  | $57.90$ | $58.30$
IAAN Yeh et al. (2019) |  |  | $-$ | $64.70$
Dialog-RNN Majumder et al. (2019) |  IEMOCAP Busso et al. (2008) and AVEC Schuller et al. (2012) datasets |  IEMOCAP: 4 Emotion labels; AVEC: 4 dimentional emotion labels | $62.75$ | $63.40$
Dialog-GCN Ghosal et al. (2019) |  IEMOCAP Busso et al. (2008), AVEC Schuller et al. (2012), and MELD Poria et al. (2019) datasets |  IEMOCAP: 4 Emotion labels; AVEC: 4 dimentional emotion labels; MELD: 7 Emotion labels | $64.18$ | $65.25$
ANA Huang et al. (2019a) |  EmoContext dataset Chatterjee et al. (2019) |  4 Emotion labels | $82.86$ | $-$
EB+ (2nd iter., extended) |  OS dialogue dataset |  32 Emotion labels + 9 Intent labels | $63.86$ | $65.00$
Table 7: Comparison of the performance results of the EB+ classifier (2nd
iter., extended) with performance of the state-of-the-art dialogue emotion
classifiers. F1-score reported here is the macro-F1 score.

##  Appendix G Distribution of emotions and intents in the OSED dataset

Figure 9 shows the distribution of emotion and intent labels in the resultant
1M OSED dataset. It could be noted that intent categories take prominence over
individual emotion classes, which is expected since one or more intents could
be utilized when responding to emotions in dialogue.

Figure 9: Distribution of emotions and intents in the OSED dataset. The
numbers are in the log scale.

##  Appendix H Example dialogues from OSED along with EB+ labels and
confidence scores.

Table 8 denotes some example dialogues taken from the OSED dataset along with
EB+ labels predicted for each dialogue turn and their confidence values.

\- I have to catch a flight (Anticipating, 0.85)
---
\- Okay , Robert (Acknowledging, 0.73)
\- Completely empty . (Lonely, 0.73)
\- Cheers . How hungry I was . Dig in . (Encouraging, 0.78)
\- Thank you for coming , Lleó . Thanks for coming to see me on your birthday.
(Grateful, 0.77)
\- I appreciate that . Gavina has broken her left foot and you ’ve come to
visit her . (Grateful, 0.97)
\- I ’ve been wanting to tell you something . I ’d admire you so much The
stand you ’re taking . You didn ’t take the easy way out . (Impressed, 0.88)
\- Not setting up one of your friends . I couldn ’t do what you did . So looks
like you ’re the one .. .. teaching me . What real character and integrity is
all about . I love you son . (Proud, 0.43)
\- Could you please hurry up ? (Suggesting, 0.69)
\- I ’m doing my best , miss . I have a feeling Elena will be here any minute
. (Hopeful, 0.48)
\- Let her ! (Encouraging, 0.51) \- We ’re not doing anything wrong. (Content,
0.47)
\- Done ! Wonderful ! (Acknowledging, 0.69)
\- Here is your money . My sister will show you to the door . (Trusting, 0.62)
Table 8: Example dialogues from the OSED dataset along with EB+ annotations
and confidence scores.

##  Appendix I Visualizations of emotion-intent flow patterns in the OSED
dataset

Figure 10 shows emotion-intent exchanges between adjacent dialogue turns in
the OSED dataset. Figure 11 denotes some additional emotion-intent flow
patterns in the OSED dataset, when the beginning dialogue emotion is one of
anticipating, trusting, afraid or disgusted. They could be used to further
justify the observations made in Section 5.

Figure 10: Emotion-intent exchanges between adjacent dialogue turns in the
OSED dataset.

(a) Beginning emotion: Anticipating

(b) Beginning emotion: Trusting

(c) Beginning emotion: Afraid

(d) Beginning emotion: Disgusted

Figure 11: Emotion-intent flow visualizations in the OSED dataset when the
beginning dialogue emotion is one of anticipating, trusting, afraid or
disgusted.


$\displaystyle=\frac{2}{\sqrt{3}}\cos\theta\Big{(}1-\sin\theta\Big{)}+b_{0}^{2}\cot\psi_{0}\sin\theta~{},$
(231) $\displaystyle\mathcal{C}_{2}$
$\displaystyle=-\frac{2}{\sqrt{3}}\cos\theta\Big{(}1+\sin\theta\Big{)}-b_{0}^{2}\cot\psi_{0}\sin\theta~{}.$
(232)

The constants $\psi_{0}$ and $\theta$ are related by

$\cot\psi_{0}=\frac{2}{\sqrt{3}}\left(\frac{1}{b_{0}^{2}}\sec\theta+\frac{1}{b_{0}^{2}}\tan\theta\right)~{}.$
(233)

Through direct substitution, one can easily check that the above profile
satisfies all SUGRA equations to leading order in $\lambda$ and
$\rho_{c}^{2}$.

## References

  * (1) J. M. Maldacena and H. S. Nastase, The Supergravity dual of a theory with dynamical supersymmetry breaking, JHEP 09 (2001) 024, [hep-th/0105049].
  * (2) S. Kachru, J. Pearson, and H. L. Verlinde, Brane / flux annihilation and the string dual of a nonsupersymmetric field theory, JHEP 06 (2002) 021, [hep-th/0112197].
  * (3) I. R. Klebanov and S. S. Pufu, M-Branes and Metastable States, JHEP 08 (2011) 035, [arXiv:1006.3587].
  * (4) S. Kachru, R. Kallosh, A. D. Linde, and S. P. Trivedi, De Sitter vacua in string theory, Phys. Rev. D 68 (2003) 046005, [hep-th/0301240].
  * (5) S. Kachru, R. Kallosh, A. D. Linde, J. M. Maldacena, L. P. McAllister, and S. P. Trivedi, Towards inflation in string theory, JCAP 10 (2003) 013, [hep-th/0308055].
  * (6) I. Bena, A. Puhm, and B. Vercnocke, Non-extremal Black Hole Microstates: Fuzzballs of Fire or Fuzzballs of Fuzz ?, JHEP 12 (2012) 014, [arXiv:1208.3468].
  * (7) I. R. Klebanov and M. J. Strassler, Supergravity and a confining gauge theory: Duality cascades and chi SB resolution of naked singularities, JHEP 08 (2000) 052, [hep-th/0007191].
  * (8) R. C. Myers, Dielectric branes, JHEP 12 (1999) 022, [hep-th/9910053].
  * (9) I. Bena, M. Graña, S. Kuperstein, and S. Massai, Giant Tachyons in the Landscape, JHEP 02 (2015) 146, [arXiv:1410.7776].
  * (10) N. Nguyen, Comments on the stability of the KPV state, JHEP 11 (2020) 055, [arXiv:1912.04646].
  * (11) I. Bena, M. Grana, and N. Halmagyi, On the Existence of Meta-stable Vacua in Klebanov-Strassler, JHEP 09 (2010) 087, [arXiv:0912.3519].
  * (12) S. S. Gubser, Curvature singularities: The Good, the bad, and the naked, Adv. Theor. Math. Phys. 4 (2000) 679–745, [hep-th/0002160].
  * (13) J. Blåbäck, U. H. Danielsson, D. Junghans, T. Van Riet, and S. C. Vargas, Localised anti-branes in non-compact throats at zero and finite $T$, JHEP 02 (2015) 018, [arXiv:1409.0534].
  * (14) S. Massai, A Comment on anti-brane singularities in warped throats, arXiv:1202.3789.
  * (15) I. Bena, M. Grana, S. Kuperstein, and S. Massai, Anti-D3 Branes: Singular to the bitter end, Phys. Rev. D 87 (2013), no. 10 106010, [arXiv:1206.6369].
  * (16) F. F. Gautason, D. Junghans, and M. Zagermann, Cosmological Constant, Near Brane Behavior and Singularities, JHEP 09 (2013) 123, [arXiv:1301.5647].
  * (17) D. Cohen-Maldonado, J. Diaz, T. van Riet, and B. Vercnocke, Observations on fluxes near anti-branes, JHEP 01 (2016) 126, [arXiv:1507.01022].
  * (18) D. Cohen-Maldonado, J. Diaz, and F. F. Gautason, Polarised antibranes from Smarr relations, JHEP 05 (2016) 175, [arXiv:1603.05678].
  * (19) R. Emparan, T. Harmark, V. Niarchos, and N. A. Obers, World-Volume Effective Theory for Higher-Dimensional Black Holes, Phys. Rev. Lett. 102 (2009) 191301, [arXiv:0902.0427].
  * (20) R. Emparan, T. Harmark, V. Niarchos, and N. A. Obers, Essentials of Blackfold Dynamics, JHEP 03 (2010) 063, [arXiv:0910.1601].
  * (21) J. Armas, J. Gath, V. Niarchos, N. A. Obers, and A. V. Pedersen, Forced Fluid Dynamics from Blackfolds in General Supergravity Backgrounds, JHEP 10 (2016) 154, [arXiv:1606.09644].
  * (22) J. Armas, N. Nguyen, V. Niarchos, N. A. Obers, and T. Van Riet, Metastable Nonextremal Antibranes, Phys. Rev. Lett. 122 (2019), no. 18 181601, [arXiv:1812.01067].
  * (23) B. Michel, E. Mintun, J. Polchinski, A. Puhm, and P. Saad, Remarks on brane and antibrane dynamics, JHEP 09 (2015) 021, [arXiv:1412.5702].
  * (24) I. Bena, A. Buchel, and O. J. C. Dias, Horizons cannot save the Landscape, Phys. Rev. D 87 (2013), no. 6 063012, [arXiv:1212.5162].
  * (25) I. Bena, J. Blaback, U. H. Danielsson, and T. Van Riet, Antibranes cannot become black, Phys. Rev. D 87 (2013), no. 10 104023, [arXiv:1301.7071].
  * (26) G. S. Hartnett, Localised Anti-Branes in Flux Backgrounds, JHEP 06 (2015) 007, [arXiv:1501.06568].
  * (27) J. Blåbäck, F. F. Gautason, A. Ruipérez, and T. Van Riet, Anti-brane singularities as red herrings, JHEP 12 (2019) 125, [arXiv:1907.05295].
  * (28) I. Bena, G. Giecold, and N. Halmagyi, The Backreaction of Anti-M2 Branes on a Warped Stenzel Space, JHEP 04 (2011) 120, [arXiv:1011.2195].
  * (29) J. Armas, N. Nguyen, V. Niarchos, and N. A. Obers, Thermal transitions of metastable M-branes, JHEP 08 (2019) 128, [arXiv:1904.13283].
  * (30) S. Bhattacharyya, V. E. Hubeny, S. Minwalla, and M. Rangamani, Nonlinear Fluid Dynamics from Gravity, JHEP 02 (2008) 045, [arXiv:0712.2456].
  * (31) J. Camps and R. Emparan, Derivation of the blackfold effective theory, JHEP 03 (2012) 038, [arXiv:1201.3506]. [Erratum: JHEP 06, 155 (2012)].
  * (32) D. Gorbonos and B. Kol, A Dialogue of multipoles: Matched asymptotic expansion for caged black holes, JHEP 06 (2004) 053, [hep-th/0406002].
  * (33) T. Harmark, Small black holes on cylinders, Phys. Rev. D 69 (2004) 104015, [hep-th/0310259].
  * (34) M. Cvetic, G. W. Gibbons, H. Lu, and C. N. Pope, Ricci flat metrics, harmonic forms and brane resolutions, Commun. Math. Phys. 232 (2003) 457–500, [hep-th/0012011].
  * (35) I. Bena and S. Kuperstein, Brane polarization is no cure for tachyons, JHEP 09 (2015) 112, [arXiv:1504.00656].
  * (36) R. Gregory and R. Laflamme, Black strings and p-branes are unstable, Phys. Rev. Lett. 70 (1993) 2837–2840, [hep-th/9301052].
  * (37) J. L. Hovdebo and R. C. Myers, Black rings, boosted strings and Gregory-Laflamme, Phys. Rev. D 73 (2006) 084013, [hep-th/0601079].
  * (38) J. E. Santos and B. Way, Neutral Black Rings in Five Dimensions are Unstable, Phys. Rev. Lett. 114 (2015) 221101, [arXiv:1503.00721].
  * (39) L. Lehner and F. Pretorius, Black Strings, Low Viscosity Fluids, and Violation of Cosmic Censorship, Phys. Rev. Lett. 105 (2010) 101102, [arXiv:1006.5960].
  * (40) P. Figueras, M. Kunesch, and S. Tunyasuvunakool, End Point of Black Ring Instabilities and the Weak Cosmic Censorship Conjecture, Phys. Rev. Lett. 116 (2016), no. 7 071102, [arXiv:1512.04532].
  * (41) J. Armas and E. Parisini, Instabilities of Thin Black Rings: Closing the Gap, JHEP 04 (2019) 169, [arXiv:1901.09369].
  * (42) J. Polchinski, String theory. Vol. 2: Superstring theory and beyond.  Cambridge Monographs on Mathematical Physics. Cambridge University Press, 12, 2007.
  * (43) C. P. Herzog, I. R. Klebanov, and P. Ouyang, Remarks on the warped deformed conifold, in Modern Trends in String Theory: 2nd Lisbon School on g Theory Superstrings Lisbon, Portugal, July 13-17, 2001, 2001.  hep-th/0108101.
  * (44) R. Minasian and D. Tsimpis, On the geometry of nontrivially embedded branes, Nucl. Phys. B 572 (2000) 499–513, [hep-th/9911042].