# Empathic Conversations: A Multi-level Dataset of Contextualized
Conversations

Damilola Omitaomu1, Shabnam Tafreshi211footnotemark: 1, Tingting Liu3,1, Sven
Buechel4,
Chris Callison-Burch1, Johannes Eichstaedt5, Lyle Ungar1, João Sedoc6
1University of Pennsylvania; 2University of Maryland:ARLIS; 3National
Institute on Drug Abuse;
4Friedrich-Schiller-Universität Jena; 5Stanford University; 6New York
University
These authors equally contributed to this work. Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

Empathy is a cognitive and emotional reaction to an observed situation of
others. Empathy has recently attracted interest because it has numerous
applications in psychology and AI, but it is unclear how different forms of
empathy (e.g., self-report vs counterpart other-report, concern vs. distress)
interact with other affective phenomena or demographics like gender and age.
To better understand this, we created the Empathic Conversations dataset of
annotated negative, empathy-eliciting dialogues in which pairs of participants
converse about news articles. People differ in their perception of the empathy
of others. These differences are associated with certain characteristics such
as personality and demographics. Hence, we collected detailed characterization
of the participants’ traits, their self-reported empathetic response to news
articles, their conversational partner other-report, and turn-by-turn third-
party assessments of the level of self-disclosure, emotion, and empathy
expressed. This dataset is the first to present empathy in multiple forms
along with personal distress, emotion, personality characteristics, and
person-level demographic information. We present baseline models for
predicting some of these features from conversations.

##  1 Introduction

Figure 1: A schematic of data collection. First, (Step 0) we collect the
demographic and personality information of participants using a survey, then
(Step 1) each participant writes an essay about the news article and submits
the Batson survey Batson et al. (1987) to assess the self-reported level of
empathy and distress the person felt toward the article. Next, (Step 2) two
participants converse about an article, and at the end of their conversation,
(Step 3) they rate other-report counterpart perceived empathy. Finally, (Step
4) other annotators (third-person assessment) label the conversational turns
for Empathy, Self-Disclosure, Emotion, and Emotion Polarity, and dialogue
acts.

Humans are an irreducibly social species. The complex social environment
requires us to quickly and accurately process cues we received and
correspondingly generate reactions (Preston and De Waal, 2002). Affective
states (embodied feelings, short term emotions, and longer-term moods) are
particularly potent contextual cues and elements of the human experience, as
they substantially impact almost every phase of our cognitive functioning and
social interactions, including attention, perception, memory, and behavioral
reactions. Thus, when interacting with conversational agents, their ability to
understand and interact in emotionally appropriate ways has become more
important to facilitate a successful social interaction, as its the basis for
humans to feel seen, understood, and generate trust. One important way to
understand and translate others’ emotional expression is empathy.

Designing conversational agents informed by and responsive to empathy is
important to effectively communicate with users and thus emerging area of
research. For example, empathy is critical for clinical applications of agents
such as automated behavioral therapy agents for self-destructive and unhealthy
behaviors (Fitzpatrick et al., 2017). Many studies show that effective
gesturing of robots should include natural language understanding of empathy
Fung (2015). In the field of embodied and virtual agents, empathy is critical
and actively studied (McQuiggan and Lester, 2007; Paiva et al., 2017; Yalcin
and DiPaola, 2018).

However, agents in AI and more generally human-computer interaction, have yet
to successfully implement this complex emotional-motivational state. This is
in part due to the lack of training data sets in which empathy can be modelled
in the full complexity with which humans encounter it.

In this paper, we present a novel dataset that includes dyadic (two person)
text conversations of crowd workers about news articles, thus complementing
the first-person statements of the Empathic Reactions data by Buechel et al.
(2018).111https://github.com/wwbp/empathic_reactions To obtain a more
comprehensive empathy rating, we assessed our conversation from three
different angles, including self-report empathy assessed from two aspects
(compassion and distress); other-report second-person perceived empathy of
counterpart in the conversation; and third-person turn-level annotation of
empathy, self-disclosure, emotion, and emotional polarity (Figure 1).
Additionally, we added assessments of demographics and personality, which have
been found to be correlated with empathy perception, to provide a rich
psychological constructs. Our Empathic Conversations dataset222The dataset is
available by request. provides a rich psychologically founded dataset with
detailed analysis.333Our research was performed as a registered protocol under
IRB# 826448.

Our contributions are as follows:

  1. 1.

We present an empathetic conversation dataset grounded in responses to news
articles; this includes empathy and distress of each conversant along with
their personality and demographic information.

  2. 2.

We build models to predict self-report empathy and distress; second-person
perceived empathy; and turn-level Empathy, Emotion/Emotional Polarity, and
Self-Disclosure.

  3. 3.

We show the usefulness of the different levels of annotation for predicting
multiple levels of empathy.

The rest of this article is structured as follows: Section 2 describes the
related resources and modeling of empathy and distress; Section 3 explains the
dataset statistics, annotation process, and challenges; Section 4 provides
deep-learning models for empathy and distress; Section 5 describes the
findings in the annotation process and the models we built for empathy and
distress; Section 7 concludes this study and suggests future directions;
Section 8 describes the ethical factors we considered during the annotation
process.

##  2 Related Work

Empathy:  To build and evaluate the empathic conversations between human and
AI agents, how human produce and process empathy should be aware. While
conceptualized slightly differently by different theorists, emotional empathy
includes two different types of processes: process of responding to others’
distress, and the process of experiencing the feelings on behalf of the
observed person (Goubert et al., 2005; Chen, 2018). The latter one was usually
termed as ‘empathic concerns’ and here we use interchangeably with ‘empathy’
in our study Lin and McFatter (2012). For the responses to distress, it
involves the representations of self and other in generating the a similar
(negative) affective reactions (e.g., increased personal distress evoked by
others’ distress); for the empathy, it is a feeling of concern, sympathy,
warm, tenderness, and compassionate (e.g., feel others’ feelings (Buechel et
al., 2018; Lin and McFatter, 2012)). To assess these two important
underpinnings of empathy in human-AI interaction, we utilized a first-person
survey, the Batson’s Empathic Concern – Personal Distress Scale Batson et al.
(1987) (as suggested in Buechel et al. (2018)).

Empathy has also been found to be related to personality. For example,
agreeableness and conscientiousness have been found to be the most important
and consistent predictors of empathy in a survey research conducted across
four countries (Melchers et al., 2016). To better understand empathy in human-
AI conversational interactions, we also included self-report personality
traits of participants.

Modeling text-based empathy:  Prior work on modeling text-based empathy
focuses on the following objectives and definitions: Litvak et al. (2016);
Fung et al. (2016) studied the empathetic concern which is to share others’
emotions in conversations; Xiao et al. (2015, 2016); Gibson et al. (2016)
modeled empathy based on the ability of a therapist to adapt to the emotions
of their clients; and Zhou and Jurgens (2020) quantified empathy in
condolences in social media using appraisal theory.

Several works analyze and model text-based empathy: Khanpour et al. (2017)
proposed a neural network (NN) model to predict empathy in health-related
posts. Buechel et al. (2018) modeled empathy and distress in crowdworkers’
written reactions to (empathy-evoking) news articles. In a follow-up study,
Sedoc et al. (2020) developed a mixed-level feed-forward NN model that learned
word-level empathy and distress from the above text-level ratings, resulting
in a lexicon of empathy and distress words. Recent work by Hosseini and
Caragea (2021) studied whether each person in an online health community seeks
or wants to provide empathy. Empathy was also studied in the context of
virtual and embodied agents (Yalcin and DiPaola, 2018; Paiva et al., 2017).

Dialogue Systems with Empathy Capabilities: To date, most dialogue systems
with empathy capabilities are trained on conversations grounded in personal
situations in order for agents to learn to seem more engaging and empathetic.

The Empathetic Dialogues (ED) dataset (Rashkin et al., 2019) consists of 25k
personal conversations. Each dialogue is grounded in specific emotional
situations where a speaker feels a certain emotion towards a circumstance and
receives empathetic responses from the other speaker. ED has served as a
foundational dataset for building more empathetic dialog models (e.g., Lin et
al., 2019; Majumder et al., 2020; Li et al., 2020; Smith et al., 2020; Gao et
al., 2021). There have been several efforts to increase the size of ED with
filtered found data. Welivita et al. (2021) curated a ‘silver-standard’
dataset Emotional Dialogues in OpenSubtitles (EDOS) of 1M dialogues annotated
in semi-automated way into nine emotion classes. The persona-based empathetic
conversation (PEC) dataset is mined from Reddit comments resulting in 355k
empathetic conversations with persona information (Zhong et al., 2020).

Another set of work aimed to create a dataset that is motivated by
psychological theory of empathy. Sharma et al. (2020) introduced the EmPathy
In Text-based, asynchrOnous MEntal health conversations (EPITOME) dataset in
an effort to detect empathy in textual conversations in a theoretically-
grounded way. This dataset was used in subsequent work to rewrite counselor
text in a more empathetic and effective manner (Sharma et al., 2022).

Our dataset is akin to these datasets in that we capture empathy in
conversations; however, we ground these conversations by news articles and
capture multiple views and dimensions of empathy. Similar to the EPITOME
dataset, our EC dataset is motivated by psychological theory.

Several works are grounded in personal context to make agents more engaging
for users (Li et al., 2016; Vijayakumar et al., 2016; Mazaré et al., 2018).
They often use another emotion-annotated dataset is the DailyDialog (DD)
dataset (Li et al., 2017). It consists of daily communications crawled from
different websites used for English learners to practice English dialog in
daily life. DD is annotated manually with emotion and the purpose of the
communication.

##  3 Data Acquisition and Annotation Methods

In our work, we study how a range of personality traits affect conversations,
specifically empathetic conversations. Our dataset consists of 500
conversations between crowd workers chatting through a text interface about a
selection of articles from Buechel et al. (2018). We use the top 100 articles
of negative events which elicited the most empathy and personal distress. We
note that this study analyzes the negative empathy, in addition, we cannot
measure whether our annotators’ reactions were natural/wild or made up. They
were aware that they would read negative events, but they did not know the
event topic. Hence, wild or made up reactions are possible. We quantify the
empathy in conversations that are grounded in these news articles. In the
following subsections, we explain the collection procedure (Figure 1) for our
Empathic Conversations dataset. Table 1 details the total number of
annotations.

###  3.1 Data Collection and Annotation Methodology

#### Step 0: Recruitment and Demographic / Personality Information

The acquisition of participants was set up as a crowdsourcing task on
MTurk.com pointing to a Qualtrics questionnaire. We collected demographic and
personality information, including the widely used Big Five (OCEAN)
personality traits John et al. (1999)444We use the Ten Item Personality
Inventory (TIPI; Gosling et al., 2003). and Interpersonal Reactivity Index
(IRI; Davis, 1980). After the participants filled out background information
dealing with demographics and personality, they then read a random selection
of five news articles selected from the topmost 100 empathic articles. After
reading each news article, each participant was then asked to rate their level
of empathy and distress before summarizing their thoughts and feelings about
the article. Each message was reviewed manually to filter out the responses
that deviated from the task. Of the 110 workers that did the HIT, 92 workers
performed it appropriately555Completed the survey and wrote proper essays and
did not randomly fill out surveys. This is the same criteria as Buechel et al.
(2018). and were offered a qualification to participate in the second
crowdsourcing task on MTurk.

#### Step 1: Reading News Articles

Before every conversation, each pair of crowd workers were asked to read a
news article from the topmost empathy eliciting articles of negative events
from Buechel et al. (2018).666Available at
https://drive.google.com/file/d/1A-7XiLxqOiibZtyDzTkHejsCtnt55atZ/view. The
annotation process was set up as a task on Amazon Mechanical Turk. Workers
were grouped in pairs. For each of the 100 articles we collected 5
conversations.

#### Step 2: Essay and Batson Survey

After reading the article, crowd workers were asked to write an essay (limit
300-800 characters) just as in Buechel et al. (2018). During this phase,
participants rated their empathy and distress level using the Batson scale and
summarized their thoughts and feelings through a Qualtrics survey.

#### Step 3: The Conversation

Next, participants were asked to express their empathy towards the article in
an online text conversation with each other (Figure 1). We collected dialogues
using the ParlAI platform to interact with Amazon Mechanical Turk
participants, who were paired together to have a $\geq$ 15 turn conversation
about the article. We provide a snapshot of an example news article and crowd
workers’ conversation (Figures 6 and 5 in Appendix A). Workers were paid $1
per HIT completed.

#### Step 4: Other-report Empathy Rating

Further, we obtained other-report empathy by asking each annotator to rate the
level and nature of empathy they perceived in each other (second-person) on a
scale of 1-7. Other-report counterparty empathy is collected after the
conversation has been completed.

#### Step 5: Turn Level Annotations of Conversations

Next, we collected third-person annotations of every conversational utterance
from crowd workers. Turn level annotations on the collected conversations were
conducted using Amazon Mechanical Turk and the same workers that participated
in the collection of the conversations. The workers were asked to analyze the
following aspects of each turn: Empathy, Emotion, Emotional Polarity, and
Self-Disclosure. In Appendix A, Figure 4 illustrates a snapshot of the
interface we presented to the workers. In each HIT, workers were asked to
annotate only one aspect (e.g., only Empathy) for an entire conversation.
Workers were paid $0.25 for each HIT completed with a $0.75 bonus for good
work.

Finally, 54 conversations (1,400 out of 5821 individual turns) were labeled by
dialogue act, by 1 annotator for each turn, using the annotation manual from
Jurafsky and Shriberg (1997). These were sufficiently difficult to crowdsource
such that we had research assistants annotate the turns. After training, 92%
agreement was reached on annotations.777The remaining 8% were ambiguous.
Subsequently, the conversational turns were coded without redundancy; however,
any unclear situations were discussed. Many turns for an individual contained
multiple dialog acts. As expected, 30.5% of utterances contained opinion
statements, 17.4% included agreement, and 7.3% were factual statements.

###  3.2 Task Implementation

The main issue during this stage was the wait time for workers to be matched
with other workers. We implemented several solutions to improve wait time
efficiency. First, we sent out specific times when HITs would be posted and
then noted which 10 minute intervals workers should join the HIT. If the wait
time exceeded the 10 minutes to match with someone, we compensated the workers
for the wait times. Despite all these efforts to efficiently use worker’s
time, there were still cases where the wait time exceeded 10 minutes, hence we
had to dismiss those workers. Second, we implemented a sign-up sheet that
would allow us to specifically post HITs when most workers stated they would
be available. Only 25 / 92 workers filled out this sign-up sheet and there was
a direct correlation between whether a worker joined outside the specified
times and whether they completed the sheet. Additionally, based on feedback
from early runs, we reduced the minimum number of required conversational
turns by workers to 15 turns. We offered bonuses to incentivize longer turns:
15-20 turns a $1.50 bonus and 20+ turns a $2 bonus.

The main challenge with collecting Turn Level Annotations was to find the best
description for the categories to offer a more uniform understanding, since
workers had different viewpoints on the meaning of Empathy and Self-Disclosure
when annotating. The workers who completed the Turn Level Annotation were the
same as the ones who participated in the initial conversation
collection.888$<$ 3% of turn-level annotations are by conversants in the
conversation and can easily be removed without effect, since turn-level
annotations are 3-way redundantly annotated.

###  3.3 Dataset Statistics

The Hub |
---|---
New Articles | 100
Conversations | 500
The Spokes |
Person-level |
Personality & Demographic Info | 79
Post-article /pre-conversation |
Essays - Batson (Empathy/Distress) | 1,000
Post-conversation Perceived Empathy | 1,000
Turn level Annotations |
(Empathy, Self-Disclosure, Emotion) | 5,821
Turn - level dialog Acts | 1,400

Table 1: Corpus statistics detailing the number of annotations.

In this subsection, we detail the basic statistics of the collected dataset
(Table 1 and Figure 2). This dataset consists of 500 conversations that each
took an average of 30 minutes (min=12, max=65) to be completed. 75 US-based
Mechanical Turk workers participated in this study. Each worker was paired by
time priority (i.e. the first 2 workers to accept the HIT were paired). As a
result, the same workers were paired together in multiple conversations. The
75 workers completed an average of 13 conversations (min=1, max=88). The
highest frequency of separate conversations that two workers held together was
11, within the 251 distinct pairs of worker conversations; 4 other worker
pairs had 10 conversations together. The average age of the workers is 35
years old (min=19, max=62) with an average income of 58,000 (min=5,000,
max=165,000). Furthermore, gender was more male-dominated with 44 male workers
and 31 female workers. The workers come from various educational and ethnic
backgrounds with almost half having a 4 year bachelor’s degree and more than
half being White (for further details see Tables 7 and 8 in Appendix C of the
datasheet).

Figure 2: Empathic Conversation dataset combines person-, conversation-, and
turn-level information. Furthermore, there are multiple first-, second-, and
third-person perspectives. Gray shading indicates text components of the
dataset.

Data analysis is based on the qualification survey data. Each worker had to
take the survey before entering into any conversation. The perceived
counterparty empathy score was rated at the end of the conversation. Since the
workers were told to discuss their thoughts and feelings about the article,
they had a lot of freedom to steer the discussion. Each conversation was
manually examined to ensure that the workers were having a valid conversation.
We noticed that the conversations are of high quality with the workers drawing
in personal relations to the topic. One concern was that workers would focus
on the factual information within the articles instead of expressing their
opinions and feelings. Interestingly this did not occur. Another concern was
that the conversation would be controlled by the first speaker and turn into
more of an interrogation, resulting in “plain” responses from the
“interrogated.” We did not observe this in the majority of conversations; both
workers contributed equally to the substance of the conversation; however, we
noticed instances in which the conversations did not have much substance.
Nonetheless, the distribution of empathy scores among the workers was
clustered toward the higher end (Figure 3).

#### Annotator Agreement

Figure 3: Perceived empathy level counts.

The overall inter-annotator Agreement (IAA) measured using Krippendorff’s
alpha (Hayes and Krippendorff, 2007) was 0.492. Table 2 shows the breakdown of
IAA by Empathy, Emotion, Emotional Polarity, and Self-Disclosure. These alpha
statistics are calculated by micro-averaging per conversation, and thus, none
of the individual annotations are more than the overall Krippendorff’s alpha.
Low agreement is normal for fine-grained emotion(-like) annotation problems
because this is a subjective task De Bruyne et al. (2020); Troiano et al.
(2021); Demszky et al. (2020); Davani et al. (2021); however, when using mean
absolute deviation filtering Zhao et al. (2020) and macro Krippendorff’s alpha
computation, all aspect alpha values are above 0.7.999We believe this is
common “alpha hacking” and leave it up to the users of the dataset. Empathy
has the lowest IAA which indicates the variety of this phenomenon among
different personalities (note again that micro-average alphas are expected to
be significantly lower). Emotional Polarity and emotion have the highest IAA.
This is expected because all articles are about negative events. Self-
Disclosure has an average IAA among all these quantities.101010See the
Appendix B for further discussion.

Category | Average IAA
---|---
Empathy | 0.274
Emotion | 0.389
Self-Disclosure | 0.335
Emotional Polarity | 0.442
Overall | 0.492
Table 2: Turn level annotation inter-annotator agreement (IAA) using
Krippendorff’s alpha by category.

###  3.4 Multi-level Correlations Analysis

One of the main contributions of the empathic conversations dataset is that we
can look at interaction effects between various empathy measures, emotion,
demographics, and personality. For all correlation statistical significance
testing, we use a p-value $<0.05$ calculate via bootstrap analysis with
multiple hypothesis testing correction.

First, we asked how are the multiple views of empathy correlated? We found an
asymmetry in that Person 1 (the initiator of the conversation) had a lower
correlation between their actual empathy and the perceived empathy (Pearson
correlation of 0.125) whereas Person 2 had a correlation of 0.35. There is a
similar relationship for the correlation between perceived empathy and first-
person distress where Person 1 distress is correlated to Person 2 perceived
empathy of Person 1 at 0.125 whereas the other way is 0.262. This low
correlation is consistent with prior work showing that people do not know how
they appear Sun and Vazire (2019). We found a weak correlation (0.13) between
turn-level Empathy and perceived empathy; however, there is no significant
correlation between turn-level Empathy and first-person empathy.

Next, we asked what is the relationship between demographic and empathy
levels? We found that women showed higher distress than men; however, the
empathy level was not statistically significantly different. Similarly, there
was no significant difference in turn-level aspects. The perceived empathy in
women was higher than men ($0.19$). We found that older participants ($>35$)
showed lower first-person empathy and distress; however, they had higher turn-
level Emotion and Self-Disclosure. Higher income individuals ($>58,000$)
showed the same pattern as age with only one difference which is that their
distress was much lower ($-1.08$). Higher education was correlated with higher
first- and second-person empathy and distress, but no significant difference
in turn-level aspects. For this analysis, we found that there were
insufficient racial set sizes to find statistically significant results. The
only statistically significant other-report empathy difference was due to
education level.

Psych Construct / Observation |  | Self-report
---
Empathic Concern
| Self-report
---
Personal Distress
| Other-report
---
Perceived Empathy
| Turn-level
---
Empathy
| Turn-level
---
Emotion
| Turn-level
---
Emotional Polarity
| Turn-level
---
Self-disclosure
Big 5 |  |  |  |  |  |  |
Openness | $\uparrow$ | $\uparrow$ | $\uparrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$ | $\downarrow$
Conscientiousness | $\uparrow$ |  | $\uparrow$ | $\uparrow$ |  |  |
Extroversion | $\downarrow$ | $\downarrow$ |  $\uparrow$$\uparrow$ |  | $\uparrow$ | $\uparrow$ |  $\uparrow$$\uparrow$
Agreeableness | $\uparrow$ |  | $\uparrow\uparrow$ |  |  |  |
Stability | $\uparrow$ | $\downarrow$ |  |  |  |  |
IRI |  |  |  |  |  |  |
Perspective-taking | $\uparrow$ |  | $\uparrow$ |  |  |  |
Distress |  | $\uparrow$ | $\downarrow$ |  |  |  |
Fantasy | $\uparrow$ | $\uparrow$ |  |  |  |  |
Empathic concern | $\uparrow$ | $\uparrow$ | $\uparrow$ |  |  |  |

Table 3: Relationship between personality traits and various empathy
perspectives.

What is the correlation between empathy levels and personality? Extroversion
and Agreeableness are the most significant personality traits to increase
perceived empathy (Table 3). In the case of Extroversion, we observe a
disconnect between self-report and other-report perceived empathy. As
expected, higher Openness leads to higher self-report empathy and distress and
higher other-report perceived empathy; however, lower turn-level Emotion,
Empathy, and Self-Disclosure. Higher Conscientiousness follows a similar
pattern of increased self- and other-report empathy; however, only the turn-
level Empathy is higher. Surprisingly, Emotional Stability seems to have no
discernible effect. We conclude that to influence other-report empathy only
two personality traits matter.

Higher perspective-taking correlates with higher self- and other-report
empathy. Higher IRI distress correlates to higher self-report distress and
lower perceived empathy. Higher IRI fantasy corresponds to higher self- and
other-report empathy and distress. Although most results are consistent with
expectations, they help validate the annotation and guide possible
personalities for a conversational agent.

Finally, we examined the impact of simple textual features on other-report
perceived empathy. We provide statistics such as correlations of empathy score
with respect to word count (total words they used), word diversity, word
frequency, pronoun usage, and determiner usage. The average word count among
all the conversations is 249 (min=52, max=1010), which has a weak positive
correlation of 0.074. Word Frequency has a correlation of 0.291. Pronoun usage
has a correlation of 0.468. Determiner has a correlation of -0.051. Thus the
pronoun usage has the highest correlation with the perceived empathy scores
given by the workers. In conversations, whenever the worker was able to relate
to the topic of the article personally, the worker was able to have a higher
empathy. This finding is consistent with previous work on language
emotionality and word statistics (Tausczik and Pennebaker, 2010).

##  4 Modeling Empathy and Distress

We developed the baseline models to show the impact of our dataset in
predicting empathy, distress, and perceived empathy in text. First, we modeled
empathy in the turn-levels in each conversation. Second, we modeled the
perceived counterparty empathy in conversation level. Lastly, we modeled self-
report Batson scale empathy and distress.111111The reason for choices in pre-
trained LM (i.e., RoBERTa) and NN architecture like bi-rnn is our observations
in several successful results that are obtained in prior text classification
studies such as sentiment and emotion classification.

###  4.1 Modeling Empathy

#### Task setup

We implemented models to predict Empathy, Emotion, Emotion polarity, and Self-
Disclosure for each turn in the turn-level conversations. We developed two
models: we trained a Gated Bi-RNN with attention layer (Bi-rnn-Att) and we
fine-tuned the RoBERTa-base pretrained language model with our dataset. We use
fasttext model (300-dimensions) in Bi-rnn-Att to represent the context
features for each turn, then we concatenate these context features with
numerical values of Empathy, Emotion, Emotion Polarity, and Self-Disclosure.
For example these numerical values to model empathy are: Emotion, Emotion
Polarity, and Self-Disclosure. We refer to these numerical values features in
the entire of this section. To evaluate our dataset with transformer models we
fine-tuned RoBERTa-base pretrained LM, we used [CLS] token for fine-tuning.
There are a total of 5821 individual turns in all turn-level conversations. We
split the data randomly into 70%/15%/15% for train/dev/test, respectively.
Training and tuning conditions are provided in the Appendix.

#### Results

Table 4) demonstrates the results. Best result is obtained by fine-tuning the
RoBERTa-base model. We can observe that when features are added to each model
in Bi-rnn-att condition it improved the results for each metric, and the
results of Bi-rnn-att-features are comparable with RoBERTa-base model for each
metric. Particularly we can observe the impact of these numeric features for
modeling Emotion, Emotion Polarity and self-disclosure. One observation is the
significant impact of features in modeling Emotion, Emotion Polarity, and
Self-disclosure. In all these models numerical value of Empathy is among the
features that is concatenated with the context features. As future study it is
worth to analyze the impact of this feature in more isolated setup.

Conditions | Empathy | Emo | Emo-pol | Self-dis
---|---|---|---|---
Bi-rnn-att | 0.575 | 0.369 | 0.339 | 0.370
Bi-rnn-att-features | 0.613 | 0.706 | 0.609 | 0.638
RoBERTa-base | 0.771 | 0.814 | 0.812 | 0.769

Table 4: Model performance for predicting turn-level Empathy, Emotion, Emotion
Polarity, and Self-Disclosure in Pearson r for each turn in turn-level
conversations. “Features” is the numeric value of each of these quantities
concatenated to the encoded sentences (e.g., for empathy we concatenated the
numeric values of Emotion, Emotion Polarity, and Self-Disclosure as features).

###  4.2 Modeling Perceived Counterparty Empathy

#### Task setup

We used the same NN architecture described in subsection 4.1. All models were
trained to predict Perceived Counterparty Empathy for the entire conversation.

#### Results

We found that the Perceived Counterparty Empathy of person 2 is more
predictable than person 1 (Table 5). Table 5 shows the results that achieved
when person 1 scored the Perceived Counterparty Empathy from person 2 and vise
versa.

Model | (Person 1) | (Person 2)
---|---|---
Bi-rnn-Att | 0.268 | 0.115

Table 5: Model performance for predicting other-report perceived empathy in
Pearson r for two people scoring at the end of the conversation.

###  4.3 Modeling Empathy & Distress in Essays

#### Task setup

We fine-tuned RoBERTa-base with essays that both person 1 and 2 wrote after
reading the article. For this series of experiments, we split the data into
60%/10%/30% for train/dev/test, respectively. The reason for this choice of
split is this dataset is smaller relative to conversation dataset, hence,
bigger split for test set was necessary to have a reasonable numebr of
datapoints in test set.

#### Results

Table 6 shows the models’ performance in predicting empathy and distress based
on the context of the essays person 1 and 2 wrote after reading the article.

Model | Empathy | Distress | Mean
---|---|---|---
RoBERTa-base | 0.560 | 0.665 | 0.612

Table 6: Model performance for predicting empathy and distress in Pearson r
for essays written by annotators after reading the articles.

##  5 Discussion

We demonstrate the modeling of Emotion, Emotional Polarity, Self-Disclosure,
Empathy and Distress. Further, personality and demographic variables are
important when personalized empathy is the goal of an application.

The empirical results we obtained from Deep NN models suggest that our dataset
is sufficient for modeling empathy. Further, our results indicate that
emotion, polarity, and self-disclosure improve the modeling of empathy.
Despite the complexity of empathy and the difficulties some people may have
being empathetic or expressing feelings of empathy, our results in modeling
empathy are significant.

Perceived Counterparty Empathy is also a significant feature of this dataset.
This feature can be used to measure quantities in downstream tasks or
applications, especially, ones that measure the amount of empathy. It is
interesting that the model performs much better on Person 1, who initiates the
conversation, than Person 2 (Table 5). The reason for such a result is that
Person 1 is on average less empathetic than Person 2. This is likely to be
because the first person to read the article and complete the survey starts
the conversation.

##  6 Applications & Future Work

The many facets of the Empathic Conversation dataset make it useful for number
of applications and future research. Here are a few of these:

  * •

Developing conversational agents that are perceived as empathetic will lead to
more successful interactions; for example, expressing empathy is crucial for
behavioral therapy agents, and more generally, understanding empathy in a
relational framework is critical (van Dijke et al., 2020).

  * •

Enabling personality traits in conversational agents is promising, as
conversational agents displaying certain personality traits are more likely to
be perceived as empathetic, and hence more successful in their interactions
(Costa et al., 2014). Agreeableness, and perhaps also openness and sociability
are likely to be beneficial (Hojat et al., 2005).

  * •

Training empathic agents to make use of context (here, both the news articles
and the demographics and personalities of the people conversing) may enable
agents to express empathy in more natural and convincing ways.

  * •

There are noticeable differences between how people experience their own
empathy, that of the people that they are conversing with, and conversations
that other people are having. Teasing apart the linguistic markers of these
different perceptions is key to understanding them.

This dataset will facilitate future work including a better understanding of
how demographics and personality correlate with empathy and distress in
conversations.

##  7 Conclusion

Empathy and distress are crucial components in human-computer interaction and
important traits of conversational AI agents. In this work, we provide an
empathic conversations dataset grounded in news articles, where 2 people
express their empathy about the plight of a third party around a shared real-
world topic of conversation. Such data provides a new perspective on empathic
conversations.Further, at the end of the conversation, each worker rated the
amount of empathy they perceived from their counterpart. This is the first
attempt to collect empathic conversations with these characteristics. Further,
we collected demographic variables and personality of the authors of the
conversations and we analyzed the correlation and importance of personality
with empathy and distress. This is particularly important for personalized
empathic AI agents and other applications in human-computer interactions.

##  8 Ethical Considerations

There are two main ethical concerns regarding our dataset: the first is the
possibility of disclosing private information121212We follow HIPPA guidelines
https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-
identification/index.html. and the second is the possibility of misuse of the
data for manipulation of others (e.g., a malicious influence bot). Regarding
data privacy, all identifiable information was removed from the released
dataset including Amazon Mechanical Turk IDs. All IP information, as well as
the time of collection, have been removed. We also read all of the
conversations to ensure that no personally identifiable information was
present. This is especially important given the high level of self-disclosure
in our dataset.

Our research was performed as a registered protocol under IRB# 826448.
Participants were told that these conversations and their demographic,
personality surveys, and essays would be used for research purposes and
distributed externally for research purposes.

> Your response to the survey will be used publicly for academic research.
> Your personally identifiable information will be stored on our secure server
> and never shared with third parties. We will use your data as part of
> publicly available research dataset, and will only report de-identified
> information, so no one can identify you as an individual person.

Nonetheless, to mitigate this risk we make this data accessible only to
research communities and we emphasize the risk involved in using this dataset.
The corpus will be available under an Academic License (free of cost).
Researchers will download a pdf and will supply their intended use. The
process will require the institution to sign the agreement. The license will
be a standard FDP data transfer form. It should be easy for institutions to
obtain a signature.

The training models could potentially be misused to predict empathy,
demographics, and personality for persuasion, given that the Empathic
Conversations dataset is sufficiently rich. By distributing the dataset with
clear requirements this risk will be mitigated.

## References

  * Batson et al. (1987) C Daniel Batson, Jim Fultz, and Patricia A Schoenrade. 1987.  Distress and empathy: Two qualitatively distinct vicarious emotions with different motivational consequences.  _Journal of personality_ , 55(1):19–39.
  * Buechel et al. (2018) Sven Buechel, Anneke Buffone, Barry Slaff, Lyle Ungar, and João Sedoc. 2018\.  Modeling empathy and distress in reaction to news stories.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4758–4765.
  * Carletta (1996) Jean Carletta. 1996.  Assessing agreement on classification tasks: The kappa statistic.  _Computational Linguistics_ , 22(2):249–254.
  * Chen (2018) Jun Chen. 2018.  Empathy for distress in humans and rodents.  _Neuroscience bulletin_ , 34(1):216–236.
  * Costa et al. (2014) Patrício Costa, Raquel Alves, Isabel Neto, Pedro Marvao, Miguel Portela, and Manuel Joao Costa. 2014.  Associations between medical student empathy and personality: a multi-institutional study.  _PloS one_ , 9(3):e89254.
  * Craggs and Wood (2005) Richard Craggs and Mary McGee Wood. 2005.  Squibs and discussions: Evaluating discourse and dialogue coding schemes.  _Computational Linguistics_ , 31(3):289–296.
  * Davani et al. (2021) Aida Mostafazadeh Davani, Mark Díaz, and Vinodkumar Prabhakaran. 2021.  Dealing with disagreements: Looking beyond the majority vote in subjective annotations.  _arXiv preprint arXiv:2110.05719_.
  * Davis (1980) Mark H Davis. 1980.  _Interpersonal Reactivity Index_.  Edwin Mellen Press.
  * De Bruyne et al. (2020) Luna De Bruyne, Orphee De Clercq, and Veronique Hoste. 2020.  An emotional mess! deciding on a framework for building a Dutch emotion-annotated corpus.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1643–1651, Marseille, France. European Language Resources Association.
  * Demszky et al. (2020) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054, Online. Association for Computational Linguistics.
  * Eugenio and Glass (2004) Barbara Di Eugenio and Michael Glass. 2004.  The kappa statistic: A second look.  _Computational linguistics_ , 30(1):95–101.
  * Fitzpatrick et al. (2017) Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017.  Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (woebot): a randomized controlled trial.  _JMIR mental health_ , 4(2):e19.
  * Fung (2015) Pascale Fung. 2015.  Robots with heart.  _Scientific American_ , 313(5):60–63.
  * Fung et al. (2016) Pascale Fung, Dario Bertero, Yan Wan, Anik Dey, Ricky Ho Yin Chan, Farhad Bin Siddique, Yang Yang, Chien-Sheng Wu, and Ruixi Lin. 2016.  Towards empathetic human-robot interactions.  In _International Conference on Intelligent Text Processing and Computational Linguistics_ , pages 173–193. Springer.
  * Gao et al. (2021) Jun Gao, Yuhan Liu, Haolin Deng, Wei Wang, Yu Cao, Jiachen Du, and Ruifeng Xu. 2021\.  Improving empathetic response generation by recognizing emotion cause in conversations.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 807–819, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gebru et al. (2018) Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2018.  Datasheets for datasets.  _arXiv preprint arXiv:1803.09010_.
  * Gibson et al. (2016) James Gibson, Dogan Can, Bo Xiao, Zac E Imel, David C Atkins, Panayiotis Georgiou, and Shrikanth Narayanan. 2016.  A deep learning approach to modeling empathy in addiction counseling.  _Commitment_ , 111:21.
  * Gosling et al. (2003) Samuel D Gosling, Peter J Rentfrow, and William B Swann Jr. 2003.  A very brief measure of the big-five personality domains.  _Journal of Research in personality_ , 37(6):504–528.
  * Goubert et al. (2005) Liesbet Goubert, Kenneth D Craig, Tine Vervoort, Stephen Morley, Michael JL Sullivan, Williams de CAC, A Cano, and Geert Crombez. 2005.  Facing others in pain: the effects of empathy.  _Pain_ , 118(3):285–288.
  * Hayes and Krippendorff (2007) Andrew F Hayes and Klaus Krippendorff. 2007.  Answering the call for a standard reliability measure for coding data.  _Communication methods and measures_ , 1(1):77–89.
  * Hojat et al. (2005) Mohammadreza Hojat, Marvin Zuckerman, Mike Magee, Salvatore Mangione, Thomas Nasca, Michael Vergare, and Joseph S. Gonnella. 2005.  Empathy in medical students as related to specialty interest, personality, and perceptions of mother and father.  _Personality and Individual Differences_ , 39(7):1205 – 1215.
  * Hosseini and Caragea (2021) Mahshid Hosseini and Cornelia Caragea. 2021.  It takes two to empathize: One to seek and one to provide.
  * John et al. (1999) Oliver P John, Sanjay Srivastava, et al. 1999.  _The Big-Five trait taxonomy: History, measurement, and theoretical perspectives_ , volume 2.  University of California Berkeley.
  * Jurafsky and Shriberg (1997) Dan Jurafsky and E. Shriberg. 1997.  Switchboard swbd-damsl shallow-discourse-function annotation coders manual.
  * Khanpour et al. (2017) Hamed Khanpour, Cornelia Caragea, and Prakhar Biyani. 2017.  Identifying empathetic messages in online health communities.  In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 246–251.
  * Krippendorff (2004) Klaus Krippendorff. 2004.  Reliability in content analysis: Some common misconceptions and recommendations.  _Human communication research_ , 30(3):411–433.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. 2016.  A persona-based neural conversation model.  _arXiv preprint arXiv:1603.06155_.
  * Li et al. (2020) Qintong Li, Hongshen Chen, Zhaochun Ren, Pengjie Ren, Zhaopeng Tu, and Zhumin Chen. 2020.  EmpDG: Multi-resolution interactive empathetic dialogue generation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4454–4466, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017.  Dailydialog: A manually labelled multi-turn dialogue dataset.  _arXiv preprint arXiv:1710.03957_.
  * Lin and McFatter (2012) Hung-Chu Lin and Robert McFatter. 2012.  Empathy and distress: Two distinct but related emotions in response to infant crying.  _Infant Behavior and Development_ , 35(4):887–897.
  * Lin et al. (2019) Zhaojiang Lin, Andrea Madotto, Jamin Shin, Peng Xu, and Pascale Fung. 2019.  MoEL: Mixture of empathetic listeners.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 121–132, Hong Kong, China. Association for Computational Linguistics.
  * Litvak et al. (2016) Marina Litvak, Jahna Otterbacher, Chee Siang Ang, and David Atkins. 2016.  Social and linguistic behavior and its correlation to trait empathy.  In _Proceedings of the Workshop on Computational Modeling of People’s Opinions, Personality, and Emotions in Social Media (PEOPLES)_ , pages 128–137.
  * Majumder et al. (2020) Navonil Majumder, Pengfei Hong, Shanshan Peng, Jiankun Lu, Deepanway Ghosal, Alexander Gelbukh, Rada Mihalcea, and Soujanya Poria. 2020.  MIME: MIMicking emotions for empathetic response generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8968–8979, Online. Association for Computational Linguistics.
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018\.  Training millions of personalized dialogue agents.  _arXiv preprint arXiv:1809.01984_.
  * McQuiggan and Lester (2007) Scott W McQuiggan and James C Lester. 2007.  Modeling and evaluating empathy in embodied companion agents.  _International Journal of Human-Computer Studies_ , 65(4):348–360.
  * Melchers et al. (2016) Martin C Melchers, Mei Li, Brian W Haas, Martin Reuter, Lena Bischoff, and Christian Montag. 2016.  Similar personality patterns are associated with empathy in four different countries.  _Frontiers in psychology_ , 7:290.
  * Paiva et al. (2017) Ana Paiva, Iolanda Leite, Hana Boukricha, and Ipke Wachsmuth. 2017.  Empathy in virtual agents and robots: a survey.  _ACM Transactions on Interactive Intelligent Systems (TiiS)_ , 7(3):1–40.
  * Preston and De Waal (2002) Stephanie D Preston and Frans BM De Waal. 2002.  Empathy: Its ultimate and proximate bases.  _Behavioral and brain sciences_ , 25(1):1–20.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381, Florence, Italy. Association for Computational Linguistics.
  * Sedoc et al. (2020) João Sedoc, Sven Buechel, Yehonathan Nachmany, Anneke Buffone, and Lyle Ungar. 2020.  Learning word ratings for empathy and distress from document-level user responses.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 1664–1673, Marseille, France. European Language Resources Association.
  * Sharma et al. (2022) Ashish Sharma, Inna W Lin, Adam S Miner, David C Atkins, and Tim Althoff. 2022.  Human-ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support.  _arXiv preprint arXiv:2203.15144_.
  * Sharma et al. (2020) Ashish Sharma, Adam Miner, David Atkins, and Tim Althoff. 2020.  A computational approach to understanding empathy expressed in text-based mental health support.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5263–5276, Online. Association for Computational Linguistics.
  * Smith et al. (2020) Eric Michael Smith, Mary Williamson, Kurt Shuster, Jason Weston, and Y-Lan Boureau. 2020.  Can you put it all together: Evaluating conversational agents’ ability to blend skills.  _arXiv preprint arXiv:2004.08449_.
  * Sun and Vazire (2019) Jessie Sun and Simine Vazire. 2019.  Do people know what they’re like in the moment?  _Psychological science_ , 30(3):405–414.
  * Tausczik and Pennebaker (2010) Yla R. Tausczik and James W. Pennebaker. 2010.  The psychological meaning of words: LIWC and computerized text analysis methods.  _Journal of Language and Social Psychology_ , 29(1):24–54.
  * Troiano et al. (2021) Enrica Troiano, Sebastian Padó, and Roman Klinger. 2021.  Emotion ratings: How intensity, annotation confidence and agreements are entangled.  In _Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 40–49, Online. Association for Computational Linguistics.
  * van Dijke et al. (2020) Jolanda van Dijke, Inge van Nistelrooij, Pien Bos, and Joachim Duyndam. 2020.  Towards a relational conceptualization of empathy.  _Nursing Philosophy_ , 21(3):e12297.
  * Vijayakumar et al. (2016) Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016.  Diverse beam search: Decoding diverse solutions from neural sequence models.  _arXiv preprint arXiv:1610.02424_.
  * Welivita et al. (2021) Anuradha Welivita, Yubo Xie, and Pearl Pu. 2021.  A large-scale dataset for empathetic response generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1251–1264, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Xiao et al. (2016) Bo Xiao, Chewei Huang, Zac E Imel, David C Atkins, Panayiotis Georgiou, and Shrikanth S Narayanan. 2016.  A technology prototype system for rating therapist empathy from audio recordings in addiction counseling.  _PeerJ Computer Science_ , 2:e59.
  * Xiao et al. (2015) Bo Xiao, Zac E Imel, Panayiotis G Georgiou, David C Atkins, and Shrikanth S Narayanan. 2015.  ” rate my therapist”: automated detection of empathy in drug and alcohol counseling via speech and language processing.  _PloS one_ , 10(12):e0143055.
  * Yalcin and DiPaola (2018) Özge Nilay Yalcin and Steve DiPaola. 2018.  A computational model of empathy for interactive agents.  _Biologically inspired cognitive architectures_ , 26:20–25.
  * Zhao et al. (2020) Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020.  Designing precise and robust dialogue response evaluators.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 26–33, Online. Association for Computational Linguistics.
  * Zhong et al. (2020) Peixiang Zhong, Chen Zhang, Hao Wang, Yong Liu, and Chunyan Miao. 2020.  Towards persona-based empathetic conversational models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6556–6566, Online. Association for Computational Linguistics.
  * Zhou and Jurgens (2020) Naitian Zhou and David Jurgens. 2020.  Condolences and empathy in online communities.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 609–626.

##  Appendix A Turn-Level Annotations

Figure 4: Turn Level Annotation Category and Rating Descriptions

Figure 5: Illustration of the empathetic conversation between two workers
about a particular news article. Figure 6: Illustration of a news article
provided to workers. Figure 7: Screenshot of the other-report empathy provided
to workers.

##  Appendix B Detailed Turn-Level Annotation Breakdown

As seen in Figure 4, there are several turn-level annotations by crowd
workers. After several pilots we focused on independent tasks with full
conversational history after each annotation. This led to the highest inter-
annotator agreement. We break down the average per-conversation Krippendorff’s
alpha in order to understand which aspects are most difficult to annotate. In
Table 2 we observe that the empathy IAAs are quite low. This is notable, since
annotations are less consistent when there are conversations with extremely
low Empathy and Self-Disclosure. Also, by averaging only per-conversation, we
are showing a worse IAA than overall which explains why none of the individual
annotations score higher than the overall Krippendorff’s alpha. Nonetheless,
this offers insight into which aspects of annotation are difficult. In an
annotation with trained research assistants, we found the same trend.

One common concern about our turn-level annotation is that the Krippendorff’s
alpha is below 0.667. As Craggs and Wood (2005) suggest,

> When inferring reliability from agreement, a common error is to believe that
> there are a number of thresholds against which agreement scores can be
> measured in order to gauge whether or not a coding scheme produces reliable
> data. Most commonly this is Krippendorff’s decision criterion, in which
> scores greater than 0.8 are considered satisfactory and scores greater than
> 0.667 allow tentative conclusions to be drawn Krippendorff (2004). The
> prevalent use of this criterion despite repeated advice that it is unlikely
> to be suitable for all studies Carletta (1996); Eugenio and Glass (2004);
> Krippendorff (2004) is probably due to a desire for a simple system that can
> be easily applied to a scheme. Unfortunately, because of the diversity of
> both the phenomena being coded and the applications of the results, it is
> impossible to prescribe a scale against which all coding schemes can be
> judged.

##  Appendix C Dataset Demographic Statistics

There were 44 workers who self-identified as male and 31 who self-identified
as female workers who participated in the conversations. We included an option
for gender non-disclosure/other; however, this was not selected by
participants. Tables 7 and 8 illustrate the distribution of education, and
race.

Education
---
Less than a high school diploma | 0
High School diploma | 8
Technical/Vocational School | 2
Some college but no degree | 13
Two year associate degree | 9
Four year bachelor’s degree | 34
Postgradute or professional degree | 9
Table 7: Distribution of the education level of corpus participants.
Race/Ethnicity
---
White | 56
Hispanic or Latino | 8
Black or African American | 6
Native American or American Indian | 0
Asian/Pacific Islander | 4
Other | 1
Table 8: Distribution of the race/ethnicity of corpus participants.

##  Appendix D Sample of Conversation

Table 9 and Table 10 display the differences in the first three turns of two
conversations about the same article.

Person 1 |  This article just brings home how tough cancer is and how anyone at any time can find themselves having to deal with this terrible illness.
---|---
Person 2 |  My thoughts exactly. I feel that this disease really can impact anybody from any class.
Person 1 |  Exactly! This is something that can’t be stopped with money or status or anything like that and brings us all down to the same level.
Person 2 |  I just with there were more clear sets of information regarding cancer. There seems to be too much misinformation.
Person 1 |  There does seem to be a lot of misinformation. But I think it’s in part because even the same kind of cancer doesn’t act the same way with everyone. Each cancer experience is so different and yet in the end, it’s the same disease. I think that is what makes it so frustrating.
Person 2 |  Right but I guess I mean more-so related to basic causes and how a person even gets cancer. It just seems so random and leaves a ton of people fearful that they will get it.
Table 9: First three turns of an example conversation where workers had high self-report empathy. Person 1 |  what did you think about this article
---|---
Person 2 |  it was interesting
Person 1 |  It was… cancer is such a sad subject no matter what
Person 2 |  yes it is sad to see in anyone
Person 1 |  I don’t get why it has not been cured yet
Person 2 |  probably deliberate
Table 10: First three turns of an example conversation where both workers had
low self-report empathy.

##  Appendix E NN models - training condition

During training the Bi-RNN model, we use fasttext pretrained embedding with
300-dimensions, dropout of 0.2%, and 3 fully connected layers were used after
attention layer. Adam optimizer was used to minimize cross-entropy loss with a
learning rate of 0.0001 for the fully connected layer. A batch size of 32 was
used with single NVIDIA GTX 1080 Ti. We fine-tune RoBERTa-base for 10 epoch
using Adam optimizer with the learning rate 0.00001 was used to minimize MSE
loss. Each of the experiments was repeated 5 times for RNN models and 3 times
on average for RoBERTa models. We report the best model performance.

##  Appendix F Datasheet for Empathic Conversations Dataset

Gebru et al. (2018) stated that the objective to have datasheet is to clearly
describe the process of collection, distribution, and maintenance of a
dataset. Thus, below, we provide a datasheet for our Empathic Conversations
dataset, as structured and suggested by Gebru et al. (2018). Further, this
datasheet could be useful to individuals who participated in this study and to
policy makers and advocates. If a question is not applicable to our dataset,
the answer is n.a.

###  F.1 Motivation

  * •

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.
Empathic Conversations Grounded in News Stories was created be used to better
train models to improve their perceived empathy in a conversation.

  * •

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?
The data was collected at the University of Pennsylvania.

  * •

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number
This data was partially funded by João Sedoc’s Microsoft Research Dissertation
Grant.

###  F.2 Composition

  * •

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.
Each instance is a breakdown of the conversation had between two workers on
Amazon Mechanical Turk over a specific news article. Beyond this, there are
personality, demographic, turn-level empathy, self-disclosure, emotion, and
dialog acts.

  * •

How many instances are there in total (of each type, if appropriate)?
A total of 500 conversations from 100 articles with 5 conversations per
article.

  * •

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).
n.a.

  * •

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.
Each instance contains a conversation.

  * •

Is there a label or target associated with each instance? If so, please
provide a description.
Each conversation is accompanied by the corresponding survey information taken
from each participant in the conversation, and the article that the
conversation is focused around.

  * •

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.
Yes, the Amazon Mechanical Turk worker IDs and IP addresses collected from
qualtrics were excluded to preserve the privacy of the workers. Additionally
the qualtrics survey timing was excluded. All conversations were reviewed and
no private information is present.

  * •

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.
One relationship between the instances is that multiple conversations
consisted of the same participants, but the participants were not aware of
that. Another is that some participants referenced different articles that
they already had conversations on.

  * •

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.
n.a.

  * •

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.
n.a.

  * •

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.
n.a.

  * •

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.
The dataset does not contain confidential information since all information
was scraped from news stories.

  * •

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.
The dataset does not contain data that might be offensive, insulting,
threatening, or might otherwise cause anxiety.

###  F.3 Collection Process

  * •

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If data was reported by
subjects or indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.
Conversations and Annotations of the conversations were collected through
Amazon Mechanical Turk.

  * •

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?
Articles used for the conversations were from Empathic News Reactions
(https://github.com/wwbp/empathic_reactions and
https://drive.google.com/file/d/1A-7XiLxqOiibZtyDzTkHejsCtnt55atZ/view).
Crowdworkers were recruited on Amazon mechanical turk. During phase 1 they
completed personality tests and entered demographic information. Then
afterwards crowdworkers were matched to read the articles, write essays/fill
out surveys and chat. After ending the conversation, they rated the other
person’s empathy. Finally, turn-level annotations of Empathy, Emotion, Self-
Disclosure was done via AMT crowdworkers and dialogue acts were labeled by
research assistants.

  * •

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?
The 100 articles used were sampled from the Empathic News Reactions
(https://github.com/wwbp/empathic_reactions and
https://drive.google.com/file/d/1A-7XiLxqOiibZtyDzTkHejsCtnt55atZ/view). The
conversations with the highest empathy and distress scores were selected.

  * •

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?
Subsequent demographic information is listed in Tables 7 and 8, and Section
3.2 were performed by crowd workers found through Amazon Mechanical Turk.

  * •

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.
Dataset was collected during the timeframe of June 2019-November 2019 for the
conversations and turn-level annotations: May 2020-August 2020 and February
2021-May 2022.

  * •

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.
The IRB approved our study. We stated all of the goals of the research and
submitted all the surveys and screenshots of the interactions.

  * •

Does the dataset relate to people? If not, you may skip the remainder of the
questions in this section.
Yes, some news articles were about specific individuals. Also, the
conversations are between two people and their personality/demographic
information is collected.

  * •

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?
Directly using Amazon Mechanical Turk

  * •

Were the individuals in question notified about the data collection? If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.
Yes the participants were consented into the study

  * •

Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.
Participants were informed that their personality, demographic, essays, and
conversations will be used for research purposes by people different from us,
the collector of these information. The wording we used to communicate with
the participants are provide in 8 of the main paper.

  * •

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis)been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.
Analysis of the potential impact of the dataset is laid out in Section 8.

###  F.4 Preprocessing/cleaning/labeling

  * •

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? If so, please provide a
description. If not, you may skip the remainder of the questions in this
section.
The following step were taken to process the data: 1) Gathering New Articles:
News Articles were selected from the Empathy New Reactions
dataset131313https://github.com/wwbp/empathic_reactions and articles
https://drive.google.com/file/d/1A-7XiLxqOiibZtyDzTkHejsCtnt55atZ/view with
100 articles with the highest empathy and distress ratings, 2) Labeling for
the Turn Level Annotation collection: Empathy, Emotion, Self-Disclosure, and
Emotional Polarity were the annotation categories for the turns in each
conversation.

  * •

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.
The raw unprocessed data (consisting of conversations) is saved on a secure
server as per the IRB requirements.

  * •

Is the software used to preprocess/clean/label the instances available? If so,
please provide a link or other access point.
This is available by request.

###  F.5 Uses

  * •

Has the dataset been used for any tasks already? If so, please provide a
description.
This dataset is used to model empathy and distress and to fine-tune a
conversation empathic model; results are provided in this article. The dataset
has also been used to create a empathic chatbot.

  * •

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.
Yes, available by request.

  * •

What (other) tasks could the dataset be used for?
The author’s suggest that this dataset be used to enhance tasks in which
empathy is a module and this dataset would be distributed for research
purposes.

  * •

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?
The demographic information of the participants in the conversations is known,
so demographic fairness checks can be used.

  * •

Are there tasks for which the dataset should not be used? If so, please
provide a description.
The creation of malicious bots.

###  F.6 Distribution

  * •

How will the dataset will be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)?
The dataset will be distributed via requests and for research purposes.

  * •

When will the dataset be distributed?
Via requests and for research purposes.

  * •

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.
The corpus will be available under an Academic License (free of cost).
Researchers will download a pdf and will supply their intended use. The
process will require the institution to sign the agreement. The license will
be a standard FDP data transfer form. It should be easy for institutions to
obtain a signature.

  * •

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.
Third parties may not redistribute the data.

  * •

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.
n.a.

###  F.7 Maintenance

  * •

Who is supporting/hosting/maintaining the dataset?
The authors of the paper provide maintenance and support.

  * •

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?
By email<EMAIL_ADDRESS>
  * •

Is there an erratum? If so, please provide a link or other access point.
n.a.

  * •

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?
The authors keep a mailing list of the users of this dataset and all changes
will be communicated by the authors of this paper.

  * •

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)? If so, please describe these limits and explain how they will be
enforced.
n.a.

  * •

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.
Authors provide support for all versions that are in use.

  * •

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
other users? If so, please provide a description.
From the ethical perspective the same protocol that is developed for this work
should be used in addition to any new code of ethics if applicable. Documents
are available and can be distributed to the eligible people who want to extend
the work.