# The Impact of the Filter Bubble—A Simulation Based Framework for Measuring
Personalisation Macro Effects in Online Communities

Thomas Gottron

Felix Schwagereit

Institute for Web Science and Technologies,
University of Koblenz-Landau, Germany<EMAIL_ADDRESS>Institute for
Web Science and Technologies,
University of Koblenz-Landau, Germany<EMAIL_ADDRESS>
###### Abstract

The term _filter bubble_ has been coined to describe the situation of online
users which—due to filtering algorithms—live in a personalised information
universe biased towards their own interests. In this paper we use an agent-
based simulation framework to measure the actual risk and impact of filter
bubble effects occurring in online communities due to content or author based
personalisation algorithms. Observing the strength of filter bubble effects
allows for opposing the benefits to the risks of personalisation. In our
simulation we observed, that filter bubble effects occur as soon as users
indicate preferences towards certain topics. We also saw, that well connected
users are affected much stronger than average or poorly connected users.
Finally, our experimental setting indicated that the employed personalisation
algorithm based on content features seems to bear a lower risk of filter
bubble effects than one performing personalisation based on authors.

##  1 Introduction

Eli Pariser coined the term filter bubble in [8]. The filter bubble represents
the personal, unique universe of online information a user lives in. The
boundaries of this universe, what gets in and what remains outside the reach
of the user, says Pariser, depends on the personalisation algorithms in social
networks and search engines. The personalisation algorithms consider a user’s
interaction with information to predict which information she most likely
considers relevant. As a consequence the algorithms will show such information
at higher ranks in a result list more likely or, in the most extreme case,
even exclusively, i.e. blocking other information from the sight of the user.
Thus, the personalisation algorithms effectively operate as filters on the
information the users perceive.

The genuine aim of the personalisation algorithms is to get an idea of the
topics the users are actually interested in and to improve their user
experience by preferably showing them contents that match their interests. On
the downside, this leads to a situation, where the user might be confronted
with information that is so highly personalised, that she will not see any
content that does not match her interests or represents opinions and facts
apart from her own point of view. For an individual user this might lead to a
skewed and biased perception of the world.

The risks and problems of the filter bubble effect have been discussed widely
and in different contexts. Several approaches aim at ensuring diversified
results in search scenarios or for particular information needs [5]. The
actual risk of observing filter bubble effects as well as their strength,
instead, has hardly been analysed.

In this paper we use an agent-based simulation to measure the impact of
personalisation algorithms on the perception of users in social networks.
Here, we are not interested in the effects on the level of individual users,
but rather on the macro level of an entire community. To this end, we combine
an established generative network model [1] and a state of the art topic
modelling approach [3] to simulate a social network structure and the messages
the users create and consume in this network. The resulting model consists of
a set of agents, each of which with her individual range of topics she is
interested in. The agents have their individual network context of friends.
They send messages to and receive messages from these friends. By involving
algorithms for personalisation the simulation allows to observe the
development of appropriate macro-level metrics which measure the benefit and
risk of personalised filters. The metrics to measure the benefits include
classical information retrieval metrics, such as precision and recall. The
negative impact is measured by analysing the reduction of the active social
context to only a few friends or a reduced used of vocabulary. Finally, the
number of perceived messages that fall into the core field of interest of the
users can be read in both ways: positive as the users are not bothered with
off-topic messages and negative because their perception of other topics of
peripheral interest is reduced or even filtered out entirely.

We considered two possible algorithms for personalisation: one based on the
content of messages and one based on the author of messages. In our simulation
we observed, that both personalisation approaches lead to filter bubble
effects w.r.t. the focus of the perceived messages being shifted towards the
core interests of the users. However, we also observed that the
personalisation algorithm based on content lead to the same benefits, but with
a lower impact on the downsides, e.g. the active social network does not
become as thin, neither the used vocabulary as sparse.

The contributions we make in this paper are the following:

  * •

We describe a model for simulating the generation, dissemination and rating of
messages based on established generative models for social network
construction and topic models.

  * •

We investigate the strength of filter bubble effects on a macro level when
introducing personalisation filters in the content dissemination of a social
network.

  * •

We analyse which user groups of an online community are affected most by
filter bubble effects.

  * •

We compare two prototypical filtering methods under the aspect of risks and
benefits of filter bubble effects.

We proceed as follows. In Section 2 we will review related work in the field
of community simulation as well as the analysis of effects of personalisation
and diversification in information provision. Afterwards we present the
general framework of our simulation. Its implementation and the choice of
model parameters is presented in 4. In Section 5 we describe the two
personalisation algorithms we have analysed and in Section 6 the metrics we
use to measure filter bubble effects. The actual experiments in the form of
simulation runs are discussed in Section 7. We present our results in 8 and
discuss them in detail in 9. We end in Section 10 with a summary of our
findings and a look at future work.

##  2 Related Work

The importance and the effects of information propagation and perception in
social networks and information portals has been analysed in various contexts.
Serendipity discovery of non-relevant items in digital libraries [12] shows a
general benefit of being inspiring for seeking information and thinking out of
the box. The impact on political processes and democracy was discussed in [2].
A psychological study addressed the effects of information presentation and
forgetting processes in social networks [4].

Simulation and probabilistic models as tools for the analysis of online
communities has been established in recent years. Schwagereit et al. use
simulations to support policy modelling and strategic decisions [11]. The
dynamics of popularity among news stories was investigated and described by
Hogg and Lerman [6, 7]. Self-enforcing content generation mechanisms and their
representation as stochastic processes were investigated in [14].

In our simulation, we use the preferential attachment model of Barabasi and
Alberts [1] to construct a social network. Preferential attachment is a
probabilistic model to generate graph structures which exhibit typical
features of social networks. We use topic models to represent the interests of
agents. Latent-Dirichlet-Allocation (LDA) [3] can be considered a state of the
art topic modelling approach. LDA is a generative, probabilistic model to
describe the topic composition of text documents. To this end each document is
represented as distribution over $k$ global topics. This distribution is
independent from the distribution of other documents but follows the same
parameters. The topics themselves are modelled as distributions over terms.
Various extensions have shown, that LDA is also suitable to model sentiments
and opinions as well as authors.

To the best of our knowledge there is no systematic large scale analysis of
filter bubble effects, neither based on real world data nor based on
simulations.

##  3 The Simulation Model

For our analysis we make use of an agent-based simulation. The initial step
prior to starting the actual simulation consists of the setup of the network
with a fixed number of agents, each of which has a profile of her individual
interests. The network structure reflects the friendship relations between
agents and will provide the basis for how messages are disseminated among the
agents. We do not consider changes in the network structure during the
simulation of the communities interaction. An agent’s profile of interest will
serve to describe the messages she will write as well as for the decision
whether or not a message lies within her core field of interests.

The idea of the simulation is focussing on an iterative creation,
dissemination, filtering and rating of messages published in a social network
plus a step of learning for updating the filters based on the user feedback.
The overall process is depicted in Figure 1 and we illustrate the high level
steps before going into the technical details of the implementation in the
next section.

Figure 1: High level iterative process of the agent-based simulation.

Creation

Each agent creates messages. The number of messages an agent creates depends
on how verbose she is. The content of the messages themselves depends on an
agent’s interests.

Dissemination

The messages are disseminated along the edges of the social network. This
means that for each agent the set of incoming messages is determined by the
messages created by her friends.

Filtering

The personalisation algorithms operate on this set of incoming messages. The
messages are ranked according to the predicted relevance and the result list
is cut off at a given threshold. Each agent has an individual instance of the
personalisation algorithm which is specifically trained on the basis of her
behaviour.

Rating

The agents are presented with the messages the personalisation algorithms
deemed interesting for them. At this point the agents interact with the
messages of other agents, i.e. they rate them as relevant or as irrelevant.
This decision also depends on whether or not the message is in the core field
of interest of a agent.

Learning

The personalisation algorithms receive the information which of the messages
were rated as relevant. This feedback of the agents is used by the
personalisation algorithms to update their individual relevance model for the
agents.

Iterating this cycle many times allows the personalisation algorithms to learn
from the agent’s feedback. In this way they can adapt to the agent’s
interests. At some point in time we would expect to see filter bubble effects,
such as a high ratio of messages in the core field of interest, the active
social network of the agents being reduced to a few contacts or a reduction of
the perceived vocabulary.

##  4 Implementation

The general process of the simulation is given as pseudo code in Algorithm 1.
The phase of the initial setup and the iteration phase with the steps of
content generation, dissemination, filtering, rating and the learning phase
are outlined. The next sections illustrate the relevant parts of this process,
i.e. the concrete implementation of the methods in this algorithms.

Input: Number of agents $N$, personalisation algorithm $F$, Number $I$ of
iterations

// Agent Setup

for  _$n=1\ldots N$_ do

// Create an agent and her instance of the personalisation algorithm

$a\leftarrow\textsc{initialiseAgent}()$;

$F_{a}\leftarrow F.\textsc{instance}()$;

$A\leftarrow A\cup\\{a\\}$;

// Network Setup

$\textit{Network}\leftarrow\textsc{constructNetwork}()$;

for  _$i=1\ldots I$_ do

// Content creation

foreach  _$a\in A$_ do

$\textit{out}_{a}\leftarrow a.\textsc{generateMessages}()$;

// Dissemination

foreach  _$a\in A$_ do

foreach  _$b\in\textit{Network}.\textsc{friendsOf}(a)$_ do

$\textit{in}_{a}\leftarrow\textit{in}_{a}\cup\textit{out}_{b}$;

// Filtering

foreach  _$a\in A$_ do

$\textit{filtered}_{a}\leftarrow F_{a}.\textsc{filter}(\textit{in}_{a})$;

// Rating

foreach  _$a\in A$_ do

foreach  _$c\in\textit{filtered}_{a}$_ do

if  _ $a.\textsc{isRelevant}(c)$_ then

$\textit{relevant}_{a}\leftarrow\textit{filtered}_{a}\cup\\{c\\}$;

// Learning

foreach  _$a\in A$_ do

$F_{a}.\textsc{learnInterests}(\textit{filtered}_{a},\textit{relevant}_{a})$;

Algorithm 1 Simulation

###  4.1 Initialising an Agent and her Interests

When initialising an agent with the method initialiseAgent, the interests of
the agents are modelled in a probabilistic manner by randomly assigning a
distribution over topics. The more probable topics in this distribution
represent the core field of interest of the agent. This distribution will be
used in both the generative process when an agent writes messages as well as
in the perception phase when an agent decides whether or not a message lies
within her core field of interests.

To implement the topic models of the agents we employ Latent Dirichlet
Allocation (LDA) [3]. To this end, each agent is assigned a random
distribution over $k$ global topics which is drawn from a Dirichlet
distribution with prior $\alpha$. The entries in the prior vector $\alpha$
influence how the distribution over the topics looks like. High entries in
$\alpha$ generate distributions in which more topics are covered equally
likely. Low values in $\alpha$ generate topic distributions in which only few
topics have a high probability of occurring. The global topics themselves are
randomly initialised as well. They also follow a Dirichlet distribution. Here
the prior $\beta$ influences how the distributions of terms within each topic
look like. Thus, the topic distribution of an agent is independent from the
distribution of the other agents but follows the same parameters. The same is
true for the topic distributions. As we are not interested in the actual
contents, we use an artificial vocabulary and artificial topics, which have
the same probabilistic characteristics as real world topics. We chose the
values of $\alpha$ and $\beta$ from literature to model focussed topics and a
not too wide coverage of too many topics by an individual agent. The concrete
settings for the Dirichlet priors, as well as the number of agents and
vocabulary terms we used in our simulation runs are summarised in Table 1.

Additionally we model the verbosity of each agent individually. To this end,
we provide each agent $a$ with the number $\lambda_{a}$ of messages we expect
her to produce on average. The value of $\lambda_{a}$ is chosen randomly when
setting up the agent and follows a $\chi$ distribution with mean $e$. To have
realistic values for how many messages we can expect the user to write we
estimated the value of $e$ from statistics on how many tweets people write per
day111http://edudemic.com/2012/12/14-twitter-statistics-you-may-not-know/.

###  4.2 Social Network Setup

We construct the social network of the agents randomly in constructNetwork
using the preferential attachment model [1]. In this generative model a social
network graph is constructed starting from a nucleus of very few completely
connected nodes. Iteratively the graph is extended with new nodes, each of
which establishing the same amount of edges to existing nodes. The probability
of connecting to an existing edge is proportional to the number of edges it
already has. In our settings, we started with a nucleus of ten nodes and had
each newly created node connect to five other nodes.

###  4.3 Content Creation

The first step in the method generateMessages is that the agent randomly
decides how many messages she is going to write in this iteration. The number
of generated messages follows a Laplace distribution with parameter
$\lambda_{a}$, which has been assigned to each agent $a$ during
initialization.

Once it is decided how many messages will be generated, the agents write each
individual message $c$ based on their profile of interests. For the sake of
simplicity, in our simulation each message has an equal length of 10 words. To
determine the actual words we used the generative process behind the LDA
model. This means that for each word of the messages, the model randomly
chooses first a topic (according to the agent’s topic distribution) and in a
second step randomly selects a word from this topic (according to the term
distribution of the corresponding topic).

###  4.4 Rating

The agents need to be able to interact with the messages in the social network
in order to give feedback on what is relevant to them. To this end, in the
method isRelevant each message the agent perceives is first classified into
either falling into her field of core interests or into the category of
peripheral interest. The core interest of an agent is defined by the set of
LDA topics which covers a high fraction of her individual topic distribution.
The other topics describe her peripheral interests. The decision if a message
is in the field of core interests is based on the odds of the agent to have
constructed such a message solely from her core interests topics versus solely
from her peripheral interests. If these odds are higher than a given threshold
$\theta$, a message is considered to be of core interest to an agent,
otherwise it belongs to her peripheral interests. For our simulation we set
the value _cov_ of core topic coverage to 80% and the odds threshold $\theta$
to 2. In preliminary experiments we found that with these settings an agent
assigns more than 90% of her own messages and about 5-20% of the messages of
any random other agent to be in her core interest. However, as long as the
decision on what is part of the core interest is consistent the overall model
will behave very similar.

The actual act of rating a message as relevant then depends on two
probabilities $p_{\textit{core}}$ and $p_{\textit{peripheral}}$. Not all
messages of core interest will be considered relevant. Neither will all
messages of peripheral interest be considered irrelevant. As we have no
empirical data on which to base these probabilities will we keep them variable
and explore the behaviour of the entire simulated community depending on these
two probabilities.

Table 1: Parameters and their values in our simulation. $|V|$ | Size of vocabulary, i.e. number of words | 10,000
---|---|---
$\alpha_{i}$ | Parameter vector entries in Dirichlet prior for the topic distributions of the agents | 0.01
$\beta_{i}$ | Parameter vector entries in Dirichlet prior for the term distributions of the topics | 0.001
$N$ | Network size, i.e. number of users | 10,000
$m$ | Minimum number of friends per user | 5
$k$ | Number of topics | 100
$l$ | Message length | 10
$e$ | Prior for the expected value of the number of messages a user creates | 2.42
$p_{\textit{core}}$ | Probability to rate a message of core interest as relevant | variable
$p_{\textit{peripheral}}$ | Probability to rate a message of peripheral interest as relevant | variable
_cov_ | Core topic coverage | 0.8
$\theta$ | Threshold odds for core topic | 2.0
$b$ | Cut off rank position in the filters | 20

##  5 Personalisation Algorithms

The personalisation algorithms need to implement the two methods filter and
learnInterests to work in the context of our simulation framework. This
essentially complies with the operation modes of these algorithm. On the one
hand, the algorithm needs to rank a set of messages in decreasing order of
predicted interest for a user. It operates as a filter if the resulting
ranking is cut off at a certain position $b$ and all entries below that
position are discarded and hidden from the user. On the other hand it needs to
interpret the interactions of the users with the messages which indicate
relevance in order to derive the user’s interests.

We implemented two prototypical, orthogonal personalisation algorithms. One
algorithm is based on the content and one on the authors of a message. Both
are based on established methods to predict if a given message with a specific
content or from a specific author will be rated as relevant by a user.

###  5.1 Content Based Personalisation

A message is characterised by its individual words. This assumption underlies
many information retrieval models. The terms are signals that either
contribute or oppose the probability of the message to be relevant. The
information provided by the user via her interaction with the contents can
help to understand which kind of signal is emitted by which term. This
assumption underlies the concept of relevance feedback in search systems.

We implemented our content based personalisation algorithm following the
principles of relevance feedback in the binary independence model of
probabilistic retrieval [9]. This means, that for each document the user has
seen, we keep track of whether or not she has rated it to be relevant. This
relevance is directly mapped onto the single terms. And for each term $t_{i}$
we can estimate the probability of being relevant (using a random variable
$R$) by:

$p(R|t_{i})=\frac{cr(t)+C_{V}}{cr(t)+cn(t)+2C_{V}}$ (1)

In this case $cr_{V}(t_{i})$ is the number of times a user has rated a message
containing term $t_{i}\in V$ as relevant. Likewise, $cn_{V}(t_{i})$ counts how
often a message with this term has not been rated as relevant. The constant
$C_{V}$ is a smoothing parameter to overcome zero probabilities for unseen
events.

Using these estimates allows to formulate a probability of relevance for a new
incoming message. To this end the message $c$ is broken up into its individual
terms. The probability of relevance is given by the product of the
probabilities of relevance over the single terms:

$p(R|c)=\prod_{t_{i}\in c}P(R|t_{i})$ (2)

This probability can be computed for each incoming message. The messages are
then sorted in decreasing order of probability following the probability
ranking principle [10].

###  5.2 Author Based Personalisation

Instead of focussing in the contents, an alternative paradigm is to consider
the authors of a message as indicative signal for relevance. This is mainly
motivated by the strong social component of online communities: it makes sense
to consider that it is rather the news of specific people than specific
contents the users want to see. Again we can model this in a probabilistic
way. Depending on how many times messages of one specific author $a$ have been
seen and rated as relevant allows for estimating the probability of messages
of this author to be relevant. Given that the author is the only signal
considered we can then rank all incoming messages according to the
probabilities:

$p(R|a)=\frac{cr_{A}(a)+C_{A}}{cr_{A}(a)+cn_{A}(a)+2C_{A}}$ (3)

Here, $cr_{A}(a)$ counts the number of times a user has rated a message of
author $a\in A$ as relevant, while $cn_{A}(a)$ counts how often the messages
with this author were not relevant. Again we use a smoothing constant $C_{A}$.

##  6 Measuring Filter Bubble Effects

To measure the impact of personalisation we have to consider two kinds of
effects. On the one hand we have to measure the benefits of the
personalisation on incoming content items, on the other hand we have to
measure the filter bubble effect. The improvement in the ranking of incoming
items can be measured relatively easy by considering classical information
retrieval metrics. Precision measures the ratio of relevant items among the
retrieved, i.e. here among the seen messages. IR systems and also their
personalisation components strive to improve precision. Thus, an increase in
precision at the individual level of single agents due to the use of a
personalisation filter is sign for the benefits of the personalisation
algorithms. To observe this behaviour on the macro-level we use average
precision over all agents.

To answer our questions of the strength of the filter bubble we need different
metrics. To the best of our knowledge, metrics for filter bubble effects have
not been developed so far. However, there are clear signals for users being in
a filter bubble: (1) a reduction of the active social context, (2) a reduction
of the vocabulary a user perceives and (3) users perceives messages mainly or
only from the fields of her core interests.

To measure the active social context it is sufficient to count the number of
distinct friends from which an agent perceives messages. If we denote the set
of authors for a set of messages by
$A(\textit{filtered}_{a})=\\{b\in\textit{Network}:\exists
c\in\textit{filtered}_{a}:c.\textsc{author}=b\\}$, then we can define the
active social context of any given agent $a$ by:

$ASC(a)=\frac{|A(\textit{filtered}_{a})|}{|\textit{Network}.\textsc{friends}(a)|}$
(4)

This value measures the active social context for one agent only. To observe
macro effects, we aggregate and average the values over all agents in the
simulation:

$ASC=\frac{1}{|A|}\cdot\sum_{a\in A}ASC(a)$ (5)

However, even a low value of this metric does not necessarily indicate a
poorer social context. After all it is possible, that the context is small at
each time step, but constantly changing with every iteration. Figuratively
speaking, a user might received messages always just from a small circle of
her friends, but this circle is composed of different friends in each
iteration.

Thus, we need to take the change rate of the social context over a longer time
into consideration. This can be achieved by extending $ASC$ to operate on the
authors seen over a longer time period (e.g. the last 10 iterations). We will
refer to this values as $ASC_{h}$. Comparing $ASC$ to $ASC_{h}$ we can
estimate the change rate.

The reduction of active vocabulary can be measured in an equivalent way. We
defined the according functions as $AV$ and $AV_{h}$.

The ratio $CR$ of core messages in the perceived messages finally can be
computed similar to average precision. If we define $\textit{core}_{a}$ as the
set of all messages in $\textit{filtered}_{a}$ which are in the core interest
of an agent, then we get:

$CR=\frac{|\textit{core}_{a}|}{|\textit{filtered}_{a}|}$ (6)

As we interested in which parts of the online community are affected stronger
by filter bubble effects we stratified the agents depending on their node
degree. We divided the agents in five strata. The first stratum consisted of
the agents with the highest node degree which covered 20% of all network
edges. The next stratum contained all agents providing the next 20% of the
edges and so on.

##  7 Simulation Experiments

As stated above we could estimate most parameters in our model from public
statistics of online communities or by using values which have been reported
in related work. One parameter we could not estimate are the probabilities
$p_{\textit{core}}$ and $p_{\textit{peripheral}}$ with which the users rate
messages of their core or peripheral interest as relevant. Thus, we kept these
parameters variable and observed how this user behaviour influences the filter
bubble effect.

This lead to a setup where we ran for each personalisation algorithm
independent simulations over 441 parameter settings to cover the parameter
space of the probabilities $p_{\textit{core}}$ and $p_{\textit{peripheral}}$.
We iterated over these probabilities from 0 to 1 in steps of 0.05. Each
simulation run was initiated with a new random network and random assignment
of topic distributions to the users. Then we ran the simulation for 100
iterations—a value which in preliminary experiments had shown that at this
point the values become stable.

##  8 Results

(a) Global Average

(b) Top Stratum

Figure 2: $CR$ values for content based personalisation.

(a) Global Average

(b) Top Stratum

Figure 3: $CR$ values for author based personalisation.

Given that we iterate in small steps over a 2-dimensional parameter space we
will present the result as three dimensional surface plots. This allows to get
a very good visual impression of the impact of the user behaviour (i.e. how
likely they rate messages of core and peripheral interest as relevant) on the
filter bubble effect.

The first question we want to address is the impact on the ratio of messages
of core interest the agents receive. Figure 2 shows the value of the $CR$
metric over the range of different settings for $p_{\textit{core}}$ and
$p_{\textit{peripheral}}$ when operating with a content based personalisation.
The left Figure 2(a) shows the average $CR$ for the entire online community.
We can see very clearly the diagonal ridge where the two probabilities are
about the same value. Above and below this ridge the $CR$ values form a
plateau. This means that the ratio of messages in the core interest field of
an agent is higher if the probability for rating messages as relevant is
higher in the class of core interest than in the category of peripheral
messages. This behaviour is somewhat expected as the personalisation filters
learn what are the core interests of the users. The plateau, however,
indicates that the ratio of core interest messages does not grow arbitrarily
high. On the entire community, the values of $CR$ reach an average of at most
73% if the users rate more core interest messages as relevant and at least 13%
if the users mainly rate peripheral interest messages as relevant. The right
plot in Figure 2(b) shows the values for the top stratum of highly connected
agents in the social network. While the general behaviour is the same, the
effects are much stronger. The values in the lower plateau are set around 4%
and the high plateau values about 87%. This means that especially the well
connected users are affected strongest by the filtering effects. They hardly
get to see messages from outside their core field of interest.

Figure 3 shows the same plots for the personalisation algorithm based on
message authors. Again the general behaviour is very similar. The effect,
however, is slightly less strong. The $CR$ values in the top stratum reaches
only 80%.

Let us look at the impact on the social network next. In Figure 4 we compare
the active social network ($ASC_{h}$) of the top stratum for content based
(Figure 4(a)) and author based personalisation (Figure 4(b)). Now we can see a
clear difference between the two methods. While the author based
personalisation immediately restricts the number of people from which messages
are received as soon as the agents give any feedback, the content based
personalisation is not that extreme. The $ASC_{h}$ values are higher in
general and the decline is much slower, when users give feedback. Furthermore,
as long as they do not excessively rate messages as relevant (with a
probability of more than 60%) the active social context remains relatively
broad. This is also reflected in the change rate of the perceived authors.

(a) Content base personalisation

(b) Author base personalisation

Figure 4: Active social network $ASC_{h}$ for the top stratum.

A similar behaviour can be observed on the side of the vocabulary in Figure
5222Note the changed scale for the values. Given the size of the vocabulary,
the few messages an agent receives after filtering can only cover a small
fraction of the vocabulary anyway.. This is intriguing, as content based
personalisation in the end relies on single terms. However, filtering messages
based on the preferred terms restricts the overall perceived terms less than
when filtering messages based on their author.

(a) Content base personalisation

(b) Author base personalisation

Figure 5: Active vocabulary $AV_{h}$ for the top stratum.

The positive aspects of the personalisation can be seen when looking at
average precision in Figure 6. We can see that precision improves with the
probability of the users giving feedback on what they consider relevant. This
is valid for both, the content and the author based personalisation. However,
the shape of these plots partially depends directly also on the probabilities
$p_{\textit{core}}$ and $p_{\textit{peripheral}}$. If those probabilities are
close to zero, the agents rate no or hardly any message as relevant.
Accordingly precision is low by definition. However, if they tend to consider
messages relevant at a high probability they will rate nearly everything as
relevant and, thus, precision will automatically be very high. The most
interesting aspect of the plot are the regions where $p_{\textit{core}}$ and
$p_{\textit{peripheral}}$ have very different values. In these cases the
algorithms really need to bring the right category of messages (core interest
or peripheral interest) to the agent’s attention such that they have the right
basis of messages to rate with a higher probability as relevant. Given the
high values in these regions and the observations made for the $CR$ values,
this is achieved by both approaches

(a) Content base personalisation

(b) Author base personalisation

Figure 6: Average precision for the top stratum.

##  9 Discussion

Our simulation model allowed us to make some quite interesting observations on
the macro level of online communities:

  1. 1.

Filter bubble effects occur when using personalisation algorithms. We observed
for both types of personalisation the effect, that users have a biased
perception based on their own interests. This happens as soon as they indicate
a preference of their core interest by rating messages in this field as
relevant with a different probability than all other messages. However, this
after all is the aim of the personalisation algorithms. So this can be
considered the central trade off between using personalisation for improving
the user experience and not using personalisation for the sake of unbiased
message perception.

Furthermore, on average the filter bubble effect is not that strong that a
user cannot get out of the filter bubble any more. Users still receive
messages from outside their core field of interest which technically allows
them re-adjust the personalisation algorithms to potentially new interests.

  2. 2.

Better connected users tend to be more prone to filter bubble effects than the
average user. This can be easily explained with these users receiving far more
messages. Accordingly they are also likely to receive more relevant messages
falling into field of their core interest and from which the personalisation
algorithms can chose. The algorithms are simply less under pressure to also
suggest less relevant messages.

However, this bears a certain risk as well: the well connected users represent
hubs and are typically considered influential users. If their perception of
the world is biased, this bias may affect also their social context and be
propagated through the online community.

  3. 3.

Author based personalisation causes a stronger filter bubble effect than
content based personalisation under the aspects of active social context and
active vocabulary. Our experiments showed a much stronger restriction of the
active social network and perceived vocabulary when using the author based
filtering algorithm. This is partially interesting as it might influence the
potential to actually get out of the filter bubble.

Given that both approaches behave similar under the aspect of focusing on the
users interests and achieving comparable precision values, the content based
personalisation approach seems to have a lower impact on the filtering of the
social network and vocabulary. Thus, one conclusion might be that content
based personalisation seems the better option as it causes a less strong
filter bubble effect. However, this observation needs to be checked and
confirmed with other personalisation algorithms.

##  10 Summary and Future Work

In this paper we proposed a model to measure the macro effects of
personalisation in online communities with respect to filter bubble effects.
We combined established topic and network models to build an agent-based
simulation of an online community and investigated the impact of different
personalisation algorithms on the diversity of topics, people and vocabulary
the users perceive. We observed that filter bubble effects occur as soon as
the users give some indications about the topics they are interested in.
However beyond the filtering of topics we also observed that in our setting
author based personalisation lead to stronger restrictions and thus stronger
filtering effect on the social network and vocabulary.

In future work we will investigate other personalisation algorithms and
consider flexible network structures. Also networks formed on the basis of
common interests of the agents will be considered. One suitable candidate
might be the social-circles model [13].

## References

  * [1] Albert-László Barabási and Réka Albert.  Emergence of scaling in random networks.  Science, 286:509–512, 1999.
  * [2] Solon Barocas.  The price of precision: voter microtargeting and its potential harms to the democratic process.  In Proceedings of the first edition workshop on Politics, elections and data, PLEAD ’12, pages 31–36, New York, NY, USA, 2012. ACM.
  * [3] David M. Blei, Andrew Y. Ng, and Michael I. Jordan.  Latent dirichlet allocation.  Journal of Machine Learning Research, 3:993–1022, 2003.
  * [4] A. Coman and W. Hirst.  Cognition through a social network: The propagation of induced forgetting and practice effects.  Journal of Experimental Psychology: General, 141(2):321, 2012.
  * [5] M. Drosou and E. Pitoura.  Search result diversification.  ACM SIGMOD Record, 39(1):41–47, 2010.
  * [6] Tad Hogg and Kristina Lerman.  Social dynamics of digg.  In Proceedings of the Fourth International Conference on Weblogs and Social Media, ICWSM 2010, Washington, DC, USA, May 23-26, 2010. The AAAI Press, 2010.
  * [7] Kristina Lerman and Tad Hogg.  Using stochastic models to describe and predict social dynamics of web users.  ACM TIST, 3(4):62, 2012.
  * [8] E. Pariser.  The filter bubble: What the Internet is hiding from you.  Penguin Press HC, 2011.
  * [9] S.E. Robertson and K. Spärck Jones.  Relevance weighting of search terms.  Journal of the American Society for information Science, 27(3):129–146, 1976.
  * [10] Stephen Robertson.  The Probability Ranking Principle in IR.  The Journal of documentation, 33(4):294–304, 1977.
  * [11] Felix Schwagereit, Sergej Sizov, and Steffen Staab.  Finding optimal policies for online communities with cosimo.  In Proceedings of the WebSci10: Extending the Frontiers of Society On-Line, April 26-27th, 2010, Raleigh, NC: US, 2010.
  * [12] Elaine G. Toms and Lori McCay-Peet.  Chance encounters in the digital library.  In Proceedings of the 13th European conference on Research and advanced technology for digital libraries, ECDL’09, pages 192–202, Berlin, Heidelberg, 2009. Springer-Verlag.
  * [13] Douglas R. White, Nataša Kejžar, Constantino Tsallis, Doyne Farmer, and Scott White.  Generative model for feedback networks.  Phys. Rev. E, 73(1):016119, Jan 2006.
  * [14] Dennis M. Wilkinson.  Strong regularities in online peer production.  In Proceedings of the 9th ACM conference on Electronic commerce, EC ’08, pages 302–309, New York, NY, USA, 2008. ACM.


# Token Sequence Labeling vs. Clause Classification for
English Emotion Stimulus Detection

Laura Oberländer  Roman Klinger
Institut für Maschinelle Sprachverarbeitung, University of Stuttgart
Pfaffenwaldring 5b, 70569 Stuttgart, Germany
<EMAIL_ADDRESS>
###### Abstract

Emotion stimulus detection is the task of finding the cause of an emotion in a
textual description, similar to target or aspect detection for sentiment
analysis. Previous work approached this in three ways, namely (1) as text
classification into an inventory of predefined possible stimuli (“Is the
stimulus category A or B?”), (2) as sequence labeling of tokens (“Which tokens
describe the stimulus?”), and (3) as clause classification (“Does this clause
contain the emotion stimulus?”). So far, setting (3) has been evaluated
broadly on Mandarin and (2) on English, but no comparison has been performed.
Therefore, we analyze whether clause classification or token sequence labeling
is better suited for emotion stimulus detection in English. We propose an
integrated framework which enables us to evaluate the two different approaches
comparably, implement models inspired by state-of-the-art approaches in
Mandarin, and test them on four English data sets from different domains. Our
results show that token sequence labeling is superior on three out of four
datasets, in both clause-based and token sequence-based evaluation. The only
case in which clause classification performs better is one data set with a
high density of clause annotations. Our error analysis further confirms
quantitatively and qualitatively that clauses are not the appropriate stimulus
unit in English.

00footnotetext:  This work is licensed under a Creative Commons Attribution
4.0 International License. License details:
http://creativecommons.org/licenses/by/4.0/.

##  1 Introduction

Research in emotion analysis from text focuses on classification, i.e.,
mapping sentences or documents to emotion categories based on psychological
theories (e.g., Ekman (1992), Plutchik (2001)). While this task answers the
question _which_ emotion is expressed in a text, it does not detect the
textual unit, which reveals _why_ the emotion has been developed. For
instance, in the example _“Paul is angry because he lost his wallet.”_ it
remains hidden that _lost his wallet_ is the reason for experiencing the
emotion of anger. This stimulus, e.g., an event description, a person, a state
of affairs, or an object enables deeper insight, similar to targeted or
aspect-based sentiment analysis (Jakob and Gurevych, 2010; Yang and Cardie,
2013; Klinger and Cimiano, 2013; Pontiki et al., 2015, 2016, i.a.). This
situation is dissatisfying for (at least) two reasons. First, detecting the
emotions expressed in social media and their stimuli might play a role in
understanding why different social groups change their attitude towards
specific events and could help recognize specific issues in society. Second,
understanding the relationship between stimuli and emotions is also compelling
from a psychological point of view, given that emotions are commonly
considered responses to relevant situations Scherer (2005).

Models which tackle the task of detecting the stimulus in a text have seen
three different problem formulations in the past: (1) Classification into a
predefined inventory of possible stimuli Mohammad et al. (2014), similarly to
previous work in sentiment analysis Ganu et al. (2009), (2) classification of
precalculated or annotated clauses as containing a stimulus or not (Gui et
al., 2016, i.a.), and (3) detecting the tokens that describe the stimulus,
e.g., with IOB labels (Ghazi et al., 2015, i.a.). We follow the two settings
in which the stimuli are not predefined categories (2+3, cf. Figure 1).

Clause-based Classification:
No Stimulus Stimulus [ She’s pleased at ] [ how things have turned out . ]
Token Sequence Labeling:
O O O O B I I I I O She ’s pleased at how things have turned out .

Figure 1: Different formulations for emotion stimulus detection. Figure 2:
Framework for emotion stimulus detection. Tokens t are split into clauses for
clause class. Mapping ensures that both methods result in clause
classifications $(\textbf{t},\textbf{l})_{i}$ and token sequences with labels
$(\textbf{c},\textbf{y})_{j}$.

These two settings have their advantages and disadvantages. The clause
classification setting is more coarse-grained and, therefore, more likely to
perform well than the token sequence labeling setting, but it might miss the
exact starting and endpoints of a stimulus span and needs clause annotations
or a syntactic parse with the risk of error propagation. The token sequence
labeling setting might be more challenging, but has the potential to output
more exactly which tokens belong to the stimulus. Further, sequence labeling
is a more standard machine learning setting than a pipeline of clause
detection and classification.

These two different formulations are naturally evaluated in two different ways
and have not been compared before, to the best of our knowledge. Therefore, it
remains unclear which task formulation is more appropriate for English.
Further, the most recent approaches have been evaluated only on Mandarin
Chinese, with the only exception being the EmotionCauseAnalysis dataset being
considered by Fan et al. (2019), but not in comparison to token sequence
labeling. No other English emotion stimulus data sets have been tackled with
clause classification methods. We hypothesize that clauses are not appropriate
units for English, as Ghazi et al. (2015) already noted that: “such
granularity [is] too large to be considered an emotion stimulus in English”. A
similar argument has been brought up during the development of semantic role
labeling methods: Punyakanok et al. (2008) stated that “argument[s] may span
over different parts of a sentence”.

Our contributions are as follows: (1) we develop an integrated framework that
represents different formulations for the emotion stimulus detection task and
evaluate these on four available English datasets; (2) as part of this
framework, we propose a clause detector for English which is required to
perform stimulus detection via clause classification in a real-world setting;
(3) show that token sequence labeling is indeed the preferred approach for
stimulus detection in most available English datasets; (4) show in an error
analysis that this is mostly because clauses are not the appropriate unit for
stimuli in English. Finally, (5), we make our implementation and annotations
for both clauses and tokens available at http://www.ims.uni-
stuttgart.de/data/emotion-stimulus-detection.

The remainder of the paper is organized as follows. We first introduce our
integrated framework of stimulus detection which enables us to evaluate clause
classification and token sequence labeling in a comparable manner (Section 2).
We then turn to the experiments (Section 3) in which we analyze results on
four different English data sets. Section 4 discusses typical errors in
detail, which leads to a better understanding of how stimuli are formulated in
English. We conclude in Section 6.

##  2 An Integrated Framework for Stimulus Detection

The two approaches for open-domain stimulus detection, namely, clause
classification and token sequence labeling, have not been compared on English.
We propose an integrated framework (Figure 2) which takes tokens t as input,
splits this sequence into clauses and classifies them (clause detection can be
bypassed if manual annotations of clauses are available). The token sequence
labeling does not rely on clause annotations. The output, either clauses c
with classifications y ($\textbf{y}\in\\{\textrm{yes},\textrm{no}\\}^{n}$) or
tokens t with labels l are then mapped to each other to enable a comparative
evaluation. We explain these steps in the following subsections.

###  2.1 Clause Extraction

The clause classification methods rely on representing an instance as a
sequence of clauses. Clauses in English grammar are defined as the smallest
grammatical structures that contain a subject and a predicate, and can express
a complete proposition Kroeger (2005). We show our algorithm to detect clauses
in Algorithm 1.

Input: text

Output: Clauses $\mathbf{c}$

1 $\mathbf{t}$ $\leftarrow$ tokenize(text)

tree $\leftarrow$ parse($\mathbf{t}$)

// constituency parse

gaps $\leftarrow\\{0,|\mathbf{t}|\\}$

// potential clause bounds

segments $\leftarrow\varnothing$

// initial. set of segments

2 foreach  _node $n$ in tree_ do

3 if  _label(n) $\in$ S, SBAR, SBARQ, INV, SQ_

4 $\ell$ $\leftarrow$ first token leaf that $n$ governs

5 $r$ $\leftarrow$ last token leaf that $n$ governs

6 gaps = gaps $\cup\;\\{$idxℓ, idxr \+ 1$\\}$

7foreach  _adjacent pair $(i,j)$ in sort(gaps)_ do

8 segments = segments $\cup{}\;\mathbf{t}[i:j]$

9repeat

10 foreach  _$s_{i}$ in segments_ do

11 if  _$s_{i}\sim=$ /^[^A-za-z0-9]+$/_

12 $s_{i-1}=s_{i-1}\mathbin{\|}s_{i}$

13 segments = segments$\setminus s_{i}$

14 if  _$|s_{i}|\leq 3$_

15 $s_{i+1}=s_{i}\mathbin{\|}s_{i+1}$

16 segments = segments$\setminus s_{i}$

17until  _convergence_

return _segments_

Algorithm 1 Clause Extraction

To mark the segments that would potentially approximate clauses, we rely on
the constituency parse tree of the token sequence (Line 1). For that reason,
we use the Berkeley Neural Parser Kitaev and Klein (2018). As illustrated by
Feng et al. (2012) and Tafreshi and Diab (2018) we also do that by segmenting
the constituency parse tree of the instance (Line 1) at the borders of
constituents labeled as clause-type Bies et al. (1995). We then join the
segments until convergence heuristically based on punctuation (Line 1). We
illustrate the algorithm in the example in Figure 3.

[.S [.SBARQ a b ] [.N c ] ] gaps = {0, $|a\;b\;c|$} = {0, 3} Go over nodes
tagged S, SBAR, … On node SBARQ (a b) Add $idx_{\ell}$ (0) to gaps, new gaps:
{0, 3} Add $idx_{r}+1$ (2) to gaps, new gaps: {0, 3, 2} On node S (a b c) Add
$idx_{\ell}$ (0) to gaps, new gaps: {0, 3, 2} Add $idx_{r}+1$ (3) to gaps, new
gaps: {0, 3, 2} segments = $\varnothing$ For each pair $i,j$ in sorted gaps
({0, 2, 3}) i=0, j=2 Append tokens[0:2] (a b) to segments, new segments: [a b]
i=2, j=3 Append tokens[2:3] (c) to segments, new segments: [a b, c] Return
segments: [a b, c]

Figure 3: Example for the application of Algorithm 1.

###  2.2 Stimulus Detection

Our goal is to compare sequence labeling and clause classification. To
attribute the performance of the model to the formulation of the task, we keep
the differences between the models at a minimum. We therefore first discuss
the model components and then how we put them together.

Our models are composed of four layers. As Embedding Layer, we use pretrained
embeddings to embed each token in the instance $s=t_{1}\ldots t_{n}$ to obtain
$\vec{e}_{1},\ldots,\vec{e}_{n}$. For the Encoding Layer, we use a
bidirectional LSTM which outputs a sequence of hidden states
$\vec{h}_{1},...,\vec{h}_{n}$. In an additional Attention Layer, each word or
clause is represented as the concatenation of its embedding and a weighted
average over other words or clauses in the instance:
$\vec{u}_{i}=[\vec{h}_{i};\sum_{j=1}^{n}a_{i,j}\cdot\vec{h}_{j}]$. The weights
$a_{i,j}$ are calculated as the dot-product between $\vec{h}_{i}$ and every
other word, and by normalizing the scores using softmax
$\vec{a}_{i}=\operatorname{softmax}(\vec{h}_{i}^{T}\cdot\vec{h}_{j})$. We
concatenate all representations to obtain the final representation vector
$\vec{s}$. The Output Layer is different for the two different task
formulations (sequence labeling vs. single softmax). For the case of the
single softmax, the input to the classifier is the representation of the
clause obtained on the previous layer and the classifier output is defined as
$\vec{o_{i}}=\operatorname{softmax}(W\cdot\operatorname{ReLU}(\operatorname{Dropout}(h(\vec{s}))))$.
When labels are not predicted independently from each other but rather in a
sequential manner, we use a linear-chain conditional random field Lafferty et
al. (2001). It takes the sequence of probability vectors from the previous
layer $\vec{u_{1}},\vec{u_{2}},\ldots$ and outputs a sequence of labels
$\vec{y_{1}},\vec{y_{2}},\ldots$. The score of the labeled sequence is defined
as the sum of the probabilities of individual labels and the transition
probabilities:
$s(y_{1:n})=\sum_{i=1}^{n}\vec{u}_{i}(y_{i})+\sum_{i=2}^{n}T[y_{i-1},y_{i}]$,
where the matrix $T$ that contains the transition probabilities between one
label and another (i.e., $T[i,j]$ represents the probability that a token
labeled $i$ is followed by a token labeled $j$). At prediction time, the most
likely sequence is chosen with the Viterbi algorithm Viterbi (1967).

With these components, we can now put together the actual models which we use
for stimulus detection. We compare three different models, one for token
sequence labeling (SL) and two for clause classification (CC). The model
architectures are illustrated in Figure 4.

##### Token Sequence Labeling (SL).

In this model, we formulate emotion stimulus detection as token sequence
labeling with the IOB alphabet Ramshaw and Marcus (1995). As embeddings, we
use word-level GloVe embeddings Pennington et al. (2014). The sequence-to-
sequence architecture comprises a bidirectional LSTM, an attention layer and
the CRF output layer.

##### Independent Clause Classification (ICC).

This model, similarly proposed by Cheng et al. (2017), takes

Figure 4: Comparable model architectures.

the clauses from the clause detector (or from annotated data) and classifies
them as containing the stimulus or not. The model has a similar architecture
to the one before, with the exception of the final classifier, which is a
single softmax to output a single label. The training objective is to minimize
the cross-entropy loss. This model does not have access to clauses other than
the one it predicts for.

##### Joint Clause Classification (JCC).

In this model, the neural architecture we employ is slightly different from
before to enable it to make a prediction for clauses in the context of all
clauses. It comprises multiple LSTM modules as word-level encoders, one for
each clause. The LSTM at the word-level encodes the tokens of one clause into
one representation. The next layer is a clause-level encoder based on two
bidirectional LSTMs, where the clause representations are learned and updated
by integrating the relations between multiple clauses. After we obtain the
final clause representation for each clause, we perform sequence labeling with
a CRF _on the clause level_. The training objective is to minimize the
negative log-likelihood loss across all clauses. This implementation follows
the architecture by Xia et al. (2019), with the change of the upper layer,
which is, in our case, an LSTM clause encoder and not a transformer, to keep
the architecture comparable across our different formulations. Therefore, this
is comparable to all other hierarchical models proposed for the task Ding et
al. (2019); Xu et al. (2019); Xia and Ding (2019).

###  2.3 Mapping between Task Formulations

The last component of our integrated framework maps the different
representations of each formulation of emotion stimulus detection between each
other, namely clause classifications to token sequence labeling and vice
versa. We obtain clause classifications from token label sequences
($T\rightarrow C$ in Figure 2) by accepting any clause that has at least one
token being labeled as B or I as a stimulus clause. The other way around,
clause classes are mapped to tokens ($C\rightarrow T$) in such a way that the
first token of a stimulus clause is a B and all the remaining tokens in the
respective clause are I. Tokens from clauses that do not correspond to a
stimulus all receive O labels.

##  3 Experiments and Results

We now put the models to use to understand the differences between sequence
labeling and clause classification for English emotion stimulus detection and
the suitability of clauses as the unit of analysis.

Data set | Size | Stimuli | Tokens | Clauses
---|---|---|---|---
|  |  | $\mu$ | $\sigma$ |  $\mu$S/I |  $\mu$S/C | Total | w. S |  $\mu$ I |  $\mu$ w. all S/I
EmotionStimulus | 2,414 | $820$ | 7.29 | 5.20 | 0.12 | 0.11 | $5,818$ | $1,117$ | 2.41 | 0.05
ElectoralTweets | 4,056 | $2,427$ | 6.22 | 4.00 | 0.20 | 0.17 | $13,612$ | $3,295$ | 3.36 | 0.34
GoodNewsEveryone | 5,000 | $4,798$ | 7.27 | 3.67 | 0.55 | 0.50 | $9,190$ | $6,301$ | 1.84 | 0.52
Emotion Cause Ana. | 2,655 | $2,580$ | 8.48 | 5.20 | 0.20 | 0.10 | $19,473$ | $2,897$ | 7.33 | 0.33
Table 1: Data sets available for the Emotion Stimulus Detection task in
English. Size: number of annotated instances, Stimuli : number of instances
with stimuli annotated; $\mu$, $\sigma$: mean/standard deviation of length of
stimuli in tokens; $\mu$S/I: mean number of stimulus tokens per instance;
$\mu$S/C: mean number of stimulus tokens per clause; Total: total number of
clauses, w. S: number of clauses that contain a stimulus; $\mu$ I: average
number of clauses per instance; $\mu$ w. all S/I: average number of clauses in
which all tokens correspond to annotated stimuli.

###  3.1 Data Sets

We base our experiments on four data sets.111Corpora which we do not consider
for our experiments are discussed in the related work section. For each data
set, we report the size, the number of stimulus annotations and statistics for
tokens and clauses in Table 1.

#####  EmotionStimulus.

This data set proposed by Ghazi et al. (2015) is constructed based on
FrameNet’s _emotion-directed_
frame.222https://framenet2.icsi.berkeley.edu/fnReports/data/frameIndex.xml?frame=Emotion_directed
The authors used FrameNet’s annotated data for 173 emotion lexical units,
grouped the lexical units into seven basic emotions using their synonyms and
built a dataset manually annotated with both the emotion stimulus and the
emotion. The corpus consists of 820 sentences with annotations of emotion
categories and stimuli. The rest of 1,594 sentences only contain an emotion
label. For this dataset, we see the lowest average number of clauses for which
all tokens correspond to a stimulus ($\mu$ w. all S/I in Table 1). This result
shows that the stimuli annotations rarely align with the clause boundaries.

#####  ElectoralTweets.

Frame Semantics also inspires a dataset of social media posts Mohammad et al.
(2014). The corpus consists of 4,056 tweets of which 2,427 contain emotion
stimulus annotations on the token level. The annotation was performed via
crowdsourcing. The tweets are the shortest instance type in length and have a
higher average of clauses per instance than the GoodNewsEveryone or the
EmotionStimulus datasets. They also show the same mean of stimulus tokens per
instance as EmotionCauseAnalysis with a slightly higher mean for the number of
clauses in which all tokens correspond to stimulus annotations.

#####  GoodNewsEveryone.

The data set by Bostan et al. (2020) consists of news headlines. From a total
of 5000 instances, 4,798 contain a stimulus. The headlines have the shortest
stimuli in token count. Similar to the ElectoralTweets, they also have a high
average stimulus token density in clauses. This set has the lowest mean number
of clauses per instance ($\mu$ I in Table 1).

#####  EmotionCauseAnalysis

Gao et al. (2017) comparably annotate English and Mandarin texts on the clause
level and the token level. In our work, we use the English subset, which is
the only English corpus annotated for stimuli both at the clause level and at
the token level. This dataset has the fewest instances without stimuli among
all the others. It also has the longest instances and stimuli. The mean of
stimuli tokens annotated per clause is comparable to EmotionStimulus despite
having a higher mean of stimuli tokens per instance. In the upcoming
experiments, we use the clause annotations and not automatically recognized
clauses with Algorithm 1 as input to our framework.

|  | Intrinsic | Extrinsic
---|---|---|---
| IAA | Stimuli vs. Anno. Clauses | Extra. vs. Anno. Clauses | Stimuli vs. Extra. Clauses
Dataset | $\kappa$ | Exact | Left | Right | Precision | Recall | F1 | Exact | Left | Right
EmotionCauseAnalysis | 0.60 | 0.60 | 0.35 | 0.86 | 0.77 | 0.75 | 0.76 | 0.59 | 0.36 | 0.84
GoodNewsEveryone | 0.77 | 0.62 | 0.29 | 0.90 | 0.87 | 0.76 | 0.80 | 0.61 | 0.27 | 0.89
EmotionStimulus | 0.59 | 0.47 | 0.83 | 0.11 | 0.86 | 0.72 | 0.76 | 0.17 | 0.26 | 0.07
ElectoralTweets | 0.63 | 0.56 | 0.39 | 0.63 | 0.82 | 0.78 | 0.80 | 0.54 | 0.43 | 0.60
Table 2: Evaluation of Clause Detection. Note that for EmotionCauseAnalysis,
the clauses stem from the annotation provided in the original data and not
from our automatic detection method.

###  3.2 Clause Identification Evaluation

Before turning to the actual evaluation of the emotion stimulus detection
methods, we evaluate the quality of the automatic clause detection. For an
intrinsic evaluation, we annotate 50 instances from each test corpus in each
data set with two annotators trained on the clause extraction task in two
iterations. The two annotators are graduate students and have different
scientific backgrounds: computational linguistics (A1) and computer science
with a specialization in computer vision (A2). Each student annotated 50
instances of each dataset from the datasets we use in the same order. As an
environment for the annotation process, we used a simple spreadsheet
application. We did this small annotation experiment as an inner check for our
understanding of the clause extraction task. None of the annotators is a
native English speaker; A1 is a native speaker of a Romance language, and A2 a
German speaker. The inter-annotator agreement is shown in Table 2. We achieve
an acceptable average agreement of $\kappa$=.65.

We now turn to the question if annotated clauses (as an upper bound to an
automatic system) align well with annotated stimuli (Stimuli vs. Anno. Clauses
in Table 2). The evaluation is based on recall (i.e., measuring for how many
stimuli a clause exists), either for the whole stimulus (exact), or for the
left or the right boundary. We see that except for the corpus EmotionStimulus,
the right boundaries match better than the left.

Turning to extracted clauses instead of annotated ones (Extra. vs. Anno.
Clauses) we first evaluate the automatic extraction algorithm. We obtain
$\textrm{F}_{1}$ values between 0.76% and 0.80%, which we consider acceptable
though they also show that error propagation could occur.

For the actual extrinsic evaluation, if clause boundaries are correctly found
for annotated stimuli (Stimuli vs. Extra. Clauses), we see that the results
are only slightly lower than for the gold annotations, except for
EmotionStimulus. Therefore, we do not expect to see error propagation due to
an imperfect extraction algorithm for most data sets.

These results suggest that clauses are not an appropriate unit for stimuli in
English. Still, we do not know yet if the clause detection task’s simplicity
outweighs these disadvantages in contrast to token sequence labeling. We turn
to answer this in the following.

Figure 5: Results of the three different models across four different datasets

###  3.3 Stimulus Detection Evaluation

####  3.3.1 Evaluation Procedure

We evaluate the quality of all models with five different measures. Motivated
by the formulation of _clause classification_ , we (1) evaluate the prediction
on the clause level with precision, recall, and $\textrm{F}_{1}$. For the
_sequence labeling_ evaluation, we use four variations. (2)  _Exact_ , where
we consider a consecutive token sequence to be correct if a gold annotation
exists that exactly matches, (3)  _Relaxed_ , where an overlap of one token
with a gold annotation is sufficient, (4)  _Left-Exact_ and (5)  _Right-Exact_
, where at least the most left/right token in the prediction needs to have a
gold-annotated counterpart.

One might argue that sequence labeling evaluation is unfair for the clause
classification, as it is more fine-grained than the actual prediction method.
However, for transparency across methods and analysis of advantages and
disadvantages of the different methods, we use this approach in addition to
clause classification evaluation.

We split the data for each set randomly into three sets: $80\%$ train, $10\%$
dev, and $10\%$ test. We use dropout with a probability of 0.5, train with
Adam Kingma and Ba (2015) with a base learning rate of 0.003, and a batch size
of 10. At test time, we select the model with the best validation accuracy
after 50 epochs with a patience of 10 epochs. All models use embedding sizes
of 300 and hidden state sizes of 100 Pennington et al. (2014). We do not tune
hyperparameters for any of the architectures and implement all models with the
AllenNLP library Gardner et al. (2018).

|  | SL | ICC | JCC |
---|---|---|---|---|---
$\textrm{Ann.}\over\textrm{Pred.}$ | Error types | ET | GNE | ES | ECA | ET | GNE | ES | ECA | ET | GNE | ES | ECA | Sum
| Early stop | 0 | 4 | 1 | 3 | 0 | 6 | 2 | 7 | 0 | 5 | 1 | 4 | 33
| Late stop | 11 | 9 | 10 | 8 | 19 | 30 | 7 | 25 | 17 | 31 | 13 | 22 | 202
| Early start & stop | 0 | 3 | 0 | 1 | 9 | 11 | 5 | 1 | 6 | 10 | 3 | 2 | 51
| Early start | 152 | 16 | 0 | 6 | 192 | 73 | 9 | 164 | 220 | 58 | 3 | 159 | 1052
| Late start | 28 | 3 | 0 | 1 | 3 | 8 | 1 | 0 | 2 | 7 | 1 | 0 | 54
| Late start & stop | 2 | 1 | 0 | 0 | 0 | 2 | 0 | 1 | 0 | 1 | 0 | 1 | 8
| Contained | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 2 | 3
| Multiple | 143 | 189 | 11 | 260 | 47 | 24 | 9 | 11 | 37 | 34 | 4 | 0 | 769
| Surrounded | 9 | 10 | 0 | 5 | 19 | 31 | 28 | 43 | 22 | 33 | 26 | 28 | 254
| False Negative | 231 | 160 | 59 | 228 | 126 | 112 | 10 | 97 | 85 | 50 | 1 | 81 | 1240
/ | False Positives | 10 | 18 | 2 | 14 | 78 | 92 | 11 | 38 | 60 | 73 | 4 | 26 | 426
| All | 586 | 413 | 83 | 526 | 493 | 389 | 82 | 388 | 449 | 302 | 56 | 325 | 4092
Table 3: Counts for each error type for each model across all data sets.

####  3.3.2 Results

We now study the performance of the different models on the English data sets.
Figure 5 summarizes the results. (Precision and recall values are available in
Table 7 in Appendices.)

##### Which of the modeling approaches performs best on English data?

If we only compare the absolute numbers in $\textrm{F}_{1}$, we see that the
clause classification evaluation (Class) shows the highest result across all
models and data set. The only exception is the EmotionStimulus data, in which
the Left-Exact evaluation is slightly higher. When we rely on this evaluation
score, we see that the token sequence labeling method shows a superior result
to the classification methods in two data sets, namely GoodNewsEveryone and
EmotionCauseAnalysis. On ElectoralTweets and EmotionStimulus, the results are
en par across all methods with this evaluation measure. We find this
surprising to some degree, as this evaluation is more natural for the
classification tasks (ICC and JCC) than for sequence labeling (SL), which
requires the mapping step.

As this suggests that clauses are not the appropriate unit, it is worth
comparing these results with the Exact evaluation measure, which evaluates on
the token-sequence level. We observe that token sequence labeling outperforms
both clause classification methods on three of the four data sets, with
ElectoralTweets being the only exception with the shortest textual instances
and the highest number of clauses in which all tokens correspond to stimulus
annotation (see Table 1). Therefore, we conclude that token sequence labeling
is superior to clause classification on (most of our) English data sets.

##### Do clause classification models perform better on the left or the right
side of the stimulus clause?

Given the evaluation of the clause detection, we expect the right boundary to
be better found for GoodNewsEveryone and EmotionCauseAnalysis and the left
boundary for EmotionStimulus. Surprisingly, this is not entirely true – the
right boundary is found with higher $\textrm{F}_{1}$ on all data sets, not
only on those where the clauses are better aligned with the stimulus’ right
boundary. Nevertheless, the effect is more reliable for GoodNewsEveryone, as
expected.

##### Does token sequence labeling perform better on the left or the right
side of the stimulus clause?

We can ask this similar question for token sequence labeling, though it might
be harder to motivate than in the classification setting. Non-surprisingly,
such a clear pattern cannot be observed. For ElectoralTweets and
EmotionCauseAnalysis, the difference between the left and right match is
minimal. For GoodNewsEveryone, it can be observed to a lesser extent than for
the classification approaches, and for EmotionStimulus, the left boundary is
better found than the right boundary. It seems that for the longer sequences
in EmotionStimulus and EmotionCauseAnalysis, the beginning of the stimulus
span is easier to find than for shorter sequences.

##### Is joint prediction of clause labels beneficial?

This hypothesis can be confirmed; however, the differences are of a different
magnitude depending on the data set. For GoodNewsEveryone, the effect is more
substantial than for the other corpora. ElectoralTweets shows the smallest
difference.

Err | Example | Model | Data set
---|---|---|---
|  Steve talked to me a lot — about being abandoned — and the pain — that caused. | JCC | ECA
| No what I told about the way — they treated you and me — made him angry. | SL | ECA
|  Fuck Mitt Romney — and Fuck Barack Obama — … God got me !!!!! | ICC | ET
|  Maurice Mitchell wants — you to do more than vote. | ICC | GNE
| And he started — to despair that his exploration was going — to be entirely unsuccessful … | ICC | ECA
|  Deeply ashamed of my wayward notions, — I tried my best to contradict myself. | ICC | ES
| Anyone else find it weird — I get excited about stuff like the RNC tonight ?! — # polisciprobs | SL | ET
|  Doesn’t he do it well — said the girl following with admiring eyes, — every movement of him. | JCC | ECA
| If he feared — that some terrible secret might evaporate from them, — it was a mania with him. | SL | ECA
| I was furious — because the Mac XL wasn’t real said Hoffman. | SL | ECA
| With such obvious delight in food, it ’s hard — to see how Blanc remains so slim. | SL | ES
| Triad Thugs Use Clubs to Punish Hong Kong ’ s Protesters. | JCC | GNE
|  I’m glad to see you — so happy Lupin | ICC | ES
Figure 6: Examples for error types for different models and data sets.
Extracted clauses are separate by —.

##  4 Error Analysis

In the following, we analyze the error types made by the different models on
all data sets and investigate in which ways SL improves over the ICC and JCC
models. We hypothesize that the higher flexibility of token-based sequence
labeling leads to different types of errors than the clause-based
classification models.

For quantitative analysis, we define different error types, illustrated in
Table 3 with different symbols as abbreviations. The top bar illustrates the
gold span, while the bottom corresponds to the predicted span. The error types
illustrated with symbols  and  correspond to false positives;  are false
negatives. All other error types correspond to either both false positive and
false negative in a strict evaluation setting or true positives in one of the
relaxed evaluation settings.

##### Do ICC and JCC particularly miss starting or end points of the stimulus
annotation?

We see in Table 3 that for Late stop , CC models make considerably more
mistakes across all datasets. ICC does so on ET and ECA, while JCC makes more
mistakes on GNE and ES. For data sets in which stimulus annotations end with a
clause, errors of this type are less likely. These results are more prominent
for Early start & stop .

##### Do all methods have similar issues with finding the whole consecutive
stimulus?

We see this in the error type Multiple . When the CC models make this mistake,
it can be attributed to the automatic fine-grained clause extraction, which
can cause a small clause within a gold span to become a false negative.
However, we see that SL shows higher numbers of this issue than CC. This
result is also reflected in the surprisingly low number of Contained () – if
the prediction is completely inside a gold annotation, the gold annotation
tends to be long, and this increases the chance that it is (wrongly) split
into multiple predictions.

##### How do the error types differ across models?

The Early Start (& Stop) and Surrounded (, , ) counts show differences across
the different types of models. Presumably, the clause classification models do
have difficulties in finding the left boundary, and they are more prone to
“start early” than the token sequence labeling models. This might be due to
gold spans starting in the middle of a clause which is predicted to contain
the stimulus.

##### How do the error types differ across data sets?

The results and error types differ across data sets (see particularly , , ).
This points out what we have seen in the evaluation already: The structure of
a stimulus depends on the domain and annotation. The least challenging data
set is EmotionStimulus with the lowest numbers of errors across all models.
This result is caused by most sentences having similar syntactic trees, all
stimuli are explicit and mostly introduced in a similar way.

For qualitative analyses, Figure 6 shows one example of each type of error
described above. In the first example, the JCC model does not learn to include
the second part of the coordination – “and the pain”. In the second example,
similarly, the SL model misses the right part of the coordination. For most
cases of independent clauses that we inspect, we see a common pattern for both
types of models, which is that the prediction stops while encountering
coordinating conjunctions. In the sixth example, the prediction span includes
the emotion cue. This issue could be solved by doing sequence labeling instead
or by informing the model of the presence of other semantic roles. These
examples raise the following question: would improved clause segmentation lead
to improvements for the clause-classification models across all data sets?

##  5 Related Work

The task of detecting the stimulus of an expressed emotion in text received
relatively little attention.

Next to the corpora we mentioned so far, the REMAN corpus Kim and Klinger
(2018) consists of English excerpts from literature, sampled from Project
Gutenberg. The authors consider triples of sentences as a trade-of between
longer passages and sentences. Further, Neviarouskaya and Aono (2013)
annotated English sentences on the token level.

Besides English and Mandarin, Russo et al. (2011) developed a method for the
identification of Italian sentences that contain an emotion cause phrase. Yada
et al. (2017) annotate Japanese sentences on newspaper articles, web news
articles, and Q&A sites. Table 8 in Appendices shows which corpora and methods
have been used and compared in previous work for the available English and
Chinese sets. We see that the methods applied on the Chinese sets are not
evaluated on the English sets.

Lee et al. (2010) firstly investigated the interactions between emotions and
the corresponding stimuli from a linguistic perspective. They publish a list
of linguistic cues that help in identifying emotion stimuli and develop a
rule-based approach. Chen et al. (2010) build on top of their work to develop
a machine learning method. Li and Xu (2014) implement a rule-based system to
detect the stimuli in Weibo posts and further inform an emotion classifier
with the output of this system. Other approaches to develop rules include
manual strategies Gao et al. (2015), bootstrapping Yada et al. (2017) and the
use of constituency and dependency parsing Neviarouskaya and Aono (2013).

All recently published state-of-the-art methods for the task of emotion
stimulus detection via clause classification are evaluated on the Mandarin
data by Gui et al. (2016). They include multi-kernel learning Gui et al.
(2016) and long short-term memory networks (LSTM) Cheng et al. (2017). Gui et
al. (2017) propose a convolutional multiple-slot deep memory network (ConvMS-
Memnet), and Li et al. (2018) a co-attention neural network model, which
encodes the clauses with a co-attention based bi-directional long short-term
memory into high-level input representations, which are further passed into a
convolutional layer. Ding et al. (2019) proposed an architecture with
components for “position augmented embedding” and “dynamic global label” which
takes the relative position of the stimuli to the emotion keywords and use the
predictions of previous clauses as features for predicting subsequent clauses.
Xia et al. (2019) integrate the relative position of stimuli and evaluate a
transformer-based model that classifies all clauses jointly within a text.
Similarly, Yu et al. (2019) proposes a word-phrase-clause hierarchical
network. The transformer-based model achieves state of the art, however, it is
shown that the RNN based encoders are very close in performance Xia et al.
(2019). Therefore, we use a comparable model that is grounded on the same
concept of a hierarchical setup with LSTMs as encoders. Further, there is a
strand of research which jointly predicts the clause that contains the emotion
stimulus together with its emotion cue Wei et al. (2020); Fan et al. (2020).
However, the comparability of methods across data sets has been limited in
previous work, as Table 8 in the appendices shows.

##  6 Conclusion

We contributed to emotion stimulus detection in two ways. Firstly, we
evaluated emotion stimulus detection across several English annotated data
sets. Secondly, we analyzed if the current standard formulation for stimulus
detection on Mandarin Chinese is also a good choice for English.

We find that the domain and annotation of the data sets have a large impact on
the performance. The worst performance of the token sequence labeling approach
is obtained on the crowdsourced data set ElectoralTweets. The well-formed
sentences of EmotionStimulus pose fewer difficulties to our models than tweets
and headlines. We see that the sequence labeling approaches are more
appropriate for the phenomenon of stimulus mentions in English. This shows in
the evaluation of the comparably coarse-grained clause level and is also
backed by our error analysis.

For future work, we propose closer investigation of whether other smaller
constituents might represent the stimulus better for English and a check of
whether the strong results for the sequence labeling hold for other languages.
Notably, the clause classification setup has its benefits, and this might lead
to a promising setting as joint modeling or as a filtering step to finding
parts of the text which might contain a stimulus mention. Another step is to
investigate if the emotion stimulus and the emotion category classification
benefit from joint modeling in English as it has been shown for Mandarin Chen
et al. (2018).

## Acknowledgments

This research has been conducted within the project SEAT (Structured Multi-
Domain Emotion Analysis from Text, KL 2869/1-1), funded by the German Research
Council (DFG). We thank Enrica Troiano, Evgeny Kim, Gabriella Lapesa, and Sean
Papay for fruitful discussions and feedback on earlier versions of the paper.

## References

  * Bies et al. (1995) Ann Bies, Mark Ferguson, Karen Katz, Robert MacIntyre, Victoria Tredinnick, Grace Kim, Mary Ann Marcinkiewicz, and Britta Schasberger. 1995.  Bracketing guidelines for Treebank II style Penn Treebank project.  Online: http://languagelog.ldc.upenn.edu/myl/PennTreebank1995.pdf.
  * Bostan et al. (2020) Laura Ana Maria Bostan, Evgeny Kim, and Roman Klinger. 2020.  GoodNewsEveryone: A corpus of news headlines annotated with emotions, semantic roles, and reader perception.  In _Proceedings of The 12th Language Resources and Evaluation Conference_ , pages 1554–1566, Marseille, France. European Language Resources Association.
  * Chen et al. (2018) Ying Chen, Wenjun Hou, Xiyao Cheng, and Shoushan Li. 2018.  Joint learning for emotion classification and emotion cause detection.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 646–651, Brussels, Belgium. Association for Computational Linguistics.
  * Chen et al. (2010) Ying Chen, Sophia Yat Mei Lee, Shoushan Li, and Chu-Ren Huang. 2010.  Emotion cause detection with linguistic constructions.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_ , pages 179–187, Beijing, China. Coling 2010 Organizing Committee.
  * Cheng et al. (2017) Xiyao Cheng, Ying Chen, Bixiao Cheng, Shoushan Li, and Guodong Zhou. 2017.  An emotion cause corpus for chinese microblogs with multiple-user structures.  _ACM Transactions on Asian and Low-Resource Language Information Processing_ , 17(1):6:1–6:19.
  * Ding et al. (2019) Zixiang Ding, Huihui He, Mengran Zhang, and Rui Xia. 2019.  From independent prediction to reordered prediction: Integrating relative position and global label information to emotion cause identification.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 6343–6350.
  * Ekman (1992) Paul Ekman. 1992.  An argument for basic emotions.  _Cognition & emotion_, 6(3-4):169–200.
  * Fan et al. (2019) Chuang Fan, Hongyu Yan, Jiachen Du, Lin Gui, Lidong Bing, Min Yang, Ruifeng Xu, and Ruibin Mao. 2019.  A knowledge regularized hierarchical approach for emotion cause analysis.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5614–5624, Hong Kong, China. Association for Computational Linguistics.
  * Fan et al. (2020) Chuang Fan, Chaofa Yuan, Jiachen Du, Lin Gui, Min Yang, and Ruifeng Xu. 2020.  Transition-based directed graph construction for emotion-cause pair extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3707–3717, Online. Association for Computational Linguistics.
  * Feng et al. (2012) Song Feng, Ritwik Banerjee, and Yejin Choi. 2012.  Characterizing stylistic elements in syntactic structure.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 1522–1533, Jeju Island, Korea. Association for Computational Linguistics.
  * Ganu et al. (2009) Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009.  Beyond the stars: Improving rating predictions using review text content. In _Twelfth International Workshop on the Web and Databases (WebDB 2009)_.
  * Gao et al. (2015) Kai Gao, Hua Xu, and Jiushuo Wang. 2015.  A rule-based approach to emotion cause detection for chinese micro-blogs.  _Expert Systems with Applications_ , 42(9):4517–4528.
  * Gao et al. (2017) Qinghong Gao, Jiannan Hu, Ruifeng Xu, Gui Lin, Yulan He, Qin Lu, and Kam-Fai Wong. 2017.  Overview of NTCIR-13 ECA task.  In _Proceedings of the 13th NTCIR Conference on Evaluation of Information Access Technologies_ , pages 361–366, Tokyo, Japan.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018.  AllenNLP: A deep semantic natural language processing platform.  In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_ , pages 1–6, Melbourne, Australia. Association for Computational Linguistics.
  * Ghazi et al. (2015) Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015.  Detecting emotion stimuli in emotion-bearing sentences.  In _International Conference on Intelligent Text Processing and Computational Linguistics_ , pages 152–165. Springer.
  * Gui et al. (2017) Lin Gui, Jiannan Hu, Yulan He, Ruifeng Xu, Qin Lu, and Jiachen Du. 2017.  A question answering approach for emotion cause extraction.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1593–1602, Copenhagen, Denmark. Association for Computational Linguistics.
  * Gui et al. (2016) Lin Gui, Dongyin Wu, Ruifeng Xu, Qin Lu, and Yu Zhou. 2016.  Event-driven emotion cause extraction with corpus construction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1639–1649, Austin, Texas. Association for Computational Linguistics.
  * Jakob and Gurevych (2010) Niklas Jakob and Iryna Gurevych. 2010.  Extracting opinion targets in a single and cross-domain setting with conditional random fields.  In _Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing_ , pages 1035–1045. Association for Computational Linguistics.
  * Kim and Klinger (2018) Evgeny Kim and Roman Klinger. 2018.  Who feels what and why? annotation of a literature corpus with semantic roles of emotions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1345–1359. Association for Computational Linguistics.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Kitaev and Klein (2018) Nikita Kitaev and Dan Klein. 2018.  Constituency parsing with a self-attentive encoder.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2676–2686, Melbourne, Australia. Association for Computational Linguistics.
  * Klinger and Cimiano (2013) Roman Klinger and Philipp Cimiano. 2013.  Bi-directional inter-dependencies of subjective expressions and targets and their value for a joint model.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 848–854, Sofia, Bulgaria. Association for Computational Linguistics.
  * Kroeger (2005) Paul R. Kroeger. 2005.  _Analyzing grammar: An introduction_.  Cambridge University Press.
  * Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In _International Conference on Machine Learning_ , pages 282–289.
  * Lee et al. (2010) Sophia Yat Mei Lee, Ying Cohen, Shoushan Li, and Chu-Ren Huang. 2010.  Emotion cause events: Corpus construction and analysis.  In _Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10)_ , pages 1121–1128, Valletta, Malta. European Language Resources Association (ELRA).
  * Li and Xu (2014) Weiyuan Li and Hua Xu. 2014.  Text-based emotion classification using emotion cause extraction.  _Expert Systems with Applications_ , 41(4):1742–1749.
  * Li et al. (2018) Xiangju Li, Kaisong Song, Shi Feng, Daling Wang, and Yifei Zhang. 2018.  A co-attention neural network model for emotion cause analysis with emotional context awareness.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4752–4757, Brussels, Belgium. Association for Computational Linguistics.
  * Mohammad et al. (2014) Saif Mohammad, Xiaodan Zhu, and Joel Martin. 2014.  Semantic role labeling of emotions in tweets.  In _Proceedings of the 5th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 32–41, Baltimore, Maryland. Association for Computational Linguistics.
  * Neviarouskaya and Aono (2013) Alena Neviarouskaya and Masaki Aono. 2013.  Extracting causes of emotions from text.  In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_ , pages 932–936, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Plutchik (2001) Robert Plutchik. 2001.  The nature of emotions human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice.  _American Scientist_ , 89(4):344–350.
  * Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad AL-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, Véronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia Loukachevitch, Evgeniy Kotelnikov, Nuria Bel, Salud María Jiménez-Zafra, and Gülşen Eryiğit. 2016.  SemEval-2016 task 5: Aspect based sentiment analysis.  In _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 19–30, San Diego, California. Association for Computational Linguistics.
  * Pontiki et al. (2015) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015.  SemEval-2015 task 12: Aspect based sentiment analysis.  In _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 486–495, Denver, Colorado. Association for Computational Linguistics.
  * Punyakanok et al. (2008) Vasin Punyakanok, Dan Roth, and Wen-tau Yih. 2008.  The importance of syntactic parsing and inference in semantic role labeling.  _Computational Linguistics_ , 34(2):257–287.
  * Ramshaw and Marcus (1995) Lance Ramshaw and Mitch Marcus. 1995.  Text chunking using transformation-based learning.  In _Third Workshop on Very Large Corpora_ , pages 82–94.
  * Russo et al. (2011) Irene Russo, Tommaso Caselli, Francesco Rubino, Ester Boldrini, and Patricio Martínez-Barco. 2011.  EMOCause: An easy-adaptable approach to extract emotion cause contexts.  In _Proceedings of the 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA 2.011)_ , pages 153–160, Portland, Oregon. Association for Computational Linguistics.
  * Scherer (2005) Klaus R. Scherer. 2005.  What are emotions? And how can they be measured? _Social Science Information_ , 44(4):695–729.
  * Tafreshi and Diab (2018) Shabnam Tafreshi and Mona Diab. 2018.  Sentence and clause level emotion annotation, detection, and classification in a multi-genre corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , pages 1246–1251, Miyazaki, Japan. European Language Resources Association (ELRA).
  * Viterbi (1967) Andrew J. Viterbi. 1967.  Error bounds for convolutional codes and an asymptotically optimum decoding algorithm.  _IEEE Transactions on Information Theory_ , 13(2):260–269.
  * Wei et al. (2020) Penghui Wei, Jiahao Zhao, and Wenji Mao. 2020.  Effective inter-clause modeling for end-to-end emotion-cause pair extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3171–3181, Online. Association for Computational Linguistics.
  * Xia and Ding (2019) Rui Xia and Zixiang Ding. 2019.  Emotion-cause pair extraction: A new task to emotion analysis in texts.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1003–1012, Florence, Italy. Association for Computational Linguistics.
  * Xia et al. (2019) Rui Xia, Mengran Zhang, and Zixiang Ding. 2019.  RTHN: A RNN-Transformer Hierarchical Network for Emotion Cause Extraction.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)_ , pages 5285–5291, Macao, China. International Joint Conferences on Artificial Intelligence.
  * Xu et al. (2019) Bo Xu, Hongfei Lin, Yuan Lin, Yufeng Diao, Lian Yang, and Kan Xu. 2019\.  Extracting emotion causes using learning to rank methods from an information retrieval perspective.  _IEEE Access_ , 7:15573–15583.
  * Xu et al. (2017) Ruifeng Xu, Jiannan Hu, Qin Lu, Dongyin Wu, and Lin Gui. 2017.  An ensemble approach for emotion cause detection with event extraction and multi-kernel svms.  _Tsinghua Science and Technology_ , 22(6):646–659.
  * Yada et al. (2017) Shuntaro Yada, Kazushi Ikeda, Keiichiro Hoashi, and Kyo Kageura. 2017.  A bootstrap method for automatic rule acquisition on emotion cause extraction.  In _2017 IEEE International Conference on Data Mining Workshops (ICDMW)_ , pages 414–421.
  * Yang and Cardie (2013) Bishan Yang and Claire Cardie. 2013.  Joint inference for fine-grained opinion extraction.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1640–1649, Sofia, Bulgaria. Association for Computational Linguistics.
  * Yu et al. (2019) Xinyi Yu, Wenge Rong, Zhuo Zhang, Yuanxin Ouyang, and Zhang Xiong. 2019.  Multiple level hierarchical network-based clause selection for emotion cause extraction.  _IEEE Access_ , 7:9071–9079.

##  Appendix A Appendix

|  | SL Evaluation | CC Evaluation
---|---|---|---
|  | Exact | Relaxed | Left-Exact | Right-Exact | Clause
Data | Model | P | R | $\textrm{F}_{1}$ | P | R | $\textrm{F}_{1}$ | P | R | $\textrm{F}_{1}$ | P | R | $\textrm{F}_{1}$ | P | R | $\textrm{F}_{1}$
EmotionStimulus | SL | 69 | 73 | 71 | 69 | 74 | 72 | 100 | 74 | 85 | 100 | 59 | 74 | 81 | 83 | 82
ICC | 03 | 26 | 05 | 10 | 100 | 18 | 03 | 26 | 05 | 05 | 44 | 09 | 82 | 70 | 73
JCC | 05 | 12 | 07 | 21 | 42 | 30 | 12 | 10 | 12 | 17 | 46 | 24 | 84 | 80 | 82
ElectoralTweets | SL | 15 | 07 | 10 | 41 | 30 | 35 | 52 | 09 | 15 | 42 | 11 | 17 | 100 | 40 | 57
ICC | 12 | 47 | 19 | 22 | 100 | 37 | 13 | 47 | 21 | 21 | 74 | 32 | 59 | 59 | 59
JCC | 14 | 30 | 19 | 25 | 54 | 34 | 15 | 48 | 23 | 28 | 47 | 35 | 59 | 57 | 58
EmotionCauseAnalysis | SL | 16 | 20 | 18 | 42 | 60 | 49 | 76 | 29 | 41 | 83 | 23 | 36 | 99 | 68 | 80
ICC | 05 | 35 | 09 | 14 | 100 | 24 | 05 | 35 | 09 | 13 | 82 | 22 | 79 | 64 | 67
JCC | 06 | 29 | 10 | 18 | 40 | 25 | 11 | 15 | 12 | 35 | 29 | 31 | 82 | 68 | 74
GoodNewsEveryone | SL | 39 | 30 | 34 | 66 | 92 | 77 | 79 | 30 | 44 | 86 | 86 | 86 | 96 | 99 | 97
ICC | 15 | 29 | 19 | 37 | 100 | 54 | 15 | 29 | 20 | 48 | 92 | 63 | 71 | 67 | 68
JCC | 16 | 25 | 19 | 40 | 90 | 55 | 17 | 25 | 20 | 54 | 82 | 65 | 82 | 93 | 87

Figure 7: Results of the three different models across the five different datasets |  |  | Data sets and Annotation Approach
---|---|---|---
|  |  | Categorical Class. |  |  |  |  |
|  |  | & Sequence Lab. | Sequence Labeling | Clause Class.
|  |  | ET | ES | REMAN | GNE | ECA | EDCE
| Models | Papers | (en) | (en) | (en) | (en) | (en) | (zh)
Methods | CRF | Ghazi et al. (2015) | $-$ | + | $-$ | $-$ | $-$ | $-$
BiLSTM-CRF | Kim and Klinger (2018) | $-$ | $-$ | + | $-$ | $-$ | $-$
BiLSTM-CRF | Bostan et al. (2020) | $-$ | $-$ | $-$ | + | $-$ | $-$
SVM | Mohammad et al. (2014) | + | $-$ | $-$ | $-$ | $-$ | $-$
CRF | Gao et al. (2017) | $-$ | $-$ | $-$ | $-$ | + | $-$
LSTM | Cheng et al. (2017) | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
JMECause | Chen et al. (2018) | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
multi-kernel SVM | Xu et al. (2017) | $-$ | $-$ | $-$ | $-$ | $-$ | +
Multi-Kernel | Gui et al. (2016) | $-$ | $-$ | $-$ | $-$ | $-$ | +
ConvMS-Memnet | Gui et al. (2017) | $-$ | $-$ | $-$ | $-$ | $-$ | +
CANN | Li et al. (2018) | $-$ | $-$ | $-$ | $-$ | $-$ | +
PAE-DGL | Ding et al. (2019) | $-$ | $-$ | $-$ | $-$ | $-$ | +
HCS | Yu et al. (2019) | $-$ | $-$ | $-$ | $-$ | $-$ | +
Ranking | Xu et al. (2019) | $-$ | $-$ | $-$ | $-$ | $-$ | +
Hierarchical BiLSTM | Xia and Ding (2019) | $-$ | $-$ | $-$ | $-$ | $-$ | +
| RTHN | Xia et al. (2019) | $-$ | $-$ | $-$ | $-$ | $-$ | +
| Our work | Ours (2020) | + | + | $-$ | + | + | $-$
| TransECPE | Fan et al. (2020) | $-$ | $-$ | $-$ | $-$ | $-$ | +
| RankCP | Wei et al. (2020) | $-$ | $-$ | $-$ | $-$ | $-$ | +
Figure 8: Mapping of previous state-of-the-art methods to data sets. +
indicates that we are aware of a publication which reports on the method being
evaluated on the respective data set and a $-$ indicates our assumption that
no reported results exist with the respective method being evaluated on the
respective data set. ET corresponds to ElectoralTweets, ES to EmotionStimulus,
GNE to GoodNewsEveryone, whereas the other data set are as being introduced
above.