#  M-Sense: Modeling Narrative Structure in Short Personal Narratives Using
Protagonist’s Mental Representations

Prashanth Vijayaraghavan, Deb Roy

###### Abstract

Narrative is a ubiquitous component of human communication. Understanding its
structure plays a critical role in a wide variety of applications, ranging
from simple comparative analyses to enhanced narrative retrieval,
comprehension, or reasoning capabilities. Prior research in narratology has
highlighted the importance of studying the links between cognitive and
linguistic aspects of narratives for effective comprehension. This
interdependence is related to the textual semantics and mental language in
narratives, referring to characters’ motivations, feelings or emotions, and
beliefs. However, this interdependence is hardly explored for modeling
narratives. In this work, we propose the task of automatically detecting
prominent elements of the narrative structure by analyzing the role of
characters’ inferred mental state along with linguistic information at the
syntactic and semantic levels. We introduce a Stories dataset of short
personal narratives containing manual annotations of key elements of narrative
structure, specifically climax and resolution. To this end, we implement a
computational model that leverages the protagonist’s mental state information
obtained from a pre-trained model trained on social commonsense knowledge and
integrates their representations with contextual semantic embed-dings using a
multi-feature fusion approach. Evaluating against prior zero-shot and
supervised baselines, we find that our model is able to achieve significant
improvements in the task of identifying climax and resolution.

## Introduction

Narratives are the fundamental means by which people organize, understand, and
explain their experiences in the world around them. Researchers in the field
of psychology maintain that the default mode of human cognition is a narrative
mode (Beck 2015). Humans share their personal experiences by picking specific
events or facts and weaving them together to make meaning. These are referred
to as personal narratives, a form of autobiographical storytelling that gives
shape to experiences. Polkinghorne (1988) suggested that personal narratives,
like other stories, follow broad characteristics involving: (a) typically a
beginning, middle, and end, (b) specific plots with different characters and
settings, or events. Often, characters learn something or change as a result
of the situation or a conflict and resolution, but not always. Some of these
characteristics provide the basis for the organizational framework of a story,
commonly referred to as the narrative structure or the storyline. The growing
amount of personal narrative text information in the form of social media
posts, comments, life stories, or blog posts presents new challenges in
keeping track of the storyline or events that form the defining moments of the
narrative. Several recent works (Dore et al. 2018; Yuan et al. 2017; Chung,
Lee, and Glass 2017; Kočiskỳ et al. 2018; Mostafazadeh et al. 2017) have made
efforts to advance the research in narrative comprehension. However, the
development of computational models that automatically detect and interpret
different structural elements of a narrative remains an open problem.
Discovery of structural elements of a narrative has many applications in: (a)
retrieval of narratives based on similar dramatic events or concepts instead
of keywords (McCabe, Allyssa, and Peterson 1991; Finlayson and Winston 2006;
Marchionini, Liebscher, and Lin 1991), (b) linking related stories that form a
narrative thread towards theme generation (Berman and Slobin 2013), (c)
summarization of stories (Lehnert 1981; Papalampidi et al. 2020) and (d) story
ending prediction or generation (Chen, Chen, and Yu 2019; Li et al. 2013;
Mostafazadeh et al. 2017), (e) commonsense reasoning (Goodwin et al. 2012;
Gordon, Bejan, and Sagae 2011), to list a few.

Figure 1:  (Left) Freytag’s Pyramid. (Right) Highlights of climax and
resolution for a sample personal narrative.

Several narrative theories have been proposed such as Freytag (Freytag 1894),
Prince (Prince 2012), Bruner (Bruner 1991, 2009), Labov & Waletzky (Labov and
Waletzky 1997), to name a few. These theories explain different elements of a
narrative structure containing typical orderings between them. Certain
elements of the narrative structure are correlated across different narrative
theories. For example, Bruner’s ‘breach in canonicity’ (Bruner 1991) could
correspond to (a) Freytag’s ‘climax’ – referring to the ‘turning point’ of the
fortunes of the protagonist (Abrams and Harpham 2014) or (b) Labov’s ‘most
reportable event’ (MRE) – describing the event that has the greatest effect
upon the goals, motivations and emotions of the characters (participants) in
the narrative (Labov and Waletzky 1997; Labov 2006). Shorter narratives tend
to consist mostly of complicating actions that culminate in the MRE or climax
and instances of events that reach a ‘resolution’ stage indicated by a swift
drop in dramatic tension, while the other structural elements are more likely
to occur in longer narratives. Figure 1 (Left) shows Freytag’s pyramid
containing the key elements of the narrative structure and Figure 1 (right)
contains highlights of climax and resolution for a sample personal narrative.
Thus, our work aims to leverage computational approaches at the intersection
of information retrieval, NLP, and aspects of psychology, and model the key
elements of narrative structure – MRE and resolution. As an operating
definition, we consider an MRE to be contained in a sentence(s) based on the
following criteria – it is an explicit event that can be reported as the
summary of the story and occurs at the highest tension point of the story.
Similarly, an event qualifies as ‘resolution’ if it usually occurs after the
MRE and resolves the dramatic tension in the narrative.

Recently, Papalampidi et al. (Papalampidi, Keller, and Lapata 2019) introduced
a dataset consisting of movie screenplays and plot synopsis annotated with
turning points. Few attempts have been made at annotating elements of high-
level narrative structures (Li et al. 2017) and automatically extracting them
from free text. Ouyang et al. (Ouyang and McKeown 2015)’s study on predicting
MRE in narratives is the closest work to the problem considered in this paper.
While most of these methods rely on syntactic, semantic, surface-level affect,
or narrative features obtained using hand-engineering or pre-trained semantic
embedding methods to model narrative structure, we investigate the role of a
protagonist’s psychological states in capturing the pivotal events in the
narrative and their relative importance in identifying the elements of
narrative structure – Climax and Resolution. We find a basis for this study in
prior theoretical frameworks (Murray 2003; Ryan 1986; Ouyang and McKeown 2014;
Lehnert 1981; Schafer 2016) that emphasize (a) how narrative structure
organizes the use of psychological concepts (e.g. intentions, desires and
emotions) and mediates all the human interactions and their social behavior,
and (b) how protagonist’s mental states (both implicit and explicit
inferences, also imputed by readers) and psychological trajectory correlate
with the classic dramatic arc of stories. Thus, to obtain the protagonist’s
mental states, we refer to a recent work (Vijayaraghavan and Roy 2021; Sap et
al. 2019; Rashkin et al. 2018) that learns to embed characters’ mental states
using an external memory module. Our contributions are summarized below:

  * •

A Stories111Short for STructures Of Reddit PEsonal Stories corpus containing a
collection of Reddit Personal Narratives with fine-grained annotations of
prominent structural elements of a narrative – climax and resolution.

  * •

An end-to-end neural network for modeling narrative structure, referred to as
M-sense222Short for Mental State Enriched Narrative Structure modEl, that
allows for integration of protagonist’s mental state representations with
linguistic information via multi-feature fusion.

  * •

Experiments that analyze the impact of our modeling choices for short personal
narratives. Specifically, we gauge the influence of incorporating mental state
embeddings and report an improvement in $F_{1}$ scores of $\sim 11\%$ and
$\sim 13\%$ over the base model for predicting climax and resolution
respectively.

## Related Work

There is a large body of prior work that focuses on different aspects of
narrative comprehension. Computational analysis of narratives operates at the
level of characters and plot events. Examples include plot-related studies –
story plot generation, plot summarization, detecting complex plot units,
modeling event schemas and narrative chains and movie question-answering;
character-based studies – inferring character personas or archetypes,
analyzing inter-personal relationships and emotion trajectories, identifying
enemies, allies, heroes; story-level analysis – story representation,
predicting story endings, modeling story suspense, and creative or artistic
storytelling, to list a few.

Several studies have analyzed the literature in narratology and formulated
different goals and annotation labels associated with narratives for modeling
their structure. Elson’s (Elson 2012) Story Intention Graph (SIG) provided an
annotation schema to capture timelines as well as beliefs, intentions, and
plans of story characters. The annotations in this approach are similar to
story generation methods described in Belief-Desire-Intention agents (Rao,
Georgeff et al. 1995) and intention-based story planning (Riedl and Young
2010). Previous studies like (Gordon, Bejan, and Sagae 2011) have analyzed
personal weblog stories containing everyday situations. Rahimtoroghi et al.
(Rahimtoroghi et al. 2014) and Swanson et al. (Swanson et al. 2014) used a
subset of Labov’s categories, including orientation, action, and evaluation in
such personal weblog narratives. Black and Wilensky (1979) evaluate the
functionality of story grammar in story understanding, As explained earlier,
(Papalampidi, Keller, and Lapata 2019)’s dataset for analyzing turning points
is a valuable addition in this area of work. Moreover, there have been
consistent efforts (Jorge et al. 2018, 2019) that study the link between
Information Retrieval (IR) and narrative representations from a given text.
These include works that exploit narrative structure in movies for IR (Jhala
2008), detect and retrieve narratives in health domain (patient communities &
medical reports) (Johnson et al. 2008; Rokach, Maimon, and Averbuch 2004;
Dirkson, Verberne, and Kraaij 2019; Koopman, Cripwell, and Zuccon 2017),
identify narrative structures in news stories (Boyd, Blackburn, and Pennebaker
2020; Levi et al. 2020) or generate summaries from screenplays or novels
(Papalampidi et al. 2020), to name a few. Given this wide spectrum of work, we
leverage mental state representation models that are pre-trained using social
commonsense knowledge aggregated using IR and text mining techniques. We
employ the ensuing mental state embeddings in tandem with contextual semantic
embeddings towards our primary objective of identifying elements of high-level
narrative structure – climax and resolution. We also conduct a detailed
analysis of the outcome and the contribution of the protagonist’s
psychological state trajectory to our task.

## Dataset Collection

Figure 2 presents our data collection pipeline. First, we collect Reddit posts
from two communities: /r/offmychest and /r/confession using the PushShift API
333https://pushshift.io/. Next, we filter the collected data to retain only
those posts that do not contain tags like “[Deleted]”, “NSFW” 444NSFW – not
safe for work or “over_18”. Finally, we further narrow down the aggregated
posts using a Bert-based story classifier. The pipeline is described in the
Appendix.

Figure 2:  Illustration of our data collection pipeline.

### Annotation

Here, we explain the annotation process involved in the construction of our
Stories dataset. Table 1 shows the descriptive statistics of our dataset.

Dataset Statistics
---
#Total Narratives | 63,258
#Annotated Narratives | 2,382
#Total Sentences | 42,614
#Climax Sentences | 5,173
#Resolution Sentences | 4,502
Table 1: Statistics of our annotated Stories dataset.

#### Setup

We created a user interface for MTurk workers to make the annotation procedure
convenient for capturing key elements of the narrative structure – climax and
resolution. The user interface allows the workers to highlight parts of the
text that qualify as climax and resolution using red and green colors
respectively. Three annotators were mainly involved in the annotation process.
Each worker is presented a sampled text from the Reddit personal narrative
corpus. Additionally, the workers are provided with an option of selecting
checkboxes: “No Climax” or “No Resolution”. This caters to those personal
stories that don’t contain a climax or resolution.

#### Agreements

Once the data is collected using our annotation setup, we measure the inter-
annotator agreement (IAA) at the sentence-level. For sentence-level agreement,
we use the following metrics: (i) Fleiss’s kappa $(\kappa)$ (Fleiss and Cohen
1973), (ii) mean annotation distance ($\mathcal{D}$), i.e., the distance
between two annotations for each category, normalized by story length
(Papalampidi, Keller, and Lapata 2019).

#### Analysis

We study the appearance of climax and resolution sentences by estimating their
mean position normalized by the story length. We present the distribution of
the position of both the structural elements in Figure 3. While the average
position for climax (0.61) coincides with the peak, we observe that the
resolution contents occur later in the story. Table 2 shows the sentence-level
IAA measures for each narrative element. We observe that substantial agreement
is achieved for both the climax and resolution. Clearly, we obtain higher
agreement values for resolution than the climax. Figure 4a displays sample
annotations (e.g. multi-sentence or non-contiguous highlights; no resolution)
from our Stories dataset.

Figure 3:  Distributions of mean climax & resolution sentence positions. Metric | Climax | Resolution
---|---|---
Percentage Agreement | 0.736 | 0.807
Fleiss’s Kappa $(\kappa)$ | 0.646 | 0.756
| Mean Annotation Distance $(\%\mathcal{D})$
---
1.764 | 1.590
Table 2:  Sentence-level inter-annotator agreement.

##  M-sense: Modeling Narrative Structure

In this paper, we explore different modeling and analysis methods for
understanding narratives and automatically extracting text segments that act
as key elements of narrative structure, particularly climax and resolution.
The models are provided a narrative text $T$ with $L$ sentences,
$T=[S_{1},S_{2},...,S_{L}]$, as input. Here, each sentence $S_{i}$ contains
$N_{i}$ words $\\{w_{1}^{i},w_{2}^{i},..,w_{N_{i}}^{i}\\}$ from vocabulary
$\mathcal{V}$. Towards automatic detection of structural elements, we
formulate it as a sentence labeling task where the goal is to predict a label
$\hat{y}_{i}\in\\{None,Climax,Resolution\\}$ for each sentence $S_{i}$, based
on the story context. Beyond linguistic features extracted from narratives, we
focus on a dominant aspect in which a narrative is formed or presented, that
is an account of characters’ mental states – motives and emotions. Thus, we
leverage transfer learning from pretrained models trained to infer characters’
mental states from a narrative. We implement a multi-feature fusion based
learning model, M-sense, that potentially encapsulates syntactic, semantic,
characters’ mental state features towards our overall goal of predicting
climax and resolution in short personal narratives. Our M-sense model consists
of the following components:

Ensemble Sentence Encoders, which computes per-sentence linguistic & mental
state embeddings.

Fusion layer, which integrates the protagonist’s mental state information with
the extracted linguistic features.

Story Encoder, which maps the fused encodings into a sequence of
bidirectionally contextualized embeddings.

Interaction layer, which estimates state transition across sequential context
windows to identify the boundaries.

Classification layer, which involves linear layers to eventually calculate the
label probabilities.

Figure 4:  (a) Sample annotations of climax (Red) and Resolution (Green) by
one of the annotators. (b) Illustration of our M-sense model. Note that
$h^{i1}=h^{i};h^{i2}=\hat{h}^{i};h^{i3}=\tilde{h}^{i}$ relate to semantics
$(xSem)$, intents $(xIntent)$ and emotional reactions $(xReact)$ of the
$i^{th}$ sentence respectively.

### Ensemble Sentence Encoders

In this work, we aim to exploit both linguistic and mental state features for
an enhanced model for narratives.

#### Extracting Linguistic Representations

Pretrained general purpose sentence encoders usually capture a hierarchy of
linguistic information such as low-level surface features, syntactic features
and high-level semantic features. Given a narrative text with $L$ sentences
$T=[S_{1},S_{2},...,S_{L}]$, this component outputs hidden representations for
sentences $H_{sents}=[h^{1},h^{2},...,h^{L}]$ using different encoding
methods.In our Msense model, we use a token-level Bert-based sentence encoder
(more in the Appendix). Here, each sentence is prepended with a special
$[CLS]$ token at the beginning of each sentence and appended with a $[SEP]$
token at the end of each sentence in the narrative. We apply both position and
segment embeddings and feed to the pre-trained Bert model as:

$\scalebox{0.85}{$H=[h_{[CLS]}^{1},..,h_{N_{1}}^{1},h_{[SEP]}^{1},..,h_{[CLS]}^{i},..,h_{N_{i}}^{i},..,h_{[SEP]}^{L}]$}\\\
\scalebox{0.85}{$=\textsc{Bert}(T)$}$ (1)

The hidden representation of the $i^{th}$ $[CLS]$ token from the top Bert
layer is extracted as the semantic embedding of the $i^{th}$ sentence.
However, we drop the subscript $[CLS]$ from $h^{i}_{[CLS]}$ and denote the
output semantic embeddings as: $H^{xSem}_{sents}=[h^{1},h^{2},...,h^{L}]$.

#### Incorporating Protagonist’s Mental Representation

Prior studies have established how the progression of a story is as much a
reflection of a sequence of a protagonist’s motivation and emotional states as
it is the workings of an abstract grammar (Palmer 2002; Mohammad 2013; Alm and
Sproat 2005). We follow a recent work (Vijayaraghavan and Roy 2021) that
implements a Nemo model, a variant of a Transformer-based encoder-decoder
architecture to embed and explain characters’ (or entities’) mental states. We
extract the embeddings of intents and emotional reactions of the protagonist
for a given sentence in the narrative conditioning on the prior story context.
Figure 4b contains the overview of the Nemo architecture. The computation of
mental state embeddings are facilitated by a knowledge enrichment module that
consolidates commonsense knowledge about social interactions and an external
memory module that tracks entities’ mental states. Using prior context
$(S_{<i})$, entity $(e_{j})$ and mental state attribute information
($m\in\\{xIntent,xReact\\}$ representing intent and emotional reaction
respectively), we use the encoder, $\textsc{StoryEntEnc}(\cdot)$, in this
trained model to obtain entity-aware mental state representation of the
current sentence $S_{i}$. The encoding process in the Nemo model is given by:

$\scalebox{0.86}{$(\hat{H}_{xIntent}^{i},\tilde{H}_{xReact}^{i})=\textsc{StoryEntEnc}(S_{i},S_{<i},e_{j},m);$}\\\
\scalebox{0.86}{$\forall m\in\\{xIntent,xReact\\}$}\vspace{-0.5cm}$ (2)

where $e_{j}\in\mathcal{E}$ is the entity,
$(\hat{H}_{xIntent}^{i},\tilde{H}_{xReact}^{i})$ is the resulting entity-aware
intent and emotion representation of the $i^{th}$-sentence given the story
context. In this work, we use the narrator (“I” or “self” in the personal
narratives) as the protagonist. We only utilize the hidden representations of
the $[CLS]$ token from both $(\hat{H}_{xIntent}^{i},\tilde{H}_{xReact}^{i})$
for subsequent processing steps. We denote these intent and emotion
representation as: $H_{sents}^{xintent}=[\hat{h}^{1},..,\hat{h}^{L}]$ and
$H_{sents}^{xReact}=[\tilde{h}^{1},..,\tilde{h}^{L}]$ respectively.

### Transformer-based Fusion Layer

Given multiple sentence-level embeddings, we apply a fusion strategy to derive
a unified sentence embedding for our classification task. Let
$h^{ik};\forall{k\in\\{1,...,K\\}}$ denote different per-sentence latent
vectors. In our case, $K=3$ and
$h^{i1}=h^{i};h^{i2}=\hat{h}^{i};h^{i3}=\tilde{h}^{i}$ are embeddings related
to semantics $(xSem)$, intents $(xIntent)$ and reactions $(xReact)$ of the
$i^{th}$ sentence respectively. Drawing ideas from the literature of
multimodal analysis (Urooj et al. 2020), we treat the multiple latent vectors
as a sequence of features by first concatenating them together. We introduce a
special token $[FUSE]$ 555$[FUSE]$ is similar to the commonly used $[CLS]$
token. that accumulates the latent features from different sentence encodings.
The final hidden representation of $[FUSE]$ token obtained after feeding them
to a Transformer layer is the fused output sentence representation:
$h_{fuse}^{i}=\textsc{Tf}(\parallel_{k=0}^{K}{h^{ik}})$, where Tf refers to
the transformer encoder layer and $h^{i0}$ (i.e. when $k=0$) is set to the
trainable $[FUSE]$ vector.

### Story Encoder

We apply Transformer layers on the top of the sentence representations to
extract narrative-level features. We refer it as Inter-sentence Transformer.
Intuitively, the transformer layer focuses on possibly different sentences in
the narrative, and produces context-aware sentence embeddings. This is given
as:

$\begin{array}[]{c}\scalebox{0.8}{$\hat{H}^{l}=LayerNorm(\hat{C}^{l-1}+\textsc{Mha}(\hat{C}^{l-1})$}\\\
\scalebox{0.8}{$\hat{C}^{l}=LayerNorm(\hat{H}^{l}+\textsc{Ffl}(\hat{H}^{l}))$}\\\
\scalebox{0.8}{$C_{sents}=[c^{1},c^{2},...,c^{L}]=\hat{C}^{n_{L}}$}\end{array}$
(3)

where $\hat{C}^{0}=PE(H_{sents})$, $PE$ refers to the positional encoding,
$LayerNorm$ refers to layer normalization operation, Mha is the multi-head
attention operation and Ffl is a feed-forward layer (Vaswani et al. 2017). The
superscript $l$ indicates the depth of the stacked Transformer layers. The
output from the topmost layer, $l=n_{L}$, is our contextualized sentence
embeddings $C_{sents}$.

### Interaction layer

In this layer, we compute the transition of state across sentences by
measuring similarity metrics in the embedding space between sequential context
windows and concatenating them with contextualized embeddings. By choosing
windows of size $s$, we compute the left $(c_{left}^{i})$ and right
$(c_{right}^{i})$ context information for the $i^{th}$ sentence by computing
the mean sentence embedding within that window. Finally, we get the
interaction-feature enhanced context-aware embeddings:
$E_{sents}=[e^{1},e^{2},...,e^{L}]$.

### Classification layer

The resulting embeddings are mapped to a $C$-dimensional output using a
softmax-based classification layer. Here, $C=3$ is the number of labels. This
step is given as: $\hat{y}_{i}=softmax(f_{s}(e^{i}))$.

## Experiments

We conduct experiments to study the following research questions:

RQ1: How does our model compare with other baselines for identifying climax
and resolution in short narratives?

RQ2: How do various model components contribute to the overall performance? To
what extent do mental state representations play a role for our classification
task?

### Overall Predictive Performance (RQ1)

#### Baselines

We compare our model with a set of carefully selected zero-shot (see the
Appendix section) & supervised baselines, shown as follows.

Random baseline, which assigns labels (Climax, Resolution or None) to
sentences randomly.

Distribution baseline, which picks sentences that lie on the peaks of the
empirical distributions for climax and resolution in our training set as
explained earlier.

Heuristic baseline, which labels the sentences as climax or resolution based
on heuristics. While we use the sentence that is the closest semantic
neighbour of the post title as climax, the last sentence in the narrative is
labeled as the resolution (as explained in the Appendix section.).

A recent work (Wilmot and Keller 2020) has explored surprise a measure of
suspense in narratives. (Ely, Frankel, and Kamenica 2015)’s surprise is
defined as the amount of change from previous sentence to the current sentence
in the narrative (see the Appendix). We encode the sentences in the story
using the following approaches and eventually compute suspense measures for
our classification task.

GloVeSim (Pennington, Socher, and Manning 2014), Bert (Devlin et al. 2018),
Use (Cer et al. 2018), which computes semantic embeddings using average word
vectors (using GloVe) or Transformer-based models.

StoryEnc (Papalampidi, Keller, and Lapata 2019), which uses the hierarchical
RNN based language model to encode sentences in the story.

StoryEntEnc (Vijayaraghavan and Roy 2021), which encodes the sentences in the
story from the protagonist’s perspective. Here, we denote intent and emotion
embeddings as $(E_{int}=H_{sents}^{xIntent})$ and
$(E_{emo}=H_{sents}^{xReact})$ respectively.

Cam and Tam (Papalampidi, Keller, and Lapata 2019) consist of bidirectional
LSTM model with the latter model having an additional interaction layer to
compute boundaries between the topics in each story.

M-sense–Fusion, which is a variant of our M-sense model without mental state
embeddings.

M-sense, which is our complete model incorporating protagonist’s mental
representation.

Models | $\boldsymbol{F_{1}\uparrow}$ | $\boldsymbol{D\downarrow}$
---|---|---
| C | R | C | R
Random | 0.196 | 0.143 | 29.05 | 30.57
Distribution | 0.274 | 0.315 | 15.79 | 14.42
Heuristic | 0.217 | 0.147 | 23.74 | 26.82
GloVeSim | 0.312 | 0.344 | 12.06 | 11.65
$\textsc{Bert}_{tok}$ | 0.408 | 0.441 | 9.37 | 8.09
$\textsc{Bert}_{sent}$ | 0.352 | 0.366 | 10.88 | 9.73
$\textsc{Use}_{sent}$ | 0.379 | 0.391 | 10.42 | 9.58
StoryEnc | 0.410 | 0.438 | 8.81 | 7.46
$E_{int}$ | 0.437 | 0.462 | 8.19 | 6.94
$E_{emo}$ | 0.429 | 0.475 | 8.43 | 6.67
Tam | $0.565_{\pm 0.022}$ | $0.609_{\pm 0.008}$ | 5.90±3.18 | 5.02±2.82
Cam | $0.578_{\pm 0.019}$ | $0.604_{\pm 0.0032}$ | 6.58 ±4.02 | 5.44±3.05
M-sense | 0.694*±0.0027 | 0.743*±0.0015 | 4.15±1.84 | 3.20±1.06
Table 3:  Evaluation Results of different models to detect Climax and
Resolution. We report $F_{1}$ score per class & percent $(D)$ for these
models. We use $\uparrow,\downarrow$ to indicate if higher/ lower values mean
better performance respectively. * refers to significance $(p<0.05)$ over TAM
using a paired T-Test.

#### Results

Table 3 outlines the results of our evaluation. We report the performance of
simple baselines, of which the distribution baseline turns out to be the
strongest. The heuristic baseline performs slightly better than the random
baseline. This suggests that the Reddit post title contains relevant signal to
predict the climax while the last sentence heuristic for resolution is only as
good as a random classifier.

Applying suspense-based approaches with different sentence embedding methods
yields relative improvement over the simple baselines in terms of both the
evaluation metrics. As expected, sentence-level $\textsc{Bert}/\textsc{Use}$
performs worse than its token-level counterpart. We attribute this variation
in performance to the lack of any story context information for computing
latent embedding, thereby affecting the assessment of state changes in the
narrative. However, sentence-level Use’s ability to produce better similarity
estimates gives it a slight advantage over sentence-level Bert. Notably,
sentence representations obtained from models trained on stories
$(\textsc{StoryEnc},\textsc{StoryEntEnc})$ recorded comparable to improved
results over other sentence embedding methods. Strikingly, computing surprise
using protagonist mental state embeddings exhibit an overall enhanced
classification capability. We find that the intent embedding $(E_{int})$ helps
achieve the best zero-shot performance for detecting climax. A competitive
outcome for resolution is obtained using protagonist’s emotion representation
$(E_{emo})$. We compare our complete M-sense model with the best performing
prior models such as $\textsc{Cam},\textsc{Tam}$ (Papalampidi, Keller, and
Lapata 2019) applied for similar tasks. As we can see, supervised fine-tuning
approaches easily beat the earlier results obtained using zero-shot methods.
Finally, our M-sense model achieves an absolute improvement of $\sim 20.07\%$
and $\sim 22\%$ for climax and resolution prediction respectively.

Model Variants | $\boldsymbol{F_{1}}\uparrow$
---|---
| C | R
M-sense | 0.688 | 0.738
Sentence Encoder Variants |  |
w/ Sentence-level Bert | 0.665 | 0.709
w/ Sentence-level Use | 0.677 | 0.726
Story Encoder Variant |  |
w/o Story Encoder | 0.620 | 0.653
w/ Inter-Sentence Rnn | 0.659 | 0.705
Interaction Layer Variant |  |
w/o Interaction Layer | 0.654 | 0.716
Fusion Layer Variants |  |
–w/o Fusion Layer | 0.614 | 0.640
–w/o $E_{int}$ | 0.638 | 0.703
–w/o $E_{emo}$ | 0.652 | 0.687
Table 4:  We report $F_{1}$ score per class with non-default modeling choices
for each component of our model.

### Ablation Study (RQ2)

To evaluate the contributions of each component in our M-sense model, we
conduct an ablation study using the validation set. For this study, we compare
our best performing M-sense model with alternative modeling choices for each
of the components. Table 4 shows the results of our study. We modify one
component at a time and report their performance using $F_{1}$ metric. This
involves either replacing a component (denoted by “w/”) or removing a
component (denoted by “w/o” to refer without the component). For eg. “w/
Sentence-level Bert” refers to replacing token-level Bert in our M-sense model
with sentence-level Bert as our sentence encoder; “w/o $E_{emo}$” indicates
the removal of protagonist’s emotion state embedding from the fusion layer.

Influence of Mental State Embeddings: In this study, we examine the necessity
of a fusion layer and probe the influence of protagonist’s mental state
embeddings towards our classification task. Notably, the results in Table 4
validate the benefits of introducing the fusion layer and demonstrate the
relative performance gains obtained with intent and emotion embeddings. In the
absence of a fusion layer, we observe that the performance drop is $\sim 11\%$
and $\sim 13\%$ for predicting climax and resolution respectively. The loss of
the protagonist’s intent information impacts the climax prediction more. This
is analogous to the effect emotion information has on resolution prediction.

### Analysis and Discussion

Effect of Story Length: Here, we compare the performance of different sentence
encoders with and without fusion layer for detecting climax in narratives with
varying length. Figure 5 shows the results of this analysis. We observe that
the token-level Bert outperforms sentence-level Bert and Use encoders for
narratives containing up to $13-14$ sentences, but the performance gradually
degrades beyond 14 sentences. Sentence-level Use encoder produces stable and
relatively better outcomes for longer narratives (story length $>14$). With
the introduction of mental state representation through fusion layer, the
$F_{1}$ score improved significantly irrespective of the sentence encoder
used.

Figure 5:  Performance of sentence encoders for detecting climax in story with
varying length.

Error Analysis: In order to estimate why our model augmented with mental state
representation performs better, we conduct error analysis between our full
M-sense model and the model without mental representation fusion
($\textsc{M-sense}-Fusion$). For those narratives where the latter model fails
to predict correctly, we gauge the patterns emerging out of the following
analysis: (a) Using VADER666https://github.com/cjhutto/vaderSentiment (Hutto
and Gilbert 2014) a normalized, weighted composite sentiment score is computed
for each sentence in the narrative, and (b) Using state classification
(Rashkin et al. 2018; Vijayaraghavan and Roy 2021), we assess Maslow’s
motivation or intent categories associated with sentences predicted as climax
or resolution in the narrative and analyze for any pattern related to ground
truth climax/resolution sentences. For predicting resolution, the
$\textsc{M-sense}-Fusion$ makes $\sim 28\%$ more mistakes than M-sense model
for narratives with homogeneous endings (i.e. narratives having same sentiment
sentences in the neighbourhood of resolution closer to the end of the story).
$\textsc{M-sense}-Fusion$ model is unable to discern clearly and predicts a
different sentence as resolution. Based on our analysis (b), there is a clear
pattern that M-sense gains significantly over the $\textsc{M-sense}-Fusion$
when the ground-truth climax sentences belong to “Esteem” and “Love/Belonging”
categories. Our attention analysis results are shown in the Appendix section.

## Task: Modeling Movie Turning Points

Given that our work is primarily focused on modeling narrative structure in
personal narratives, we analyze how we can apply such a model towards
identifying climax and resolution in movie plot synopsis. (Papalampidi,
Keller, and Lapata 2019) introduced a Tripod dataset containing a corpus of
movie synopses annotated with turning points (TPs). By testing our model on
this dataset, we evaluate our model’s performance on an out-of-domain dataset.
The dataset identified five major turning points in the movie synopses and
screenplay, referring to them as critical events that prevent the narrative
from drifting away. By their definitions for each of these categories
(Papalampidi, Keller, and Lapata 2019), TP4 and TP5 align clearly with our
usage of climax and resolution from prior narrative theories. Due to this
alignment, it is relevant to use our model to predict these two categories in
the Tripod dataset. However, we focus on the movie plot synopses in this work
and use the cast information collected from IMDb as a part of this dataset.

We first apply our M-sense trained on our Stories corpus directly and evaluate
its zero-shot performance (referred as Zs). We assume the protagonist in the
movie to be the top character from the IMDb cast information. Though this may
not always be true, it measures how our model fares on this dataset for
predicting TP4 and TP5. Further, we use sentence-level Use-based sentence
encoder as some of the wiki plot synopses are longer than what can be
accommodated by our token-level Bert model. Additionally, we also fine-tune
our model with the training set of the Tripod dataset. This is denoted by
$\textsc{M-sense}(FT)$.

### Results

We display our model’s performance compared to the best performing Tam
reported in the original work (Papalampidi, Keller, and Lapata 2019). Tam with
TP views implemented separate encoders for each of the categories and computed
different representations for the same sentences acting as different views
related to each TP. Similarly, Tam+Entities enriched the model with entity
information by applying co-reference resolution and obtain entity-specific
representations. Table 5 shows our model’s results compared to the prior
proposed approaches for modeling turning points in plot synopses. We find that
our model in zero-shot settings outperforms a supervised TAM+TP views model,
though it falls slightly behind the best supervised model.Also, we restrict
our model for predicting only two of the five major turning point labels.
Finally, our fine-tuned model outperforms the best performing model,
significantly reducing the mean annotation distance by an average of $\sim
20\%$ on both the turning point labels. Thus, we are able to achieve
remarkable improvement on an out-of-domain dataset even with assumptions on
protagonist information. Therefore, we demonstrate that our Msense model can
predict climax and resolution in stories beyond just personal narratives,
albeit limited by story length at this point.

Methods | TP4 | TP5
---|---|---
TAM+TP views | 6.91 | 4.26
TAM+Entities | 5.23 | 3.48
$\textsc{M-sense}(ZS)$ | 6.62 | 4.54
$\textsc{M-sense}(FT)$ | 4.17 | 2.38
Table 5:  Results of evaluation on Tripod dataset. We report Mean Annotation
Distance (%) $D$ results for identifying TP4 and TP5 relevant to this work.

## Conclusion

Towards modeling high-level narrative structure, we construct a dataset of
personal narratives from Reddit containing annotations of climax and
resolution sentences. Next, we introduce a deep neural model, referred to as
M-sense, that learns to effectively integrate protagonist’s psychological
state features with linguistic information towards improved modeling of
narrative structure. We experimentally confirm that our model outperforms
several zero-shot and supervised baselines and benefits significantly from
incorporating protagonist’s mental state embeddings. Our model is able to
achieve $\sim{20\%}$ higher success in prediction task than the previous
methods.We believe that our work will advance the research in understanding
the larger dynamics of narrative communication and aid future efforts towards
developing AI tools that can interact with users though stories.

## References

  * Abrams and Harpham (2014) Abrams, M. H.; and Harpham, G. 2014.  _A glossary of literary terms_.  Nelson Education.
  * Alm and Sproat (2005) Alm, C. O.; and Sproat, R. 2005.  Emotional sequencing and development in fairy tales.  In _International Conference on Affective Computing and Intelligent Interaction_ , 668–674. Springer.
  * Beck (2015) Beck, J. 2015.  Life’s stories.  _The Atlantic_ , 1.
  * Berman and Slobin (2013) Berman, R. A.; and Slobin, D. I. 2013.  _Relating events in narrative: A crosslinguistic developmental study_.  Psychology Press.
  * Boyd, Blackburn, and Pennebaker (2020) Boyd, R. L.; Blackburn, K. G.; and Pennebaker, J. W. 2020.  The narrative arc: Revealing core narrative structures through text analysis.  _Science advances_ , 6(32): eaba2196.
  * Bruner (1991) Bruner, J. 1991.  The narrative construction of reality.  _Critical inquiry_ , 18(1): 1–21.
  * Bruner (2009) Bruner, J. S. 2009.  _Actual minds, possible worlds_.  Harvard university press.
  * Cer et al. (2018) Cer, D.; Yang, Y.; Kong, S.-y.; Hua, N.; Limtiaco, N.; John, R. S.; Constant, N.; Guajardo-Cespedes, M.; Yuan, S.; Tar, C.; et al. 2018.  Universal sentence encoder.  _arXiv preprint arXiv:1803.11175_.
  * Ceran et al. (2012) Ceran, B.; Karad, R.; Mandvekar, A.; Corman, S. R.; and Davulcu, H. 2012.  A semantic triplet based story classifier.  In _2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining_ , 573–580. IEEE.
  * Chen, Chen, and Yu (2019) Chen, J.; Chen, J.; and Yu, Z. 2019.  Incorporating structured commonsense knowledge in story completion.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 6244–6251.
  * Chung, Lee, and Glass (2017) Chung, Y.-A.; Lee, H.-Y.; and Glass, J. 2017.  Supervised and unsupervised transfer learning for question answering.  _arXiv preprint arXiv:1711.05345_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dirkson, Verberne, and Kraaij (2019) Dirkson, A.; Verberne, S.; and Kraaij, W. 2019.  Narrative Detection in Online Patient Communities.  In _Text2Story@ ECIR_ , 21–28.
  * Dore et al. (2018) Dore, R. A.; Amendum, S. J.; Golinkoff, R. M.; and Hirsh-Pasek, K. 2018.  Theory of mind: A hidden factor in reading comprehension?  _Educational Psychology Review_ , 30(3): 1067–1089.
  * Edunov et al. (2018) Edunov, S.; Ott, M.; Auli, M.; and Grangier, D. 2018.  Understanding back-translation at scale.  _arXiv preprint arXiv:1808.09381_.
  * Eisenberg and Finlayson (2017) Eisenberg, J.; and Finlayson, M. 2017.  A simpler and more generalizable story detector using verb and character features.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 2708–2715.
  * Elson (2012) Elson, D. K. 2012.  Detecting story analogies from annotations of time, action and agency.  In _Proceedings of the LREC 2012 Workshop on Computational Models of Narrative, Istanbul, Turkey_ , 91–99.
  * Ely, Frankel, and Kamenica (2015) Ely, J.; Frankel, A.; and Kamenica, E. 2015.  Suspense and surprise.  _Journal of Political Economy_ , 123(1): 215–260.
  * Finlayson and Winston (2006) Finlayson, M. A.; and Winston, P. H. 2006.  Analogical retrieval via intermediate features: The Goldilocks hypothesis.
  * Fleiss and Cohen (1973) Fleiss, J. L.; and Cohen, J. 1973.  The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.  _Educational and psychological measurement_ , 33(3): 613–619.
  * Freytag (1894) Freytag, G. 1894.  _Die technik des dramas_.  S. Hirzel.
  * Goodwin et al. (2012) Goodwin, T.; Rink, B.; Roberts, K.; and Harabagiu, S. 2012.  UTDHLT: COPACETIC system for choosing plausible alternatives.  In _* SEM 2012: The First Joint Conference on Lexical and Computational Semantics–Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , 461–466.
  * Gordon, Bejan, and Sagae (2011) Gordon, A.; Bejan, C.; and Sagae, K. 2011.  Commonsense causal reasoning using millions of personal stories.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 25.
  * Gordon and Swanson (2009) Gordon, A.; and Swanson, R. 2009.  Identifying personal stories in millions of weblog entries.  In _Third international conference on weblogs and social media, data challenge workshop, San Jose, CA_ , volume 46, 16–23.
  * Gordon et al. (2013) Gordon, A. S.; Huangfu, L.; Sagae, K.; Mao, W.; and Chen, W. 2013.  Identifying Personal Narratives in Chinese Weblog Posts.
  * Hearst (1997) Hearst, M. A. 1997.  Text Tiling: Segmenting text into multi-paragraph subtopic passages.  _Computational linguistics_ , 23(1): 33–64.
  * Hutto and Gilbert (2014) Hutto, C.; and Gilbert, E. 2014.  Vader: A parsimonious rule-based model for sentiment analysis of social media text.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 8.
  * Jhala (2008) Jhala, A. 2008.  Exploiting structure and conventions of movie scripts for information retrieval and text mining.  In _Joint International Conference on Interactive Digital Storytelling_ , 210–213. Springer.
  * Johnson et al. (2008) Johnson, S. B.; Bakken, S.; Dine, D.; Hyun, S.; Mendonça, E.; Morrison, F.; Bright, T.; Van Vleck, T.; Wrenn, J.; and Stetson, P. 2008.  An electronic health record based on structured narrative.  _Journal of the American Medical Informatics Association_ , 15(1): 54–64.
  * Jorge et al. (2018) Jorge, A.; Campos, R.; Jatowt, A.; and Nunes, S. 2018.  First international workshop on narrative extraction from texts (Text2Story 2018).  In _ECIR_ , 833–834.
  * Jorge et al. (2019) Jorge, A. M.; Campos, R.; Jatowt, A.; and Bhatia, S. 2019.  The 2 nd International Workshop on Narrative Extraction from Text: Text2Story 2019.  In _European Conference on Information Retrieval_ , 389–393. Springer.
  * Kingma and Ba (2014) Kingma, D. P.; and Ba, J. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_.
  * Kočiskỳ et al. (2018) Kočiskỳ, T.; Schwarz, J.; Blunsom, P.; Dyer, C.; Hermann, K. M.; Melis, G.; and Grefenstette, E. 2018.  The narrativeqa reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6: 317–328.
  * Koopman, Cripwell, and Zuccon (2017) Koopman, B.; Cripwell, L.; and Zuccon, G. 2017.  Generating clinical queries from patient narratives: a comparison between machines and humans.  In _Proceedings of the 40th international ACM SIGIR conference on Research and development in information retrieval_ , 853–856.
  * Labov (2006) Labov, W. 2006.  Narrative pre-construction.  _Narrative inquiry_ , 16(1): 37–45.
  * Labov and Waletzky (1997) Labov, W.; and Waletzky, J. 1997.  Narrative analysis: Oral versions of personal experience.
  * Lehnert (1981) Lehnert, W. G. 1981.  Plot units and narrative summarization.  _Cognitive science_ , 5(4): 293–331.
  * Levi et al. (2020) Levi, E.; Mor, G.; Shenhav, S.; and Sheafer, T. 2020.  CompRes: A Dataset for Narrative Structure in News.  _arXiv preprint arXiv:2007.04874_.
  * Li et al. (2017) Li, B.; Cardier, B.; Wang, T.; and Metze, F. 2017.  Annotating high-level structures of short stories and personal anecdotes.  _arXiv preprint arXiv:1710.06917_.
  * Li et al. (2013) Li, B.; Lee-Urban, S.; Johnston, G.; and Riedl, M. 2013.  Story generation with crowdsourced plot graphs.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 27.
  * Marchionini, Liebscher, and Lin (1991) Marchionini, G.; Liebscher, P.; and Lin, X. 1991.  Authoring hyperdocuments: Designing for interaction.  _Interfaces for Information Retrieval and Online Systems. Greenwood Press, New York, NY_ , 119–131.
  * McCabe, Allyssa, and Peterson (1991) McCabe, A.; Allyssa, M.; and Peterson, C. 1991.  _Developing narrative structure_.  Psychology Press.
  * Mohammad (2013) Mohammad, S. 2013.  From once upon a time to happily ever after: Tracking emotions in novels and fairy tales.  _arXiv preprint arXiv:1309.5909_.
  * Mostafazadeh et al. (2017) Mostafazadeh, N.; Roth, M.; Louis, A.; Chambers, N.; and Allen, J. 2017.  Lsdsem 2017 shared task: The story cloze test.  In _Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics_ , 46–51.
  * Murray (2003) Murray, M. 2003.  Narrative psychology and narrative analysis.
  * Ouyang and McKeown (2014) Ouyang, J.; and McKeown, K. 2014.  Towards Automatic Detection of Narrative Structure.  In _LREC_ , 4624–4631.
  * Ouyang and McKeown (2015) Ouyang, J.; and McKeown, K. 2015.  Modeling reportable events as turning points in narrative.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 2149–2158.
  * Palmer (2002) Palmer, A. 2002.  The construction of fictional minds.  _Narrative_ , 10(1): 28–46.
  * Papalampidi et al. (2020) Papalampidi, P.; Keller, F.; Frermann, L.; and Lapata, M. 2020.  Screenplay Summarization Using Latent Narrative Structure.  _arXiv preprint arXiv:2004.12727_.
  * Papalampidi, Keller, and Lapata (2019) Papalampidi, P.; Keller, F.; and Lapata, M. 2019.  Movie Plot Analysis via Turning Point Identification.  _arXiv preprint arXiv:1908.10328_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in neural information processing systems_ , 8026–8037.
  * Pennington, Socher, and Manning (2014) Pennington, J.; Socher, R.; and Manning, C. D. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , 1532–1543.
  * Piper (2018) Piper, A. 2018.  Fictionality.
  * Polkinghorne (1988) Polkinghorne, D. E. 1988.  _Narrative knowing and the human sciences_.  Suny Press.
  * Prince (2012) Prince, G. 2012.  _A grammar of stories: An introduction_ , volume 13.  Walter de Gruyter.
  * Rahimtoroghi et al. (2014) Rahimtoroghi, E.; Corcoran, T.; Swanson, R.; Walker, M. A.; Sagae, K.; and Gordon, A. S. 2014.  Minimal narrative annotation schemes and their applications.  In _7th Workshop on Intelligent Narrative Technologies_.
  * Rao, Georgeff et al. (1995) Rao, A. S.; Georgeff, M. P.; et al. 1995.  BDI agents: from theory to practice.  In _Icmas_ , volume 95, 312–319.
  * Rashkin et al. (2018) Rashkin, H.; Bosselut, A.; Sap, M.; Knight, K.; and Choi, Y. 2018.  Modeling naive psychology of characters in simple commonsense stories.  _arXiv preprint arXiv:1805.06533_.
  * Riedl and Young (2010) Riedl, M. O.; and Young, R. M. 2010.  Narrative planning: Balancing plot and character.  _Journal of Artificial Intelligence Research_ , 39: 217–268.
  * Rokach, Maimon, and Averbuch (2004) Rokach, L.; Maimon, O.; and Averbuch, M. 2004.  Information retrieval system for medical narrative reports.  In _International Conference on Flexible Query Answering Systems_ , 217–228. Springer.
  * Ryan (1986) Ryan, M.-L. 1986.  Embedded narratives and tellability.  _Style_ , 319–340.
  * Sap et al. (2019) Sap, M.; Le Bras, R.; Allaway, E.; Bhagavatula, C.; Lourie, N.; Rashkin, H.; Roof, B.; Smith, N. A.; and Choi, Y. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 3027–3035.
  * Schafer (2016) Schafer, S. B. 2016.  _Exploring the Collective Unconscious in the Age of Digital Media_.  IGI Global.
  * Swanson et al. (2014) Swanson, R.; Rahimtoroghi, E.; Corcoran, T.; and Walker, M. 2014.  Identifying narrative clause types in personal stories.  In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , 171–180.
  * Urooj et al. (2020) Urooj, A.; Mazaheri, A.; Shah, M.; et al. 2020.  MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , 4648–4660.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , 5998–6008.
  * Vijayaraghavan and Roy (2021) Vijayaraghavan, P.; and Roy, D. 2021.  Modeling human motives and emotions from personal narratives using external knowledge and entity tracking.  In _Proceedings of the Web Conference 2021_ , 529–540.
  * Wilmot and Keller (2020) Wilmot, D.; and Keller, F. 2020.  Suspense in short stories is predicted by uncertainty reduction over neural story representation.  In _Proceedings of the 58th annual meeting of the association for computational linguistics_ , 1763–1788.
  * Yang et al. (2016) Yang, Z.; Yang, D.; Dyer, C.; He, X.; Smola, A.; and Hovy, E. 2016.  Hierarchical attention networks for document classification.  In _Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies_ , 1480–1489.
  * Yuan et al. (2017) Yuan, X.; Wang, T.; Gulcehre, C.; Sordoni, A.; Bachman, P.; Subramanian, S.; Zhang, S.; and Trischler, A. 2017.  Machine comprehension by text-to-text neural question generation.  _arXiv preprint arXiv:1705.02012_.

##  Appendix A Appendix

### Dataset Collection

The first stage in this pipeline is dedicated to ingesting posts from Reddit.
To collect natural first-person stories, we rely on Reddit communities
comprising user-generated textual accounts of happy events, long-standing
baggage, recent trauma, life experiences, adventurous encounters or guilt, and
redemption episodes. To this end, we aggregate posts from two communities:
/r/offmychest and /r/confession using the PushShift API
777https://pushshift.io/. The Pushshift API provides access to a database of
all Reddit posts made since Reddit’s launch as a social platform. We obtain
$\sim 440,000$ posts from this step.

Next, we filter the collected data to retain only those posts that do not
contain tags like “[Deleted]”, “NSFW” 888NSFW – not safe for work or
“over_18”. Relying on Prince’s definition (Prince 2012) of a minimal story to
comprise a starting state, a state-changing event, and an ending state, we
eliminate posts containing less than three sentences. The subsequent stages in
the pipeline are explained in the sections below.

### Story Classifier

The aggregated data consists of a wide variety of contents some of which do
not qualify as personal narratives. In order to separate such non-narrative
content from the collected data, we develop a story classifier that takes
textual content as input and predicts the likelihood of the input text being a
story.

#### Story vs. Non-Story Dataset

We gather a diverse collection of first-person blog text drawn randomly from
the Spinn3r Blog Dataset containing everyday situations (Gordon and Swanson
2009). Consistent with our filtering approach for Reddit posts, we follow a
similar length criterion and sample $\sim 1,500$ blog posts. Further, we
randomly selected $\sim 1,500$ texts from our Reddit posts corpus. Together,
we obtain a total of $\sim 3,000$ posts to be annotated by MTurk workers.

For each post, annotators were instructed to read the textual content and
choose one among the three labels: Story, Non-Story or Unsure (Gordon et al.
2013). We define these categories as follows – (a) Story: Non-fictional
narratives that people share with each other about their own life experiences.
They contain a sequence of causally or temporally related events with the
narrator being a participant; (b) Non-Story: Texts that are not primarily
personal stories or don’t give account of past events. They may or may not
contain texts from first-person point of view but include opinion pieces,
excerpts from news articles, recipes, technical explanations, facts, questions
or some random discussion, personal advice, to list a few. When the annotators
are uncertain about the right label, they are allowed to select the “Unsure”
option. The three workers reached unanimous agreement on 76% of the cases. We
use the majority vote when such an agreement is not reached. Of the 3,000
posts, 1,197 posts were tagged as “Story”, 1,173 as “Non-Story” and remaining
as “Unsure”.

#### Model

We introduce a story classifier that separates non-fictional narratives from
non-narrative textual content. Prior work have used feature engineering to
extract features like Tf-Idf, Semantic Triplets, VerbNet & coreference
resolution chain based character features (Ceran et al. 2012; Eisenberg and
Finlayson 2017) for this task. A work by Piper (Piper 2018) specifically used
linguistic aspect of text to measure fictionality, i.e., distinguish works of
fiction from non-fiction. In our work, we use a pretrained Bert model for our
classification task. Given an input text, the goal is to predict if the text
qualifies as a story or not. We formulate the input text as
$T=\\{S_{1},S_{2},...,S_{n}\\}$, where $S_{i}$ is the $i^{th}$ sentence of the
text. Following (Devlin et al. 2018), we tokenize the input text and
concatenate all tokens as a new sequence,
$\\{[CLS],S_{1},[SEP],S_{2},[SEP],...,S_{n-1},$ $[SEP],S_{n},$ $[SEP]\\}$,
where $[CLS]$ is a special token used for classification and $[SEP]$ is a
delimiter. Each token is initialized with a vector by summing the
corresponding token and position embedding from pretrained Bert, and then
encoded into a hidden state. Finally, we get $[H_{[CLS]},H_{S_{1}},$
$H_{S_{2}},..,H_{S_{n}},H_{[SEP]}]$ as an encoding output. We concatenate the
$[CLS]$-token representation from the last four layers of the model for our
classification task. We apply two linear layers on top of the concatenated
output representation with a sigmoid activation function at the final linear
layer. We optimize the binary-cross entropy loss and choose the model with
least loss on the validation set as our final story classifier. We evaluate
this model on the held-out test set. In Table 6a, we report the $F_{1}$ score,
and compare our approach to other baselines. The best performing model
achieves an $F_{1}$-score of 0.79. Finally, we feed the Reddit posts to the
trained story classifier and obtain a probability score, $p$, that indicates
the likelihood of the post being a story. Furthermore, to increase the
reliability of our data, we discard all those posts with probability score
lesser that a chosen threshold $\delta$, i.e. $p<\delta$. In our work, we set
$\delta$ to 0.75. This procedure yields a total of 63,258 stories, referred to
as Reddit Personal Narratives dataset.

Models | P | R | F1
---|---|---|---
Sem. Triplet (Ceran et al. 2012) | 0.64 | 0.47 | 0.46
VerbNet (Eisenberg and Finlayson 2017) | 0.71 | 0.66 | 0.68
Han (Yang et al. 2016) | 0.75 | 0.73 | 0.74
Bert (Devlin et al. 2018) | 0.81 | 0.78 | 0.79
Table 6:  Performance of our BERT-based story classifier on the annotated
dataset.

#### Agreement

We analyze the discrepancies in the annotated data to gain insights about the
potential challenges in the annotation process.For climax, we note that the
annotators get confused with sentences that involve events contributing to
rising action (or Labov’s complicating action) which eventually culminate in a
climax (or Labov’s MRE). Further, we observe that the differences are finer in
many instances and hence harder to reliably detect. Though we achieve higher
agreement on resolution category, the annotation gets less accurate with
ambiguities in resolution and aftermath/endings, especially when narratives
don’t have a clear resolution. Interestingly, the annotators are able to
discern between the two interest categories despite the high cognitive load
and complexity involved in detecting them from unstructured user generated
content.

### M-SENSE Model

#### Token-BERT

Token-level Bert: Since Bert produces output vectors that are grounded to
tokens instead of sentences, we undertake an input processing step that
involves insertion of a special $[CLS]$ token at the beginning of each
sentence and a $[SEP]$ token at the end of each sentence in the input
sequence. we feed the entire narrative text $T$ to the input processing step
to get the following sequence:
$\\{[CLS],w_{1}^{1},w_{2}^{1},..,w_{N_{1}}^{1},[SEP],$
$[CLS],w_{1}^{2},w_{2}^{2},..,w_{N_{2}}^{2},[SEP],..,[SEP],$
$[CLS],w_{1}^{L},w_{2}^{L},$ $..,w_{N_{L}}^{L},[SEP]\\}$, where $w_{j}^{i}$ is
the $j^{th}$ in $i^{th}$ sentence in the narrative. The multiple $[CLS]$
symbols will aggregate the features for sentences taking the context into
consideration. Next, we apply alternating segment embeddings indicative of
different sentences in our input textual narrative. Given a narrative with
four sentences, $[S_{1},S_{2},S_{3},S_{4}]$, we assign the segment embeddings
as $[E_{A},E_{B},E_{A},E_{B}]$.

### Zero-shot Approaches

In addition to our M-sense model and its variants, we experiment with zero-
shot methods that utilize either simple heuristics or suspense-based
approaches to model narrative structure.

#### Heuristic-based Approaches

In addition to different modeling approaches, we experiment with simple
heuristics for automatically labeling the sentences in story. This method
assumes that the title of the Reddit post provides the summary of the post and
hence, could refer to the MRE/climax of the narrative. We use a pretrained
sentence embedding model and compute the semantic similarity between the each
sentence in the narrative and the original title of the Reddit post. We
compute Use embeddings of sentences to identify the nearest neighbor of the
post title and label it as the climax. Next, we assign the last sentence as
the resolution because it is more like for the cognitive tension to drop and
reach a resolution at the near end of the narrative.

#### Suspense-based Approaches

A recent work (Wilmot and Keller 2020) has explored surprise and uncertainty
reduction as a measure of suspense in narratives by considering sentences as
the primary unit of processing. In our work, we mainly focus on surprise
values based on consequential state change in narratives. Intuitively, a large
change in any particular state indicates increased surprise at that point in
the narrative. (Ely, Frankel, and Kamenica 2015)’s surprise is defined as the
amount of change from previous sentence to the current sentence in the
narrative. Peaks in such measures could reflect potential events where
protagonist faces key obstacles and these may act as the defining moments of
narrative structure. Thus, we examine different sentence embedding techniques
to compute state changes including change in protagonist’s mental state
representations in our experiments to recognize suspenseful states. This will
act as a relevant baseline to determine the effectiveness of semantic and
mental state features.

### Training & Hyperparameters

Though we approach narrative structure model as a sentence classification
task, we divide the collected data based on number of narratives into train,
validation and test sets at 70-10-20 split. In our sentence labeling task,
each sentence will be accompanied with the entire narrative context. On the
training set, we perform data augmentation by replacing sentences in narrative
context with their paraphrases. The paraphrases are generated using a back-
translation approach (Edunov et al. 2018) based on pretrained
English$\leftrightarrow$German translation model. We limit the number of such
modified sentences to $20\%$ of the story length. We tune the hyperparameters
using grid search and the best configuration is obtained based on validation
set performance. Our best configuration consists of a two-layer story encoder
$(n_{L}=2)$ and a single transformer-based fusion layer with 12 heads. For the
window size in the interaction layer, it is set to two sentences, i.e., $s=2$.
We also added a dropout with rate 0.2 in order to prevent overfitting. We
optimize using Adam (Kingma and Ba 2014) at a learning rate of $\alpha=0.0001$
and a batch size of 32 with PyTorch (Paszke et al. 2019) being our model
implementation framework. We report the score over an average of 3 runs. All
our experiments were conducted on multiple NVIDIA Tesla P100 GPUs each with
12GB memory.

### Results

We conducted ablation study in Section Ablation Study (RQ2). Table 4 shows the
results of our study. We describe some of the aspects below.

Importance of Contextualization: Next, we evaluate the contribution of a story
encoder to our classification task. We observe that the performance drops
significantly (by $\sim 10\%$ and $\sim 12\%$ for climax and resolution
prediction respectively) without a story encoder. The importance of
contextualizing the story sentences is established as we see a marked
improvement of $\sim 8\%$ in the average overall performance with the
introduction of an inter-sentence Rnn story encoding layer. Still, this
performance lags behind our default M-sense setting with Transformer-based
story encoding layer. We find the story encoder being relevant even if the
inter-sentence dependencies are captured using the token-level Bert model. We
attribute this to the task-specific inter-sentence relationships being
unearthed as we fine-tune our model.

Choice of Sentence Encoder: From the results, it is clear that token-level
Bert generally performs better the sentence-level Bert variant. This is
unsurprising as the sentence-level approach produces embeddings without story
context. Such an approach results in the loss of fine-grained inter-sentence
token dependencies that the token-level Bert can extract. Experiments also
suggest that sentence-level Use model trained on textual similarity tasks
results in better $F_{1}$ scores than the Bert counterpart.

Impact of Interaction Layer: The addition of an interaction layer yields an
average $\sim 4\%$ gain in performance for identifying climax and resolution.
The advantage of introducing an interaction layer has been studied in prior
studies (Hearst 1997; Papalampidi, Keller, and Lapata 2019) and we find the
performance improvement to be congruous with these studies.

Figure 6:  Attention analysis of two stories with climax and resolution
sentences related to Maslow’s categories – Esteem (top) and Love/Belonging
(bottom).

### Attention Analysis

We conduct attention analysis on those narratives containing sentences
belonging to “Esteem” and “Love/ Belonging” categories. Specifically, we study
the functioning of Transformer-based Fusion layer for aggregating multiple
latent embeddings – Semantic $(Sem)$, Intent $(Int)$, and Emotion $(Emo)$. We
then visualize the attention heatmaps from the fusion layer corresponding to
these predicted climax and resolution sentences by computing the average
attention score map over all heads. Figure 6 displays the visualized attention
map for sample stories belonging to the above mentioned categories. Since,
$[FUSE]$ is the aggregated output over all the three latent embeddings. We
note that the attention map has high attention scores between intent $(Int)$
and $[FUSE]$ vectors for stories related to “Esteem” motivation category,
while more weight is assigned for emotion $(Emo)$ in samples associated with
“Love/Belonging” category.