#  Nece: Narrative Event Chain Extraction Toolkit

Guangxuan Xu2 , Paulina Toro Isaza111footnotemark: 1 , Moshi Li3, Akintoye
Oloko1
Bingsheng Yao4, Aminat Adebiyi1, Yufang Hou1, Nanyun Peng2, Dakuo Wang1
1 IBM Research 2University of California, Los Angeles
3Northeastern University 4Rensselaer Polytechnic Institute
{gxu21<EMAIL_ADDRESS>{ptoroisaza, YHou<EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Nece is an event-based text analysis toolkit built for narrative documents.
Nece aims to provide users open and easy accesses to an event-based summary
and abstraction of long narrative documents through both a graphic interface
and a python package, which can be readily used in narrative analysis,
understanding, or other advanced purposes. Our work addresses the challenge of
long passage events extraction and temporal ordering of key events; at the
same time, it offers options to select and view events related to narrative
entities, such as main characters and gender groups. We conduct human
evaluation to demonstrate the quality of the event chain extraction system and
character features mining algorithms. Lastly, we shed light on the toolkit’s
potential downstream applications by demonstrating its usage in gender bias
analysis and Question-Answering tasks.

##  1 Introduction

Narrative is a common literary form typically associated with accounts of
related events covering both fictional (fairy-tales and novels) and non-
fictional (news reports or memoirs) stories (Carey and Snodgrass, 2006).
Despite the prevalence of long narratives, they can be difficult for humans to
quickly process and extract needed information from.

Nece is built to address this difficulty by identifying events in narrative
texts and building temporal event chains that are clearly displayed through an
intuitive UI system. With the observation that events, time, and characters
are the backbones of the narrative style texts, we choose to use temporal
event chains with character features to abstract the key plot of narrative.
Nece is specifically designed to mine character features, including the name,
gender, and importance of the characters in the narrative, so that it can
create separate event-chains for the respective character group. We also build
algorithms to find salient events and provide options to view only a chain of
salient events that are relevant to main plot in order to save users from
being flooded by generic events.

Figure 1: The salient events window displays a temporal chain of key events,
aiming to offer a succinct event-based representation of the narrative plot
Figure 2: The gender page shows event chains for Male, Female, and Mix(plural
noun inclusive of all genders). Right hand side showcases a statistics table
for most polarized events for both gender meansured by Odds Ratio.

The Nece toolkit is an ensemble of existing SOTA NLP tools and creative novel
algorithms that clusters and creates the event chains. Nece uses AllenNLP’s
Semantic Role Labeling model (Gardner et al., 2017) to extract events along
with the corresponding subjects and direct objects of those events; it also
adopts the BookNLP (Bamman et al., 2014) narrative processing pipeline to
extract character information. BookNLP (Bamman et al., 2014) is a powerful
pipeline combining co-reference resolution, semantic role labeling, and
character clustering algorithms to extract character information in literary
texts. Nece builds on BookNLP to add temporal event-chain construction a
interactive UI system. Nece uses the ECONET (Han et al., 2021) models to
predict pairwise temporal relations; but different from the EventPLUS (Ma et
al., 2021) pipeline that focuses on sentences-level temporal relation
extraction, Nece outputs a global temporal ranking for global events in a
document to construct event chain. StoryAnalyzer (Mitri, ) is another
narrative analysis toolkit built on CoreNLP (Manning et al., 2014); it also
hosts a graphic interface with features of entities coloring, summary of top
characters, and event identification of time and dates. While there is overlap
between StoryAnalyzer and Nece, such as the main character summary feature,
Nece’s support for temporal event chains is distinct, and its extensions for
salient event chains at the story, gender, and character level are also unique
and novel.

We present Nece, the first system with an easily accessible online interface
for narrative event-chain extraction by character. Human evaluation is
conducted at the system level to demonstrate high performance of our character
features, temporal relations, and salient events extraction systems. We also
demonstrate Nece’s direct application in Question-Answering systems. Lastly, a
demo website is available for the public to explore narrative events for
analyses such as gender bias. 111The Nece toolkit is licenced under MIT
license and will be available for public usage from https://moshi-
li.github.io/demo/.

##  2 System Design

Researchers and developers can use Nece through a Python package available on
GitHub to output the full processing results of a narrative document. The
general public can also easily access the toolkit through an online interface
to process narrative texts of their choice and interactively find
visualizations, analyses, and summaries of the uploaded document.

###  2.1 System Pipeline

Figure 3: The system design pipeline for Nece Toolkit; orange coloring denotes
models and code from existing packages; blue coloring represents our novel
algorithms.

The Nece Toolkit’s backend has multiple components running sequentially for
each input story. However, each component module can be run individually if
the necessary intermediate files already exist. The main component modules are
summarized in Figure3 where the orange coloring represents an existing toolkit
and the blue coloring represents our new contributions. Some key components
are explained in more detail below.

#1. Character Extraction: A modified version of BookNLP Bamman et al. (2014)
extracts and clusters characters and performs co-reference resolution to link
pronouns to characters. We use the BookNLP output to count the combined name
mentions and pronoun mentions for characters to arrive at their relative
importance category, and a high-level categorization as primary, secondary,
and tertiary characters.

#2. Event Extraction and Semantic Role Labeling: We identify events with
AllenNLP’s Gardner et al. (2017) Semantic Role Labeling (SRL) model which
extracts verb events and their related arguments such as subject and direct
object. Additionally, we determine salient events, those most important to
main plot, using our novel algorithm based on the tf-idf (term
frequency–inverse document frequency) algorithm.

#5. Temporal Event Ordering: The extracted verb events are filtered and read
into ECONET Han et al. (2021) to find the temporal relationship between pairs
of events. A custom ranking algorithm then orders all events based on their
pairwise relationships.

#6. Character Gender Extraction: The character and co-reference resolution
output from Step #1 is used in combination with NLTK’s parts of speech tagging
to extract gender features of characters. Due to limitations of current
method, we only offer an imperfect gender categorization as female, male,
group/non-binary, or unknown (See Limitation Limitations)

#7. Salient events are defined as events that are relevant, or essential, to
the main narrative of the story as defined by (Zhang et al., 2021). To find
the salient events, we trained an inverse document frequency(idf) dictionary
on a fairy-tale corpus (Xu et al., 2022) to calculate a tf-idf based salience
score. Inverse document frequency can filter out generic events, and document
frequency can be used to identify locally important events in the analyzed
text. Our experiments 3.3 show that a weighted tf-idf score for the stem words
of events successfully identifies salient events.

Figure 4: The character interface provides event-chains for top-five
characters;

#8. Temporal Event Chains of Characters: While temporal relations between
events are much more complicated than a linear event chain, we still choose an
imperfect global event chain representation due to its simplicity in
aggregating and summarizing hundreds of events. We acknowledge, as pointed out
by (Ning et al., 2018), not all events are aligned on the same temporal axis
and thus are not comparable. However, we show through human evaluation that
our simplified event chain is a more accurate temporal model than using
sequential ordering based merely on the event’s position in the narrative.

#9. Statistic Calculations: These aggregate statistics offer an overview of
the narrative document at the character-level, event-level, and story-level.
The statistics include metrics for quantifying bias between characters of
different narrative importance or between male and female characters. It
displays the importance category as reported in # 1; The bias metric is the
odds ratio as the as calculated using (Sun and Peng, 2021; Monroe et al.,
2017). For example, in a given story, the occurrence of the event ’kill’ has
an odds ratio of four from male to female characters. This means that male
characters are four times more likely than female characters to be involved in
an event involving killing. We apply a common correction, Haldane-Anscombe
(Lawson, 2004), to account for cases in which one group has no observed counts
of the event.

Method | Before | After | F1 | # TrueLabel
---|---|---|---|---
prec | rec | prec | rec | micro | macro | before | after
Random (ref.) | 0.503 | 0.847 | 0.510 | 0.160 | 0.504 | 0.437 | 264 | 49
Sequential | 1.0 | 0.843 | 0.0 | 0.0 | 0.843 | 0.457 | 264 | 49
Nece(ECONET) | 0.931 | 0.995 | 0.979 | 0.727 | 0.939 | 0.898 | 264 | 49
Table 1: Random refers to random ordering of two pairs; Sequential refers to a
heuristic that puts events that are in earlier sequential order in the text as
temporally earlier; Baseline1 is a temporal relations model that we test;
Nece(ECONET) is our model’s performance. Best scores are emboldened.Due to the
imbalanced dataset favoring BEFORE relation, the sequential heuristics gives
good accuracy, compared to random. However, our temporal model far outperforms
heuristics in macro-f1, and performs better in accuracy.

###  2.2 Online Interface

The online toolkit interface takes in story text as input and creates a
visualization of the event chains and maps the events back to the text in a
dynamic graphical interface. As shown in Figure1, annotated story text
occupies the main screen and users can scrolled up and down for full content;
on the lower left part of the screen is the event chain where clicking on the
event block directs the user to the annotated event in the text corpus. For
the Gender and Character feature windows, a statistics overview is offered on
the right hand section of the screen.

####  2.2.1 Salient Event Chain

The first feature of our interface, salient event chain extraction and
visualization, is depicted in Figure 1. While a complete event chain can be
helpful to a get holistic and comprehensive view of the story, long event
chains can be hard to digest for users. Thus, we focus on the salient events
of which main characters are participants. The event chain is organized in the
temporal order by which the event happens. When clicking on one of the events,
the user is taken to the location of the event in the text with the subject
and objects of the event highlighted.

####  2.2.2 Gender

Gender representation in print media has attained increasing public attention:
according to (McCabe et al., 2011), no more than 33% of children’s books
published in any given year contain female main characters, the number is up
to 100% for male characters. Following the call to emphasize gender fairness
by explicitly identifying the gender of main characters of
books222https://www.kirkusreviews.com/news-and-features/articles/unmaking-
white-default/, we present event chains for different gender groups, as shown
in Figure 2. We use different shapes to denote the argument type of the
character(subject or direct object) of the event; different coloring denotes
the gender of the subject argument of events. The statistics section displays
a contingency table of the story-level event statistics for the most gender
polarized events based on odds ratio.

####  2.2.3 Character

The character feature, shown in Figure 4, displays option tabs for the top 5
characters in the narrative as determined by number of appearances. Clicking
on the tab brings the respective event chains and a statistics table
summarizing the importance of characters by mentions in the story.

##  3 Evaluation of System on Fairy-Tale Corpus

###  3.1 Experiment Setup

Dataset Experiments are conducted on the FairytaleQA dataset, a corpus of 278
open-source fairy-tales (Xu et al., 2022). First, we evaluate the event
ordering model compared to heuristics-based method of ordering events using
f-score based metrics. Next, we conduct in-group annotations for four
dimensions of the output event chain, and we adopt a precision-based metric to
validate the quality of our event chain. Lastly, we shed light on Nece’s
application potential by showing its usage in bias analysis and Question-
Answering tasks.

###  3.2 Human Evaluation of Temporal Ordering

The Nece toolkit incorporates ECONET(MATRES)333ECONET model trained on MATRES
dataset; https://github.com/PlusLabNLP/ECONET (Han et al., 2021) as the engine
to predict temporal relations between events. ECONET is a BERT-based model
(Devlin et al., 2019) trained with a joint-objective of temporal relation
mask-filling and contrastive loss from discriminating corrupted temporal
tokens. While (Han et al., 2021) reports a high performance of 0.793 F1-score
on the MATRES (Ning et al., 2018) dataset, we conduct another round of human
evaluation of the model’s performance on literary stories to validate the
quality of our temporal ordering model in our domain.

We randomly selected three stories from the FairytaleQA corpus, totaling 358
event-pairs. We manually annotated each adjacent pairs of events as golden
labels, of which 264 are before, 49 are after, and 45 are vague. Then, we
conduct experiments on the temporal ordering model with those annotations as
ground truth and compare our Nece toolkit with two heuristics based models and
a neural-based baseline.

Event-chain | precision | # samples
---|---|---
Salient Event | 0.758 | 195
Subject Character | 0.86 | 196
Subject Gender | 0.90.7 | 196
Temporal-Order | 0.94.3 | 196

Table 2: Evaluation of the event chain; We conduct experiments to evaluate 4
aspects of the event chain, including its salient event extraction, character
extraction, gender extraction.

The ECONET model classifies events into four labels: BEFORE, AFTER,
SIMULTANEOUS, and VAGUE. For the practical purpose of constructing an event-
chain, we apply the imperfect heuristic of converting all SIMULTANEOUS and
VAGUE labels to BEFORE. The results are shown in Table1. The ECONET model
performs better in both micro and macro-F1 than assuming a sequential ordering
based simply on position in narrative. The macro-F1 score on our corpus is
higher than the reported score in (Han et al., 2021) because our evaluation
data is imbalanced in favor of the BEFORE label which is easier for the model.
Also, notably, our model has high precision in predicting the AFTER label but
low recall. One caveat of the ECONET model lies in its inability to detect
VAGUE events where two events are incomparable. About 12.5% of total event
pairs are annotated VAGUE, but none of these pairs is predicted to be VAGUE by
the temporal model. An example of vague relations, is the following,

This godmother of hers, who was a fairy, said to her, “You wish you could go
to the ball…”

“Said” and “wish” lie in two separate timelines and thus are ambiguous to
compare. VAGUE relations are common if we compare relations within the same
sentence; to avoid incomparable event pairs, we also tried running the
temporal relation prediction model only on salient events so that in-sentence
events comparison no longer happens. Besides helping the performance of the
temporal relation extraction model, constructing salient event chains can also
provide more concise representations of the narrative. We acknowledge that, as
pointed out by (Ning et al., 2018), not all events can be compared along the
same temporal axis. Even so, temporal ordering of events based on extracted
temporal relations still results in higher accuracy than relying on heuristics
as shown in Table 1.

###  3.3 Evaluation of the Four Dimensions of an Event Chain

We conduct experiments to evaluate four dimensions of the event chain: salient
events identification, character extraction, gender extraction, and temporal
ordering. Each story contains hundreds of events and manually coding each on
the four dimensions is laborious. To measure the quality of the event chain,
and for practical reasons, we adopt a precision based method to ask humans to
judge whether the output event is correct in each step on the respective four
dimensions.

We perform in-group annotation on three randomly selected stories from the
FairyTale corpus (Xu et al., 2022) consisting of around 200 events and 600
data points. We annotate based on the following procedure: for salient events,
we annotate one or more salient events in each sentence to examine if the
toolkit’s prediction correctly matches the human-identified salient events;
for character extraction, we annotate the name of the character that is the
subject argument of the event to compare with model’s character coding; for
gender prediction, we annotate the gender of the event’s subject argument;
lastly, to annotate temporal ordering, at each step of the event chain, we ask
the annotator whether the current event’s position is reasonable given its
previous and subsequent events in the chain. Table2 shows the precision of the
evaluation results along the four dimensions.

The evaluation results are shown in Table 3.3; we observe a high score for
temporal-order consistent with the high precision observed in pair-wise
temporal ordering experiments in Table 1; subject gender prediction, a
relatively easy task solvable through gendered pronoun identification and co-
reference resolution, enjoys the second highest score; subject character
identification is difficult, given the number of possible characters in a
narrative, but still achieves good performance largely thanks to the
robustness of the BookNLP toolkit (Bamman et al., 2014); salient event
identification using the tf-idf algorithm gives decent performance but still
has room for improvement.

##  4 Downstream Tasks

###  4.1 Gender Bias Analysis

For each narrative text, Nece can automatically construct a table showing
events that have a biased distribution between male and female characters. For
example, in the right section of Figure 2, the statistics table shows that
“know” and “tell” are two event disproportionately associated with female
characters, while “kill” and “realize” are more likely to be male associated
events. Kill could be a stereotype that associates male with violence. Event-
based automatic bias flagging using Nece can be a critical industrial
application to protect unaware children or teenager against gender bias from
storybooks.

###  4.2 Question-Answering

Nece toolkit is also helpful in enhancing a Question-Answering model’s
performance on certain narrative events related questions. Xu et al. (2022)
leverages a SOTA sequence-to-sequence model for the baseline QA-task
experiment on the FairytaleQA dataset which contains 10,580 QA-pairs annotated
by education experts. The education experts split each story into multiple
sections which contain multiple paragraphs. Then, they provided related story
section(s) and a category of the question that belongs to an established
framework along with each QA-pair they annotated. Among the six categories
(character, setting, action, feeling, causal relationship, outcome resolution,
and prediction), we notice that action, causal relationship and outcome
resolution are specifically about a certain event or events that are causally
related.

Fine-tune | baseline event train split |  | event valid split
---
\+ event sentences
Predict |  | baseline event
---
test split
| event test split
---
\+ event sentence
| baseline event
---
test split
| event test split
---
\+ event sentence
| all
---
event QA
0.762 | 0.765 | 0.721 | 0.732
| causal
---
relationship
0.725 | 0.747 | 0.676 | 0.687
| outcome
---
resolution
0.664 | 0.668 | 0.645 | 0.693
action | 0.793 | 0.797 | 0.765 | 0.678

Table 3:  The QA experiment result demonstrates our toolkit is able to help
models make better predictions on event-related QA-pairs in the FairytaleQA
dataset.

By leveraging Nece, we extract events from related story sections of each QA-
pair and further identify the sentence(s) in the story closely related to the
question based on word vector similarity. We hypothesize that providing
correct context for the question would help the model to better locate and
extract the correct answer. We conduct two experiments to examine whether the
additional information provided by Nece can improve model performance.

We select those extracted event-related QA-pairs from the original train,
valid, and test split, forming three new subsets of 4432/536/513 QA-pairs
respectively. We leverage Rouge-L as the evaluation metric to compare the
generated answer with two human annotated ground-truth answers. We first fine-
tune a baseline T5-base model with the event-related train split to
concatenate with the sentence most relevant to the original input question
during prediction. By comparing with the baseline prediction setting where no
additional information is provided, we discover that the model’s prediction
performance on all three event-related categories see improvement.

Then, we use Nece and the event-related validation split to fine-tune another
model where we concatenate the most closely related sentence and its adjacent
sentences as additional information to the original input. We perform the same
two prediction experiments and also observe improvement on the model’s
performance on three event-related categories. Detail of the results are shown
in Table 3. The hyper-parameters of the QA experiments are in Appendix A.

##  5 Conclusion

In conclusion, we present Nece, the first system online interface that
extracts temporal event chains for narrative documents with built in
functionalities for character feature extraction and analysis. Our pipeline is
hosted on a public server which is powered by GPU to run fast inference for
large neural models. We hope that Nece can assist research and work in event-
based analysis or understanding of novels, short stories, fairy-tales, and
other wide ranges of narrative documents. While bias analysis and Question-
Answering are two example downstream tasks, the Nece system can be leveraged
for many more potential applications.

## Limitations

Our toolkit is primarily limited by the accuracy of the existing models it
leverages such as BookNLP, ECONET, and AllenNLP Semantic Role Labeling.
BookNLP’s entity extraction and co-reference resolution in particular misses
many character mentions because it has a hard time mapping nominal mentions of
characters ("the girl") with the corresponding character names ("Cinderella").
ECONET was trained on a dataset sourced from news articles, so it may not
yield the same high performance on other domains of narrative such as literary
and fictional stories. Gender extraction of non-binary characters is limited
by few examples in the development corpus and existing models’ predisposition
to label all uses of the pronoun "them" as plural. Thus, the current
implementation is limited to comparisons of female and male characters which
perpetuates a gender binary. Another limitation lies in defining narrative
events as action verbs in a sentence. Not all narrative events are described
explicitly through action verbs.

## Acknowledgements

We’d like to thank Daniel Firebanks-Quevedo for his work on the gender feature
extraction.

## References

  * Bamman et al. (2014) David Bamman, Ted Underwood, and Noah A. Smith. 2014.  A bayesian mixed effects model of literary character.  In _ACL_.
  * Carey and Snodgrass (2006) Gary Carey and Mary Ellen Snodgrass. 2006.  _A multicultural dictionary of literary terms_.  & Co.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ArXiv_ , abs/1810.04805.
  * Gardner et al. (2017) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017\.  Allennlp: A deep semantic natural language processing platform.
  * Han et al. (2021) Rujun Han, Xiang Ren, and Nanyun Peng. 2021.  Econet: Effective continual pretraining of language models for event temporal reasoning.  In _EMNLP_.
  * Lawson (2004) Raef Lawson. 2004.  Small sample confidence intervals for the odds ratio.  _Communications in Statistics - Simulation and Computation_ , 33(4):1095–1113.
  * Ma et al. (2021) Mingyu Derek Ma, Jiao Sun, Mu Yang, Kung-Hsiang Huang, Nuan Wen, Shikhar Singh, Rujun Han, and Nanyun Peng. 2021.  Eventplus: A temporal event understanding pipeline.  In _NAACL_.
  * Manning et al. (2014) Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 55–60, Baltimore, Maryland. Association for Computational Linguistics.
  * McCabe et al. (2011) Janice McCabe, Emily Fairchild, Liz Grauerholz, Bernice A. Pescosolido, and Daniel Tope. 2011.  Gender in twentieth-century children’s books: Patterns of disparity in titles and central characters.  _Gender & Society_, 25(2):197–226.
  * (10) Mike Mitri.  Story analyzer.
  * Monroe et al. (2017) Burt L. Monroe, Michael P. Colaresi, and Kevin M. Quinn. 2017.  Fightin’ words: Lexical feature selection and evaluation for identifying the content of political conflict.  _Political Analysis_ , 16(4):372–403.
  * Ning et al. (2018) Qiang Ning, Hao Wu, and Dan Roth. 2018.  A multi-axis annotation scheme for event temporal relations.  In _ACL_.
  * Sun and Peng (2021) Jiao Sun and Nanyun Peng. 2021.  Men are elected, women are married: Events gender bias on wikipedia.  _ArXiv_ , abs/2106.01601.
  * Xu et al. (2022) Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Li, Nora Bradford, Branda Sun, Tran Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022.  Fantastic questions and where to find them: FairytaleQA – an authentic dataset for narrative comprehension.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 447–460, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2021) Xiyang Zhang, Muhao Chen, and Jonathan May. 2021.  Salience-aware event chain modeling for narrative understanding.  In _EMNLP_.

##  Appendix A QA Experiment Hyper-parameters

The model backbone for all models fine-tuned for the QA experiment is T5-base.
We maintain the same learning rate of $5e^{-5}$. For the model fine-tuned on
event-related train split, we set $epochs=4$. For the model fine-tuned on
event-related valid split with additional event information (with about $1/8$
amount of data than the previous setting), we set $epoch=32$.