# KGAP: Knowledge Graph Augmented Political
Perspective Detection in News Media

Shangbin Feng♠ Zilong Chen♠11footnotemark: 1 Wenqian Zhang♠11footnotemark: 1
Qingyao Li♠ Qinghua Zheng♠ Xiaojun Chang♢ Minnan Luo♠
Xi’an Jiaotong University♠ University of Technology Sydney♢
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>These authors
contributed equally to this work. Corresponding author.

###### Abstract

Identifying political perspectives in news media has become an important task
due to the rapid growth of political commentary and the increasingly polarized
political ideologies. Previous approaches focus on textual content and leave
out the rich social and political context that is essential in the perspective
detection process. To address this limitation, we propose KGAP, a political
perspective detection method that incorporates external domain knowledge.
Specifically, we construct a political knowledge graph to serve as domain-
specific external knowledge. We then construct heterogeneous information
networks to represent news documents, which jointly model news text and
external knowledge. Finally, we adopt relational graph neural networks and
conduct political perspective detection as graph-level classification.
Extensive experiments demonstrate that our method consistently achieves the
best performance on two real-world perspective detection benchmarks. Ablation
studies further bear out the necessity of external knowledge and the
effectiveness of our graph-based approach.

KGAP: Knowledge Graph Augmented Political
Perspective Detection in News Media

Shangbin Feng♠††thanks:  These authors contributed equally to this work.
Zilong Chen♠11footnotemark: 1 Wenqian Zhang♠11footnotemark: 1 Qingyao Li♠
Qinghua Zheng♠ Xiaojun Chang♢ Minnan Luo♠††thanks:  Corresponding author.
Xi’an Jiaotong University♠ University of Technology Sydney♢
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

The past decade has witnessed dramatic changes of political commentary (Wilson
et al., 2020) in two major ways. Firstly, the popularity of social media has
led to a dramatic increase in political discussions on online media and social
networks. Besides, ever-increasing political polarization has made it hard for
journalists and news agencies to remain impartial in nature. Detecting
political perspectives in the news media would help alleviate the issue of
“echo chamber” (Barberá et al., 2015), where only a single viewpoint is
reiterated in closely formed communities and further deepens the divide. That
being said, political perspective detection has become a pressing task which
calls for further research efforts.

Previous news bias detection methods have focused on analyzing the textual
content of news articles. Recurrent neural networks (Yang et al., 2016) and
pre-trained language models (Devlin et al., 2018) are adopted by Li and
Goldwasser (2021) to analyze news content for perspective analysis. Jiang et
al. (2019) uses convolutional neural networks with GloVe word embeddings
(Pennington et al., 2014) for political perspective detection and achieves the
best result in the hyperpartisan news detection task in SemEval 2019 (Kiesel
et al., 2019). Li and Goldwasser (2021) leverages the attention mechanism and
entity mentions in text and achieves state-of-the-art results.

Figure 1: External knowledge of social and political entities in news articles
that is essential for political perspective detection.

However, these text-based methods fail to incorporate pervasive social and
political context in news articles, while human readers rely on external
knowledge as background information to facilitate commonsense reasoning and
perspective detection. For example, Figure 1 presents a news article that
contains real-world entities such as political organizations, elected
officials and geographical locations in the United States. External knowledge
of these entities informs the reader that the mentioned individuals are
republicans and mostly come from conservative states, which helps to identify
the liberal stance expressed in criticizing them. This reasoning demonstrates
that political perspective detection relies on social and political context as
background information in addition to news content. That being said, political
perspective detection methods should leverage external knowledge to reason
beyond text and identify implicit perspectives.

Figure 2: Overview of our proposed approach KGAP.

In light of these challenges, we propose KGAP (Knowledge Graph Augmented
Political perspective detection), which leverages domain-specific external
knowledge to incorporate background information and augment argument
reasoning. Specifically, we firstly construct a political knowledge graph to
represent the external knowledge that serves as background information for
political narratives. We then construct heterogeneous information networks to
represent news documents, which jointly model textual content and external
knowledge in the knowledge graph. Finally, we adopt relational graph neural
networks for graph representation learning and conduct political perspective
detection as graph-level classification. Our main contributions are summarized
as follows:

  * •

To the best of our knowledge, we construct the first domain-specific knowledge
graph of contemporary U.S. politics with 1,071 entities and 10,703 triples to
serve as external knowledge for political perspective detection. We will
publicize the political knowledge graph upon acceptance to facilitate research
in related tasks.

  * •

We propose to leverage domain knowledge and model news articles as
heterogeneous information networks for political perspective detection. Our
approach is end-to-end, inductive, and effectively framing the task as graph-
level classification to incorporate external knowledge.

  * •

We conduct extensive experiments to evaluate KGAP and competitive baselines.
KGAP consistently outperforms state-of-the-art methods on two real-world
datasets. Further studies demonstrate the necessity of external knowledge and
explore the effect of knowledge graphs, text models, and the news graph
structure in KGAP.

##  2 Related Work

###  2.1 Political Perspective Detection

The task of political perspective detection is generally studied in two
different settings: social media and news media. For stance detection in
social media, many works have studied the problem as text classification to
identify stances in specific posts based on their textual content. They used
techniques such as sentiment analysis (Jiang et al., 2011) and neural
attention networks (Du et al., 2017). Other approaches explored the problem of
identifying stances of social media users instead of individual posts. Darwish
et al. (2020) conducted user clustering to identify their stances. Magdy et
al. (2016) proposed to predict user perspectives on major events based on
network dynamics and user interactions.

For political perspective detection in news media, news documents are often
classified into several perspective labels. Previous works have proposed to
leverage semantic information in news articles with bias features (Horne et
al., 2018) and various deep language encoders (Li and Goldwasser, 2021; Yang
et al., 2016; Jiang et al., 2019). Later approaches have tried to enrich the
textural content with the graph structures of online communities that interact
with different news outlets. Baly et al. (2020) proposes to leverage media
sources and design a triplet loss for political perspective detection. Li and
Goldwasser (2019) supplemented news articles with a network of Twitter users
and their interaction. In this paper, we focus on political perspective
detection in news media and propose to incorporate political external
knowledge.

###  2.2 Knowledge-aware NLP

Knowledge graphs (KGs) are often leveraged as external knowledge sources in
various tasks in natural language processing. Yu et al. (2021) leverages
knowledge graphs for open-domain question answering. Hu et al. (2021)
incorporates KGs in fake news detection with graph neural networks, knowledge
graph embedding techniques and Wikipedia description of KG entities. Liu et
al. (2021) proposes to extract KG subgraphs and combine textual context and
mentioned KG entities as input to language models. Zhu et al. (2021) fuses
commonsense statements in KGs with textual context using a transformer-based
encoder-decoder architecture. In this paper, we construct a politics-specific
knowledge graph and build on these works to incorporate political KGs into
stance detection. Specifically, we explore the novel approach of leveraging
knowledge graph embeddings (Bordes et al., 2013) as initial node features to
enable knowledge-text interaction from a structural perspective.

##  3 Methodology

Figure 2 presents an overview of our knowledge-aware political perspective
detection approach KGAP. Specifically, we firstly construct a domain-specific
knowledge graph of contemporary politics to serve as external knowledge for
perspective detection. We then transform news articles into heterogeneous
information networks, which jointly models textual content and related
external knowledge. Finally, we adopt gated relational graph convolutional
networks to learn graph representations and conduct political perspective
detection.

###  3.1 Knowledge Graph Construction

Existing knowledge graphs (KGs) such as ConceptNet (Speer et al., 2017),
Freebase (Bollacker et al., 2008), and YAGO (Tanon et al., 2020) have been
leveraged in various NLP tasks such as misinformation detection (Hu et al.,
2021), knowledge-aware question answering (Yu et al., 2021), and emotion
detection (Zhu et al., 2021). However, the task of political perspective
detection demands politics-related external knowledge to facilitate argument
reasoning, while existing KGs are too generic to be compatible or effective.
In this paper, we propose to construct a political knowledge graph and hope to
complement the scarce literature in domain-specific knowledge graph
construction. The design goals of our political knowledge graph are summarized
as follows:

  * •

domain-specific: the knowledge graph should focus on closely related political
background knowledge for efficient knowledge infusion.

  * •

diversified sources: the knowledge graph should draw from both generic
information sources and political expert knowledge.

  * •

scalable: the knowledge graph construction process should be able to expand in
scope to cope with unseen entities and task-specific settings.

  * •

adaptable: the knowledge graph construction process should be applicable to
different nations, time periods, and political systems.

Bearing these design principles in mind, we first select active U.S. political
actors in the past decade as entities and determine ten types of political
relations to ensure the KG is domain-specific. We then draw from Wikipedia
pages of political actors as generic knowledge and two political think tanks,
AFL-CIO111aflcio.org/scorecard and Heritage
Action222heritageaction.com/scorecard, as political expert knowledge to
determine triples and ensure the KG leverages diversified sources. Based on
the initial entities and relations, we then leverage co-reference resolution
(Lee et al., 2018) to expand the political KG by identifying new entities in
Wikipedia documents. As a result, we construct a scalable KG while ensuring
it’s domain-specific for political perspective detection. Our political KG
focuses on U.S. politics in the past decade to better align with news corpus,
while our proposed construction process is adaptable to other nations and time
periods by starting with corresponding political actors. As a result, we
obtain a political knowledge graph with 1,071 entities and 10,703 triples to
serve as external knowledge. A complete list of entities, relations and more
KG construction details could be found in the technical appendix.

###  3.2 News Graph Construction

Graphs and graph neural networks have become increasingly involved in NLP
tasks such as misinformation detection (Hu et al., 2021) and question
answering (Yu et al., 2021). In this paper, we construct news graphs to
jointly model textual content and external knowledge. We also propose to
leverage knowledge graph embeddings as node attributes to enable structural
knowledge-text interaction. Specifically, we firstly determine the nodes in
the news graph:

  * •

document node: we use one document node to represent the news article. We use
pre-trained RoBERTa (Liu et al., 2019) to encode news title and use it as node
attribute $v^{d}$.

  * •

paragraph node: we use one node to represent each paragraph and use pre-
trained RoBERTa for node features $v^{p}$.

  * •

entity node: we use one node for each entity in our political KG. We propose
the novel approach of using TransE (Bordes et al., 2013) KG embeddings as node
attributes $v^{e}$.

After determining the nodes in the news graph, we construct a heterogeneous
graph by connecting them with three types of edges:

  * •

doc-para edge: the document node is connected with each paragraph node by doc-
para edges. This ensures that every paragraph contributes to the overall
perspective analysis.

  * •

para-para edge: each paragraph node is connected with paragraphs that precede
and follow it. These edges preserve the sequential flow of the original news
document.

  * •

para-ent edge: each paragraph node is connected with mentioned entities in our
political KG. We conduct co-reference resolution and adopt TagMe (Ferragina
and Scaiella, 2011) to align text with entities. These edges aim to
incorporate external knowledge for stance detection.

We denote these three edges as $R=\\{doc-para,para-para,para-ent\\}$. As a
result, we obtain news graphs for each news article, which jointly model
textual content and external knowledge to enable knowledge-text interaction
from a structural perspective. We then transform node attributes $v^{d}$,
$v^{p}_{i}$, and $v^{e}_{i}$ with fully connected layers to obtain the initial
node features, where $i$ denotes the $i$-th node in the set. Specifically, for
document node and paragraph nodes:

$x_{0}^{(0)}=\phi(W_{S}\cdot v^{d}+b_{S}),\ \ \ x_{i}^{(0)}=\phi(W_{S}\cdot
v^{p}_{i}+b_{S})$ (1)

where $W_{S}$ and $b_{S}$ are learnable parameters and $\phi$ denotes Leaky-
ReLU. Similarly, we use another fully connected layer to obtain initial
features for entity nodes:

$x_{i}^{(0)}=\phi(W_{E}\cdot v^{e}_{i}+b_{E})$ (2)

where $W_{E}$ and $b_{E}$ are learnable parameters. In the following, we use
$x^{(0)}$ to denote initial node features for graph neural networks.

Method | Text+ | SemEval |  AllSides
---|---|---|---
Acc | MaF | Acc | MaF
CNN_GloVe |  | $79.63$ | $N/A$ | $N/A$ | $N/A$
CNN_ELMo |  | $84.04$ | $N/A$ | $N/A$ | $N/A$
HLSTM_GloVe |  | $81.58$ | $N/A$ | $N/A$ | $N/A$
HLSTM_ELMo |  | $83.28$ | $N/A$ | $N/A$ | $N/A$
HLSTM_Embed |  | $81.71$ | $N/A$ | $76.45$ | $74.95$
HLSTM_Output |  | $81.25$ | $N/A$ | $76.66$ | $75.39$
BERT |  | $84.03$ | $82.60$ | $81.55$ | $80.13$
MAN_GloVe | ✓ | $81.58$ | $79.29$ | $78.29$ | $76.96$
MAN_ELMo | ✓ | $84.66$ | $83.09$ | $81.41$ | $80.44$
MAN_Ensemble | ✓ | $86.21$ | $84.33$ | $85.00$ | $84.25$
KGAP_RGCN | ✓ | $89.22$ | $84.41$ | 86.98 | 86.53
KGAP_GRGCN | ✓ | 89.56 | 84.94 | $86.02$ | $85.52$
Table 1: Model performance on datasets SemEval and Allsides. $N/A$ denotes
that the result is not reported in the related work. Text+ indicates whether
the method leverages more than textual content. Acc and MaF denote accuracy
and macro-averaged F1-score.

###  3.3 Learning and Optimization

We propose to adopt gated relational graph convolutional networks (gated
R-GCNs) to learn representations for the news graphs and conduct political
perspective detection as graph-level classification. For the $l$-th layer of
gated R-GCN, we firstly aggregate messages from neighbors as follows:

$u_{i}^{(l)}=\Theta_{s}\cdot x_{i}^{(l-1)}+\sum_{r\in R}\sum_{j\in
N_{r}(i)}\frac{1}{|N_{r}(i)|}\Theta_{r}\cdot x_{j}^{(l-1)}$ (3)

where $u_{i}^{(l)}$ is the hidden state for the $i$-th node in the $l$-th
layer, $N_{r}(i)$ is node $i$’s neighborhood of relation $r$, $\Theta_{s}$ and
$\Theta_{r}$ are learnable parameters. We then calculate gate levels:

$a_{i}^{(l)}=\sigma(W_{A}\cdot[u_{i}^{(l)},x_{i}^{(l-1)}]+b_{A})$ (4)

where $\sigma(\cdot)$ is the sigmoid function, $[\cdot,\cdot]$ denotes the
concatenation operation, $W_{A}$ and $b_{A}$ are learnable parameters. We then
apply the gate to $u_{i}^{(l)}$ and $x_{i}^{(l-1)}$:

$x_{i}^{(l)}=tanh(u_{i}^{(l)})\odot
a_{i}^{(l)}+x_{i}^{(l-1)}\odot(1-a_{i}^{(l)})$ (5)

where $x_{i}^{(l)}$ is the output of the $l$-th gated R-GCN layer and $\odot$
denotes the Hadamard product operation.

After applying a total of $L$ gated R-GCN layers, we obtain the learned node
representations $x^{(L)}$. We then apply average pooling on paragraph node
representations:

$v^{g}=\frac{1}{s}\sum_{i=1}^{s}x_{i}^{(L)}$ (6)

where $s$ denotes the total amount of paragraphs in the news document. We then
transform $v^{g}$ with a softmax layer:

$\hat{y}=softmax(W_{O}\cdot v^{g}+b_{O})$ (7)

where $\hat{y}$ is the predicted perspective label, $W_{O}$ and $b_{O}$ are
learnable parameters. We then derive the loss function $L$:

$L=-\sum_{D}\sum_{i=1}^{Y}y_{i}log(\hat{y_{i}})+\lambda\sum_{w\in\theta}w^{2}$
(8)

where $Y$ is the number of stance labels, $y$ denotes stance annotation, $D$
represents the data set, $\theta$ denotes all learnable parameters in our
proposed model, and $\lambda$ is a hyperparameter.

##  4 Experiments

Dataset | # News Articles | # class | Class Distribution | Average # Sentence | Average # Word
---|---|---|---|---|---
SemEval | 645 | 2 | 407 / 238 | 27.11 | 494.29
Allsides | 10,385 | 3 | 4,164 / 3,931 / 2,290 | 49.96 | 1040.05

Table 2: Details of two datasets SemEval and Allsides.

###  4.1 Dataset

We evaluate our approach on the same benchmark datasets as in previous works
(Li and Goldwasser, 2019, 2021), namely SemEval and Allsides. Dataset details
are are presented in Table 2. SemEval is the training data set from the
SemEval 2019 Task 4: Hyperpartisan News Detection (Kiesel et al., 2019).
Allsides is a larger and more diversified political perspective detection
dataset collected in Li and Goldwasser (2019). We follow the same settings in
Li and Goldwasser (2019, 2021) so that our results are directly comparable
with previous works.

###  4.2 Baselines

We compare our method with the following competitive baselines on the SemEval
and Allsides benchmarks:

  * •

CNN (Jiang et al., 2019) is the first place solution from the SemEval 2019
Task 4 contest (Kiesel et al., 2019). It uses GloVe (CNN_GloVe) and ELMo
(CNN_ELMo) word embeddings with convolutional layers for stance prediction.

  * •

HLSTM (Li and Goldwasser, 2019) stands for hierarchical long short-term memory
networks (Yang et al., 2016). It encodes news with GloVe (HLSTM_GloVe) and
ELMo (HLSTM_ELMo), hierarchical LSTMs, and aggregate with self-attention for
political perspective detection.

  * •

HLSTM_Embed and HLSTM_Output (Li and Goldwasser, 2021) use Wikipedia2Vec
(Yamada et al., 2018) and BERT-inspired masked entity models to learn entity
representations and concatenate them with word embeddings or document
representation for stance detection.

  * •

BERT (Devlin et al., 2018) Pre-trained BERT is fine-tuned on the specific task
of political perspective detection.

  * •

MAN (Li and Goldwasser, 2021) is a political perspective detection method that
is pre-trained with social and linguistic information and fine-tuned with news
bias labels.

Figure 3: Statistics of entity mentions and para-ent edges under generic and
domain-specific knowledge graphs. The line, shadow and line segment indicates
the average, distribution and range of entity mentions and para-ent edges on
two benchmark datasets.

###  4.3 Implementation

We implement our proposed knowledge-aware political perspective detection
approach with pytorch (Paszke et al., 2019), pytorch lightning (Falcon and The
PyTorch Lightning team, 2019), pytorch geometric (Fey and Lenssen, 2019), and
the HuggingFace transformers library (Wolf et al., 2020). We present the
hyperparameter settings of our proposed knowledge-aware political perspective
detection approach in Table 3 to facilitate reproduction. We follow the same
hyperparameter settings in Table 3 to compare with baselines and conduct
ablation studies unless stated otherwise.

Hyperparameter | SemEval / Allsides
---|---
RoBERTa embedding size |  768
TransE embedding size |  200 / 768
gated R-GCN hidden size |  512
optimizer |  Adam
learning rate |  $10^{-3}$
batch size |  16
dropout |  0.5 / 0.6
maximum epochs |  50 / 150
$L2$-regularization $\lambda$ |  $10^{-5}$
gated R-GCN layer count $L$ |  2

Table 3: Hyperparameter settings of our model.

###  4.4 Experiment Results

We run KGAP with R-GCN (KGAP_RGCN) and gated R-GCN (KGAP_GRGCN) and report
model performance in Table 1, which demonstrates that:

  * •

KGAP consistently outperforms all competitive baselines on both SemEval and
Allsides, including the state-of-the-art method MAN (Li and Goldwasser, 2021).

  * •

Two methods that leverage information other than textual content, namely MAN
and ours, outperform other baselines. Our method further incorporates external
knowledge in political knowledge graphs and outperforms MAN.

  * •

For methods that use word embeddings, ELMo often outperforms their GloVe
counterparts. This suggests that text modeling is still essential in political
perspective detection.

Figure 4: Average model performance and standard deviation when triples in our
constructed political knowledge graph (a-d) and para-ent edges in our news
graphs (e-h) are gradually removed.

In conclusion, KGAP achieves state-of-the-art performance on two widely
adopted benchmarks. In the following, we study the effect of external
knowledge, textual content, and graphs in our method and the task of political
perspective detection. We run all experiments five times to ensure a
consistent evaluation and report the average performance as well as standard
deviation if necessary.

###  4.5 Knowledge Study

####  4.5.1 Knowledge Density

We study the effect of knowledge density with two different settings. We
gradually remove triples in our constructed political KG and report model
performance in Figure 4 (a) - (d). We then gradually remove para-ent edges in
news graphs and report model performance in Figure 4 (e) - (h). In both
settings, model performance drops significantly with very little knowledge,
while quickly saturating with $10\%$ to $20\%$ external knowledge. Another
performance boost is often witnessed at $100\%$ external knowledge, which
argues for complete KGs. Besides, the effect of external knowledge is greater
on the smaller dataset SemEval, which suggests that our knowledge-aware
approach is especially effective with limited data.

####  4.5.2 Knowledge Type

To examine whether our constructed political knowledge graph is essential in
our model’s performance, we compare it with various generic knowledge graphs
(Speer et al., 2017; Bollacker et al., 2008; Tanon et al., 2020) that are
widely adopted in NLP tasks. Using TagMe (Ferragina and Scaiella, 2011) as a
tool for text-entity alignment, Figure 3 illustrates how many KG entities are
mentioned in news articles and how many para-ent edges are captured in
constructed news graphs. It shows that fewer entities and para-ent edges are
provided by our politics-specific knowledge graph, which indicates that our KG
adds little computational burden to the news graphs. We then report model
performance paired with different knowledge graphs in Table 4. It is
demonstrated that our domain-specific knowledge graph outperforms generic
knowledge graphs in political perspective detection, which indicates
successful domain-specific KG construction and proves the necessity and
efficiency of our political knowledge graph.

KG | SemEval | AllSides
---|---|---
Acc | MaF | Acc | MaF
ConceptNet | $87.81$ | $82.65$ | $85.59$ | $85.08$
FreeBase | $87.98$ | $82.73$ | $85.84$ | $85.32$
YAGO | $87.30$ | $81.78$ | $85.18$ | $84.66$
Ours | 89.56 | 84.94 | 86.02 | 85.52
Table 4: Average model performance when using generic and domain-specific knowledge graphs as external knowledge. KGE Method | SemEval | AllSides
---|---|---
Acc | MaF | Acc | MaF
TransE | $89.56$ | $84.94$ | $86.02$ | $85.52$
TransR | $88.54$ | $83.45$ | $85.15$ | $84.61$
DistMult | $88.51$ | $83.63$ | $84.47$ | $83.90$
HolE | $88.85$ | $83.68$ | $84.78$ | $84.24$
RotatE | $88.84$ | $84.04$ | $85.61$ | $85.11$
Table 5: Average model performance with entity node attributes derived by
different knowledge graph embedding techniques.

####  4.5.3 Knowledge Graph Embedding

In this paper, we propose the novel approach of leveraging knowledge graph
embeddings as initial node features to facilitate knowledge-text interaction
from a structural perspective. We further explore this idea with other
knowledge graph embedding techniques (Lin et al., 2015; Yang et al., 2014;
Nickel et al., 2016; Sun et al., 2019) and report model performance in Table
5. It is demonstrated that model performance on both datasets does not change
significantly, indicating that our approach does not rely on specific
knowledge graph embedding techniques. We further train TransE (Bordes et al.,
2013) to different extents and report performance in Figure 5. It is
illustrated that as little as 10 epochs of TransE training would lead to
better task performance than random initialization (0 epoch), while our
approach is generally effective with knowledge graph embeddings with more
training epochs.

Figure 5: Average model performance and standard deviation when TransE KG
embeddings are trained for different epochs. Figure 6: Average performance of
text models (base, lighter) and combining them with our news graph structure
(graph, darker).

###  4.6 Text Study

We observed in Section 4.4 that text modeling plays an important role in
political perspective detection. We further explore how different text
analysis techniques perform on their own and when combined with our news
graphs. Specifically, we encode news text with text analysis techniques and
conduct political perspective detection with fully connected layers for text-
only settings. We then substitute the paragraph node attributes with these
methods for graph-based settings. Figure 6 demonstrates that advanced language
models consistently outperform classic methods, indicating the correlation
between model performance and text processing ability. Besides, Figure 6 shows
that language modeling methods perform better when combined with our proposed
news graph, indicating the necessity of incorporating structural information
and external knowledge in political perspective detection.

###  4.7 Graph Study

We propose the novel approach of constructing news graphs to jointly model
textual content and external knowledge while conducting political perspective
detection as graph-level classification. To examine the effect of news graphs,
we firstly remove three types of edges and report model performance in Table
6. It is demonstrated that all three types of edges contribute to model
performance, while para-ent edges are most essential on the smaller SemEval
since they help incorporate external knowledge to boost task performance on
limited data. Besides, we adopt R-GCN and gated R-GCN for graph representation
learning on our constructed news graphs. We further explore the effect of GNNs
by changing to other GNN layers and report performance in Table 7, which
indicates the necessity of GNN heterogeneity in our approach.

Ablation Setting | SemEval | AllSides
---|---|---
Acc | MaF | Acc | MaF
no doc-para edges | $89.03$ | $84.23$ | $-$ | $-$
no para-para edges | $88.41$ | $83.14$ | $82.52$ | $81.83$
no para-ent edges | $86.71$ | $80.60$ | $85.54$ | $85.02$
full graph | 89.56 | 84.94 | 86.02 | 85.52

Table 6: Average model performance when three different types of edges in the
news graphs are respectively removed.

GNN Type | Het. | SemEval | AllSides
---|---|---|---
Acc | MaF | Acc | MaF
GCN |  | $87.20$ | $81.63$ | $86.91$ | $86.43$
GAT |  | $89.00$ | $84.41$ | $84.12$ | $83.33$
GraphSAGE |  | $89.12$ | $84.47$ | $86.77$ | $86.24$
R-GCN | ✓ | $89.22$ | $84.41$ | 86.98 | 86.53
Gated R-GCN | ✓ | 89.56 | 84.94 | $86.02$ | $85.52$

Table 7: Average model performance with various heterogeneous and homogeneous
GNNs. Het. indicates heterogeneous GNNs.

##  5 Conclusion

In this paper, we propose an end-to-end, inductive and graph-based approach to
leverage domain-specific external knowledge in political perspective
detection. We firstly construct a domain-specific knowledge graph as external
knowledge for political perspective detection. We then construct news graphs
to jointly model news content and external knowledge. Finally, we adopt
relational graph neural networks to learn graph representations and framing
the task as graph-level classification. Extensive experiments demonstrate that
KGAP consistently outperforms state-of-the-art methods on two widely adopted
perspective detection benchmarks. Further ablation studies indicate the
necessity of external knowledge and examine the effect of knowledge, text, and
graph in KGAP.

## References

  * Baly et al. (2020) Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020.  We can detect your bias: Predicting the political ideology of news articles.  _arXiv preprint arXiv:2010.05338_.
  * Barberá et al. (2015) Pablo Barberá, John T Jost, Jonathan Nagler, Joshua A Tucker, and Richard Bonneau. 2015.  Tweeting from left to right: Is online political communication more than an echo chamber?  _Psychological science_ , 26(10):1531–1542.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , pages 1247–1250.
  * Bordes et al. (2013) Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ , 26.
  * Darwish et al. (2020) Kareem Darwish, Peter Stefanov, Michaël Aupetit, and Preslav Nakov. 2020.  Unsupervised user stance detection on twitter.  In _Proceedings of the International AAAI Conference on Web and Social Media_ , volume 14, pages 141–152.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Du et al. (2017) Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017.  Stance classification with target-specific neural attention networks.  International Joint Conferences on Artificial Intelligence.
  * Falcon and The PyTorch Lightning team (2019) William Falcon and The PyTorch Lightning team. 2019.  PyTorch Lightning.
  * Ferragina and Scaiella (2011) Paolo Ferragina and Ugo Scaiella. 2011.  Fast and accurate annotation of short texts with wikipedia pages.  _IEEE software_ , 29(1):70–75.
  * Fey and Lenssen (2019) Matthias Fey and Jan Eric Lenssen. 2019.  Fast graph representation learning with pytorch geometric.  _arXiv preprint arXiv:1903.02428_.
  * Horne et al. (2018) Benjamin D Horne, Sara Khedr, and Sibel Adali. 2018.  Sampling the news producers: A large news and feature data set for the study of the complex media landscape.  In _Twelfth International AAAI Conference on Web and Social Media_.
  * Hu et al. (2021) Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. 2021.  Compare to the knowledge: Graph neural fake news detection with external knowledge.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 754–763.
  * Jiang et al. (2011) Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011.  Target-dependent twitter sentiment classification.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , pages 151–160.
  * Jiang et al. (2019) Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019\.  Team bertha von suttner at semeval-2019 task 4: Hyperpartisan news detection using elmo sentence representation convolutional network.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 840–844.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019.  Semeval-2019 task 4: Hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 829–839.
  * Lee et al. (2018) Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018.  Higher-order coreference resolution with coarse-to-fine inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics.
  * Li and Goldwasser (2019) Chang Li and Dan Goldwasser. 2019.  Encoding social information with graph convolutional networks forpolitical perspective detection in news media.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2594–2604.
  * Li and Goldwasser (2021) Chang Li and Dan Goldwasser. 2021.  Using social and linguistic information to adapt pretrained representations for political perspective identification.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4569–4579.
  * Lin et al. (2015) Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015.  Learning entity and relation embeddings for knowledge graph completion.  In _Twenty-ninth AAAI conference on artificial intelligence_.
  * Liu et al. (2021) Rui Liu, Zheng Lin, Yutong Tan, and Weiping Wang. 2021.  Enhancing zero-shot and few-shot stance detection with commonsense knowledge graph.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3152–3157.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Magdy et al. (2016) Walid Magdy, Kareem Darwish, Norah Abokhodair, Afshin Rahimi, and Timothy Baldwin. 2016.  # isisisnotislam or# deportallmuslims? predicting unspoken views.  In _Proceedings of the 8th ACM Conference on Web Science_ , pages 95–106.
  * Nickel et al. (2016) Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016.  Holographic embeddings of knowledge graphs.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32:8026–8037.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proc. of NAACL_.
  * Řehůřek and Sojka (2010) Radim Řehůřek and Petr Sojka. 2010.  Software Framework for Topic Modelling with Large Corpora.  In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_ , pages 45–50, Valletta, Malta. ELRA.  http://is.muni.cz/publication/884893/en.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Sun et al. (2019) Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019.  Rotate: Knowledge graph embedding by relational rotation in complex space.  _arXiv preprint arXiv:1902.10197_.
  * Tanon et al. (2020) Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. 2020.  Yago 4: A reason-able knowledge base.  In _European Semantic Web Conference_ , pages 583–596. Springer.
  * Wilson et al. (2020) Anne E Wilson, Victoria A Parker, and Matthew Feinberg. 2020.  Polarization in the contemporary political and media landscape.  _Current Opinion in Behavioral Sciences_ , 34:223–228.  Political Ideologies.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yamada et al. (2018) Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2018.  Wikipedia2vec: An efficient toolkit for learning and visualizing the embeddings of words and entities from wikipedia.  _arXiv preprint arXiv:1812.06280_.
  * Yang et al. (2014) Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014.  Embedding entities and relations for learning and inference in knowledge bases.  _arXiv preprint arXiv:1412.6575_.
  * Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016\.  Hierarchical attention networks for document classification.  pages 1480–1489.
  * Yu et al. (2021) Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2021.  Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering.  _arXiv preprint arXiv:2110.04330_.
  * Zhu et al. (2021) Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021.  Topic-driven and knowledge-aware transformer for dialogue emotion detection.  _arXiv preprint arXiv:2106.01071_.

##  Appendix A Limitation

We identified two minor limitations of KGAP:

  * •

We noticed that there are only a few para-ent edges in the news graphs of
certain news articles. We examined these news articles and found that some
entity mentions are not captured by the entity linking tool TagMe. This issue
might be addressed by using better entity linking tools.

  * •

Political facts change over time, for example, politicians in the United
States may represent different districts due to redistricting. Our knowledge
graph does not model such changes for now, while this issue may be addressed
by introducing temporal knowledge graphs.

##  Appendix B Knowledge Graph Details

We construct a political knowledge graph (KG) of contemporary U.S. politics to
serve as domain-specific external knowledge for political perspective
detection, while our approach could be extended to political scenarios in
other countries. Table 8 presents entity and relation types in our KG.

For elected office, we consider the White House, House of Representatives,
Senate, Supreme Court and governorship. For time period, we consider the 114th
to 117th congress while our KG could be similarly extended to other time
periods. We then retrieve the presidents, supreme court justices, senators,
congresspersons, and governors that overlap with these time periods from
Wikipedia333https://www.wikipedia.org/ and U.S. Congress
website444https://www.congress.gov/. For states, we consider the 50 U.S.
states. For political parties, we consider the Democratic Party, the
Republican Party and independents. For the first five types of relations
(affiliated_to, from, appoint, overlap_with and member_of), we conduct co-
reference resolution to identify new entity mentions, connect the entities and
expand the KG. For political ideology, we use two entities to represent
liberal conservative values. We then resort to the legislator scoreboards at
AFL-CIO555aflcio.org/scorecard and Heritage
Action666heritageaction.com/scorecard. These scoreboards score U.S.
legislators from 0 to 100 to evaluate how liberal or conservative they are. We
partition the score range into strongly favor (90 - 100), favor (75 - 90),
neutral (25 - 75), oppose (10 - 25) and strongly oppose (0 - 10) to connect
entities with these five relations. Our political KG draws from both generic
information sources and political expert knowledge.

As a result, we construct a political knowledge graph with 1,071 entities and
10,703 triples to serve as external knowledge, which is domain-specific,
scalable, adaptable and draws from diversified information sources. Our
political KG could also be helpful for related tasks such as disinformation
detection. We submit the constructed knowledge graph as supplementary material
to facilitate reproduction.

Entity Type | Example | Relation Type | Example
---|---|---|---
elected office | the U.S. Senate | affiliated_to | (Joe Biden, affiliated_to, Democratic Party)
time period | 117th congress | from | (Ted Cruz, from, Texas)
president | Joe Biden | appoint | (Donald Trump, appoint, Amy Coney Barrett)
supreme court justice | Amy Coney Barrett | overlap_with | (Joe Biden, overlap_with, 117th congress)
senator | Elizabeth Warren | member_of | (Dianne Feinstein, member_of, the U.S. Senate)
congressperson | Nancy Pelosi | strongly_favor | (Bernie Sanders, strongly_favor, liberal_values)
governor | Ron DeSantis | favor | (Joe Cunningham, favor, liberal_values)
state | Massachusetts | neutral | (Henry Cuellar, neutral, liberal_values)
political party | Republican Party | oppose | (Lamar Alexander, oppose, liberal_values)
political ideology | liberal values | strongly_oppose | (Ted Cruz, strongly_oppose, liberal_values)

Table 8: List of entities and relations in our collected political knowledge
graph.

##  Appendix C Dataset Details

We evaluate our knowledge-aware political perspective detection approach on
SemEval (Kiesel et al., 2019) and Allsides (Li and Goldwasser, 2019), two
widely adopted benchmarks in previous works. We present dataset details in
Table 2. For SemEval, we follow the same 10-fold evaluation settings and the
exact folds as in Li and Goldwasser (2021). For Allsides, we follow the same
3-fold evaluation settings and the exact folds as in Li and Goldwasser (2021),
while a few news urls have expired and we could not retrieve the original news
content, leading to minor differences. In this way, our method’s performance
are directly comparable with previous state-of-the-art approaches (Li and
Goldwasser, 2019, 2021).

##  Appendix D Experiment Details

###  D.1 GNN Setting

We train our approach with both relational graph convolutional networks
(R-GCN) and gated relational graph convolutional networks (gated R-GCN), while
our approach does not rely on any specific GNN architecture. All experiments
in Section 4.4, 4.5 and 4.6 are conducted with gated R-GCN.

###  D.2 Figure 4

For better visualization effects in Figure 4, we leave out news articles whose
number of entity mentions or para-ent edges is five times greater than the
average value. This approximation does not compromise the main conclusion that
our domain-specific knowledge graph involves less computational burden while
performing better than generic knowledge graphs.

###  D.3 Table 4

The Allsides benchmark does not explicitly provide news titles. As a result,
there are no doc-para edges in the constructed news graphs for this dataset,
and thus the “no doc-para edges" setting could not be conducted on Allsides.

###  D.4 Figure 6

For Word2Vec word embeddings, we use the gensim library (Řehůřek and Sojka,
2010). For GloVe word embeddings, we use the pre-trained
vectors777https://nlp.stanford.edu/projects/glove/ from Stanford NLP. For ELMo
word embeddings, we use the AllenNLP implementation (Peters et al., 2018). For
BERT and RoBERTa, we use pre-trained models in the transformers library (Wolf
et al., 2020) to encode news content and serve as textual features. For all
the base settings, we use Leaky-ReLU and two fully connected layers with 512
as hidden layer size.


# Improving Automatic Quotation Attribution in Literary Novels

Krishnapriya Vishnubhotla1,4, Frank Rudzicz2,4,1, Graeme Hirst1 Adam Hammond3
1Department of Computer Science, University of Toronto
2Faculty of Computer Science, Dalhousie University
3Department of English, University of Toronto
4Vector Institute for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Current models for quotation attribution in literary novels assume varying
levels of available information in their training and test data, which poses a
challenge for in-the-wild inference. Here, we approach quotation attribution
as a set of four interconnected sub-tasks: character identification,
coreference resolution, quotation identification, and speaker attribution. We
benchmark state-of-the-art models on each of these sub-tasks independently,
using a large dataset of annotated coreferences and quotations in literary
novels (the Project Dialogism Novel Corpus). We also train and evaluate models
for the speaker attribution task in particular, showing that a simple
sequential prediction model achieves accuracy scores on par with state-of-the-
art models111Code and data can be found at https://github.com/Priya22/speaker-
attribution-acl2023.

##  1 Introduction

We focus on the task of automatic quotation attribution, or speaker
identification, in full-length English-language literary novels. The task
involves attributing each quotation (dialogue) in the novel to the character
who utters it. The task is complicated by several factors: characters in a
novel are referred to by various names and aliases (Elizabeth, Liz, Miss
Bennet, her sister); these aliases can change and be added over the course of
the novel; and authors often employ differing patterns of dialogue in the
text, whereby quotations are sometimes attached to the speaker explicitly via
a speech verb, and at other times require keeping track of character turns
over multiple paragraphs. The development of automated methods has also been
hindered by the paucity of annotated datasets on which models can be trained
and evaluated.

Existing methods for quotation attribution fall into one of two groups: those
that directly attribute the quotation to a named character entity and those
that treat it as a two-step process in which quotations are first attached to
the nearest relevant mention of a character and mentions are then resolved to
a canonical character name via a coreference resolution model. We contend that
most use-cases of a quotation attribution system involve resolving the speaker
mention to one among a list of character entities. Thus, the usability of
these systems is very much dependent on their ability to compile such a list
of character entities and to resolve each attributed mention to an entity from
this list.‘

Here, we use the Project Dialogism Novel Corpus Vishnubhotla et al. (2022), a
large dataset of annotated coreferences and quotations in literary novels, to
design and evaluate pipelines of quotation attribution. Our analysis shows
that state-of-the-art models are still quite poor at character identification
and coreference resolution in this domain, thus hindering functional quotation
attribution.

##  2 Background and Prior Work

Elson and McKeown (2010) introduce the CQSA corpus, which contains quotations
from excerpts from 4 novels and 7 short-stories that are annotated for the
nearest speaker mention, which can be named (e.g., Elizabeth), or nominal (her
friend). On average, only 25% of the attributions in CQSA are to a named
entity.

In contrast, He et al. (2013) link quotations directly to entities, and a list
of characters and aliases is required for attribution. This list is generated
with a named entity recognition (NER) model to obtain entity terms, which are
then grouped together using Web resources such as Wikipedia.

The GutenTag package from Brooke et al. (2015) contains modules for generating
character lists and identifying speakers in literary texts. The former is
based on the LitNER model Brooke et al. (2016a), which bootstraps a classifier
from a low-dimensional Brown clustering of named entities from Project
Gutenberg texts. The speaker attribution model is a simple rule-based approach
that identifies the nearest named entity.

Sims and Bamman (2020) annotate the first 2000 tokens of 100 novels from the
LitBank dataset111https://github.com/dbamman/litbank. Quotations are linked to
a unique speaker from a predefined list of entities. LitBank also contains
annotations for coreference for these tokens Bamman et al. (2020). The BookNLP
package222https://github.com/booknlp/booknlp from the same group contains pre-
trained models for NER, coreference resolution, and speaker attribution,
although the latter is only at the mention-level.

Cuesta-Lazaro et al. (2022) attempt to reconcile the differences in pre-
requisites and methodologies of prior attribution systems by proposing a
modularization of the task into three sub-tasks: quotation identification,
character identification, and speaker attribution. They evaluate baselines for
each component, propose a new state-of-the-art method for speaker attribution,
and quantify the relative importance of each module in an end-to-end pipeline.
Their speaker attribution module, however, considers only named mentions in
the text as candidate speakers, leading to a lower performance on implicit and
anaphoric quotations. Neither their dataset of 15 novels nor their model for
speaker attribution have been made public, precluding comparison with our work
below.

In our work, we follow this modular formulation, with some key differences:
(a) we evaluate an additional sub-task of coreference resolution, allowing us
to (b) test an attribution model that can work with both named and pronominal
candidate mentions surrounding a quotation; and (c) we evaluate our models on
a publicly available dataset.

##  3 Dataset: PDNC

We briefly describe here the Project Dialogism Novel Corpus Vishnubhotla et
al. (2022). PDNC consists of 22 full-length English novels, published in the
19th and 20th centuries, annotated with the following information:

Characters: A list of characters in the novel. This includes characters who
speak, are addressed to, or referred to multiple times in the novel. Each
character is identified by a main name (e.g., Elizabeth Bennet), as well as a
set of aliases (Liz, Lizzie, Eliza). We do not distinguish between the two,
and treat each character entity as identifiable by a set of names (so that
Elizabeth Bennet, Liz, Lizzie, Eliza forms one character entity).

Quotations: Each uttered quotation in the novel is annotated with its speaker
and addressee(s); with the referring expression, if any, that indicates who
the speaker is; and with internal mentions, i.e., named or pronominal phrases
within the quotation that refer to one or more character entities. The
annotations in PDNC make it ideal for evaluating several aspects of quotation
attribution in novels, including named entity recognition, coreference
resolution, and speaker attribution.

##  4 Modularization of the Task

Character identification: The goal of this sub-task is to build a list of the
unique character entities in a novel. Although NER models perform quite well
at identifying spans of text that constitute a named entity (here, a character
name), the task is complicated by the fact that characters can have multiple
aliases in the text. Moreover, some characters may be introduced and referred
to only by social titles (the policeman, the Grand Inquisitor, the little old
man, the bystander).

Coreference resolution: The goals here are to identify text spans that refer
to a character entity (which we refer to as mentions) and to link each mention
to the correct character entity or entities to which it refers. In addition to
mentions that are personal pronouns such as he, she, and them, literary texts
have an abundance of pronominal phrases that reflect relationships between
characters, such as her husband and their father. Such phrases can also occur
within quotations uttered by a character (e.g., my father), requiring
quotation attribution as a prerequisite for complete coreference resolution.

Quotation identification: Perhaps the most straightforward of our sub-tasks,
here we identify all text spans in a novel that constitute dialogue, i.e., are
uttered by a character entity or entities.

Speaker attribution: Finally, this sub-task links each identified quotation to
a named character identity. While most models are designed to solve the more
tractable and practical problem of linking quotations to the nearest relevant
speaker mention, we subsume the mention–entity linking tasks under the
coreference resolution module, equating the two tasks.

##  5 Models and Evaluation Metrics

We evaluate each of the modules of section 4 separately. In order not to
confound the evaluation with cascading errors, at each step, we “correct” the
outputs of the automated system from the previous step by using annotations
from PDNC.

###  5.1 Character Identification

We evaluate two pipelines — GutenTag and BookNLP — on their ability to
identify the set of characters in a novel, and potentially, the set of aliases
for each character. In addition, we also test the NER system from the
spaCy333https://explosion.ai/blog/spacy-v3 module as a proxy for the state-of-
the-art in NER that is not trained explicitly for the literary domain.

Character recognition (CR): For each novel, we compute the proportion of
annotated character entities that are identified as named entities of the
category ‘person’ Doddington et al. (2004). We use a simple string-matching
approach, where we try for either a direct match, or a unique match when
common prefixes such as Mr. and Sir are removed. Thus, if a particular novel
has $N$ character entities annotated, the NER model outputs a list of $K$
named ‘person’ entities, and $K^{\prime}$ of these entities are in turn
matched with $M$ out of the $N$ characters, the CR metric is calculated as
$M/N$.

Character clustering: We use the clustering evaluation metrics of homogeneity
(C.Hom), completeness (C.Comp), and their harmonic mean, v-score to evaluate
named entity clusters. Homogeneity (between 0 and 1) is the fraction of named
clusters that link to the same character entity; completeness is the number of
homogeneous clusters a single entity is distributed over (ideal value of 1).

As an example, consider the case where we have three annotated characters for
a novel: Elizabeth Bennet, Mary Bennet, and The Queen. The set of annotated
aliases for the characters are {Elizabeth Bennet, Eliza, Lizzie, Liz}, {Mary
Bennet, Mary}, and {The Queen}. Say model $M_{1}$ outputs the following entity
clusters: {Elizabeth Bennet, Eliza}, {Liz, Lizzie} and {Mary Bennet, Mary};
model $M_{2}$ outputs {Elizabeth Bennet, Mary Bennet, Eliza, Mary}, {Liz,
Lizzie}. Each model has recognized two out of the three characters in our
list; this evaluates to a CR score of $2/3$. Each of the three clusters from
model $M_{1}$ refers solely to one character entity, resulting in a
homogeneity score of 1.0. However, these three clusters are formed for only
two unique character entities, resulting in a completeness score of $1.5$
(v-score 0.6). Analogously, model $M_{2}$ has a homogeneity score of 0.5 and a
completeness score of 1.0 (v-score 0.5).

###  5.2 Coreference Resolution

We consider two pipelines for coreference resolution: BookNLP (based on Ju et
al. (2018)) and spaCy (based on Dobrovolskii (2021)). Given a text, these
neural coreference resolution models output a set of clusters, each comprising
a set of coreferent mention spans from the input.

Evaluating this module requires annotations that link each mention span in a
novel to the character entity referred to. PDNC, unfortunately, contains these
mention annotations only for text spans within quotations.

We therefore evaluate coreference resolution only on a subset of the mention
spans in a novel, extracted as follows: We first identify the set of mention
clusters from our models that can be resolved to an annotated character
entity, using the character lists from PDNC and the string-matching approach
described above. We then prune this to only include those mention spans that
are annotated in the PDNC dataset, i.e, mention spans that occur within
quotations, and evaluate the accuracy of the resolution.

Mention clustering (M-Clus): We compute the fraction of mention clusters that
can be matched to a unique (Uniq) annotated character entity rather than to
multiple (Mult) or no (None) entities.

Mention resolution (M-Res): For those mention spans within PDNC that are
identified by the model and are assigned to a cluster that can be uniquely
matched to a character entity (# Eval), we compute the accuracy of the linking
(Acc.).

###  5.3 Quotation Identification

Most models, rule-based or neural, can identify quotation marks and thus
quotations. We evaluate how many of such quoted text instances actually
constitute dialogue, in that they are uttered by one or more characters. Our
gold standard is the set of quotations that have been annotated in PDNC, which
includes quotations uttered by multiple characters and by unnamed characters
such as a crowd.

###  5.4 Speaker Attribution

The speaker-attribution part of BookNLP’s pipeline is a BERT-based model that
uses contextual and positional information to score the BERT embedding for the
quotation span against the embeddings of mention spans that occur within a
50-word context window around the quotation; the highest-scoring mention is
selected as the speaker. We supplement this approach by limiting the set of
candidates to resolved mention spans from the coreference resolution step,
thereby directly performing quotation-to-entity linking. As we see from our
results, this method, which we refer to as BookNLP+, greatly improves the
performance of the speaker attribution model by eliminating spurious candidate
spans.

We also evaluate a sequential prediction model that predicts the speaker of a
quotation simply by looking at the sequence of speakers and mentions that
occur in some window around the quotation. We implement this as a one-layer
RNN that is fed a sequence of tokens representing the five characters
mentioned most recently prior to the quotation text, one character mention
that occurs right after, and, optionally, the set of characters mentioned
within the quotation.

##  6 Experimental Setup

We evaluate the models for character identification, coreference resolution,
and quotation identification on the entire set of 22 novels in PDNC, since we
are neither training nor fine-tuning these on this dataset. For the speaker
attribution models, we define the training setup below.

We curate the set of mention candidates for each novel in the following
manner: the mention clusters generated by BookNLP are used to extract the set
of mention spans that could be successfully resolved to a character entity
from the annotated PDNC character lists for each novel. We append to this set
the annotated mention spans (within quotations) from PDNC, as well as explicit
mention spans — that is, text spans that directly match a named alias from the
character list.

Overlaps between the three sets are resolved with a priority ranking, whereby
PDNC annotations are considered to be more accurate than explicit name
matches, which in turn take precedence over the automated coreference
resolution model.

We test with 5-fold cross-validation in two ways: splitting the annotated
quotations in each novel 80/20 and splitting the set of entire novels 80/20.

##  7 Results

Model CR C.Hom C.Comp v-score spaCy 0.81 0.16 1.02 0.27 GutenTag 0.60 0.98
1.33 1.12 BookNLP 0.85 0.86 1.18 0.99

Table 1: Character identification: Average scores across all the novels in the
dataset. Column headings are defined in the text. Scores for each individual
novel are reported in Appendix B.

From Table 1, we see that the neural NER models of spaCy and BookNLP are
better at recognizing character names than GutenTag’s heuristic system (0.81
and 0.85 vs 0.60). However, the strengths of GutenTag’s simpler Brown-
clustering–based NER system are evident when looking at the homogeneity; when
two named entities are assigned as aliases of each other, it is almost always
correct. This shows the advantage of document-level named entity clustering as
opposed to local span-level mention clustering for character entity
recognition. The cluster quality metric, on the other hand, tells us that
GutenTag still tends to be conservative with its clustering compared to
BookNLP, which nonetheless is a good strategy for the literary domain, where
characters often share surnames.

M-Clus M-Res Model # Clus Uniq Mult None # Eval Acc. spaCy 1503.1 0.093 0.061
0.846 499.0 0.746 BookNLP 1662.8 0.043 0.003 0.953 1126.6 0.774

Table 2: Coreference resolution: All scores are averaged over the 22 novels in
PDNC. Column headings are defined in the text.

Model Quotations Novels BookNLP-OG 0.40 0.40 BookNLP+ (LitBank) 0.62 0.61 Seq-
RNN 0.72 0.64 BookNLP+ (PDNC) 0.78 0.68

Table 3: Accuracy on speaker attribution for the end-to-end BookNLP model
(BookNLP-OG), the restricted model with only resolved mention spans as
candidates (row 2), the sequential prediction model, and the restricted model
trained on PDNC, for the Quotations and the entire Novels cross-validation
split.

Performance of these models on the coreference resolution task is
significantly lower (Table 2). A majority of the mention clusters from both
BookNLP and spaCy’s coreference resolution modules end up as unresolved
clusters, with no containing named identifier that could be linked to a PDNC
character entity. However, when we evaluate mention-to-entity linking on the
subset of clusters that can be resolved, both systems achieve accuracy scores
of close to 0.78, although spaCy is able to resolve far fewer mentions (499 vs
1127).

The importance of the character identification and coreference resolution
tasks can be quantified by looking the performance of the speaker attribution
models (Table 3). The end-to-end pretrained BookNLP pipeline, when evaluated
on the set of PDNC quotations (which were identified with accuracy of 0.94),
achieves an accuracy of 0.42. When we restrict the set of candidate mentions
for each quotation to only those spans that can be resolved to a unique
character entity, the attribution accuracy increases to 0.61. However, the RNN
model still beats this performance with an accuracy of 0.72 on the random data
split. When BookNLP’s contextual model is trained on data from PDNC, its
accuracy improves to 0.78. These scores drop to 0.63 and 0.68 for the entire-
novel split, where we have the disadvantage of being restricted only to
patterns of mention sequences, and not speakers.

##  8 Analysis

We briefly go over some qualitative analyses of the errors made by models in
the different sub-tasks, which serves to highlight the challenges presented by
literary text and opportunities for future research.

#### Character Identification and Coreference Resolution:

We manually examine the mention clusters identified by our coreference
resolution modules that could not be matched a unique character entity as
annotated in PDNC.

We find that, by far, the most common error is conflating characters with the
same surname or family name within a novel. For example, several of the women
characters in these novels are often referred to by the names of their
husbands or fathers, prefixed with a honorific such as Mrs. or Miss. Thus Mrs.
Archer refers to May Welland in The Age of Innocence and Miss Woodhouse refers
to Emma Woodhouse in Emma. However, a surname without a title, such as Archer
or Woodhouse, generally refers to the corresponding male character. This
results in the formation of mention clusters that take the spans Miss
Woodhouse and Woodhouse to be coreferent, despite being different character
entities. We see similar issues with father–son character pairs, such as
George Emerson and Mr. Emerson in A Room With A View, and with character pairs
that are siblings.

Quotations Novels Model Exp. Rest Exp. Rest BookNLP-OG 0.64 0.28 0.63 0.28
BookNLP+ (LitBank) 0.93 0.47 0.95 0.43 Seq-RNN 0.85 0.65 0.76 0.57 BookNLP+
(PDNC) 0.98 0.70 0.97 0.53

Table 4: Attribution accuracy for the speaker attribution models, broken down
by quotation type, for the Quotations and Novels cross-validation splits.
Column Exp. refers to explicit quotations, and column Rest refers to implicit
and anaphoric quotations.

#### Speaker Attribution:

We first quantify the proportion of quotations attributed to a mention cluster
that cannot be resolved to a named character entity with the end-to-end
application of the BookNLP pipeline.

On average, 47.7% of identified quotations are assigned to an unresolved
mention cluster as the speaker. The range of this value varies from as low as
12.5% (The Invisible Man) to as high as 78.7% (Northanger Abbey). A majority
of these unresolved attributions occur with implicit and anaphoric quotations
(76.2%), where the speaker is not explicitly indicated by a referring
expression such as Elizabeth said, as opposed to explicit quotations (23.8%).

In Table 4, we break down the performance of the speaker attribution models by
quotation type. We see that even our local context–based RNN model is able to
identify the speaker of explicit quotations with a relatively high accuracy,
and that the speaker for non-explicit quotations can also generally be modeled
using the sequence of 5–6 characters mentioned in the vicinity of the
quotation. The transformer-based models are of course able to use this local
context more effectively by making use of linguistic cues and non-linear
patterns of mentions and speakers in the surrounding text. Still, our best
performing model achieves an accuracy of only 0.53 on implicit and anaphoric
quotations when applied to novels unseen in the training set (the Novels
split).

##  9 Conclusions and Future Work

In this work, we quantitatively evaluated the key components of a functional
quotation attribution system. We showed that the initial task of recognizing
characters and their aliases in a novel remains quite a challenge, but doing
so greatly improves the performance of speaker attribution by limiting the set
of candidate speakers. However, with existing coreference resolution systems,
a large portion of mention clusters (around 90%) remain unresolved, so this
remains a problem for new research.

## Limitations

There is much variation in literary writing and narrative styles, and our work
here deals with a small, curated subset of this domain. The novels we analyze
are all in the English language, and were published between the early 19th and
early 20th centuries. The authors and novels themselves are drawn from what is
considered to be the established literary canon, and are not necessarily
representative of all the works of that era, let alone literary works of other
eras. The texts we analyze are largely uniform in narrative style. We limit
ourselves to only those quotations that are explicitly indicated as such in
the text by quotation marks, thereby eliminating more-complex styles such as
free indirect discourse Brooke et al. (2016b) and stream-of-consciousness
novels. We do not deal with nuances such as letters and diary entries nor
quotations within quotations. The models we analyze for named entity
recognition and coreference resolution use a fixed, binary formulation of the
gender information conveyed by pronominal terms. Though the development of
fairer, more representative models is constrained by current datasets, we note
that there is encouraging progress being made in this area Bamman et al.
(2020); Yoder et al. (2021).

## References

  * Bamman et al. (2020) David Bamman, Olivia Lewke, and Anya Mansoor. 2020.  An annotated dataset of coreference in English literature.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 44–54.
  * Brooke et al. (2016a) Julian Brooke, Adam Hammond, and Timothy Baldwin. 2016a.  Bootstrapped text-level named entity recognition for literature.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 344–350.
  * Brooke et al. (2015) Julian Brooke, Adam Hammond, and Graeme Hirst. 2015.  GutenTag: an NLP-driven tool for digital humanities research in the Project Gutenberg corpus.  In _Proceedings of the Fourth Workshop on Computational Linguistics for Literature_ , pages 42–47.
  * Brooke et al. (2016b) Julian Brooke, Adam Hammond, and Graeme Hirst. 2016b.  Using models of lexical style to quantify free indirect discourse in modernist fiction.  _Digital Scholarship in the Humanities_ , 32:234–250.
  * Cuesta-Lazaro et al. (2022) Carolina Cuesta-Lazaro, Animesh Prasad, and Trevor Wood. 2022.  What does the sea say to the shore? A BERT based DST style approach for speaker to dialogue attribution in novels.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5820–5829.
  * Dobrovolskii (2021) Vladimir Dobrovolskii. 2021.  Word-level coreference resolution.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7670–7675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Doddington et al. (2004) George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004.  The Automatic Content Extraction (ACE) Program ––– tasks, data, and evaluation.  In _Language Resources and Evaluation Conference_ , volume 2, pages 837–840. Lisbon.
  * Elson and McKeown (2010) David K Elson and Kathleen R McKeown. 2010.  Automatic attribution of quoted speech in literary narrative.  In _Twenty-Fourth AAAI Conference on Artificial Intelligence_.
  * He et al. (2013) Hua He, Denilson Barbosa, and Grzegorz Kondrak. 2013.  Identification of speakers in novels.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1312–1320.
  * Ju et al. (2018) Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018.  A neural layered model for nested named entity recognition.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1446–1459, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sims and Bamman (2020) Matthew Sims and David Bamman. 2020.  Measuring information propagation in literary social networks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 642–652.
  * Vishnubhotla et al. (2022) Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hirst. 2022.  The project dialogism novel corpus: A dataset for quotation attribution in literary texts.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5838–5848, Marseille, France. European Language Resources Association.
  * Yoder et al. (2021) Michael Yoder, Sopan Khosla, Qinlan Shen, Aakanksha Naik, Huiming Jin, Hariharan Muralidharan, and Carolyn Rosé. 2021.  FanfictionNLP: A text processing pipeline for fanfiction.  In _Proceedings of the Third Workshop on Narrative Understanding_ , pages 13–23, Virtual. Association for Computational Linguistics.

##  Appendix A Implementation Details

The BookNLP pipeline is available to use as a Python package, as is spaCy,
with pretrained models for coreference resolution and speaker attribution. For
the former, these models are trained on the LitBank corpus, which is a dataset
from the literary domain. We use these pretrained models to evaluate
performance on the character identification and coreference resolution tasks.
GutenTag can be run either via a Web interface or a command-line executable
(requiring Python 2). It was designed to interface with texts from the Project
Gutenberg corpus. Some of the novels in PDNC were not found in GutenTag’s
predefined database of texts, so we exclude these when reporting average
performance metrics.

##  Appendix B Results by Novel

Tables 5 and 6 show for each novel in PDNC the per-model results for character
identification that are summarized in Table 1.

BookNLP GutenTag Novel # Chars CR # Clus C.Hom C.Comp v-score CR # Clus C.Hom
C.Comp v-score A Room With A View 63 0.83 60 0.95 1.19 1.06 0.48 35 1.00 1.17
1.08 The Age of Innocence 55 0.84 48 0.81 1.26 0.99 0.64 49 1.00 1.40 1.17
Alice’s Adventures in Wonderland 51 0.67 34 0.97 1.03 1.00 0.25 14 1.00 1.08
1.04 Anne of Green Gables 113 0.87 102 0.92 1.08 0.99 0.19 25 1.00 1.14 1.06
Daisy Miller 10 1.00 13 1.00 1.30 1.13 0.80 12 1.00 1.50 1.20 Emma 18 0.89 17
0.71 1.09 0.86 0.89 27 1.00 1.69 1.26 A Handful of Dust 104 0.82 94 0.89 1.15
1.01 $-$ $-$ $-$ $-$ $-$ Howards End 55 0.95 64 0.89 1.27 1.05 0.49 33 0.97
1.23 1.08 Night and Day 50 0.94 53 0.77 1.17 0.93 0.62 40 0.97 1.30 1.11
Northanger Abbey 20 0.90 12 0.75 1.00 0.86 0.85 23 0.96 1.29 1.10 Persuasion
35 0.86 29 0.79 1.28 0.98 0.77 28 0.96 1.08 1.02 Pride and Prejudice 74 0.81
62 0.85 1.10 0.96 0.35 30 0.90 1.35 1.08 Sense and Sensibility 24 0.83 25 0.56
1.17 0.76 0.79 26 0.96 1.39 1.14 The Sign of the Four 35 0.94 32 0.72 1.05
0.85 0.60 28 1.00 1.33 1.14 The Awakening 22 0.82 17 0.88 1.07 0.97 0.77 21
0.95 1.25 1.08 The Gambler 27 0.70 22 0.91 1.18 1.03 0.59 22 1.00 1.38 1.16
The Invisible Man 31 0.94 40 0.95 1.36 1.12 0.61 32 1.00 1.68 1.25 The Man Who
Was Thursday 30 0.80 35 0.97 1.55 1.19 0.53 23 1.00 1.44 1.18 The Mysterious
Affair at Styles 30 0.80 25 0.88 1.05 0.96 0.70 28 0.96 1.35 1.12 The Picture
of Dorian Gray 43 0.88 43 0.98 1.14 1.05 0.56 27 1.00 1.12 1.06 The Sport of
the Gods 37 0.81 34 0.94 1.23 1.07 0.54 28 0.96 1.50 1.17 The Sun Also Rises
51 0.86 51 0.96 1.23 1.08 $-$ $-$ $-$ $-$ $-$ Mean 44.5 0.85 41.45 0.86 1.18
0.99 0.60 27.55 0.98 1.33 1.12

Table 5: Results of character identification for each novel with BookNLP and GutenTag. ‘# Chars’ is the number of characters in the novel. Other headers are the same as in Table 1. Novel | # Chars | CR | # Clus | C.Hom | C.Comp | v-score
---|---|---|---|---|---|---
A Room With A View | 63 | 0.78 | 64 | 0.33 | 1.24 | 0.52
The Age of Innocence | 55 | 0.85 | 90 | 0.04 | 1.00 | 0.09
Alice’s Adventures in Wonderland | 51 | 0.80 | 44 | 0.39 | 1.00 | 0.56
Anne of Green Gables | 113 | 0.69 | 98 | 0.24 | 1.04 | 0.40
Daisy Miller | 10 | 0.90 | 3 | 0.00 | 0.00 | 0.00
Emma | 18 | 0.89 | 14 | 0.07 | 1.00 | 0.13
A Handful of Dust | 104 | 0.71 | 85 | 0.26 | 1.00 | 0.41
Howards End | 55 | 0.84 | 72 | 0.18 | 1.08 | 0.31
Night and Day | 50 | 0.88 | 52 | 0.15 | 1.00 | 0.27
Northanger Abbey | 20 | 0.90 | 15 | 0.07 | 1.00 | 0.12
Persuasion | 35 | 0.89 | 36 | 0.06 | 1.00 | 0.11
Pride and Prejudice | 74 | 0.68 | 78 | 0.17 | 1.00 | 0.29
Sense and Sensibility | 24 | 0.83 | 21 | 0.10 | 1.00 | 0.17
The Sign of the Four | 35 | 0.80 | 40 | 0.05 | 1.00 | 0.10
The Awakening | 22 | 0.86 | 24 | 0.12 | 1.00 | 0.22
The Gambler | 27 | 0.74 | 18 | 0.22 | 1.00 | 0.36
The Invisible Man | 31 | 0.84 | 37 | 0.22 | 1.00 | 0.36
The Man Who Was Thursday | 30 | 0.73 | 26 | 0.19 | 1.00 | 0.32
The Mysterious Affair at Styles | 30 | 0.87 | 29 | 0.10 | 1.00 | 0.19
The Picture of Dorian Gray | 43 | 0.86 | 32 | 0.19 | 1.00 | 0.32
The Sport of the Gods | 37 | 0.81 | 43 | 0.12 | 1.00 | 0.21
The Sun Also Rises | 51 | 0.82 | 56 | 0.32 | 1.12 | 0.50
Mean | 44.5 | 0.81 | 44.40 | 0.16 | 1.02 | 0.27
Table 6: Results of character identification for each novel with spaCy. ‘#
Chars’ is the number of characters in the novel. Other headers are the same as
in Table 1.