# Improving Quotation Attribution with Fictional Character Embeddings

Gaspard Michel†∗
<EMAIL_ADDRESS>&Elena V. Epure†
<EMAIL_ADDRESS>Hennequin†
<EMAIL_ADDRESS>Christophe Cerisara∗
<EMAIL_ADDRESS>† Deezer Research, Paris, France
∗ LORIA, CNRS, Nancy, France

###### Abstract

Humans naturally attribute utterances of direct speech to their speaker in
literary works. When attributing quotes, we process contextual information but
also access mental representations of characters that we build and revise
throughout the narrative. Recent methods to automatically attribute such
utterances have explored simulating human logic with deterministic rules or
learning new implicit rules with neural networks when processing contextual
information. However, these systems inherently lack character representations,
which often leads to errors on more challenging examples of attribution:
anaphoric and implicit quotes. In this work, we propose to augment a popular
quotation attribution system, BookNLP, with character embeddings that encode
global information of characters. To build these embeddings, we create
DramaCV, a corpus of English drama plays from the 15th to 20th century focused
on Character Verification (CV), a task similar to Authorship Verification
(AV), that aims at analyzing fictional characters. We train a model similar to
the recently proposed AV model, Universal Authorship Representation (UAR), on
this dataset, showing that it outperforms concurrent methods of characters
embeddings on the CV task and generalizes better to literary novels. Then,
through an extensive evaluation on 22 novels, we show that combining BookNLP’s
contextual information with our proposed global character embeddings improves
the identification of speakers for anaphoric and implicit quotes, reaching
state-of-the-art performance. Code and data will be made publicly available.

Improving Quotation Attribution with Fictional Character Embeddings

##  1 Introduction

Authors of literary works employ various devices to create engaging
narratives, often combining narration with character dialogues to unveil the
plot. Fictional characters portray themselves through dialogues, revealing
aspects of their personality, their own style and ideas about themselves and
the fictional world. As part of studying characters in digital humanities,
automatically identifying utterances and attributing them to characters, also
known as quotation attribution, is central Elson et al. (2010); Muzny et al.
(2017a); Labatut and Bost (2019); Sims and Bamman (2020).

Recent approaches to quotation attribution typically succeed at attributing
explicit quotes but struggle when it comes to anaphoric and implicit quotes
Muzny et al. (2017b); Cuesta-Lazaro et al. (2022); Vishnubhotla et al. (2023);
Su et al. (2023). Explicit utterances occur when the narrator indicates the
speaker of a quote with a speech verb and a named mention, while anaphoric
quotes are introduced with a speech verb and a pronoun or common noun. When no
narrative information is given on the speaker of the quote, we refer to those
as implicit quotes. Examples of such quotes are given in Figure 1.

  1. "As soon as ever Mr. Bingley comes, my dear," said Mrs. Bennet, "you will wait on him of course."

  2. "No, no. You forced me into visiting him last year, and promised if I went to see him, he should marry one of my daughters…"

  3. His wife represented to him how absolutely necessary such an attention would be from all the neighbouring gentlemen, on his returning to Netherfield.

  4. "’Tis an etiquette I despise," said he.

Figure 1: Excerpt of Pride and Prejudice by Jane Austen (1813). Quotations are
colored by quote type: explict, implicit and anaphoric. Speaker information
given by the narrator are underlined.

Attributing anaphoric and implicit quotes usually requires fine-grained
understanding of contextual information such as linking pronouns to their
canonical entity or discerning discussion patterns. Accessing information of
fictional characters (style, persona, gender) also help disambiguate
challenging examples of attribution. Variations of writing style and long-
range contextual dependencies, ubiquitous in fictional works, add an extra
layer of complexity, making attribution of non-explicit quotes a complex task
to solve even with recent tools. Sims and Bamman (2020); Vishnubhotla et al.
(2022)

While some of these shortcomings have been addressed by improving link
prediction between pronouns/common nouns and canonical characters Vishnubhotla
et al. (2023) or by exploiting better contextual representations Cuesta-Lazaro
et al. (2022); Su et al. (2023), we instead show that combining
representations encoding global stylistic information and topical preferences
of fictional characters with contextual information improves the
identification of speakers of anaphoric and implicit quotes. In particular, we
combine these embeddings with BookNLP111https://github.com/booknlp/booknlp, a
widely used NLP pipeline designed for literary texts. As explicit utterances
can easily be attributed to their speaker, we assume those quotes given, and
use them to inject global character-level information into BookNLP.

Among methods to represent characters from the words they utter, previous
works Li et al. (2023); Aggazzotti et al. (2023); Michel et al. (2024) have
explored embeddings encoding topical preferences using SBERT Reimers and
Gurevych (2019) or stylistic information using Universal Authorship
Representations (UAR) Rivera-Soto et al. (2021), a fine-tuned variant of SBERT
trained on the Authorship Verification (AV) task with millions of Reddit
users. However, considering the discrepancies between Reddit and literary
discourse, we question the effectiveness of these embeddings to represent
characters. To overcome this potential limitation, we build a variant of UAR
tailored for literary texts and evaluate those representation models on
quotation attribution and Character Verification (CV), a task similar to AV,
aiming at distinguishing characters in literary works using their utterances
within authorial bounds.

Our proposed models significantly improve CV on drama plays and novels, and
reaches state-of-the-art quotation attribution accuracy of non-explicit quotes
when used in combination with BookNLP. Besides, we show that using models
tailored for literary text leads to better quotation attribution performance.
To sum up, our contributions are:

  * •

We construct DramaCV, a corpus of English drama plays, dating from the 15th to
20th century focused on CV, and train variants of UAR on this dataset, showing
that our proposed models significantly improves CV on DramaCV and also
generalize better to novels.

  * •

Through an extensive evaluation, we show that BookNLP can be easily extended
to improve its current performance on the Project Dialogism Novel Corpus
(PDNC) Vishnubhotla et al. (2022), and reaches state-of-the-art accuracy on
non-explicit quotes when trained in combination with global character
representations computed from their explicit quotes.

##  2 Related Work

Fictional Character Embeddings. Understanding fictional characters objectives,
personas and style often comes naturally when reading novels. When accessing
information of the fictional world and interactions between characters
provided throughout the narrative, we infer mental representations of
characters that are revised when accessing new information Gernsbacher et al.
(1992); Culpeper (1996). Previous computational works on this aspect of
fictional character understanding have explored representing fictional
characters in a latent space, often targeting a specific aspect, such as
persona, style or descriptive elements. These computational representations of
characters are often used for further analysis on large literary corpora,
trying to understand character distinctiveness between authors Bamman et al.
(2014) or narrative comprehension Brahman et al. (2021). Other works focus on
using fictional characters embedings to improve character-level tasks such as
coreference resolution and character linking Li et al. (2023) or contextual
and book-level tasks such as narrative question answering Inoue et al. (2022).

Here, we propose to represent character from what characters say and how they
say it, building literary embeddings encoding style and topical preferences of
characters, derived from models trained on Authorship Verification or
Character Verification. We define Character Verification (CV) as a task
closely related to Authorship Verification, aiming at distinguishing
characters in literary works, within authorial bounds. Most works focusing on
CV analyze drama characters because of the availability of large annotated
corpora and aim at capturing syntactic, lexical and phonological variations
that occur in their direct speech using statistical measures of
distinctiveness Dinu and Uban (2017); Vishnubhotla et al. (2019); Šeļa et al.
(2023).

With the recent introduction of PDNC Vishnubhotla et al. (2022), a corpus of
28 novels annotated with quotation attribution, analysis on fictional
characters in novels can be done at a larger scale. Michel et al. (2024)
provide a comparative study of PDNC’s characters, converting the original PDNC
quotation attribution task to a CV task and showing that UAR Rivera-Soto et
al. (2021) performs well on the task. However, this model is trained on Reddit
posts, which are inherently different from fictional character utterances,
motivating the need for an in-domain variant of UAR.

text

Quotation Attribution. Datasets of quotation attribution containing fully
annotated English novels are rather scarce, and are often limited to one or
two books. Elson and McKeown (2010) introduce the CQSC corpus and attribute
automatically extracted quotes to named entities and nominals with a
supervised mention ranking model. Instead, He et al. (2013) attribute quotes
directly to speakers with a supervised ranking system using features such as
speaker alternation patterns and character-level features He et al. (2010).
Other approaches involve sequence labeling O’Keefe et al. (2012) and Dialogue
State Tracking Cuesta-Lazaro et al. (2022). The deterministic sieve-based
model of Muzny et al. (2017b) regards quotation attribution as a two-step
process: quote-mention linking and mention-speaker linking. The NLP pipeline
dedicated to books, BookNLP, replaced the deterministic sieves with fine-tuned
language models. The link between mention and speaker is carried out by
BookNLP’s coreference resolution model, and the quote-mention linking is
predicted by a BERT model, fine-tuned on LitBank’s speaker annotations Sims
and Bamman (2020). Recently, Vishnubhotla et al. (2022) introduce the largest-
to-date corpus of quotation attribution, PDNC, and show a similar accuracy
score of around 63% for both BookNLP and the sieve-based model. However,
better results were obtained later by fine-tuning BookNLP on PDNC and
restricting the predicted coreference chains to resolved characters only
Vishnubhotla et al. (2023). SIG Su et al. (2023) adapted quotation attribution
for encoder-decoder models, showing that predicting both the speaker and the
addressees of a quote with BART reaches state-of-the-art accuracy on PDNC.

Interestingly, reported BookNLP attribution results on PDNC often approach
perfect accuracy for explicit quotes, indicating that most of the research
efforts should be put in the attribution of non-explicit quotes. Driven by
this insight, we exploit explicit quotes uttered by characters to represent
them in a latent space, and are the first to show their potential for
quotation attribution when combined with contextual representations.

##  3 Literary Character Representations

In this section, we introduce DramaCV (Drama Character Verification), a
dataset of English drama plays focused on CV. Then, we compare variants of
Universal Authorship Representation (UAR) trained on DramaCV with the original
UAR and SBERT. We finish by analyzing domain adaptation results on the CV
variant of PDNC.

| Split | Segments | Utterances | Queries | Targets/Query (avg)
---|---|---|---|---|---
| Train | 1507 | 263270 | 5392 | 5.0
Scene | Val | 240 | 50670 | 1557 | 8.8
| Test | 203 | 41830 | 1319 | 8.7
| Train | 226 | 449407 | 4109 | 90.7
Play | Val | 30 | 63934 | 917 | 55.1
| Test | 31 | 74738 | 1214 | 108.5
PDNC | Explicit | - | 6303 | 562 | 11.2
Table 1: Summary statistics of the Scene and Play instances of DramaCV by
split and the Explicit split of Michel et al. (2024) that we use to test
generalization.

Among models trained on Authorship Verification, UAR, a fine-tuned variant of
SBERT, is able to distinguish millions of Reddit authors. It learns a
representation space under which a collection of documents composed by an
author has a higher similarity with another collection from the same author
than with a collection composed by another author Rivera-Soto et al. (2021).
Although the Reddit version of UAR appears to perform well on PDNC, which is
on-par with its zero-shot domain transfer abilities Rivera-Soto et al. (2021),
we question its suitability for literary analysis and propose to train instead
a domain-specific variant of UAR tailored for literary texts. Reddit has the
advantage of spanning millions of different authors, but posts are often
written informally and exhibit strong stylistic cues such as typographical or
case errors Wegmann et al. (2022). In contrast, literary texts do not contain
such errors and often span a wider vocabulary that evolves over time, in
concordance with writing styles. This motivates the need for an in-domain UAR
for analyzing authorship of fictional characters. Besides, quotes are subject
to the writing style of authors. We would like character representations
disentangled from the author’s stylistic preferences.

###  3.1 DramaCV

Character Verification aims at answering the following problem: given two
collections of quotes that have been uttered by characters within the same
work, predict whether they were spoken by the same character or not. Although
closely linked to Authorship Attribution, we note that this framework is
different for the following reasons: 1) we do not have access to a list of
canonical characters for every literary work and 2) we compare characters
within the same story, implying that each quote is inherently influenced by
the author’s style. In comparison, Authorship Attribution aims at determining
who wrote a text among a list of predefined authors, which is also inherently
different from quotation attribution. Multiple documents come naturally
annotated with who wrote or spoke them, such as social media Baumgartner et
al. (2020) movie scripts Sang et al. (2022) or drama plays Fischer et al.
(2019). Among existing datasets, DraCor Fischer et al. (2019) contains drama
plays in various languages, but only provides Shakespeare plays for English
language. Alternatively, explicit utterances from PDNC are an additional data
source, but we wanted to evaluate the generalization of literary character
embeddings for quotation attribution to unseen novels. Hence, we instead use
the Explicit split of PDNC Michel et al. (2024) as an additional test split to
analyze generalization from drama to novels. To the best of our knowledge, no
datasets focusing on English drama and containing speaker annotation has been
publicly released.

We create DramaCV, a publicly available dataset of drama plays written by
English authors, dating from the 15th to 20th century. We extract 499 plays
from Project Gutenberg222https://gutenberg.org/ with the GutenTag software
Brooke et al. (2015) that we parsed to attribute each line to its speaker.
Then, we construct two instances of DramaCV:

Scene: We split each plays in scenes, a small segment unit of drama that is
supposed to contain actions occurring at a specific time and place with the
same characters. If a play has no <scene> tag, we instead split it in acts,
with the <act> tag. Acts are larger segment units, composed of multiple
scenes. We do not consider plays that have neither of these tags. Although
scenes usually contain a small number of characters, they represent a
challenge for CV models as participating characters are akin to discuss the
same topic. We expect a model trained on scenes to rely mostly on stylistic
cues.

Play: Here, we do not segment plays, and instead use all character lines in
the full play. Compared to the scene instance, all characters participating in
the play are considered as targets for the AV task. Here, topical preferences
of each character across the whole play can be used in addition to style.

We parsed a total of 169 plays for Scene, and 287 plays for Play. The
resulting plays are split in 80% for training, 10% for validation and 10% for
test. For validation and test, half utterances of a character are randomly
sampled for the query, and the remaining half for the target. For the train
split, we consider as queries and targets 8 utterances from the pool of a
character’s set of lines, that we randomly sample at each epoch. We provide
summary statistics for both instances of DramaCV in Table 1.

###  3.2 Experimental Setup

We train UAR from scratch using its public implementation on DramaCV. For both
instances, we use the pre-trained all-distilroberta-v1 as the base encoder,
and employ a learning rate of 2e-5 for 20 epochs. We set the dimension of the
model to $d=512$ and use a maximum sequence length of 64 tokens for all
utterances. During training, we build queries and targets with distinct
collections of 8 randomly sampled utterances to represent characters. A batch
size of 1 play is used when training on the Play split, and 8 scenes are used
when training on the Scene split. During inference, we do not restrict the
number of utterances in a collection.

A supervised contrastive objective Khosla et al. (2020) is used where in-batch
negatives considered are characters within the same segment unit. We motivate
this choice for each dataset instance:

Scene. In a scene, characters are likely to discuss similar topics. By
restricting the objective to characters within a scene, we force the model to
rely on stylistic cues to distinguish lines of characters.

Play. We want to avoid capturing the writing style of authors to distinguish
among collections of character lines. Thus, we restrict the set of in-batch
negatives to characters within the same work.

We compare our variants of UAR, $\text{UAR}_{Scene}$ and $\text{UAR}_{Play}$
against their Reddit counterpart, $\text{UAR}_{Reddit}$ and SBERT. Following
previous work Aggazzotti et al. (2023); Michel et al. (2024), we create SBERT
character embeddings by encoding each utterance in a vector with a pretrained
model333We use all-mpnet-base-v2., followed by a coordinate-wise mean of all
utterance vectors. Instead, UAR directly encodes a collection of utterances
into a $d$-dimensional vector.

For both test-splits of DramaCV, we encode a query with one of the mentioned
models, yielding a character representation, and then encode each associated
targets, which are collections of lines uttered by all characters in the same
segment unit (including other lines from the same character). We then compute
the cosine similarities between a query and the targets, and evaluate the
ranking performance of these embeddings with the Area Under the Receiver
Operating Characteristic curve (AUC) metric. We also test generalization to
novels with the Explicit split of PDNC. In this experimental setup, Michel et
al. (2024) proposed two evaluations: Character-Character (CC) and Character-
Quote (CQ). The task in CC is to answer the following problem: given a
collection of all explicit quotes of a character in a chapter, and a
collection of quotes of any type from other chapters, determine if they have
been spoken by the same character or not. The CQ experiment is constructed
with the same collections of explicit quotes, but we instead need to predict
if a single quote of any type from other chapters has been spoken by the same
character or not. The later experiment resembles quotation attribution, but no
contextual information is provided.

| DramaCV | PDNC - Exp
---|---|---
| Scenes | Plays | CC | CQ
SBERT | 72.0 (5) | 78.8 (10) | 63.8 (16) | 54.4 (5)
$\text{UAR}_{Reddit}$ | 68.6 (7) | 74.5 (11) | 81.1 (9) | 54.1 (4)
$\text{UAR}_{Scene}$ |  82.3 (6) | - | 81.3 (10) |  58.9 (4)
$\text{UAR}_{Play}$ | - |  84.2 (9) |  86.5 (8) | 58.5 (5)
Table 2: Average AUC (%) on test plays in DramaCV and novels in the Explicit
split of PDNC with standard deviations between parentheses.

###  3.3 Results

We report AUC for both test splits of DramaCV and the Explicit split of PDNC
in Table 2. Interestingly, $\text{UAR}_{Reddit}$ performs worse on both splits
compared to SBERT and in-domain UARs. $\text{UAR}_{Scene}$ increases AUC on
its training domain up to 10 points compared to SBERT, indicating that the
model is able to capture stylistic cues specific to drama that help
distinguishing utterances of characters discussing similar topics. The
increase of $\text{UAR}_{Play}$ is slightly lower, but the high AUC indicates
that it succeeds at distinguishing characters within the same play. The high
standard deviation across plays suggests that task difficulty is not
distributed uniformly among these literary works. Among best-performing plays,
we found works by canonical authors such as Shakespeare’s Hamlet, Shaw’s John
Bull’s Other Island or Ben Jonson’s The Alchemist (see complete list in
Appendix A).

Looking at generalization results to PDNC, $\text{UAR}_{Reddit}$ performs
better than SBERT, but almost equals $\text{UAR}_{Scene}$ in the CC setup. We
find significant improvement with $\text{UAR}_{Play}$ over the baselines,
suggesting that training UAR on drama allows better domain adaptation to
literary novels. We also remark that this adaptation requires less data, as
training $\text{UAR}_{Scene}$ with around 5000 drama characters is enough to
match the performance of $\text{UAR}_{Reddit}$ that has been trained with
millions of different users.

The advantages of domain adaptation can also be seen in the CQ evaluation
setup, where the two variants of UAR trained on drama show a significant
increase in AUC over the baselines. This indicates that character
representations built with our proposed UARs are more similar to embeddings of
their other quotes than representations built with $\text{UAR}_{Reddit}$ and
SBERT. However, AUC scores for the CQ experiment are low, showing that the
task of attributing quotes requires more than only comparing similarities of
character and quote embeddings.

Motivated by the ability of UAR to encode character utterances in a
representation space that contain useful, though insufficient information for
quotation attribution, we evaluate next the impact of combining these global
character embeddings with BookNLP’s contextual information.

##  4 Quotation attribution

| Non-Explicit | Explicit | Anaphoric | Implicit
---|---|---|---|---
SIG | $70.0^{+}$ | - | - | -
ChatGPT | $70.0^{+}$ | - | - | -
Unanswerable (%) | 20.9 (6.7) | 0 | 24.5 (8.1) | 19.8 (8.6)
$\text{BookNLP+}_{reimp}$ |  $68.9^{\phantom{\star\star}}$ (4.4) | 98.6 (1.2) |  $70.2^{\phantom{\star\star}}$ (7.0) |  $66.4^{\phantom{\star}}$ (5.7)
\+ SBERT |  $70.0^{\star\phantom{\star}}$ (4.5) | 98.3 (1.1) |  $70.8^{\phantom{\star\star}}$ (7.3) |  $68.1^{\star}$ (5.5)
\+ $\text{UAR}_{Reddit}$ |  $71.1^{\star\star}$ (4.7) | 98.7 (1.1) |  $71.5^{\phantom{\star\star}}$ (7.9) |  $\mathbf{69.6}^{\star}$ (5.3)
\+ $\text{UAR}_{Play}$ |  $70.9^{\star\phantom{\star}}$ (4.4) | 98.7 (1.0) |  $71.1^{\phantom{\star\star}}$ (7.9) |  $69.5^{\star}$ (5.7)
\+ $\text{UAR}_{Scene}$ |  $\mathbf{71.2}^{\star\phantom{\star}}$ (4.3) | 98.7 (1.0) |  $\mathbf{71.7}^{\star\star}$ (8.1) |  $\mathbf{69.6}^{\star}$ (5.4)
Table 3: Cross validation accuracy (%) of quotation attribution on the PDNC
dataset. Standard deviations in parentheses are calculated across splits. We
take the reported accuracy of SIG and ChatGPT from Su et al. (2023) (+).
Statistical significance against $\text{BookNLP+}_{reimp}$ from paired t-test
is denoted by ⋆ (5%) and ⋆⋆ (10%).

In this section, we first review BookNLP’s approach to quotation attribution.
BookNLP is a popular Natural Language Processing pipeline dedicated to books.
Then, we present our straightforward approach to combine BookNLP’s contextual
information with embeddings of characters computed from their explicit quotes
and finish by discussing our main results.

###  4.1 BookNLP

BookNLP’s pipeline includes among others Part-of-speech tagging, Named Entity
Recognition, Coreference Resolution and quotation attribution. All components
of the pipeline have been trained on LitBank Bamman et al. (2019, 2020); Sims
and Bamman (2020), a corpus containing 100 chapters from public domain books.
Since its quotation attribution model is not part of a publication, we will
review its details in the following paragraphs.

BookNLP carries out quotation attribution in four independents steps: 1)
tokenization and named entity recognition, 2) character name clustering, 3)
pronominal coreference resolution, and 4) quote-mention linking. The
pronominal coreference resolution step outputs mentions along with the
predicted character they refer to. Using predicted mentions in the surrounding
context of a quote, the quote-mention linking step aims at predicting which
mention is the one that refers to the true speaker of the quote. It is
achieved by a fine-tuned BERT model Devlin et al. (2019). We give further
details of the quote-mention linking step below.

Let $D=(t_{1},\dots t_{n})$ be a tokenized document and
$q=(t_{i},\dots,t_{j})$ be a character utterance in this document, starting at
token $i$ and ending at token $j$. A contextual segment is computed for each
quote, $c_{q}=[\,c_{q}^{left}\;|\,q\,|\;c_{q}^{right}\,]$, where
$c_{q}^{left}$ and $c_{q}^{right}$ are left and right contextual information
of length $w$ tokens and $[\,|\,]$ denotes concatenation. Additionally, the
full quote $q$ is replaced by a special token [QUOTE] and all other quotes
occurring in the contextual segment are replaced by the special token
[ALTQUOTE], such that the contextual segment mainly reflect narration. Each
segment $c_{q}$, containing masked quotes and narrative parts, is fed to BERT,
producing a contextual representation $\mathbf{H}_{c_{q}}$. Then an unary
compatibility score between the (masked) quote token $q$ and a mention-span
$m=(t_{m_{s}},\dots,t_{m_{e}})$ occurring in the segment is calculated as
follows:

$\displaystyle h_{m}$
$\displaystyle=\frac{1}{|m|}\sum\limits_{i=m_{s}}^{m_{e}}\mathbf{H}_{c_{q}}[i]$
(1) $\displaystyle s(q,m)$
$\displaystyle=\phi([\mathbf{H}_{c_{q}}[q]\;|\;h_{m}])$ (2)

where $\mathbf{H}_{c_{q}}[i]$ is the BERT representation of token $i$ while
$|m|$ is the length in tokens of mention $m$ and $\phi$ is a feed-forward
neural network with one hidden layer and a tanh activation. Given a set of
candidates $Y(q)=(m_{1},\dots,m_{k})$, the model is trained to optimize the
log-likelihood of mentions referring to the true speaker of $q$.

The default window size is $w=50$ tokens, and the hidden layer dimension is
$100$. During training and inference, only a maximum of $k=10$ mention
candidates are considered. In our experiments, we modify these default
parameters by increasing the narrative window size to $w=100$ tokens, and
using a relu activation and a hidden dimension of $d=512$ in the feed-forward
neural network. We also replace BERT by SpanBERT-large Joshi et al. (2020),
computing mention representations by concatenating SpanBERT embeddings of the
first and last tokens. At inference, we do not truncate the number of
candidates and instead use all candidate mentions available in the context
window.

###  4.2 Integrating Character Embeddings

Let $E(c)=(q_{i},\dots,q_{j})$ be the collection of all explicit quotes
uttered by character $c$. We embed this collection of quotes using one of the
model described in Section 3.2, yielding a vector representation $v_{c}$ for
each character. If a character has not been explicitly quoted, we set its
vector representation to zeros. Additionally, we embed each quote $q$
independently with the same model to get a vector representation $u_{q}$. We
modify Equation 2 to combine BookNLP’s contextual representation with these
character and quote representations:

$s(q,m)=\phi([\mathbf{H}_{c_{q}}[q]\;|\;h_{m}\;|\;v_{\text{entity}(m)}\;|\;u_{q}])$
(3)

where $\text{entity}(m)$ is the character predicted by the coreference
resolution step for mention $m$.

Equation 3 allows BookNLP to rely on extra-contextual information by injecting
global character information and compatibility between quote $q$ and candidate
character $c$ derived from CV models. In this work, our primary goal is to
prove the usefulness of character embeddings. We leave as future work finding
the best strategy to integrate them in systems such as BookNLP.

###  4.3 Experimental Setup

Following Su et al. (2023), we use the first version of PDNC, which contains
22 annotated novels. We focus on major and intermediate characters who are
characters that uttered at least 10 quotes in the novel. The later allows to
avoid the long tail of minor characters and to compare our results to the
current state-of-the-art, SIG and ChatGPT.

We base our implementation on BookNLP+ Vishnubhotla et al. (2023), that
restricts coreference predictions to chains that can be successfully resolved
to a character entity from the annotated PDNC character lists. We follow the
same evaluation protocol, using their provided cross-validation split at the
novel level, reporting the average accuracy over each split.

We train $\text{BookNLP+}_{reimp}$ with the modifications mentioned in Section
4.1, and add character embeddings encoded by different models using Equation
3. We use AdamW optimizer with a learning rate of 5e-6 and train each model
for 20 epochs.

###  4.4 Results

Our main results are displayed in Table 3. Our implementation of BookNLP+
reaches 68.9% accuracy on non-explicit quotes, compared to the 53% reported in
Vishnubhotla et al. (2023). Combining BookNLP+ with SBERT character embeddings
matches the accuracy of SIG and ChatGPT on non-explicit quotes. The gains are
larger for all variants of UAR, reaching 71.2% accuracy on non-explicit quotes
with our proposed $\text{UAR}_{Scene}$. As expected, we found that these gains
are largely due to the improved attribution performance of implicit quotes,
increasing the accuracy by more than 3 points with all UAR models.

|  $\Delta$ Non-Exp |  $\Delta$ Ana |  $\Delta$ Imp
---|---|---|---
SBERT | $-0.0\phantom{{}^{\star\star}}$ | $-0.0\phantom{{}^{\star}}$ | $-0.1\phantom{{}^{\star}}$
$\text{UAR}_{Reddit}$ | $-0.1\phantom{{}^{\star\star}}$ | $-0.0\phantom{{}^{\star}}$ | $-0.2\phantom{{}^{\star}}$
$\text{UAR}_{Play}$ | $-0.2^{\star\star}$ | $-0.1\phantom{{}^{\star}}$ | $-0.3^{\star}$
$\text{UAR}_{Scene}$ | $+0.1\phantom{{}^{\star\star}}$ | $+0.1\phantom{{}^{\star}}$ | $+0.1\phantom{{}^{\star}}$
Table 4: Variation of accuracy when using predicted speaker of explicit
quotes. Statistical significance from paired t-test is indicated with
${}^{\star}=5\%$, ${}^{\star\star}=10\%$

Interestingly, although our proposed variants of UAR tailored for literary
texts are better in the CV task, we only see a small – though statistically
significant – increase between $\text{UAR}_{Scene}$ and its Reddit counterpart
on non-explicit attribution accuracy. The reasons could be twofold. First,
BookNLP’s system only focuses on finding the right narrative information that
helps understanding the speaker of a quote. However, our embeddings mostly
reveal stylistic cues of characters, which are not encoded by BookNLP.
Therefore, our implementation provide external character information that
might not align perfectly with BookNLP’s contextual decisions, limiting their
combination potential. Second, we add character information based on noisy
coreference chains, which might hinder the model’s peformance. We believe
further research should be conducted in finding better combinations between
character embeddings and BookNLP.

The high standard deviations across novels reveals that the task is harder for
some novels than others. Looking at the percentage of unanswerable quotes
(quotes that do not have a candidate mention referring to their speaker in
context) provides a reason for such high variations: it ranges from 0 to 56%
per novel. Figure 2 shows the attribution accuracy on non-explicit quotes when
only considering novels that have a low percentage of unanswerable quotes.
Only 4 novels have a percentage lower than 5%, but 17 out of the 22 have a
percentage lower than 20%. As expected, we see higher performance for all
models, and the gain from UAR character embeddings is more clear.

| Non-Exp | Ana | Imp
---|---|---|---
Unanswerable (%) | 20 | 21 | 18
$\text{BookNLP+}_{reimp}$ | $69.6^{\phantom{\star\star}}$ | $72.3^{\phantom{\star\star}}$ | $67.6^{\phantom{\star}}$
\+ SBERT | $69.3^{\phantom{\star\star}}$ | $71.6^{\phantom{\star\star}}$ | $68.0^{\phantom{\star}}$
\+ $\text{UAR}_{Reddit}$ | $70.5^{\star\star}$ | $72.4^{\phantom{\star\star}}$ | $69.4^{\star}$
\+ $\text{UAR}_{Play}$ | $71.1^{\star\phantom{\star}}$ | $72.8^{\star\star}$ | $\mathbf{70.1}^{\star}$
\+ $\text{UAR}_{Scene}$ | $\mathbf{71.2}^{\star\phantom{\star}}$ | $\mathbf{73.1}^{\phantom{\star\star}}$ | $70.0^{\star}$
Table 5: Average quotation attribution accuracy (%) over the 6 additional PDNC
novels. Statistical significance from paired t-test against
$\text{BookNLP+}_{reimp}$ is denoted by ⋆ (5%) and ⋆⋆ (10%).

Let us note that $\text{UAR}_{Scene}$ is consistently better than
$\text{UAR}_{Play}$, though the latter is better at distinguishing PDNC
characters in the CC experiment. Restricting UAR to encode stylistic
information by using collections of quotes with similar topic seems to
generalize better to quotation attribution.

To construct character embeddings, we relied on gold-labeled attributions of
explicit quotes. We report in Table 4 the accuracy variation when replacing
the gold attributions with speaker predictions from $\text{BookNLP+}_{reimp}$.
Only $\text{UAR}_{Play}$ shows a statistically significant decrease of
accuracy, although this variation remains very small. This validates our
intuition that explicit quotes can be extracted in a first step and that the
eventual noise contained in the resulting character embeddings does not harm
their performance for quotation attribution.

Finally, we present quotation attribution results on 6 annotated novels from
the second release of PDNC in Table 5. We use the model trained on the first
cross-validation split to predict attributions on each of these novels. The
impact of using literary character embeddings is more clear: both of our
proposed variants of UAR show increased attribution accuracy against their
Reddit counterpart.

Figure 2: Accuracy results on non-explicit quotes averaged over subsets of
novels that have a low percentage of unanswerable quotes, with standard
deviations.

##  5 Conclusion

In this work, we evaluated fictional character embeddings constructed with the
explicit quotes they utter, showing that they improve quotation attribution
accuracy of non-explicit quotes on PDNC when used in combination with
BookNLP’s contextual representations. We created DramaCV, a corpus of English
drama plays focused on Character Verification and trained two variants of UAR
on this dataset. Our proposed models show better performance when verifying
fictional characters on both DramaCV and PDNC. Character embeddings derived
from these models are further used for quotation attribution on PDNC, reaching
state-of-the-art accuracy when combined with BookNLP+. The impact of using
literary character embeddings against embeddings derived from a model trained
on Reddit is small, though statistically significant, suggesting superiority
of in-domain character representations for quotation attribution.

We proposed a straightforward way to combine fictional character embeddings
with contextual representations, using BookNLP as an example. Our approach can
be adapted to other models: cross-attention over character embeddings could be
used for encoder-decoder models, while modality alignment Zhang et al. (2023)
is a solution for decoder-only models. We leave potential improvements to the
combination of fictional character embeddings and contextual representations
to future work.

We found that BookNLP’s attribution accuracy is not distributed uniformly
across novels, mainly caused by noisy coreference resolution resulting in many
quotes not having candidate mentions that refer to their true speaker. We
believe that including our character embeddings in the coreference resolution
model as in Li et al. (2023) might improve its performance and thus increase
overall attribution accuracy. Another potential solution could include
unmasking the closest quotes in the vicinity of a target quote that are likely
to contain addressee information. We leave better uses of fictional character
embeddings with BookNLP for future exploration.

##  6 Limitations

In this work, we focused on integrating fictional character embeddings in
encoder-only models. We have not tested with other types of models such as
encoder-decoder and decoder only, but we believe that integrating our
embeddings in those framework is possible and is likely to improve their
performance on the quotation attribution task. Our first set of experiments
corroborate this assumption, on the usefulness of our proposed character
embeddings in distinguishing fictional characters.

We also proposed models that can be used to create character representations.
These models are trained on a corpus of drama plays, which spans almost 400
years of works. It is unclear how the variation of writing style that have
occurred throughout these centuries affects the information contained in these
embeddings, and how they can generalize to new literary styles. This remark
also extends to our results on quotation attribution. We presented results on
a corpus containing novels from the 19th to early 20th century ; which is
closer in terms of writing style to DramaCV than newer novels published in the
21th century. It is thus unclear how our results generalize to these newer
novels, but we strongly believe that accurate character embeddings improve
quotation attribution regardless of the writing style of the author.

We only focused on representations encoding stylistic cues and topical
preferences and showed that they improve quotation attribution. However, this
type of character embeddings might be incomplete and other aspects of
characters can be used in addition to our proposed embeddings (personas,
gender, objective, or social status). This might allow to build accurate
representations of characters, that are likely to improve even more quotation
attribution accuracy.

## References

  * Aggazzotti et al. (2023) Cristina Aggazzotti, Nicholas Andrews, and Elizabeth Allyn Smith. 2023.  Can authorship attribution models distinguish speakers in speech transcripts? _Preprint_ , arXiv:2311.07564.
  * Bamman et al. (2020) David Bamman, Olivia Lewke, and Anya Mansoor. 2020.  An annotated dataset of coreference in English literature.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 44–54, Marseille, France. European Language Resources Association.
  * Bamman et al. (2019) David Bamman, Sejal Popat, and Sheng Shen. 2019.  An annotated dataset of literary entities.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2138–2144, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Bamman et al. (2014) David Bamman, Ted Underwood, and Noah A. Smith. 2014.  A Bayesian mixed effects model of literary character.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 370–379, Baltimore, Maryland. Association for Computational Linguistics.
  * Baumgartner et al. (2020) Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. 2020.  The pushshift reddit dataset.  _Proceedings of the International AAAI Conference on Web and Social Media_ , 14(1):830–839.
  * Brahman et al. (2021) Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021.  “let your characters tell their story”: A dataset for character-centric narrative understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1734–1752, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Brooke et al. (2015) Julian Brooke, Adam Hammond, and Graeme Hirst. 2015.  GutenTag: an NLP-driven tool for digital humanities research in the Project Gutenberg corpus.  In _Proceedings of the Fourth Workshop on Computational Linguistics for Literature_ , pages 42–47, Denver, Colorado, USA. Association for Computational Linguistics.
  * Cuesta-Lazaro et al. (2022) Carolina Cuesta-Lazaro, Animesh Prasad, and Trevor Wood. 2022.  What does the sea say to the shore? a BERT based DST style approach for speaker to dialogue attribution in novels.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5820–5829, Dublin, Ireland. Association for Computational Linguistics.
  * Culpeper (1996) Jonathan Culpeper. 1996.  Inferring character from texts: Attribution theory and foregrounding theory.  _Poetics_ , 23(5):335–361.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dinu and Uban (2017) Liviu P. Dinu and Ana Sabina Uban. 2017.  Finding a character’s voice: Stylome classification on literary characters.  In _Proceedings of the Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature_ , pages 78–82, Vancouver, Canada. Association for Computational Linguistics.
  * Elson et al. (2010) David Elson, Nicholas Dames, and Kathleen McKeown. 2010.  Extracting social networks from literary fiction.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 138–147, Uppsala, Sweden. Association for Computational Linguistics.
  * Elson and McKeown (2010) David Elson and Kathleen McKeown. 2010.  Automatic attribution of quoted speech in literary narrative.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 24(1):1013–1019.
  * Fischer et al. (2019) Frank Fischer, Ingo Börner, Mathias Göbel, Angelika Hechtl, Christopher Kittel, Carsten Milling, and Peer Trilcke. 2019.  Programmable Corpora: Introducing DraCor, an Infrastructure for the Research on European Drama.  In _Proceedings of DH2019: "Complexities", Utrecht, July 9–12, 2019_. Utrecht University.
  * Gernsbacher et al. (1992) Morton Ann Gernsbacher, H Hill Goldsmith, and Rachel RW Robertson. 1992.  Do readers mentally represent characters’ emotional states?  _Cognition & Emotion_, 6(2):89–111.
  * He et al. (2013) Hua He, Denilson Barbosa, and Grzegorz Kondrak. 2013.  Identification of speakers in novels.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1312–1320, Sofia, Bulgaria. Association for Computational Linguistics.
  * He et al. (2010) Hua He, Greg Kondrak, and Denilson Barbosa. 2010.  The actor-topic model for extracting social networks in literary narrative.
  * Inoue et al. (2022) Naoya Inoue, Charuta Pethe, Allen Kim, and Steven Skiena. 2022.  Learning and evaluating character representations in novels.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1008–1019, Dublin, Ireland. Association for Computational Linguistics.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Khosla et al. (2020) Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020.  Supervised contrastive learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 18661–18673. Curran Associates, Inc.
  * Labatut and Bost (2019) Vincent Labatut and Xavier Bost. 2019.  Extraction and analysis of fictional character networks: A survey.  _ACM Computing Surveys_ , 52(5):1–40.
  * Li et al. (2023) Dawei Li, Hengyuan Zhang, Yanran Li, and Shiping Yang. 2023.  Multi-level contrastive learning for script-based character understanding.  _Preprint_ , arXiv:2310.13231.
  * Michel et al. (2024) Gaspard Michel, Elena Epure, Romain Hennequin, and Christophe Cerisara. 2024.  Distinguishing fictional voices: a study of authorship verification models for quotation attribution.  In _Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024)_ , pages 160–171, St. Julians, Malta. Association for Computational Linguistics.
  * Muzny et al. (2017a) Grace Muzny, Mark Algee-Hewitt, and Dan Jurafsky. 2017a.  Dialogism in the novel: A computational model of the dialogic nature of narration and quotations.  _Digital Scholarship in the Humanities_ , 32:ii31–ii52.
  * Muzny et al. (2017b) Grace Muzny, Michael Fang, Angel Chang, and Dan Jurafsky. 2017b.  A two-stage sieve approach for quote attribution.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 460–470, Valencia, Spain. Association for Computational Linguistics.
  * O’Keefe et al. (2012) Timothy O’Keefe, Silvia Pareti, James R. Curran, Irena Koprinska, and Matthew Honnibal. 2012.  A sequence labelling approach to quote attribution.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 790–799, Jeju Island, Korea. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rivera-Soto et al. (2021) Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021.  Learning universal authorship representations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 913–919, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Sang et al. (2022) Yisi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022.  TVShowGuess: Character comprehension in stories as speaker guessing.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4267–4287, Seattle, United States. Association for Computational Linguistics.
  * Sims and Bamman (2020) Matthew Sims and David Bamman. 2020.  Measuring information propagation in literary social networks.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 642–652, Online. Association for Computational Linguistics.
  * Su et al. (2023) Zhenlin Su, Liyan Xu, Jin Xu, Jiangnan Li, and Mingdu Huangfu. 2023.  Sig: Speaker identification in literature via prompt-based generation.  _Preprint_ , arXiv:2312.14590.
  * Vishnubhotla et al. (2019) Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hirst. 2019.  Are fictional voices distinguishable? classifying character voices in modern drama.  In _Proceedings of the 3rd Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature_ , pages 29–34, Minneapolis, USA. Association for Computational Linguistics.
  * Vishnubhotla et al. (2022) Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hirst. 2022.  The project dialogism novel corpus: A dataset for quotation attribution in literary texts.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 5838–5848, Marseille, France. European Language Resources Association.
  * Vishnubhotla et al. (2023) Krishnapriya Vishnubhotla, Frank Rudzicz, Graeme Hirst, and Adam Hammond. 2023.  Improving automatic quotation attribution in literary novels.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 737–746, Toronto, Canada. Association for Computational Linguistics.
  * Wegmann et al. (2022) Anna Wegmann, Marijn Schraagen, and Dong Nguyen. 2022.  Same author or just same topic? towards content-independent style representations.  In _Proceedings of the 7th Workshop on Representation Learning for NLP_ , pages 249–268, Dublin, Ireland. Association for Computational Linguistics.
  * Zhang et al. (2023) Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023.  SpeechGPT: Empowering large language models with intrinsic cross-modal conversational abilities.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15757–15773, Singapore. Association for Computational Linguistics.
  * Šeļa et al. (2023) Artjoms Šeļa, Ben Nagy, Joanna Byszuk, Laura Hernández-Lorenzo, Botond Szemes, and Maciej Eder. 2023.  From stage to page: language independent bootstrap measures of distinctiveness in fictional speech.  _Preprint_ , arXiv:2301.05659.

##  Appendix A Performance on all Test Plays.

We report performance of $\text{UAR}_{Scene}$ and $\text{UAR}_{Play}$ on their
respective test split in Table 6 and Table 7.

##  Appendix B Computing Information

We used a 32-core Intel Xeon Gold 6244 CPU @ 3.60GHz CPU with 128GB RAM
equipped with 3 RTX A5000 GPUs with 24GB RAM each and a single Nvidia A40 with
45GB RAM. We trained $\text{UAR}_{Scene}$ and $\text{UAR}_{Play}$ with the
Nvidia A40, requiring gradient-checkpointing to fit everything in the GPU RAM.
The training took around 1 hour for both models. We use a single RTX A5000 for
training $\text{BookNLP+}_{reimp}$ and its variant with character embeddings.
Training on a single cross-validation split took around 6 hours for each of
the proposed methods. Inference times are shorter, where quotation attribution
on a full novel can be done in a few minutes only.

Play | Author | AUC
---|---|---
Justice | John Galsworthy | 95.6
The Alchemist | Ben Jonson | 91.8
Hamlet | William Shakespeare | 86.5
Plays : Third Series | John Galsworthy | 86.3
The works of John Dryden | John Dryden | 85.2
The Life of Timon of Athens | William Shakespeare | 84.2
King Henry the Fifth | William Shakespeare | 83.7
Night Must Fall : a Play in Three Acts | Emlyn Williams | 83.7
The Spanish Tragedy | Thomas Kyd | 83.2
The works of John Dryden | John Dryden | 83.0
The Scarlet Stigma: A Drama in Four Acts | Nathaniel Hawthorne | 82.4
King John | William Shakespeare | 81.8
The Works of Charles and Mary Lamb | Mary Lamb | 80.8
The Scornful Lady | John Fletcher | 80.8
Married Life A Comedy, in Three Acts | John Baldwin Buckstone | 80.0
The Flutter of the Goldleaf; and Other Plays | Frederick Peterson | 77.5
Single Life A Comedy, in Three Acts | John Baldwin Buckstone | 75.9
The Countess Cathleen | W. B. (William Butler) Yeats | 75.6
De Turkey and De Law: A Comedy in Three Acts | Zora Neale Hurston | 66.4
Table 6: AUC per novel on the Scene test split of DramaCV. Play | Author | AUC
---|---|---
Bride Roses | William Dean Howells | 100.0
Fanny’s First Play | Bernard Shaw | 100.0
Five Little Plays | Alfred Sutro | 98.6
John Bull’s Other Island | Bernard Shaw | 95.4
Mrs. Dot: A Farce | William Somerset Maugham | 94.5
The Earl of Essex: A Tragedy, in Five Acts | Henry Jones | 94.4
The Way of the World | William Congreve | 92.7
Plays : Fifth Series | John Galsworthy | 92.5
Representative Plays by American Dramatists: 1856-1911: The New York Idea | Langdon Elwyn Mitchell | 91.9
Plays: Lady Frederick, The Explorer, A Man of Honour | William Somerset Maugham | 90.3
Great Catherine | Bernard Shaw | 89.3
Hypolympia Or, The Gods in the Island, an Ironic Fantasy | Edmund Gosse | 87.1
The works of John Dryden | John Dryden | 87.0
The Grecian Daughter | Arthur Murphy | 85.5
Dramatic Technique | George Pierce Baker | 81.6
Lyre and Lancet: A Story in Scenes | F. Anstey | 80.2
Beaumont and Fletcher’s Works, Vol. 5 | Francis Beaumont | 79.0
King Henry VI, Part 3 | William Shakespeare | 78.1
Plays and Lyrics | Cale Young Rice | 78.0
The Easiest Way Representative Plays by American Dramatists: 1856-1911 | Eugene Walter | 77.6
Charles Di Tocca: A Tragedy | Cale Young Rice | 77.3
St. Patrick’s day, or, the scheming lieutenant : a farce in one act | Richard Brinsley Sheridan | 76.7
Deirdre of the Sorrows | John Millington Synge | 76.3
Ambrose Gwinett or, a sea-side story : a melo-drama, in three acts | Douglas William Jerrold | 75.8
Henry V | William Shakespeare | 75.3
Box and Cox: A Romance of Real Life in One Act. | John Maddison Morton | 75.0
The Recruiting Officer | George Farquhar | 74.7
Yolanda of Cyprus | Cale Young Rice | 74.3
Table 7: AUC per novel on the Play test split of DramaCV.