# Quote Erat Demonstrandum: A Web Interface for Exploring the Quotebank Corpus

Vuk Vuković  EPFL<EMAIL_ADDRESS>,  Akhil Arora  EPFL<EMAIL_ADDRESS>,  Huan-Cheng Chang  EPFL<EMAIL_ADDRESS>,  Andreas Spitz
University of Konstanz<EMAIL_ADDRESS>and  Robert West  EPFL
<EMAIL_ADDRESS>
(2022)

###### Abstract.

The use of attributed quotes is the most direct and least filtered pathway of
information propagation in news. Consequently, quotes play a central role in
the conception, reception, and analysis of news stories. Since quotes provide
a more direct window into a speaker’s mind than regular reporting, they are a
valuable resource for journalists and researchers alike. While substantial
research efforts have been devoted to methods for the automated extraction of
quotes from news and their attribution to speakers, few comprehensive corpora
of attributed quotes from contemporary sources are available to the public.
Here, we present an adaptive web interface for searching Quotebank, a massive
collection of quotes from the news, which we make available at
https://quotebank.dlab.tools.

Quote, Quote corpus, News, Search, Adaptive Interface

††journalyear: 2022††copyright: acmlicensed††conference: arXiv e-print;
††booktitle: ;††doi: 10.1145/3477495.3531696††ccs: Information systems Web
interfaces††ccs: Information systems Web searching and information
discovery††ccs: Information systems Web mining††ccs: Information systems Users
and interactive retrieval

##  1\. Introduction

Quotes of sources, politicians, athletes, or scientists play an important role
in lending credibility to news articles (4). As news stories evolve, quotes
are useful data for analyzing the spread of information through the news (11),
for determining the source of news information (18), or in fact checking and
credibility assessment (17; 3). Outside of journalistic applications,
extracted quotes from the news are also valuable in social studies, for
example through opinion mining from quotes (2).

The substance of quotes lies not just in _what_ is being said, but _by whom_
the quote is uttered since the attribution to a speaker provides context to
the words. As a result, the automated extraction and attribution of quotes
from document corpora has been the subject of ongoing research over the years
(15; 16; 9; 12). However, while methods for the extraction and attribution of
quotes are developed continuously and benefit from recent advances in natural
language processing, few of the resulting resources are available to end-
users. To address this need, we report on the development of a user interface
for searching Quotebank (19), a massive corpus of quotes that we extracted
from a decade of English news.

Contributions. We provide an interface for (faceted) search in Quotebank, a
Web-scale database of quotes from a decade of English news articles. Our tool
is accessible as a website that is geared towards end-users who would
otherwise be unable to use this resource, and it provides near-realtime query
performance that enables an interactive exploration of the Quotebank corpus.

##  2\. Related Work

Prior work that is related to Quotebank is quite sparse and can be grouped
into two categories: _corpora of quotes_ and _system demonstrations_ that make
use of or recommend quotes.

Quote corpora. Available corpora include the PolNeAR corpus (10), the Penn
Attribution Relation Corpus (14), the Speech, Thought, and Writing
Presentation corpus (6), the Rich Quotation Annotations corpus (13), and the
DirectQuote dataset (21). In contrast to Quotebank, all of the above corpora
are available only as NLP resources, not as searchable repositories that can
be used by laypeople. Furthermore, they are substantially smaller in
comparison to Quotebank, which exceeds the amount of contained quotes in these
corpora by several orders of magnitude.

System demonstrations. To the best of our knowledge, there are no other search
interfaces for large-scale corpora of quotes from the news domain.
Repositories of quotes that can be accessed and searched by laypeople remain
limited to manually curated repositories of mostly historical quotes, such as
WikiQuote.111https://www.wikiquote.org/ System demonstrations in the
information retrieval community tend to only consider quotes indirectly and
often in the wider contexts of credibility assessment, such as CredEye (17),
or fact checking for news such as BeLink (3) and the work of Miranda et al.
(8).

Figure 1. Schematic overview of the Quotebank system architecture. During pre-
processing, speaker candidates and quotes are annotated in the news article
corpus, and quotes are attributed to speakers with Quobert (19). The
attributed quote data is augmented by linking speakers to Wikidata and
subsequently loaded into Elasticsearch, which is used for indexing and
retrieval. Kibana and Logstash are used internally for monitoring and managing
indices. The web server exposes the query API to handle user queries and
directly interfaces with the cache to reduce the impact of complex queries on
system load. The JavaScript UI translates user queries to request to the web
server (for details on the UI, see Figure 2).

The likely most closely related approach to ours is by MacLaughlin et al., who
propose a system for quote recommendation based on context by using a BERT
architecture (7). However, the employed QUOTUS data set (11) is relatively
small and the tool is neither available nor suitable for the exploration of a
Web-scale news corpus that would be useful to an end-user.

##  3\. Quote Corpus

Quotebank is a dataset of 235 million unique, speaker-attributed quotes that
were extracted from 196 million English news articles (127 million of them
containing quotes) published between September 2008 and April 2020 (19). The
data was extracted with the BERT-based architecture Quobert, which utilizes
Quootstrap (16) for the extraction of training data. We make two stages of the
data available in our search interface: _article-level_ and _quote-level_.

Article-level data. In the article-level version of the data, articles are the
central unit. The data contains individual quote annotations for all articles
in the news data set, each attributed with a ranked list of the most likely
speaker candidates in the article (or _no speaker_ if no suitable candidate
could be attributed by Quobert). Additionally, the data contains context
windows surrounding each of the quote mentions in the news article text.

Quote-level data. The quote-level data contains an aggregated view of the
quotes across all articles. To generate this stage of the data, all individual
occurrences of quotes are first canonicalized and quotes with matching
canonical form are then aggregated into a single data point. Speaker
candidates for these quotes are merged by weighted consensus, i.e., by summing
over the local candidate probabilities in all individual occurrences to derive
the most likely global speaker for a given quote.

Speaker data. To improve the querying capability of Quotebank and support
faceted search, we further enrich the quote corpus with speaker information
extracted from Wikidata (20), which is one of the largest publicly available
knowledge graphs (about 97M entities). Specifically, we extract and add data
concerning the occupation, nationality, and gender of the speakers in
Quotebank.

A full JSON dump of the raw data prior to the addition of speaker data from
Wikidata is also available for
download.222https://doi.org/10.5281/zenodo.4277311 For details on the
generation, we refer the reader to Vaucher et al. (19).

##  4\. The Quotebank System

The complete architecture of the Quotebank system is shown in Figure 1. It
consists of three core components: (1) a _database system_ housing the core
storage and querying capabilities; (2) a _web server_ providing a layer of
abstraction and API on top of the the database system; and (3) a _user
interface_ responsible for delivering the results to the user through an
interactive and flexible visualization. In the following, we provide a
description of these components and highlight their key functionality.

###  4.1. Database system

The Quotebank database is built using Elasticsearch (5), which is one of the
most popular distributed, scalable, and open-source search and analytics
engines supporting full-text indexing and querying, thereby serving our
primary goal of efficiently searching through terabytes of quotes and news
content.

Our database consists of three indices: (1) article, (2) quote, and (3)
speaker, which store and index the article-level, quote-level, and speaker
data, respectively. Naïvely indexing the Quotebank corpus using Elasticsearch
would require more than $2$TB of disk space, while the storage footprint of
our optimized database is $4$ times smaller and consumes only about $500$GB.
We employ the following fundamental tried-and-true design decisions and
optimizations to reduce the storage footprint and improve the query
efficiency.

  * •

Database normalization. We follow standard principles such as removing data
redundancy, unless redundancy entails significant query speed-ups.

  * •

Choice of data types. We use data types with minimal storage requirements,
such as the `integer` type for fields supporting range queries (e.g., number
of occurrences of a quote), or `keyword` type for fields used for creating
filters (e.g., speaker nationality).

  * •

Querying-indexing trade-off. We push the complexity to the indexing phase by
aggregating all text type fields into a single field, which results in faster
querying (at the cost of slower indexing) in comparison to searching multiple
fields at query time.

(a)

(b)

Figure 2. The Quotebank user interface. (a) The search panel supports quote-
level and the article-level searches, which can be faceted by speaker
attributes. (b) The search engine result page (SERP) displays retrieved quotes
in the selected time window alongside speaker candidates and the URLs of
articles from which the quotes are sourced. A histogram shows the distribution
of quotes in the result over time.

The database system also houses _Kibana_ and _Logstash_ , which are primarily
used for internal monitoring purposes. While Kibana is used for monitoring the
status of indices and analyzing database performance statistics, Logstash
handles storage management of the logs produced by Elasticsearch.

###  4.2. Web server

The web server provides a layer of abstraction on top of the database system
and exposes the API endpoints that enable the communication between the user-
interface and the database. It is responsible for composing, validating, and
submitting user queries to the database as well as parsing and returning the
retrieved result from the database to the user. In a nutshell, the web server
prevents the users from communicating directly and in an unrestrained manner
with the database, thereby providing an added level of security.

###  4.3. User interface

The user interface (cf. Figure 2) is an adaptive web-based search platform
built with React.js,333https://reactjs.org/ which allows users to
interactively explore the Quotebank corpus. It consists of two main views: (1)
the search panel, and (2) the search engine result page.

Search panel. Figure 2a portrays the interface that is exposed to the users
for querying the Quotebank corpus. In the following, we describe the key
features of the search panel.

#### Corpus type

While the system supports queries on the quote-level data (called quotation-
centric in Figure 2a) by default, it also provides the user with the option to
query the article-level data.

#### Query type

The most straight-forward way to search the Quotebank corpus is via _text
query_ , which supports _fuzzy matching_ by default. However, the user may
also choose to utilize _exact matching_ of the query text by enabling the
checkbox option `’Enable exact` `match’`. For the article-level data, which
supports only _text queries_ , users are able to query both the quotes and the
context in which they occur in the original articles (`’Search with context’`
checkbox). To enable search faceting, the interface supports a multitude of
search filters for querying quote-level data (cf. Figure 2a), such as speaker
name or nationality, minimum number of quote occurrences, etc., thereby
providing additional refinements over and above the text query. By default,
the time window is set to the full date range of the Quotebank corpus
corresponding to September 1, 2008 and April 17, 2020, but the user may adapt
this window by using the `From` and `To` fields, respectively. All other
filter-related fields are not initialized with default values.

#### Auto-complete

To assist users in applying search filters and increase search precision,
auto-complete is implemented for each speaker-related field. Similar to most
auto-complete implementations, users are only required to input the initial
characters of a speaker’s name in the corresponding field and may choose from
the provided suggestions. The auto-complete function also provides a short
description of the speaker (extracted from Wikidata), which is displayed
alongside each suggestion to help users disambiguate and choose from the list
of potential suggestions.

Search engine result page (SERP). Figure 2b portrays the SERP, which consists
of three primary components: a summary section, a histogram of quote
occurrences, and the main result blocks.

#### Summary section

This section displays basic information about the retrieved result, such as
the number of quotes or articles that were deemed relevant to the query, and
the query time in seconds. To reduce the load on the client, the total number
of results returned from the web server is capped at 1000, corresponding to
the $1000$ most-relevant results based on their content matching score with
the user query. Additionally, this section provides a `’Share’` button, which
generates a shareable _permalink_ to the user’s query and copies it to the
clipboard. A `’Save results’` button enables the user to download the results
either as a _JSON_ or text file.

#### Histogram

The histogram, implemented in D3.js,444https://d3js.org displays the
distribution of quotes or articles that match the user query in the specified
time window. The histogram bin boundaries are automatically adjusted to
improve its aesthetics while simultaneously keeping the resource utilization
on the client side low. The main purpose of the histogram is to allow the user
to effectively visualize the trend portrayed by results relevant to her query.
While the returned results are capped at $1000$, note that the histogram is
based on all quotes or articles that match the user’s query. Thus, the counts
displayed in the histogram may exceed the number of returned documents.

#### Result block

The returned results are displayed in blocks, where each block contains either
a single relevant quote or a relevant article that contains at least one
matching quote. The font and layout of the result block is designed to
simulate a real quote as one might encounter it in a newspaper. The URLs in
each block are clickable links to the news articles in which the quote was
originally published. For quote blocks, all possible speakers of a quote are
annotated with their unique Wikidata identifiers, which are also clickable
links pointing to the Wikidata page of the speaker. Long result blocks are
collapsed to a fixed size, allowing the user to effectively browse the result
summary and only expand specific blocks if desired. In the same vein, we also
implement pagination by breaking down the result blocks into pages of at most
$10$ blocks each. The buttons `Prev` and `Next` can be used to navigate the
pages. Lastly, we also enable the user to re-rank the returned results based
on several pre-defined sorting criteria.

##  5\. Demonstration

As part of the demonstration, we encourage the reader to engage with three
visual scenarios crafted to highlight key features of the interface, namely
the exploration of (1) quote-level and (2) article-level data, and (3) a free-
roam exploration on a device of the user’s choice (e.g., tablet, phone, or
laptop). These scenarios were designed with support from an EPFL journalist,
who also helped in improving our system’s usability.

###  5.1. Quote-level exploration

In this scenario, the users may explore the Quotebank corpus to identify all
the quotes from _Donald Trump_ containing the text “great again” and that
appeared in at least $500$ distinct articles. They are further encouraged to
re-rank the results in reverse chronological order and share the results with
their colleagues by sending the _permalink_ via e-mail or messenger service.

User interaction. This scenario provides a demonstration of the `’Quotation-
centric’` (cf. Figure 2a) search panel. In addition to simply posing the text
query “great again”, the user has the option of applying multiple filters.
Specifically, the user should set the `’Minimum number of occurrences’` to
$500$, and leverage the _auto-complete_ functionality to set the input for
`’Name of speaker’` to “Donald Trump”. Lastly, the user may select the pre-
defined filter `’Date (descending)’` to re-rank the results in reverse
chronological order. A permalink of the query is obtained by clicking on the
`’Share’` button on the results page. The retrieved results for this scenario
can be accessed at this quote-level permalink.

###  5.2. Article-level exploration

In this scenario, the user is encouraged to explore the Quotebank corpus to
identify all articles published on May 19, 2018 that contain the text “gdpr”
in either the quotes or in their surrounding context. The user should also
download the results as a text file and view its contents in a text editor.

User interaction. This scenario provides a demonstration of the `’Article-
centric’` (cf. Figure 2a) search panel. In addition to posing the text query
“gdpr”, the user may vary the time window by restricting it to a particular
day, i.e., May 19, 2018. Moreover, the user analyzes the differences in the
retrieved results by toggling the `’Search with context’` checkbox. Lastly, to
facilitate later exploration and analyses, the user may download the results
corresponding to her query as a text file by clicking on the `’Save results’`
button on the results page. The retrieved results for this scenario can also
be accessed at this article-level permalink.

###  5.3. Free-roam exploration

While we provide a few example use-cases (described below) to bootstrap the
exploration in this scenario, the reader is encouraged to conduct a free-roam
exploration of Quotebank.

  * •

UC 1: Using Quotebank on their phone, users search quotes from female tennis
players of Switzerland that appear in at least 100 different articles and
share the results via messenger.

  * •

UC 2: Using Quotebank on their tablet, users search quotes related to
“science” by female journalists that appear in at least 100 different articles
and share the results via e-mail.

  * •

UC 3: Using `’Enable exact match’` to search for a quote that the user
remembers. For instance, the famous quote: “You have to dream before your
dreams can come true” to recall the name of the speaker or identify its
popularity over the past decade.

##  6\. Conclusion

In this paper, we introduced a search interface that makes the Quotebank
dataset more accessible to end users who lack the computational background to
work directly with the raw data. To support faceted search based on speaker
attributes, we further enriched the quote corpus with information from
Wikidata. Our intention is to enable the general public to explore the
Quotebank data, and we are looking forward to seeing the findings and insights
that are gained in the exploration of the data by journalists, social
scientist, and laypeople.

Future work. In the future, we plan on analyzing the search logs and
investigate users’ search patterns to determine exploration strategies and
use-cases that can aid us in further refining and augmenting the Quotebank
corpus. We are also working on disambiguation techniques for speaker
candidates during quote attribution and are preparing a stand-alone end-to-end
pipeline for attributed quote extraction.

Acknowledgments. We would like to thank Tanya Petersen for testing the
interface and sharing her expert-user insights. We are also grateful to the
members of EPFL DLAB and the students enrolled in the 2021 edition of the
Applied Data Analysis (ADA) course for their feedback. This project was partly
funded by the Swiss National Science Foundation (grant 200021_185043), the
European Union (TAILOR, grant 952215), and the Microsoft Swiss Joint Research
Center. We also gratefully acknowledge generous gifts from Facebook and
Google.

## References

  * (1)
  * Balahur et al. (2009) Alexandra Balahur, Ralf Steinberger, Erik Van der Goot, Bruno Pouliquen, and Mijail A. Kabadjov. 2009\.  Opinion Mining on Newspaper Quotations. In _WI-IAT’09 Workshops_.  https://doi.org/10.1109/WI-IAT.2009.340
  * Cao et al. (2019) Tien Duc Cao, Ludivine Duroyon, François Goasdoué, Ioana Manolescu, and Xavier Tannier. 2019\.  BeLink: Querying Networks of Facts, Statements and Beliefs. In _CIKM’19_.  https://doi.org/10.1145/3357384.3357851
  * Duncan et al. (2019) Megan Duncan, Kathleen Bartzen Culver, Douglas McLeod, and Christopher Kremmer. 2019\.  Don’t Quote Me: Effects of Named, Quoted, and Partisan News Sources.  _Journalism Practice_ 13, 9 (2019).  https://doi.org/10.1080/17512786.2019.1588148
  * Gormley and Tong (2015) Clinton Gormley and Zachary Tong. 2015.  _Elasticsearch: The Definitive Guide_ (1st ed.).  O’Reilly Media, Inc.
  * Hardy (2007) Donald E. Hardy. 2007\.  Corpus Stylistics: Speech, Writing, and Thought Presentation in a Corpus of English Writing.  _Lit. Linguistic Comput._ 22, 4 (2007).  https://doi.org/10.1093/llc/fqm030
  * MacLaughlin et al. (2021) Ansel MacLaughlin, Tao Chen, Burcu Karagol Ayan, and Dan Roth. 2021.  Context-Based Quotation Recommendation. In _ICWSM’21_.  https://ojs.aaai.org/index.php/ICWSM/article/view/18070
  * Miranda et al. (2019) Sebastião Miranda, David Nogueira, Afonso Mendes, Andreas Vlachos, Andrew Secker, Rebecca Garrett, Jeff Mitchell, and Zita Marinho. 2019.  Automated Fact Checking in the News Room. In _WWW’19 Companion_.  https://doi.org/10.1145/3308558.3314135
  * Muzny et al. (2017) Grace Muzny, Michael Fang, Angel X. Chang, and Dan Jurafsky. 2017.  A Two-stage Sieve Approach for Quote Attribution. In _EACL’17_.  https://doi.org/10.18653/v1/e17-1044
  * Newell et al. (2018) Edward Newell, Drew Margolin, and Derek Ruths. 2018\.  An Attribution Relations Corpus for Political News. In _LREC’18_.  http://www.lrec-conf.org/proceedings/lrec2018/pdf/1051.pdf
  * Niculae et al. (2015) Vlad Niculae, Caroline Suen, Justine Zhang, Cristian Danescu-Niculescu-Mizil, and Jure Leskovec. 2015\.  QUOTUS: The Structure of Political Media Coverage as Revealed by Quoting Patterns. In _WWW’15_.  https://doi.org/10.1145/2736277.2741688
  * Papay and Padó (2019) Sean Papay and Sebastian Padó. 2019.  Quotation Detection and Classification with a Corpus-Agnostic Model. In _RANLP’19_.  https://doi.org/10.26615/978-954-452-056-4_103
  * Papay and Padó (2020) Sean Papay and Sebastian Padó. 2020.  RiQuA: A Corpus of Rich Quotation Annotation for English Literary Text. In _LREC’20_.  https://aclanthology.org/2020.lrec-1.104/
  * Pareti (2015) Silvia Pareti. 2015\.  _Attribution: A Computational Approach_.  Ph.D. Dissertation. University of Edinburgh, UK.  http://ethos.bl.uk/OrderDetails.do?uin=uk.bl.ethos.679448
  * Pareti et al. (2013) Silvia Pareti, Timothy O’Keefe, Ioannis Konstas, James R. Curran, and Irena Koprinska. 2013\.  Automatically Detecting and Attributing Indirect Quotations. In _EMNLP’13_.  https://aclanthology.org/D13-1101/
  * Pavllo et al. (2018) Dario Pavllo, Tiziano Piccardi, and Robert West. 2018\.  Quootstrap: Scalable Unsupervised Extraction of Quotation-Speaker Pairs from Large News Corpora via Bootstrapping. In _ICWSM’18_.  https://aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/view/17827
  * Popat et al. (2018) Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2018.  CredEye: A Credibility Lens for Analyzing and Explaining Misinformation. In _WWW’18 Companion_.  https://doi.org/10.1145/3184558.3186967
  * Spangher et al. (2021) Alexander Spangher, Nanyun Peng, Jonathan May, and Emilio Ferrara. 2021.  ”Don’t quote me on that”: Finding Mixtures of Sources in News Articles.  _CoRR_ abs/2104.09656 (2021).  https://arxiv.org/abs/2104.09656
  * Vaucher et al. (2021) Timoté Vaucher, Andreas Spitz, Michele Catasta, and Robert West. 2021.  Quotebank: A Corpus of Quotations from a Decade of News. In _WSDM’21_.  https://doi.org/10.1145/3437963.3441760
  * Vrandecic and Krötzsch (2014) Denny Vrandecic and Markus Krötzsch. 2014.  Wikidata: A Free Collaborative Knowledgebase.  _Commun. ACM_ 57, 10 (2014).  https://doi.org/10.1145/2629489
  * Zhang and Liu (2021) Yuanchi Zhang and Yang Liu. 2021.  DirectQuote: A Dataset for Direct Quotation Extraction and Attribution in News Articles.  _CoRR_ abs/2110.07827 (2021).  https://arxiv.org/abs/2110.07827