# ChroniclingAmericaQA: A Large-scale Question Answering Dataset based on
Historical American Newspaper Pages

Bhawna Piryani<EMAIL_ADDRESS>University of
InnsbruckInnsbruckAustria ,  Jamshid Mozafari<EMAIL_ADDRESS>University of InnsbruckInnsbruckAustria and  Adam Jatowt
<EMAIL_ADDRESS>University of InnsbruckInnsbruckAustria

(2024)

###### Abstract.

Question answering (QA) and Machine Reading Comprehension (MRC) tasks have
significantly advanced in recent years due to the rapid development of deep
learning techniques and, more recently, large language models. At the same
time, many benchmark datasets have become available for QA and MRC tasks.
However, most existing large-scale benchmark datasets have been created
predominantly using synchronous document collections like Wikipedia or the
Web. Archival document collections, such as historical newspapers, contain
valuable information from the past that is still not widely used to train
large language models. To further contribute to advancing QA and MRC tasks and
to overcome the limitation of previous datasets, we introduce
ChroniclingAmericaQA, a large-scale dataset with 485K question-answer pairs
created based on the historical newspaper collection Chronicling America. Our
dataset is constructed from a subset of the Chronicling America newspaper
collection spanning 120 years. One of the significant challenges for utilizing
digitized historical newspaper collections is the low quality of OCR text.
Therefore, to enable realistic testing of QA models, our dataset can be used
in three different ways: answering questions from raw and noisy content,
answering questions from cleaner, corrected version of the content, as well as
answering questions from scanned images of newspaper pages. This and the fact
that ChroniclingAmericaQA spans the longest time period among available QA
datasets make it quite a unique and useful resource.

Question answering, heritage collections, OCR text

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
International Conference on Information Retrieval (SIGIR); July 14–18, 2024;
Washington, USA††booktitle: SIGIR 2024††isbn: 978-1-4503-XXXX-X/18/06††ccs:
Information systems Question answering††ccs: Information systems Content
analysis and feature selection

##  1\. Introduction

Question Answering (QA) and Machine Reading Comprehension (MRC) are popular
Natural Language Processing (NLP) tasks. They allow users to pose questions
and receive direct, concise answers from a given context, transforming how we
interact with the information. These tasks have seen a significant advancement
in recent years mainly due to the development of deep learning techniques and,
more recently, large language models. The availability of high-quality QA and
MRC datasets has also played a crucial role in the progress of QA and MRC
tasks.

However, a closer look at the available datasets reveals particular limitation
- they are predominantly created using synchronous document collections such
as Wikipedia and the Web. Our society maintains extensive collections of
archival documents, such as historical newspapers, which constitute our
heritage. Historical documents serve as a rich knowledge repository, capturing
the trends of different eras. They offer a unique perspective on the past, as
they are the primary sources of information related to historical events,
cultural norms, and societal attitudes (Korkeamäki and Kumpulainen, 2019). It
is beneficial to use Question Answering (QA) technologies on such documents as
well. QA models could help answer a wide range of questions, from those about
specific historical events to those seeking to understand cultural
developments over time and the minutiae of everyday life. Despite their
importance and wealth of contained information, heritage document collections
are still underutilized in the context of QA research.

However, leveraging these documents for QA is complex and poses a few
challenges. The language used in heritage documents often differs
significantly from the contemporary language in terms of vocabulary, syntax,
and the overall context of the distant times (tah, 2021; Tahmasebi et al.,
2021). These raises doubts if QA models trained on current text data can
indeed perform well on documents from distant past. Additionally, the
documents are often available only in scanned form, and the quality of the
text obtained through Optical Character Recognition (OCR) can greatly vary,
especially over long time periods, adding another layer of complexity to the
task. Finally, these documents have a complex layout, often containing
different font sizes and type throughout their content, which makes
information extraction a complex task. Figure 1 shows examples of scanned
newspaper pages and the OCRed text.

Figure 1. An example of the scanned newspaper page from Chronicling America
Collection. a) depicts the entire newspaper page published in Evening Star on
1803-02-07 in the District of Columbia, b) depicts the zoomed-in paragraph of
the newspaper page shown in (a), c) shows the original OCR text of the zoomed-
in paragraph that is available in the Chronicling America, and d) shows the
OCR text corrected by GPT 3.5 Turbo.

Despite the inherent challenges of using historical newspapers, they present
an attractive and unique opportunity for research in the field of QA. The
wealth of information in these documents makes them valuable resources that
are not readily available in typical evaluation benchmarks. To address the
challenges and utilize the potential of historical document collections, we
introduce the ChroniclingAmericaQA, a large-scale dataset based on Historical
American Newspaper Pages. As an underlying historical document collection, we
use Chronicling America111https://chroniclingamerica.loc.gov/about/, a
digitized collection of America’s Historical newspaper pages from 1756-1963.
Chronicling America is a project that was created with the aim of making a
digital, searchable engine for historical American Newspapers (200, 2007).
While previous studies have leveraged the Chronicling America collection,
their focus has predominantly been on visual content extraction (Lee et al.,
2020). In contrast, we focus on utilizing the already digitized text, and we
employ this resource for testing cutting-edge information processing
technologies on this novel and highly challenging type of text data.

We adopt an automatic approach to construct the dataset due to several
reasons. First, the historical document collection is huge, and processing
such a sheer collection of data manually is impossible; because of this, we
use GPT 3.5 (Brown et al., 2020) for correcting the digitized text available
on Chronicling America. Second, we employ a generative model to generate the
questions automatically because manual generation would be very costly. Third,
recently developed generative models have shown great performance for
automatic question generation (Du et al., 2017; Pan et al., 2019) at the same
time. The final dataset that we release comprises 485k question-answer pairs
spanning 120 years (1800-1920)222The dataset is freely available at:
https://github.com/DataScienceUIBK/ChroniclingAmericaQA.git .

To sum up, we make the following contributions in this work:

  * •

We propose a large-scale QA dataset built on historical newspaper pages, which
is the longest-spanning dataset over the period of 120 years.

  * •

We comprehensively evaluate the proposed dataset on different models including
also large language models (LLMs), establishing a reference baseline level for
the performance of question answering over historical documents.

  * •

We quantify the performance degradation of various models for question-
answering tasks due to Noisy OCR text.

Table 1. Comparison of Textual QA datasets (note that for the Corpus Size
column, the values in the parentheses represent the number of articles/pages
in the entire collection, while the values outside are the numbers of
articles/paragraphs used in the dataset creation).

Dataset | #Questions | Answer Type | Question Source | Corpus Source | Corpus Size | No. of Corpus Sources | Geographical Dispersion | Synch/Diach | Document Format
---|---|---|---|---|---|---|---|---|---
MS MARCO (Bajaj et al., 2016) | 1M |  Generative Boolean | Query logs | Web Documents | - | - | - | Synchronic | Text
SQuAD 1.1 (Rajpurkar et al., 2016) | 108K | Extractive | Crowd-sourced | Wikipedia | - | 1 | - | Synchronic | Text
SQuAD 2.0 (Rajpurkar et al., 2018) | 158K | Extractive | Crowd-sourced | Wikipedia | - | 1 | - | Synchronic | Text
NaturalQuestions (Kwiatkowski et al., 2019) | 323K | Extractive | Query logs | Wikipedia | - | 2 | - | Synchronic | Text
NewsQuizQA (Lelkes et al., 2021) | 20K | Multiple-choice | Crowd-sourced | News |  5k News Article summaries | 2 | Atlanta, London |  Diachronic (2018-2020) | Text
NewsQA (Trischler et al., 2017) | 119K | Extractive | Crowd-sourced | News | 13k News Articles | 1 | Atlanta |  Diachronic (2007-2015) | Text
ArchivalQA (Wang et al., 2022) | 532K | Extractive |  Automatically Generated | News |  98K News Articles (1.8M Articles) | 1 | New York City |  Diachronic (1987-2007) | Text
ChroniclingAmericaQA | 485K | Extractive |  Automatically Generated | News |  160K Paragraphs (121M newspaper pages) | 1,694 | Across 53 US States |  Diachronic (1800-1920) |  Scan Images & OCR Text

##  2\. Related Work

###  2.1. NLP for Historical Texts

The application of Natural Language Processing (NLP) techniques to historical
documents has gained significant attention due to the increasing availability
of digitized content. Prior works have primarily focused on various aspects of
processing historical texts such as text normalization (Robertson and
Goldwater, 2018; Bollmann, 2019; Bollmann et al., 2018), PoS Tagging
(Hardmeier, 2016), Named Entity Recognition (Ehrmann et al., 2020, 2023),
Event Detection (Sprugnoli and Tonelli, 2019; Lai et al., 2021), bias analysis
(Borenstein et al., 2023) and co-reference resolution (Krug et al., 2015).
These works have significantly contributed to our understanding of historical
documents. However, the application of QA, a critical task in NLP and
Information Retrieval (IR), is still underexplored.

The challenge of utilizing the historical text often revolves around the
quality of OCRed text and its repercussions on the NLP and IR tasks. Numerous
studies have been conducted to analyze the impact of OCR errors on diverse
tasks (Mutuvi et al., 2018; de Oliveira et al., 2023; Hamdi et al., 2022; van
Strien et al., 2020). Hamdi et al. (2022) analyzed the impact of OCR errors in
named entity recognition and linking and found that 80.75% named entities were
wrongly recognized due to OCR errors, hence they concluded that OCR errors can
negatively impact the task. Chiron et al. (2017) studied the effects of OCR
errors on digital library utilization and found that for 7% of the user
queries potential documents were missed due to wrong OCRed words. Similarly,
Mutuvi et al. (2018) also quantified the adverse effects of OCR errors on
topic modeling tasks. However, to the best of our knowledge, no such study has
been conducted for the QA task. Hence, in this research work, we perform a
comprehensive evaluation of the impact of OCR errors on QA tasks and try to
bridge this research gap.

###  2.2. QA Benchmarks

In recent years, numerous large-scale QA datasets have been introduced,
contributing significantly to the field of QA. Notable examples include SQuAD
1.1 (Rajpurkar et al., 2016), a landmark dataset for reading comprehension
derived from Wikipedia articles. It was followed by SQuAD 2.0 (Rajpurkar et
al., 2018), which expanded the challenge by incorporating unanswerable
questions. Beyond Wikipedia, datasets like NaturalQA (Kwiatkowski et al.,
2019) and MS MARCO (Bajaj et al., 2016) have utilized queries issued to Google
and Bing search engines as questions, top-ranking Wikipedia pages or other web
documents as context. NarrativeQA (Kočiský et al., 2018) broadened the scope
by using books and movie script summaries as the basis for its question-answer
pairs.

Additionally, there are datasets such as CNN/Daily Mail (Nallapati et al.,
2016), ReCoRD (Zhang et al., 2018), WhoDidWhat(Onishi et al., 2016) that use
news articles, which are designed as cloze type datasets where the goal is to
identify a missing word, rather than to answer open-ended questions.
NewsQuizQA by Lelkes et al. (2021) is a multiple-choice QA dataset derived
from summaries of 5k news articles to generate quiz-style question-answer
pairs.

When it comes to diachronic document collections, two notable QA datasets,
ArchivalQA (Wang et al., 2022) and NewsQA (Trischler et al., 2017) are
available. They have however several limitations, making our dataset different
and unique. ArchivalQA and NewsQA have been created from documents spanning
shorter and more recent periods, as well as using only a single news source.
ArchivalQA utilizes the NYT Corpus (Sandhaus, 2008) as its source, with
articles comming from the period of 1987/01-2007/06, whereas NewsQA utilizes
the CNN news articles throughout 2007/04-2015/04.

Our dataset, ChroniclingAmericaQA, uses historical newspaper pages that span
120 years ranging from 1800 to 1920. Both ArchivalQA and NewsQA use modern-day
text, which makes it easy for present-day models to understand as they are
essentially trained on such text. In contrast, the text in
ChroniclingAmericaQA is from historical newspapers, which shows the language
is archaic, which makes it more valuable for training the models to advance
the QA field. ChroniclingAmericaQA is also created from historical newspaper
documents in the public domain, making it accessible to the community, unlike
the NYT corpus used for ArchivalQA, which is not open-sourced and thus not
accessible to everyone. ChroniclingAmerica also features a more diverse and
complex language because the newspaper pages are from a relatively distant
past, and the OCR text is more complex and noisy. In Table 1, we summarize the
differences between ChroniclingAmericaQA and other related datasets.

Thus, this work aims to create a QA dataset from the Historical Newspapers
collection to foster the research of QA on Historical documents. Our dataset,
ChroniclingAmericaQA, is designed to handle the unique challenges posed by
historical newspapers, such as noisy OCR data, archaic language as well as the
overall context of the distant time that differs from the present
one333Consider the term Zeitgeist in German language used to emphasize the
spirit or essence of the particular time period or epoch.. We believe that
this dataset will be a valuable resource for researchers in the field of QA
and will contribute to the advancement of QA for historical documents and the
NLP field in general.

##  3\. Methodology

This section introduces the framework used to create the ChroniclingAmericaQA,
which is also depicted in Figure 2. The framework consists of three primary
modules: Data Collection, Data Preparation and the Question-Answer Generation
module. Each of these modules is described below.

Figure 2. Dataset Generation Framework

###  3.1. Data Collection Module

In this module, we describe the process of selecting newspaper pages used for
generating the ChroniclingAmericaQA dataset.

We chose the newspaper pages from the collection of the American Historical
Newspaper dataset Chronicling
America444https://chroniclingamerica.loc.gov/about/ available in the public
domain as our source for question generation. Chronicling America is a project
developed by the Library of Congress and the National Endowment for the
Humanities (NEH) under the National Digital Newspaper Program (NDNP). It
features over 21 million pages of historical American newspapers published
from 1756 to 1963, accessible through an online search portal and public API.
The portal offers various search strategies to access the content available on
Chronicling America. The search can be performed in three ways: by identifying
newspaper pages published in a specific state and year, conducting more
advanced searches for newspaper pages containing particular words or phrases,
and locating newspaper pages related to specific ethnic groups available in
different languages. We opted for the first search strategy, i.e., statewide
search by year, as our goal was to create a corpus of historical newspapers
that covers diverse information ranging from every significant past event and
ethnicity to some minor regional events throughout the United States.
Therefore, we decided to search for the newspaper pages published across the
53 states in the United States. Figure 3 shows the time intervals for which
newspapers are available for each state of the United States in the
Chronicling America Collection. Considering the availability of data from
different states, we opted for the period from 1800 to 1920, which spans 120
years.

Due to the enormous number of digitized newspaper pages available on
Chronicling America, including all the newspaper pages in our corpus was
impossible. Instead, we randomly selected 100 newspaper pages published per
decade in every state of the United States from 1800 to 1920. We considered
1800 to 1920 as the time period for our dataset; for this time period,
sufficiently large number of states have newspaper pages in Chronicling
America as shown in Figure 3. As a result, we extracted 39,330 newspaper pages
in total. While the Chronicling America collection spans from 1789 to 1963,
not every state has digitized newspapers for the same range of years. The
number of digitized newspaper pages available at the beginning of the 1800s is
less than in the 1900s. Therefore, the number of newspaper pages for our
corpus is lower in the early 1800s than in the late 1800s and early 1900s.

One of the challenges of digitized newspapers is the quality of OCR text. For
our purpose, the correctness of the OCR text plays a significant role, as
questions generated from noisy text will not be precise and accurate.

Figure 3. The temporal distribution of available newspapers across the states
in the Chronicling America Collection.

###  3.2. Data Preparation Module

Preparing data for question generation is a crucial step in our methodology.
The Chronicling America Collection, while being a rich source of digitized
historical newspaper pages, provides raw, unprocessed text. This text is often
noisy and contains numerous inaccuracies, as illustrated in Figure 1. This
noise and inaccuracy make the raw OCR text unsuitable for direct use in
question generation. Several studies have shown that noisy OCR text negatively
affects information retrieval and NLP tasks (Chiron et al., 2017; Bazzo et
al., 2020).

To address this issue, we post-process the raw OCR text to enhance its
quality, thereby making it more suitable for question generation. Post-
processing OCR text is challenging, as demonstrated by several studies (Nguyen
et al., 2021; Jatowt et al., 2019). Traditional methods for post-OCR error
correction, such as statistical language modeling techniques (Afli et al.,
2015), lexical approaches (Strohmaier et al., 2003), merging OCR outputs (Lund
and Ringger, 2009), are inadequate for handling historical text due to the
unique complexities associated with such data.

However, various recent studies have shown utilizing Large Language Models
(LLMs) can give promising results in correcting spelling and grammatical
errors, especially by utilizing the GPT models (Yasunaga et al., 2021; Loem et
al., 2023; Penteado and Perez, 2023). Considering these studies, we decide to
leverage the capabilities of LLMs for OCR correction and opt for the GPT 3.5
Turbo555https://platform.openai.com/docs/models/gpt-3-5-turbo model to correct
the OCR text.

Before giving the raw OCR text to the GPT model for correction, we need to
split the content of newspaper pages into paragraphs. Since the content on a
single newspaper page is diverse and long, ranging from advertisements and
articles to cartoons, processing such long text will be difficult for question
generation as well as for correcting errors in the OCRed text. Therefore, we
split the content a newspaper page into multiple paragraphs. We set the length
of a paragraph to 250 words due to the context length limitation imposed by
GPT 3.5 Turbo model666The maximum output token length of gpt-3.5-turbo-1106 is
4096..

Initially, we selected a total of 39,330 newspaper pages for the purpose of
question generation. However, due to the cost associated with using the GPT
3.5 API for OCR correction, we did not utilize all the selected newspapers for
this process. Correcting all the selected newspaper pages would have required
substantial resources. Instead, we randomly selected 20 newspaper pages from
each state across America for each decade from 1800 to 1920. This resulted in
a selection of 8,419 newspaper pages for OCR text correction. We then split
the content of these 8,419 newspaper pages into paragraphs, yielding a total
of 205,068 paragraphs. We next used the following prompt to instruct GPT 3.5
Turbo to correct these paragraphs:

”You are provided with a historical English text. The provided text has a lot
of spelling mistakes. Correct only the mistakes in the provided text and write
the corrected text in ONE paragraph. If you can not correct the mistakes,
reply, ”Not able to correct.”.

In this manner, we tasked GPT 3.5 with correcting 205,638 paragraphs. Out of
these, GPT was able to correct 163,021. We eliminated the uncorrected
paragraphs (42,599) from our corpus and utilized the corrected paragraphs for
question generation. Among the corrected paragraphs, 433 contained phrases
such as ”Most of the mistakes in the given text have been corrected. Here is
the corrected text:” and ”Errors in the given text have been corrected. Here
is the corrected text:”. We removed such 96 different phrases, cleaned the
paragraphs, and prepared them for question generation. Figure 4 illustrates
the number of corrected paragraphs per decade.

Figure 4. Numbers of Raw OCR Paragraphs and Corrected Paragraphs over Time in
our Dataset

###  3.3. Question Generation Module

The third phase of our framework involves generating questions from the
revised paragraphs. We utilize the T5-base model (Raffel et al., 2020) for
this purpose, a cutting-edge pre-trained Transformer encoder-decoder model
that has been trained on the SQuAD 1.1 (Rajpurkar et al., 2016) dataset
specifically for question generation. Our approach to question generation is
answer-aware, meaning that the model receives the answers along with their
corresponding paragraphs as inputs, and generates the questions as outputs. We
identify named entities within a paragraph using the spaCy
library777https://github.com/explosion/spaCy, marking these as answers for
question generation. Based on these identified named entities, questions are
then generated. From 163,021 cleaned paragraphs, we successfully generated
2,912,551 questions.

Question & answer cleaning: Given that we employ the T5-base model, a
generative model for question generation, the produced questions may exhibit
various issues, such as duplication, explicit answer disclosure within the
question, unresolved pronouns, and more. To address these issues, we
implemented a multi-step filtering process to refine the dataset. The process
includes:

  1. (1)

Syntactic Filtering: In this initial step, we perform five key actions to
refine the generated question-answer pairs. We start by eliminating questions
that do not conclude with a question mark, followed by removing questions that
reveal the answer within the question itself (answer leakage) and eliminating
duplicates. We also filter out questions that are excessively long or short,
as well as those with too many or too few number of named entities.
Additionally, questions with unclear pronouns are removed. This step resulted
in removing 1,221,533 questions from our initial pool of generated questions.

  2. (2)

Temporal Expression Transforming: Following syntactic filtering, we convert
the relative temporal information in questions and answers into absolute
temporal information. For this transformation, we utilize HeidelTime (Strötgen
and Gertz, 2015) temporal tagger. We use the publication date of the newspaper
to which the paragraph belongs, along with the generated question for
converting the relative temporal expressions. For instance, a question like
How many subscribers did Jackson lose on Saturday? is transformed to How many
subscribers did Jackson lose on October 60, 1832? Similarly, we also transform
the temporal information in answer to absolute temporal information. Some
examples of the transformed questions and answers are shown in Table 2. This
step ensured clarity and specificity in temporal references with our dataset.

  3. (3)

Type Matching: The third and final filtering step, involved identifying and
matching the types of questions and answers. For this purpose, we finetined
RoBERTa (Liu et al., 2019)\- a large language model using the TREC Question
classifier dataset (Li and Roth, 2002) to develop a question type
classifier888The accuracy of the question type classifier is 92.8%.. The
question classifier model classifies the question type into four major types
and 50 minor types. Similarly, for answer type classification, we utilized
spaCy’s named entity recognizer. After identifying the question and answer
type, we then matched the question-answer pairs. Question and answer pairs
whose type was not the same were eliminated. In this step, we eliminated
1,205,482 question-answer pairs.

Table 2. Examples of transformed questions-answer pairs. Underlined text shows
the transformed information.

No. | Original Question-answer pair | Transformed Question-answer pair
---|---|---
1. |  How many people died last year due to famine and disease in Mexico City? (nearly 500) |  How many people died in 1822 due to famine and disease in Mexico City? nearly 500
2. |  How many British stocks were quoted at yesterday’s prices? (Two Bank Stock) |  How many British stocks were quoted at November 15, 1830’s prices? (Two Bank Stocks)
3. |  On what day did the Board of Commissioners of Roads hold their semi-annual meetings? (Monday) |  On what day did the Board of Commissioners of Roads hold their semi-annual meetings? (October 20, 1856)
4. |  When was the Secretary of State’s report to the President made? (the 24th December, 1817) |  When was the Secretary of State’s report to the President made? (December 24, 1817)

##  4\. Dataset Analysis

###  4.1. Data Statistics

After applying all the filtering steps, we arrived at our cleaned question-
answer dataset consisting of 485K question-answer pairs derived from
historical American newspapers. In Figure 5, we show the distribution of
newspaper pages in our dataset across various states of America. We randomly
split the data into training, development, and test sets to facilitate a
comprehensive training and evaluation of the models. The training set consists
of 439,302 question-answer pairs, whereas the development and test sets
comprise 24,111 and 24,084, respectively. When randomly selecting the
questions for development and test sets, we tried to balance the question-
answer pairs across all the decades. We describe more details about the
dataset statistics in Table 3. Our entire train set is quite large, which
makes it computationally expensive to fine-tune various pre-trained models
such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019), T5 (Raffel et
al., 2019); therefore, we randomly select a subset of question-answer
(111,517) pairs from the train set and use this subset of questions for
training various models whose results are shown in Section 4.2.

Figure 5. Distribution of Newspaper pages in ChroniclingAmericaQA across the
states of America. Table 3. Basis statistics of ChroniclingAmericaQA.

| Train | Dev | Test
---|---|---|---
#QA pairs | 439,302 | 24,111 | 24,084
Average paragraph length (words) | 220.09 | 218.40 | 217.66
Average question length (words) | 11.05 | 11.30 | 11.184
Average answer length (words) | 2.01 | 1.978 | 1.981
Average questions per paragraph | 3.48 | 1.90 | 1.90
Average question per newspaper page | 54.80 | 4.07 | 4.08

We also analyze the distribution of the named entity types of answers and
questions in the dataset. In the left pie chart of Figure 6, we can observe
the name entity types of the answers, whereas the right pie chart shows the
distribution of named entity types of the question. In Figure 7, we show the
dataset’s distribution of the question types. From the figure, we can see that
the ChroniclingAmericaQA data has a rather fair distribution among different
question types. Table 4 shows a few examples from the ChroniclingAmericaQA
along with the corresponding titles of the newspapers, their publication
dates, and the states in which the newspapers were published.

Figure 6. Left: Answer’s name entity distribution, Right: Question’s named
entity distribution Figure 7. Types of questions covered in
ChroniclingAmericaQA. We show the trigram prefixes of our questions. Table 4.
Examples of questions in ChroniclingAmericaQA.

No. | Question | Answer | Newspaper Title | Publication Date | State
---|---|---|---|---|---
1. |  Which army was marching slowly towards Madrid? | French | The Portland Gazette | 1823-03-17 | Maine
2. |  Who was sheriff of the county of Hanover? | Jesse Winn | Daily Richmond Whig | 1830-08-12 | Virginia
3. |  On what day will the Anti Slavery Society of Newport hold their quarterly meeting? | November 19, 1840 | Herald of the Times | 1840-11-19 | Rhode Island
4. |  Along with Texas, in what state is the rush of immigrants prodigious? | California | The Texas Republican | 1850-02-21 | Texas
5. |  How much is the interest rate in Seattle property loans? | 10 percent | The Seattle Post-intelligencer | 1892-09-02 | Washington
6. |  Who was the agency that ran the horses for the Sioux Indians? | Red Cloud Indian | Sierra County Advocate | 1902-06-27 | New Mexico
7. | Who was fined $1 by the mayor? | Parker | The Marion Daily Mirror | 1907-07-30 | Ohio
8. | What club did Frank Bishop belong to? | the National Athletic Club | Sierra County advocate | 1914-04-01 | Montana

###  4.2. Model Performance

To evaluate our dataset, we employ three models from the transformer family,
namely BERT-base, RoBERTa-base, and T5-large. We assess our dataset using four
different variations of each model. The variations are as follows:

  * •

BERT-base999https://huggingface.co/bert-base-uncased: without any fine-tuning.

  * •

BERT-base-SQuAD101010https://huggingface.co/csarron/bert-base-uncased-
squad-v1: fine-tuned on the SQuAD 1.1 dataset.

  * •

BERT-base-ChroniclingAmericaQA: fine-tuned on the ChroniclingAmericaQA
dataset.

  * •

BERT-base-SQuAD-ChroniclingAmericaQA: fine-tuned on both SQuAD 1.1 and
ChroniclingAmericaQA datasets.

  * •

RoBERTa-base111111https://huggingface.co/FacebookAI/roberta-base: without any
fine-tuning.

  * •

RoBERTa-base-SQuAD121212https://huggingface.co/deepset/roberta-base-squad2:
fine-tuned on the SQuAD 1.1 dataset.

  * •

RoBERTa-base-ChroniclingAmericaQA: fine-tuned on the ChroniclingAmericaQA
dataset.

  * •

RoBERTa-base-SQuAD-ChroniclingAmericaQA: fine-tuned on both SQuAD 1.1 and
ChroniclingAmericaQA datasets.

  * •

T5-large131313https://huggingface.co/google-t5/t5-large: without any fine-
tuning.

  * •

T5-large-SQuAD141414https://huggingface.co/potsawee/t5-large-generation-squad-
QuestionAnswer: fine-tuned on the SQuAD 1.1 dataset.

  * •

T5-large-ChroniclingAmericaQA: fine-tuned on the ChroniclingAmericaQA dataset.

  * •

T5-large-SQuAD-ChroniclingAmericaQA: fine-tuned on both SQuAD 1.1 and
ChroniclingAmericaQA datasets.

Table 5. Model Performance of ChroniclingAmericaQA Dataset using Corrected OCR
Paragraphs and Raw OCR Paragraphs as context. Red colored numbers denote
percentage decrease when using Raw OCR paragraphs. The underlined results
depict which variation of the model gave the best performance compared to its
counterparts, whereas the bold results show the overall best performance on
the given measure.

Model | Corrected OCR Paragraph | Raw OCR Paragraphs
---|---|---
| EM | F1 | EM | F1
BERT-base | 0.12 | 2.91 | 0.08 (33%$\downarrow$) | 2.27 (22%$\downarrow$)
BERT-base-SQuAD | 44.70 | 57.24 | 27.14 (39%$\downarrow$) | 40.33 (29%$\downarrow$)
BERT-base-ChroniclingAmericaQA | 63.29 | 69.43 | 38.94 (38%$\downarrow$) | 48.65 (30%$\downarrow$)
BERT-base-SQuAD-ChroniclingAmericaQA | 63.90 | 69.92 |  39.71 (37%$\downarrow$) |  49.45 (29%$\downarrow$)
RoBERTa-base | 0.02 | 1.48 | 0.02 (0%$\downarrow$) | 1.32 (10%$\downarrow$)
RoBERTa-base-SQuAD | 47.52 | 60.55 | 29.83 (38%$\downarrow$) | 42.71 (29%$\downarrow$)
RoBERTa-base-ChroniclingAmericaQA | 63.61 | 70.72 | 40.40 (36%$\downarrow$) |  50.12 (29%$\downarrow$)
RoBERTa-base-SQuAD-ChroniclingAmericaQA | 63.93 | 71.00 |  40.42 (36%$\downarrow$) |  50.12 (29%$\downarrow$)
T5-large | 46.69 | 59.77 | 26.66 (43%$\downarrow$) | 36.90 (38%$\downarrow$)
T5-large-SQuAD | 1.50 | 9.28 | 0.8 (46%$\downarrow$) | 6.07 (34%$\downarrow$)
T5-large-ChroniclingAmericaQA | 64.28 | 69.65 |  34.49 (46%$\downarrow$) |  41.54 (40%$\downarrow$)
T5-large-SQuAD-ChroniclingAmericaQA | 64.24 | 59.77 | 34.30 (47%$\downarrow$) | 41.53 (18%$\downarrow$)

Figure 8. Comparison of Model Performance over Time.

Table 5 summarizes the performance of different transformers on the
ChroniclingAmericaQA dataset using standard QA metrics: exact match (EM) and
F1 score. We measure how well different models can answer questions from both
Corrected and Raw OCR paragraphs. From the results, it is evident that models
used without any fine-tuning performed the worst, as these models lack
specific knowledge about the question-answering task. Conversely, models fine-
tuned on the SQuAD 1.1 dataset show significant improvements, highlighting the
benefits of fine-tuning on task-specific datasets. Notably, the highest
performance levels are observed from the models trained on the
ChroniclingAmericaQA dataset itself, depicting the importance of fine-tuning
on specific datasets. However, models trained on both SQuAD and
ChroniclingAmericaQA give even better performance, suggesting that combining
domain-specific knowledge such as historical content and general task specific
knowledge can even enhance the performance of models.

We also compare the results of transformer models on both Raw OCR paragraphs
and Corrected OCR paragraphs in Table 5. From the results, we can analyze that
model performance significantly drops when the Raw OCR paragraph is given as
context, suggesting the importance of cleaning the digitized text. We also
show the performance of fine-tuned models over time in Figure 8. It can be
concluded from the figure that the performance improvement due to finetunning
on our dataset remains relatively stable over time.

In summary, we can conclude that results shown in table 5 ensure fine-tuning
models on relevant datasets significantly improve the performance.
Additionally, we can also realize that correcting the OCRed text is crucial
for getting better model performance.

###  4.3. LLM Performance

Table 6. LLMs Performance on ChroniclingAmericaQA using Corrected OCR
Paragraphs and Raw OCR Paragraphs as context. Red colored numbers denote
percentage decrease when using Raw OCR paragraphs. The bold values represent
overall best performance for a given measure on Corrected OCR Paragraphs,
whereas the underlined values represent the best performance for the given
measure on Raw OCR Paragraphs.

Model | Parameter | EM | F1 | Recall | Precision | Contains
---|---|---|---|---|---|---
LLaMA2 (Cleaned OCR Paragraphs) | 7B | 0.22 | 12.04 | 57.26 | 7.08 | 44.39
LLaMA2 (Raw OCR Paragraphs) | 7B | 0.00  (100%$\downarrow$) | 2.24  (81%$\downarrow$) | 10.45  (82%$\downarrow$) | 1.32  (81%$\downarrow$) | 5.67  (87%$\downarrow$)
LLaMA2 (Cleaned OCR Paragraphs) | 13B | 0.00 | 11.89 | 58.36 | 6.86 | 45.80
LLaMA2 (Raw OCR Paragraphs) | 13B | 0.00  (0%$\downarrow$) | 2.16  (82%$\downarrow$) | 10.59  (81%$\downarrow$) | 1.27  (81%$\downarrow$) | 5.68  (87%$\downarrow$)
LLaMA2 (Cleaned OCR Paragraphs) | 70B | 5.30 | 19.52 | 61.38 | 14.17 | 48.37
LLaMA2 (Raw OCR Paragraphs) | 70B |  0.39 (93%$\downarrow$) |  3.02 (84%$\downarrow$) |  11.19 (82%$\downarrow$) |  2.00 (86%$\downarrow$) |  6.35 (87%$\downarrow$)
Mixtral (Cleaned OCR Paragraphs) | 8x7B | 1.13 | 12.88 | 62.69 | 8.15 | 50.00
Mixtral (Raw OCR Paragraphs) | 8x7B | 0.00  (100%$\downarrow$) | 0.94  (92%$\downarrow$) | 10.06  (84%$\downarrow$) | 0.51  (94%$\downarrow$) | 5.63  (89%$\downarrow$)
Mistral (Cleaned OCR Paragraphs) | 7B | 1.17 | 16.30 | 56.35 | 10.42 | 44.16
Mistral (Raw OCR Paragraphs) | 7B | 0.09  (92%$\downarrow$) | 2.08  (87%$\downarrow$) | 8.19  (85%$\downarrow$) | 1.30  (85%$\downarrow$) | 4.33  (90%$\downarrow$)

Similarly, in addition to transformer models, we evaluate our dataset with
Large Language Models (LLMs), including the LLaMA2 (tou, 2023) family, Mixtral
8x7B (Jiang et al., 2024) and Mistral 7B (Jiang et al., 2023). Evaluating the
performance of LLMs (Guo et al., 2023) within the context of question
answering poses unique challenges. Traditionally, QA systems are assessed
using Exact Match (EM) and F1 scores. However, these metrics are not fully
appropriate for generative models such as LLMs, which often produce verbose
responses. This verbosity can result in an EM value of zero due to the
presence of extra tokens in the answer. Similarly, the F1 score, which
evaluates both recall and precision, may not provide an accurate measure of
performance under these conditions. To overcome the challenge of evaluating
LLMs, we introduce model-agnostic metrics like Token Recall and Answer String
Containment151515https://huggingface.co/spaces/evaluate-metric/squad, which
are more suitable for assessing verbose responses. Token Recall evaluates how
effectively the model’s response covers the information found in the ground
truth, while Answer String Containment assesses the degree to which the ground
truth is included within the model’s response. By using these metrics, we aim
to offer a more equitable evaluation of an LLM’s ability to generate responses
to questions.

Table 6 presents the performance of different LLMs on the ChroniclingAmericaQA
Dataset using both raw OCR and corrected paragraphs. When evaluating the
results of different LLMs, we can note that LLaMA2 70B gives significantly
high EM and F1 scores compared to its counterparts and Mistral models. This
also suggests that models with large sizes have a greater ability to
understand the nuances of historical text, leading to more accurate and
precise answers. Meanwhile, the Mixtral 8x7b model performs highly on Recall
and Contains, depicting its proficiency in extracting relevant information
from the context.

Furthermore, When examining the results of different LLMs considering the raw
OCR paragraph and corrected paragraph, we can note that performance across all
models decreased drastically when utilizing the raw OCR paragraph as context.
Models such as LLaMA2 7B and Mixtral Bx7B gave zero results on EM metric,
whereas LLaMA2 70B still showed some resistance on raw OCR paragraph, however,
with a significant drop of 93%. Furthermore, When analyzing the recall and
precision metrics results, one can note that models showed better performance
on recall, indicating their ability to capture relevant information from the
context.

Summarizing the results of Table 6, we can conclude that the performance of
LLaMA2 70B is high compared to other models for handling both raw OCR text and
corrected text. On the other hand, Mixtral 8x7B gives high results for recall
and contains metrics on corrected OCR paragraphs. However, LLaMa2 70B gives a
high recall and contains value on the raw OCR paragraph.

Overall, we can conclude that the results of both Tables 5 and 6 emphasize
that fine-tuning the model on the relevant datasets plays a major role in
increasing the model performance. Additionally, it highlights the necessity of
post-processing OCR text for improving the performance of models on historical
documents, as the results of both transformer models and LLMs degrade
drastically for raw OCR text.

###  4.4. Human Evaluation

To assess the quality of the ChroniclingAmericaQA dataset, we conducted a
manual evaluation study. We randomly selected 360 question-answer pairs for
evaluation, sampling 30 pairs from each decade represented in the dataset. Six
graduate students were asked to rate these pairs on a scale of 1 to 5, with 1
being ”very bad” and 5 being ”very good.” The evaluators were instructed to
assess the pairs based on four criteria: the readability of the question,
ensuring it is grammatically correct and flows well; the readability of the
answer, verifying its grammatical correctness and fluency; relevance, to
ascertain if the generated question pertains to the provided passage; and non-
ambiguity, to ensure the question is straightforward and clear.

Each criterion’s human evaluation results were averaged and presented in Table
7. The evaluations yielded high scores across all metrics, indicating that our
dataset is of high quality, particularly regarding the relevance and
readability of both questions and answers. The score for non-ambiguity was
also notably high, demonstrating that most questions are direct and
unambiguous.

Table 7. Human evaluation results.

Readability of Question |  Readability of Answer | Relevance |  Non-Ambiguity
---|---|---|---
4.24 | 4.29 | 4.39 | 4.18

##  5\. Use cases

Our dataset can be used in several ways. First, it provides a new benchmark
for training and evaluating QA and MRC models on historical texts, enabling
models to handle the complexities of historical text, including OCR
inaccuracies and language evaluation. The evaluation can be done for different
spans of time within the time frame of the data (1800-1920). It can be also
conditioned on different regions of USA or particular states.

Second, since our dataset contains scanned images of news papers as well as
the corresponding raw and cleaned OCR texts, it is possible to benchmark
models on each of these. Some memory institutions may still hold only scans of
historical materials, while others, such as in the case of Chronicling
America, may contain quite noisy OCRed text. Our dataset allows then
benchmarking models on these different types of inputs, offering realistic
scenarios in which the models may need to be required to operate.

Third, the dataset can be used to increase the public’s engagement with
historical documents. For example, it can help people build skills in critical
reading and language arts.

Finally, ChroniclingAmericaQA could potentially serve as an educational
resource; for example, it could be used by teachers and educators to assess
students’ reading comprehension on rare, historical materials. Note, that our
dataset generation framework can be applied on other historical news
collections.

##  6\. Conclusion

In this work, we introduce ChroniclingAmetricaQA, a large-scale question-
answering dataset comprising 485k question-answer pairs over a collection of
historical American newspapers with the objective of facilitating the
development of QA and MRC systems over historical texts. Our dataset is unique
as it is specifically based on historical newspapers covering the longest time
period of 120 years from 1800 to 1920 years, among other available datasets on
news documents. We highlight the difficulties of utilizing historical
newspapers for QA tasks, such as the poor quality of OCR text and the need for
post-processing to enhance text quality. We employed language models such as
GPT 3.5 Turbo for OCR text correction and the T5-base model for question
generation to mitigate these challenges. We also carried out a comprehensive
evaluation of our dataset on various transformer models and LLMs,
demonstrating the effectiveness of our dataset. Models trained on
ChroniclingAmericaQA consistently surpass the performance of those fine-tuned
on other datasets.

Ethical considerations Lastly, we would like to mention ethical considerations
of our work. Since the dataset has been created from temporally distant data,
there is possibility that some questions may be offensive or biased in
relation to some ethnic groups or other segments of society. We made sure that
the possibility of such questions in our dataset is low by performing post-
processing search using a set of prepared keywords coupled with careful manual
analysis. Nevertheless, this kind of issue is not specific to our dataset but
is a problem typical in many heritage materials (Borenstein et al., 2023),
including also ones in the Chronicling America repository, and should be
continuously studied and mitigated in the future.

## References

  * (1)
  * 200 (2007) 2007\.  _Reference Reviews_ 21, 7 (Sept. 2007), 52–53.  https://doi.org/10.1108/09504120710821875
  * tah (2021) 2021\.  Survey of computational approaches to lexical semantic change detection.  _Computational approaches to semantic change_ 6, 1 (2021).
  * tou (2023) 2023\.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Afli et al. (2015) Haithem Afli, Loïc Barrault, and Holger Schwenk. 2015.  OCR Error Correction Using Statistical Machine Translation.
  * Bajaj et al. (2016) Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016\.  Ms marco: A human generated machine reading comprehension dataset.  _arXiv preprint arXiv:1611.09268_ (2016).
  * Bazzo et al. (2020) Guilherme Torresan Bazzo, Gustavo Acauan Lorentz, Danny Suarez Vargas, and Viviane P. Moreira. 2020.  Assessing the Impact of OCR Errors in Information Retrieval. In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II_ (Lisbon, Portugal). Springer-Verlag, Berlin, Heidelberg, 102–109.  https://doi.org/10.1007/978-3-030-45442-5_13
  * Bollmann (2019) Marcel Bollmann. 2019.  A Large-Scale Comparison of Historical Text Normalization Systems. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 3885–3898.  https://doi.org/10.18653/v1/N19-1389
  * Bollmann et al. (2018) Marcel Bollmann, Anders Søgaard, and Joachim Bingel. 2018.  Multi-task learning for historical text normalization: Size matters. In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_ , Reza Haffari, Colin Cherry, George Foster, Shahram Khadivi, and Bahar Salehi (Eds.). Association for Computational Linguistics, Melbourne, 19–24.  https://doi.org/10.18653/v1/W18-3403
  * Borenstein et al. (2023) Nadav Borenstein, Karolina Stanczak, Thea Rolskov, Natacha Klein Käfer, Natália da Silva Perez, and Isabelle Augenstein. 2023.  Measuring Intersectional Biases in Historical Documents. In _Findings of the Association for Computational Linguistics: ACL 2023_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 2711–2730.  https://doi.org/10.18653/v1/2023.findings-acl.170
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chiron et al. (2017) Guillaume Chiron, Antoine Doucet, Mickaël Coustaty, Muriel Visani, and Jean-Philippe Moreux. 2017.  Impact of OCR errors on the use of digital libraries: towards a better access to information. In _Proceedings of the 17th ACM/IEEE Joint Conference on Digital Libraries_ (Toronto, Ontario, Canada) _(JCDL ’17)_. IEEE Press, 249–252.
  * de Oliveira et al. (2023) Lucas Lima de Oliveira, Danny Suarez Vargas, Antônio Marcelo Azevedo Alexandre, Fábio Corrêa Cordeiro, Diogo da Silva Magalhães Gomes, Max de Castro Rodrigues, Regis Kruel Romeu, and Viviane Pereira Moreira. 2023.  Evaluating and mitigating the impact of OCR errors on information retrieval.  _Int. J. Digit. Libr._ 24, 1 (jan 2023), 45–62.  https://doi.org/10.1007/s00799-023-00345-6
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017.  Learning to Ask: Neural Question Generation for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1342–1352.  https://doi.org/10.18653/v1/P17-1123
  * Ehrmann et al. (2023) Maud Ehrmann, Ahmed Hamdi, Elvys Linhares Pontes, Matteo Romanello, and Antoine Doucet. 2023.  Named entity recognition and classification in historical documents: A survey.  _Comput. Surveys_ 56, 2 (2023), 1–47.
  * Ehrmann et al. (2020) Maud Ehrmann, Matteo Romanello, Alex Flückiger, and Simon Clematide. 2020.  Overview of CLEF HIPE 2020: Named entity recognition and linking on historical newspapers. In _Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11_. Springer, 288–310.
  * Guo et al. (2023) Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. 2023\.  Evaluating Large Language Models: A Comprehensive Survey.  _arXiv preprint arXiv:2310.19736_ (2023).
  * Hamdi et al. (2022) Ahmed Hamdi, Elvys Linhares Pontes, Nicolas Sidère, Mickaël Coustaty, and Antoine Doucet. 2022.  In-Depth Analysis of the Impact of OCR Errors on Named Entity Recognition and Linking.  _Journal of Natural Language Processing_ (03 2022), 24.  https://doi.org/10.1017/S1351324922000110
  * Hardmeier (2016) Christian Hardmeier. 2016.  A Neural Model for Part-of-Speech Tagging in Historical Texts. In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , Yuji Matsumoto and Rashmi Prasad (Eds.). The COLING 2016 Organizing Committee, Osaka, Japan, 922–931.  https://aclanthology.org/C16-1088
  * Jatowt et al. (2019) Adam Jatowt, Mickael Coustaty, Nhu-Van Nguyen, Antoine Doucet, et al. 2019\.  Deep statistical analysis of OCR errors for effective post-OCR processing. In _2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_. IEEE, 29–38.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023\.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ (2023).
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024\.  Mixtral of Experts.  _arXiv preprint arXiv:2401.04088_ (2024).
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The NarrativeQA Reading Comprehension Challenge.  _Transactions of the Association for Computational Linguistics_ 6 (2018), 317–328.  https://doi.org/10.1162/tacl_a_00023
  * Korkeamäki and Kumpulainen (2019) Laura Korkeamäki and Sanna Kumpulainen. 2019.  Interacting with Digital Documents : A Real Life Study of Historians’ Task Processes, Actions and Goals.  https://doi.org/10.1145/3295750.3298931
  * Krug et al. (2015) Markus Krug, Frank Puppe, Fotis Jannidis, Luisa Macharowsky, Isabella Reger, and Lukas Weimar. 2015.  Rule-based Coreference Resolution in German Historic Novels. In _Proceedings of the Fourth Workshop on Computational Linguistics for Literature_ , Anna Feldman, Anna Kazantseva, Stan Szpakowicz, and Corina Koolen (Eds.). Association for Computational Linguistics, Denver, Colorado, USA, 98–104.  https://doi.org/10.3115/v1/W15-0711
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ 7 (2019), 452–466.  https://doi.org/10.1162/tacl_a_00276
  * Lai et al. (2021) Viet Dac Lai, Minh Van Nguyen, Heidi Kaufman, and Thien Huu Nguyen. 2021.  Event Extraction from Historical Texts: A New Dataset for Black Rebellions. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 2390–2400.  https://doi.org/10.18653/v1/2021.findings-acl.211
  * Lee et al. (2020) Benjamin Charles Germain Lee, Jaime Mears, Eileen Jakeway, Meghan Ferriter, Chris Adams, Nathan Yarasavage, Deborah Thomas, Kate Zwaard, and Daniel S. Weld. 2020.  The Newspaper Navigator Dataset: Extracting Headlines and Visual Content from 16 Million Historic Newspaper Pages in Chronicling America. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ (Virtual Event, Ireland) _(CIKM ’20)_. Association for Computing Machinery, New York, NY, USA, 3055–3062.  https://doi.org/10.1145/3340531.3412767
  * Lelkes et al. (2021) Adam D. Lelkes, Vinh Q. Tran, and Cong Yu. 2021.  Quiz-Style Question Generation for News Stories. In _Proceedings of the Web Conference 2021_ (Ljubljana, Slovenia) _(WWW ’21)_. Association for Computing Machinery, New York, NY, USA, 2501–2511.  https://doi.org/10.1145/3442381.3449892
  * Li and Roth (2002) Xin Li and Dan Roth. 2002.  Learning Question Classifiers. In _COLING 2002: The 19th International Conference on Computational Linguistics_.  https://aclanthology.org/C02-1150
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _ArXiv_ abs/1907.11692 (2019).  https://api.semanticscholar.org/CorpusID:198953378
  * Loem et al. (2023) Mengsay Loem, Masahiro Kaneko, Sho Takase, and Naoaki Okazaki. 2023.  Exploring Effectiveness of GPT-3 in Grammatical Error Correction: A Study on Performance and Controllability in Prompt-Based Methods. In _Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)_ , Ekaterina Kochmar, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Nitin Madnani, Anaïs Tack, Victoria Yaneva, Zheng Yuan, and Torsten Zesch (Eds.). Association for Computational Linguistics, Toronto, Canada, 205–219.  https://doi.org/10.18653/v1/2023.bea-1.18
  * Lund and Ringger (2009) William B. Lund and Eric K. Ringger. 2009.  Improving optical character recognition through efficient multiple system alignment. In _Proceedings of the 9th ACM/IEEE-CS Joint Conference on Digital Libraries_ (Austin, TX, USA) _(JCDL ’09)_. Association for Computing Machinery, New York, NY, USA, 231–240.  https://doi.org/10.1145/1555400.1555437
  * Mutuvi et al. (2018) Stephen Mutuvi, Antoine Doucet, Moses Odeo, and Adam Jatowt. 2018.  Evaluating the impact of OCR errors on topic modeling. In _International Conference on Asian Digital Libraries_. Springer, 3–14.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , Stefan Riezler and Yoav Goldberg (Eds.). Association for Computational Linguistics, Berlin, Germany, 280–290.  https://doi.org/10.18653/v1/K16-1028
  * Nguyen et al. (2021) Thi Tuyet Hai Nguyen, Adam Jatowt, Mickael Coustaty, and Antoine Doucet. 2021.  Survey of post-OCR processing approaches.  _ACM Computing Surveys (CSUR)_ 54, 6 (2021), 1–37.
  * Onishi et al. (2016) Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016.  Who did What: A Large-Scale Person-Centered Cloze Dataset. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , Jian Su, Kevin Duh, and Xavier Carreras (Eds.). Association for Computational Linguistics, Austin, Texas, 2230–2235.  https://doi.org/10.18653/v1/D16-1241
  * Pan et al. (2019) Liangming Pan, Wenqiang Lei, Tat-Seng Chua, and Min-Yen Kan. 2019.  Recent Advances in Neural Question Generation.  arXiv:1905.08949 [cs.CL]
  * Penteado and Perez (2023) Maria Carolina Penteado and Fábio Perez. 2023.  Evaluating GPT-3.5 and GPT-4 on Grammatical Error Correction for Brazilian Portuguese. In _LatinX in AI Workshop at ICML 2023 (Regular Deadline)_.  https://openreview.net/forum?id=EOOtS6iSE0
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21, 140 (2020), 1–67.  http://jmlr.org/papers/v21/20-074.html
  * Raffel et al. (2019) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ 21 (2019), 140:1–140:67.  https://api.semanticscholar.org/CorpusID:204838007
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, Melbourne, Australia, 784–789.  https://doi.org/10.18653/v1/P18-2124
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ Questions for Machine Comprehension of Text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , Jian Su, Kevin Duh, and Xavier Carreras (Eds.). Association for Computational Linguistics, Austin, Texas, 2383–2392.  https://doi.org/10.18653/v1/D16-1264
  * Robertson and Goldwater (2018) Alexander Robertson and Sharon Goldwater. 2018.  Evaluating Historical Text Normalization Systems: How Well Do They Generalize?. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, New Orleans, Louisiana, 720–725.  https://doi.org/10.18653/v1/N18-2113
  * Sandhaus (2008) Evan Sandhaus. 2008.  The New York Times Annotated Corpus.  https://doi.org/11272.1/AB2/GZC6PL
  * Sprugnoli and Tonelli (2019) Rachele Sprugnoli and Sara Tonelli. 2019.  Novel Event Detection and Classification for Historical Texts.  _Computational Linguistics_ 45, 2 (June 2019), 229–265.  https://doi.org/10.1162/coli_a_00347
  * Strohmaier et al. (2003) Christian M. Strohmaier, Christoph Ringlstetter, Klaus U. Schulz, and Stoyan Mihov. 2003.  Lexical postcorrection of OCR-results:the web as a dynamic secondary dictionary?  _Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings._ (2003), 1133–1137.  https://api.semanticscholar.org/CorpusID:12766761
  * Strötgen and Gertz (2015) Jannik Strötgen and Michael Gertz. 2015.  A Baseline Temporal Tagger for all Languages. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , Lluís Màrquez, Chris Callison-Burch, and Jian Su (Eds.). Association for Computational Linguistics, Lisbon, Portugal, 541–547.  https://doi.org/10.18653/v1/D15-1063
  * Tahmasebi et al. (2021) Nina Tahmasebi, Lars Borin, Adam Jatowt, Yang Xu, and Simon Hengchen (Eds.). 2021.  _Computational approaches to semantic change_.  Number 6 in Language Variation. Language Science Press, Berlin.  https://doi.org/10.5281/zenodo.5040241
  * Trischler et al. (2017) Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.  NewsQA: A Machine Comprehension Dataset. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_ , Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih (Eds.). Association for Computational Linguistics, Vancouver, Canada, 191–200.  https://doi.org/10.18653/v1/W17-2623
  * van Strien et al. (2020) Daniel Alexander van Strien, Kaspar Beelen, Mariona Coll Ardanuy, Kasra Hosseini, Barbara McGillivray, and Giovanni Colavizza. 2020.  Assessing the Impact of OCR Quality on Downstream NLP Tasks. In _International Conference on Agents and Artificial Intelligence_.  https://api.semanticscholar.org/CorpusID:215756646
  * Wang et al. (2022) Jiexin Wang, Adam Jatowt, and Masatoshi Yoshikawa. 2022.  ArchivalQA: A Large-scale Benchmark Dataset for Open-Domain Question Answering over Historical News Collections. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (¡conf-loc¿, ¡city¿Madrid¡/city¿, ¡country¿Spain¡/country¿, ¡/conf-loc¿) _(SIGIR ’22)_. Association for Computing Machinery, New York, NY, USA, 3025–3035.  https://doi.org/10.1145/3477495.3531734
  * Yasunaga et al. (2021) Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2021.  LM-Critic: Language Models for Unsupervised Grammatical Error Correction. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 7752–7763.  https://doi.org/10.18653/v1/2021.emnlp-main.611
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension.  _CoRR_ abs/1810.12885 (2018).  arXiv:1810.12885 http://arxiv.org/abs/1810.12885